Server-side Baking / Appearance: key questions answered by the Lab

On Monday May 13th 2013, Troy and Nyx Linden appeared on a segment of Designing Worlds to discuss Server-side Baking / Appearance (SSB/A), alongside Brooke and Oz Linden, who were there to discuss Materials Processing.

Troy Linden is a Senior Producer at Linden Lab, who has been working on high-level server-side baking, and Nyx Linden is a Senior Software Engineer at the Lab, who has been working with the technical aspects of SSB/A and has been very much the public face of the project. Together, they answered a series of questions on the project put to them on behalf of users (the questions having been requested in advance of the show being recorded) by the Designing Worlds hosts, Saffia Widdershins and Elrik Merlin.

The following is a summary of the questions asked and answers given.

Nyx Linden (l) and Troy Linden (r) on Designing Worlds
Nyx Linden (l) and Troy Linden (r) on Designing Worlds (image courtesy of Wildstar Beaumont / Designing Worlds)

Saffia Widdershins (SW): Let’s start with the basics: what is baking, and how is it being handled now?

Troy Linden (TL): Baking is a process where we take all the information that involves your avatar – how it looks – and we combine it to deliver a finished avatar. Currently, how it’s handled right now [is] your computer, the individual’s computer, handles all of the processing involved in determining your avatar’s appearance, and it sends all the result back to our servers. So it’s a pretty involved process and there’s a bunch of time that it takes to do all that.

SW: So how is that going to be changed in the future … and will it simplify it?

TL: Server-side baking is our new system. It’s where we actually stand up a new service that will handle all of the baking process on our end. And what this actually does is it takes the load away from your computer, the individual user’s computer, and the results are a faster, more consistent experience during the whole baking process in Second Life.

Elrik Merlin (EM): Just to be clear about this … in the new system, what will be handled by the server and what will be handled by the viewer, exactly?

TL: The new viewer will be sending the server and [be] the recipient of all the avatar data, while the server does all the calculations required. So your viewer will download the results [of the baking process] over a lot faster HTTP connection.

EM: So that’s the basics of how it works, so to speak; how would you summarise the benefits to users?

Designing Worlds hosts Saffia Widdershin and Elrik Merlin
Designing Worlds hosts Saffia Widdershin and Elrik Merlin (image courtesy of Wildstar Beaumont / Designing Worlds)

TL: Well, simply put, it’s a much faster, more reliable avatar rendering experience. So hopefully you’ll see less avatars being stuck in their clouded state as well as being stuck untextured. So they’ll actually appear the way the user actually intended much quicker.

SW: So it will be an end to that problem where you half-rez but, (laughs) your make-up is blurred so you look as though you’ve been having a really heavy night!

TL: (Laughing) That’s the plan. We’re actually seeing some great results so far, so we’re very excited.

SW: Are there likely to be any downsides? There will be less impact on peoples’ machines, is that what you’re saying, or could there me more?

Nyx Linden (NL): The one downside to the new system is, because it is such a big change from how we have done things in the past, everyone is going to have to update their viewer. It will be a mandatory update. Users who don’t update will start to see even more avatars fail to load. Fortunately, we have the viewer that people need to download released, and users who use any actively maintained third-party viewer should be able to download an update presently as well. As long as users do update, they won’t see any downsides.

EM: This is obviously nearing completion and we’re nearing implementation. Can you tell us a little about where the project is, what its current status is, and what the time scales are for introduction are going to be?

NL: Absolutely! So, we’re in a multi-stage release; at this point we have our first viewer out the door. So the next stage is that we’re going to be standing-up the service that is going to be doing all the work for rezzing your avatar. Over time we will slowly roll-out the new system across the grid. That’s going to take some time, and we’re going to be following-up through our blogs and forums to let people know how that process is going, but we want to take our time with that process, to make sure that everything is working as well as we think it is.

Continue reading “Server-side Baking / Appearance: key questions answered by the Lab”

SL projects update 19 (3): server, pathfinding, materials & more

Server Deployments – Week 19

The week 19 server deployments went ahead as scheduled and as outlined in an earlier part of this report, to wit:

SLS (Main) Channel

On Tuesday May 7th, the Main channel received the maintenance package which has been running on Magnum. This project brings some new minor features to LSL, and fixes some crash modes and well as additional LSL HTTP support, as well as the previously rolled-back new AO capabilities.

In addition, this package included an additional update which was not documented in the release notes until after the deployment had been made: “Fixed a bug: ‘Orientation data for landing point is disregarded during teleport’.

This fix means that when teleporting to a place which has a defined landing point, you will arrive facing the direct determined by that landing point, rather than arbitrarily facing east. To complement this fix, there is an upcoming viewer update which will show the heading of a parcel’s landing point on the map – but no details on when that update is liable to appear.

Pointing out the fix at the Server Beta meeting, Maestro Linden added, “There’s an exception for cases in which if you have a beacon set, where you’ll face the beacon position instead. I think this is a bug.”

This lead to a discussion on the exception, in which the wider consensus appeared to be that having an avatar facing the beacon (when set) on arrival might be the preferable to automatically having the avatar facing the direction imposed by the landing point. However, this would mean people using a beacon would arrive at the landing point facing a direction other than the land owner intended, prompting Voidpointer Linden to comment, “Yeah, seems like we would only want to have you face the beacon when going to a landing point if it’s in the same region, maybe?”

It’s currently unclear as to whether the feedback from the meeting will mean this exception is allowed to continue, or whether there will be further tweaking with the landing point code in the future,

Release Candidate Channels

On Wednesday May 8th, all three Release Candidate channels (BlueSteel, LeTigre and Magnum) received server-side support for experience permissions which was running on BlueSteel in week 18. The update, as usual, will also include the changes which are going to the Main channel on May 7th, which included the new AO capabilities – release notes (BlueSteel).

SL Viewer Updates

Beta Viewer

The beta viewer as we currently know it received what might by its final update prior to the new viewer release processing coming into effect. Version 3.5.2.275565 was released on May 9th, and the release notes provide details on changes.

Materials Processing Project Viewer

A new version of the Materials Processing viewer – 3.5.2.275470 – was released on May 8th, based on the 3.5.2 viewer code. The release also appears to include a fix for MATBUG-38 (“Material with normal map but no specular map is always specular with no adjustment possible”), and which I’d described in week 18.

MATBUG-38. appears to have been fixed in Material Processing project viewer 3.5.2
MATBUG-38. appears to have been fixed in Material Processing project viewer 3.5.2.275470, May 8th 2013.

The release may also include a fix for MATBUG-16 (Changing one material, or setting causes another material texture to be lost), which I’d previously noted in week 16.

Whether this latter issue is fixed is hard to judge, as it was somewhat intermittent to start with. and there have been reports that it was already less noticeable with the previous release of the viewer. However, at the time of writing, the bugs remain marked as Unresolved / Incomplete in the public JIRA (but could have been updated in LL’s internal JIRA).

I can say that I’ve so far been unable to reproduce either bug using the new release, having carried out a number of individual tests.

Continue reading “SL projects update 19 (3): server, pathfinding, materials & more”

SL projects update week 19 (2): Interest list, missing prims, griefing

Interest List Update

As noted in week 18, Andrew Linden has been working on fixing a bug related to Meeroos (but which I’ve seen affecting other animals as well).

If you turn your camera away from a crowd of Meeroos, wait several seconds, then turn back around… the Meeroos will be updated, but not quite in the right order. So sometimes you’ll see a head move to the new position, then a fraction of a second later the rest of the body.  So I have a theoretical fix that doesn’t crash the simulator (anymore)

Reporting on the situation at the Simulator User Group meeting on Tuesday May 7th, he said, “Right before this meeting I was rounding up some meeroos to do some testing on the beta grid. The bug is theoretically fixed, but I’ve yet to actually see it work. We’ll be testing this week.” If all goes well, the fix may well be progressing towards an RC release in the near future.

“Missing Prims”

Still no major news on this issue in terms of a lasting fix becoming available. Commenting on it again during the Simulator User Group Meeting, Andrew Linden said:

We were having trouble reproducing it on one of our more recent viewers; the viewer that will eventually go along with the recent interest list fixes, is currently stalled for a mysterious crash bug.

We think the “right click to make the object show up” bug is related to loading of object cache in the viewer and that loading code has had an overhaul in this recent viewer project. So we think the bug is fixed there, but we have yet to test it a lot. Because there is a crash bug we’re still trying to track down.

"Missing" prims - viewer-side fix stalled; code might be made available to TPVs anyway?
“Missing” prims – viewer-side fix stalled; code might be made available to TPVs anyway?

The question was asked if the code could be made available to TPVs as it is, even if prone to crashing. Doing so might provide extra eyes on the problem which may both help to resolve the issue and fix the crash issue. Andrew replied to the question by saying, “Oz asked us to put that code out on a public repo, but that was before we realized we had a crash problem.” However, he agreed to take the request back to the office and see what the reaction might be.

Other Bits – Griefing Issues

There has been a long-standing issue with objects sitting on the region borders being very hard to return to their owners, and which has become something of an exploit where mainland griefing is concerned. It had been hoped that the fix for the issue would be deployed to the grid this week, However, in reviewing matters, Andrew linden regretfully reported that “It appears that ‘return of objects at region border’ bug fix is not deployed at the moment, as far as I can tell.”  There is currently no date as to when this will be deployed,

Similarly, there is still no news as to when the particle muting capability (right-click on a particle to stop your viewer generating the particle stream in your world-view) might make a viewer-side appearance.

There are reports of a new form of particularly malicious griefing involving spinning / flashing objects which appear deliberately designed to trigger epilepsy or migraine.  At the Simulator UG meeting, Simon Linden indicated that the Lab is at least aware of this form of attack.

Blocking Banned Users’ Objects from Rendering

A suggestion has been put forward at several Simulator UG meetings where griefing has been discussed that all objects on a parcel belonging to a user banned from that parcel for griefing should no longer be rendered for other users within that parcel.

This is seen as a particularly useful option at events, etc., where time would otherwise be taken up in trying to locate and return objects which may have been left behind when banning the user, and / or in telling other people in the parcel how to mute offending objects from their view.

Commenting on this as a possible server-side capability, Andrew Linden said, “More fidelity of visibility on a per-parcel level would further complicate things that are already fairly complicated. I’m not saying it can’t be done, but it would be tricky and I would worry about lag issues… server lag as it picks up more work.”

However, after a discussion on the viability of viewer-side blocking and server-side blocking, he commented, “I’m hip… if visibility of banned owners’ objects were to be done… it would probably be easiest to do it at the server.” Even so, there would still be a range of issues to deal with, were such a capability to be put into place. As it stands, it appears to be more a case of food-for-thought than a definite step to be taken.

Forced Object Return

Another griefing attack is one which sees an object deposited in a region but which doesn’t actually trigger until after the region is restarted. This means it gets included in the region back-up – and any subsequent region restart, leaving it free to annoy.

A suggestion was put forward at the Server Beta meeting on Thursday May, 2nd to force a return of all objects in a region which are not set to the required Group as a part of the region restart process. This idea gained some support at the meeting and might have resulted in a feature request being filed. However, it is not without issues of its own – such as with regions where Group object rezzing isn’t a requirement, where it could result in a lot of items which might otherwise have been “safe” returned to their owners.

SL projects update week 19 (1): server releases, SSB/A, materials

Server Deployments Week 19

Second Life Server (Main) Channel

On Tuesday May 7th, the Main channel should receive the maintenance package which has been running on Magnum. This project brings some new minor features to LSL, and fixes some crash modes and well as additional LSL HTTP support – release notes.

 Release Candidate Channels

On Wednesday May 8th, all three Release Candidate channels (BlueSteel, LeTigre and Magnum) should receive server-side support for experience permissions which was running on BlueSteel in week 18. The update, as usual, will also include the changes which are going to the Main channel on May 7th – release notes (BlueSteel).

I was somewhat confused by the release notes for 13.05.04.275247 package, as on the one hand they suggested the new LSL AO capabilities would be absent from Agni following the RC deployments (“Changes since 13.04.19.274370 … Removed changes introduced by Second Life Server 13.04.12.273874”), while on the other, the line originally following this comment suggested otherwise.

I contacted Maestro for clarification, and he confirmed that the AO capabilities will be present on the RC channels following the deployment, and that the release notes have been revised to prevent any further confusion.

As always, there is a discussion thread in the forums for the deployments.

Server-Side Baking / Appearance

SSB/A - further viewer-side updates expected. No published timeline on testing as yet
SSB/A – further viewer-side updates expected. No published timeline on testing as yet

As noted in the last part of my week 18 report, there will be at least update to the viewer-side SSB/A code, which Nyx referred to at the Content Creation meeting on Monday May 6th as being, “Mostly reliability and some related polishing”. From the TPV Developer meeting on Friday May 3rd, the release will be addressing some additional ways in which baking can fail as well as addressing stability in general, although as Nyx noted at the TPV Developer meeting, “Nothing that we know of right now that would be mandatory for the system to work.”

The question of when constrained testing would commence again came up at the Content Creation meeting. This will see a number of regions across the main grid have the server code for SSB/A enabled in order for LL to carry out further load tests on the system ahead of the “switch being thrown” and SSB/A enabled across the entire grid. All Nyx would say on the matter is, “We don’t have a timeline to announce today.”

Whether this means there will be a formal announcement when testing does commence, remains to be seen. However, the Lab is still intending to make a formal announcement on SSB/A, mostly likely via a blog post, ahead of the service being enabled across the grid in the hope of further communicating the change, and the need to upgrade to an SSB/A-capable viewer to as many users as possible.

Outfit Changes

During her tests with SSB/A on Aditi, Kitty Barnett noted that when changing a wearable or reordering the layers of an outfit, the updated bakes could take a while to come through. Concern that this might be the case when SSB/A is enabled on the main grid prompted her to ask:

For server-side baking, would there be any issue with turning on local baking to make getting dressed more pleasant? it could be Aditi, but [when] wearing something it takes a very long time before the server bake comes through, particularly reordering clothing or taking something off … I actually just turn local baking on as soon as there’s a wearable change and it seems to be working nicely and turns itself off when the server bake comes through… but I was wondering if that would cause any undesirable issues [if the same was done on Agni]?

Local bakes already occur when using the Editing Appearance options, so Kitty’s question appears to be more broadly aimed at general outfit changes (people simply swapping individual clothing layers directly from inventory, and (possibly) using the capability in some TPVs to re-order clothing layers directly from inventory, rather than going via the appearance editor).

Responding to the question, Nyx said:

I’m not certain how you’re turning on local baking, and can’t really speak to whether there are undesirable behaviors as a result. The code to switch between the different types of bakes is rather complex and sometimes fragile, and bugs in that area can be subtle … Proceed with caution and best of luck :). Report back if it works well (or if you find interesting bugs!)

Whether such a delay will be noticeable enough to be an irritant on Agni is hard to say as we’ve yet to see the server code up and running on the main grid, so it’ll be interesting to see what further tests on Kitty’s part reveal as testing commences.

Continue reading “SL projects update week 19 (1): server releases, SSB/A, materials”

SL projects update 18 (4): servers, viewer release process, group bans and bits

Server Deployments – Week 18

As reported in the first part of this update, the SLS Main channel was rolled back to release 13.04.05.273580, as a result of a widespread performance issue.  This unfortunately saw the removal of the new LSL animation capabilities from the channel.

The issue itself is related to problems with regions locating their neighbours. “The sim were hitting the [region presence lookup] service too hard, causing stability problems.” Maestro Linden said at the Server Beta meeting on Thursday May 2nd. Release notes.

In the original notes for this week’s deployments, BlueSteel and LeTigre were scheduled to receive the same deployment package. However, this was subsequently changed so that:

  • BlueSteel received the same reversions as the Main channel due to the performance issue between neighbouring regions, but also received updates for the Experience Keys project as originally planned. Release notes.
  • LeTigre received the same code that was on the main channel in week 17, the only difference being that it fixes the performance issue that caused the Main channel to be rolled back this week.  Release notes.

Maestro described the deployment of the fix for region lookup issues as the “conservative option”, rather than deploying the fix to multiple Release Channels, “In case the changes from the other two channels have their own problems.”

Magnum received the package originally scheduled for it, described as bringing some new minor features to LSL, and fixes some crash modes as well as the fix for grid performance issue, and fixing an issue in which llDialog() messages sent to the object owner could be incorrectly throttled. Release notes.

The hope is that if all is well with the Magnum update, it is liable to be deployed to the Main channel in week 19.

SL Viewer Updates

Beta Viewer Release

A new beta version of the viewer emerged on May 2nd, using the 3.5.2 code (3.5.2.275087). This release includes the update from FMOD to FMOD Ex, and well as a number of other maintenance and other fixes as specified in the release notes.  However, it does not include the anticipated Vivox updates to improve SL Voice. These will be coming along at a later date.

The current plan is for this release to remain in the beta channel for at least one further update prior to it appearing in the release channel. As such, it is unlikely to be in the release viewer until late in week 19 or in week 20. Once it has appeared in the release viewer, LL will probably deploy the new viewer release process, and the beta channel will cease being used.

Viewer Release Process

Oz Linden
Oz Linden

As previously reported, the viewer release process will be changing in the coming weeks. As a part of this, the development viewer channel has already been deprecated; however it will still be a while before the new process is put into place, as further infrastructure changes are still required on LL’s part.

Concerns have been raised by TPV developers about a side-effect of the new process potentially being that the rate at which code becomes available to them may slow down, thus causing them to “fall behind” the LL viewer in terms of new functionality or capabilities. Stressing that this is not the intent, Oz Linden described process in further detail at the TPV Developer meeting, indicating that under the new system:

  • Viewer projects will each have their own repositories, which will be made available to TPVs (and others) once it is deemed they are “safe to share” as a project or “beta” viewer
    • While it has yet to be formally decided within LL, and may take a little time to work up to, critical bug fixes are liable to have their own repositories, from where they can be merged into other viewers
  • Users will be able to pick which beta / project viewer(s) they download from the Alternate Viewer wiki page without being tied to any specific update route (so you can download and run as many project / beta viewers as you like)
  • Once a project is believed to be of release quality, it will be put into a release candidate, built on the current viewer release code and released to a target number of users (as chosen by LL), alongside other release candidates being used by other users
  • When a user receives a release candidate viewer via the download page, the updates they receive will be offered on the basis of the release candidate they are currently running (for example, if a user is running the Materials RC viewer, they won’t be offered updates from, say, the SSB/A RC viewer)
  • After some period of time, and when LL have looked at the results, one of the release candidates will be promoted to the default download (without the viewer having to be rebuilt) and will be available on the main download page
  • The remaining viewer projects (at least those at release candidate status) will then be merged with the newly-released viewer code, and re-test and issue a further release candidate, which may in turn be selected as the next candidate for promotion to the default download.

He added that in terms of TPV’s concerns over code being made available to them, the level of co-operation which has been evident in the Sunshine project (SSB/A) has “not gone unnoticed” within LL’s management, and that the team involved has received a lot of kudos for the way they have handled interaction with TPVs. As such, it is likely that the Lab will endeavour to build on this going forward.

Continue reading “SL projects update 18 (4): servers, viewer release process, group bans and bits”

SL projects update 18 (3): more on the viewer

Viewer Release Process

Following-on from the announcement of a new process for viewer releases Oz Linden provided more information on the process as it will operate going forward during the Open-Dev meeting on Wednesday May 2nd, and gave a clarification on my original post after I’d mistakenly referred to individual viewer “release candidate channels”, rather than more correctly, “release candidates”. His comment in full reads:

Oz Linden
Oz Linden

The release candidates will be updates in the regular release channel, not separate candidate channels. The candidates will be in a named “cohort” within the channel, but the cohort name is not built into the viewer the way a channel name is, which means we will be able to move a candidate from its named cohort to the default without rebuilding it – the build we test will be exactly what we release. Candidates will be indistinguishable from the default release viewer (the one on the main download page) except for the version number and the new features.

The new process also means that officially, individual viewers will no longer be referred to by specific project channel names (such as CHUI, Materials Processing, etc.). Instead, they’ll be referred to by their full 4-part number (i.e. 3.5.1.274847 rather than the “Materials project viewer”).  This also means there is likely to be at least one update to the Bug Tracker so that the Environment box in the form will no longer allow a report to be submitted until viewer channel and version number have been recorded.

As a result of these changes, Oz indicated that the development viewer channel has already been depreciated, and that the beta channel, as mentioned in my original piece will be depreciated as the Lab switches over to the new system.  When a candidate viewer is ready for public consumption, it will appear on the  Alternate Viewers wiki page where it can be downloaded.

I’ll be making changes to my Viewer Round-up Page to match the new system as it comes into play.

Current SL Viewer Status Update

In the meantime, and in the lead-up to the new release process, there is expected to be a further beta viewer release (3.5.2 code), which will include the FMOD Ex updates in it, and for which Oz extended thanks to the Singularity team for their work.

The Mac Cocoa project is a little stalled. This is because, to quote Oz, “We’ve been stealing people [TPV devs] from Cocoa to get Materials done. The good news is that it’s working, we’re really knocking down bugs at a steady clip.”

The mesh deformer also remains stalled. This is again due to manpower issues but this time internal to LL. However, Oz again wanted to reassure people, stating of the deformer, “There are things I’ve given up on at this job, but that isn’t one of them.” Sadly, his plate is also full with the Materials project at the moment, together with things like the new viewer release process.

Related Links