As noted in the update to part 2 of this week’s report, the planned deployment of two new releases to the RC channels didn’t go as anticipated. Originally, it had been intended that BlueSteel and LeTigre would received the new threaded region crossing code while Magnum would receive Andrew Linden’s interest list code improvements.
Interest List Deployment Cancellation
The interest list deployment was cancelled after the 11th hour discovery of some bugs with the code. Speaking at the Server Beta meeting on Thursday 10th January, Maestro Linden described the main issues as – ironically – being connected with region crossings, and with object updates.
In the first, anyone crossing between regions several times in a vehicle would experience all of their non-rigged attachments disappearing from their world view, with the viewer itself eventually physically detaching them. Not only did this cause confusion as to what was happening with attachments for those experiencing the issue, it also resulted in some avatars ending up naked following a relog.
Interest list code: bugs led to deployment cancellation on Wednesday 9th January
The second problem was slightly more complicated, if potentially more rare. In it, if User A had an object on the ground and User B looked at then turned their camera away such that the object was no longer on their screen, User A could then wear the object as an attachment and teleport away; however, when User B subsequently turned their camera back to where the object had been, they would still see it on the ground despite the fact it had been taken away. What happened if User A (now wearing the object) teleported back to User B wasn’t actually tested.
As a result of both of these issues and the cancellation of the interest list deployment, Magnum received the same region crossing package as intended for BlueSteel and LeTigre.
Region Crossing Code Issues
However, the bad news did not end there, as Maestro Linden explained, “After a few hours, we saw that the [region/sim] crash rate was way too high.” As a result, the threaded region crossing code was disabled via a configuration change to the servers without the need to rollback the release. Once this had happened, region crash rates returned to “normal” levels. In all the new region crossing code was active for around five hours before being disabled once more.
Analysis of the crash rate revealed it to be linked to avatars crossing to / from heavily scripted regions. While the new code was extensively tested on Aditi, the regions there were not excessively loaded with scripts during testing, and so the problem did not manifest. However, subsequent testing with the test regions running heavy script loads did result in them also crashing, confirming the problem.
At the time of writing, Kelly Linden believes he has a fix for the issue; if so, it should hopefully find its way to the RC channels in week 3, commencing Monday 14th January.
Update: The Interest list release to Magnum has been postponed due to a last-minute bug being found. Magnum has instead received the same code as deployed to Bluesteel and LeTigre, containing the threaded region crossing update.
Server Deployments for Week 2
Server deployment recommenced this week, with a main channel roll-out on Tuesday 8th January, which saw that channel get the same code as released to the RC channels just before the Christmas / New Year break and as reported here.
Wednesday January 9th will see two major deployments to the RC channels. LeTigre and BlueSteel should be receiving a package which includes threaded region crossing code while Magnum gets Andrew Linden’s work on Interest Lists.
The threaded region crossing code should help improve simulator performance when avatars are region crossing to and from a region. However, and as previously noted, while the code did give clear improvements when crossing between regions on foot, the results were less positive when using vehicles during a recent test. Whether this has been improved as a result of those tests remains to be seen.
The Interest List work should reduce the bandwidth usage of viewers due to object updates, and should improve simulator performance, especially in sims with many connected avatars and which is busy in terms of object updates.
This is the first phase of work planned around interest lists and object caching as a part of the Shining project. Andrew’s work has primarily been focused on improving the manner in which object updates are handled, etc., in order to provide improved performance. However, even with this initial phase of the work, there should be some improvements in the actual order in which objects appear in world (those closer to your camera position appearing prior to items further away – although there is far more work to be done in this regards before the project is finished.
As it stands, there are already some updates to the work in the offing, as Andrew indicated at the Simulator User Group meeting on the 8th January, saying, “I’ve got some interest list work that didn’t quite make it into the Magnum RC … this work is almost ready for testing, so I’ll be trying to get it into a maintenance branch or something for a follow-up release.”
Interest list updates should be deployed to the Magnum RC on Weds 9th Jan.
The new interest list code will see the final removal of the “legacy cloud” layer (at around the 170-200m mark). While this has been disabled viewer-side since around the introduction of Viewer 2, the data relating to the legacy cloud layer has still been sent out by the servers, allowing viewers which still incorporate the necessary code to render the clouds in-world. However, the server-side code relating to legacy clouds has been removed from the interest list code, so viewers will no longer be able to render the clouds.
There have been some rendering issues with the last release of the beta viewer (3.4.4.268497, December 20, 2012) which had caused the Lab a slight headache in that not all tests are giving the same results. However, a further 3.4.4 release is anticipated for either Monday 7th or Tuesday 8th January, 2013, which includes various fixes. Whether these are related to the rendering issues is unclear. However, they have not as yet been merged into the Sunshine Project (Avatar baking – see below).
CHUI – Communications Hub User Interface
As reported over the holiday period, the CHUI project is moving forward, with a further update of the project viewer and several updates to the development version of the viewer, possibly the result of code refactoring work which had been indicated as being required prior to the holidays. However, as of the TPV Developer Meeting of Friday 4th January, 2013, it was unclear as to whether this refactoring work has been completed.
CHUI: TPVs may cherry-pick from the code
Currently, the code has yet to be made available to TPVs, and concerns have been raised by some TPV developers that integrating the CHUI code could be as much a headache as the Avatar Baking code. Given the work some have put into the communications elements of their own viewers, it is also possible that some might opt to cherry-pick which elements of the CHUI code they will adopt. Whether CHUI is liable to be deployed before or after the Avatar Baking project remains to be seen, as the Lab has yet to make a decision either way.
Server-side Avatar Baking
Avatar bake fail
Project Sunshine, the work to implement a new server-side baking process, kicked-off (as far as TPVs are concerned) just before Christmas. This represents a substantial code merge for TPVs, and one which is going to take TPVs a while to handle as a result, hence the reason why LL have given TPVs a long lead-time on the project, with around an eight-week window available for them to work on the code, provide feedback and assist with testing.
As mentioned in my detailed look at the new service (see link above), any deployment of the server code will be dependant upon further and significant load tests, which are viewed as essential in ensuring the new compositing service has sufficient hardware for it to support avatar baking across the entire grid. At the time that article was written, Nyx indicated that details on how the load tests would be handed had not been finalised.
Speaking at the TPV Developer Meeting, Oz indicated that these tests are still under consideration, and as such, much in the project is still up in the air in terms of unknowns. Obviously, on way in which load tests can be carried out is to have more test / development viewers available to enable greater testing of the server-side code, so overall implementation of the new service is somewhat symbiotic, and it is unlikely there will be a large-scale deployment of the service prior to TPVs being sufficiently comfortable / up-to-speed with integrating the code into their viewers.
As such, it is unlikely that there will be any major move on the Lab’s part to push the project forward much before the end of February. With regards to this, Oz commented, “Obviously, what we’d like to know is that we’ve got at least one version of all the third-party viewers that are prepared to cope with it, and that certainly getting an affirmative on as many of those as possible before we make a final call on what our target dates are would be really great. So that’s why we’re keeping the pressure on you to do that testing, as we’d rather you were ready before we were.”
Materials Processing
The materials processing project continues to move forward, although there are growing concerns over the fact that the viewer will be required to run in deferred mode (i.e. with shadows & lighting active) in order for the new capabilities to be properly rendered. This means that computers which do not have sufficient processing capabilities to run in deferred mode will not be able to render the effects of normal and specular maps, and so will not see the effects of materials processing.
When life gives you lemons, map them: on the left, a normal map, on the right, a normal and a specular map together (credit: Mind Teat Studios)
However, this does not mean that those unable to run SL reliably or reasonably with deferred rendering enabled will have their SL experience negatively impacted. The expectation is that users on such system will continue to see SL as we all see it today, regardless as to whether or not in-world objects and avatar attachments (prim, sculpt or mesh) are using the new materials capabilities.
However, this is also conditional on content creators understanding how to correctly make use of materials process as it will apply to Second Life (especially those trying to leverage the new capabilities, but who may not themselves be able to run the viewer in deferred mode), and ensuring they use underpinning diffuse maps (textures) of a suitable quality. To help ensure this, Oz Linden has stated he will give those at the Lab responsible for the Good Building Practices guide on the wiki a nudge so that it is expanded to cover materials processing.
That materials processing does require running the viewer in deferred mode has given rise to concerns as to how widely the capability will be adopted. However, the Lab has no plans to try to implement materials processing in a way which does not require deferred rendering (assuming this could be done). This is not to exclude anyone from experiencing it, but rather because the capability simply requires deferred rendering to be enabled. Whether or not the capability will introduce an additional overheads to running in deferred has yet to be fully determined.
In the meantime, the repro for the viewer-side code required for materials processing will be officially made available (it was accidentally exposed just before Christmas), some time in the next two weeks or so, and a project viewer should appear shortly thereafter. The server-side code is thought to be in, “Pretty good shape.”
The Lab is still getting back up to speed following the Christmas / New Year break, so expect further information to be forthcoming on Main and RC releases for week 2, 2013 via the Server topic of the Technology Forum.
However, as it stands, there are two projects which it is hoped will reach RC channel release in the week commencing Monday 7th January, 2013. These are Caleb Linden’s threaded region crossing code and Andre Linden’s interest list code.
As I’ve previously reported, the threaded region crossing code was subjected to a pile-on test on Aditi towards the end of 2012. The results were, on the whole, a little disappointing for those taking part – although expectations may have been set a little high. While there were some improvements noted – particularly when travelling between regions on foot and with a heavy script load – overall, there were sill issues with crossing between regions in vehicles (particularly ground vehicles).
Airbourne antics: vehicles still exhibited region crossing issues duing the threaded region pile-on test in December 2012
Issues arising from the pile-on test are still being looked at, and Caleb repeated his request that anyone noting specific issues should raise a JIRA directed for his attention. For those wishing to try out the code, the GC Test regions are still available on Aditi.
The Interest List code is still subject to receiving an OK from the Lab’s QA team. There will doubtless be an update on this – and on the planned RC releases in general – at the Simulator User Group meeting on Tuesday 8th January, 2013.
SL Viewer News
Not a lot to report on here at present. The Beta viewer reached the 3.4.4 code base Just before Christmas 2012, (3.4.4.268497, December 20, 2012). There may be a rendering issue which may require addressing and might lead to a slight delay in releases; apparently, not all tests are giving the same results, so LL are still investigating the matter. Work is continuing to update the GPU tables for the viewer; further cards have been added to the table, and several blanket entries have been removed (such as all unrecognised nVidia cards being detected as nVidia Ion GPUs).
As reported over Christmas, CHUI rolled through a number of rapid releases in its development version, and the main project version rolling to 3.4.3.268587 on December 22nd. Both the development and project versions of the viewer are on the 3.4.3 codebase, and the most recent development release was made on January 4th (3.4.3.268703). Both versions are available from the Alternative Viewers download page.
While the core of the Mac version of the viewer is built using OSX 10.7 (with Xcode 4.3.3), work is progressing in moving the viewer to OSX 10.8 Mountain Lion, which is expected to happen “very soon” according to Oz Linden, although no date is available as to when.
New Pathfinding Capability
VoidPointer Linden is working on a new flag for pathfinding characters. STAY_WITHIN_PARCEL is designed so that when set, pathfinding characters will only set goal points during wander, evade, pursuit, etc that are within the parcel they get created in. If the parcel is a non-regular shape, it is still possible a character will cross between it and neighbouring parcels (unless the navmesh is cut through the use of an exclusion volume), but goal points will only be set within the originating parcel. The code is still in development, and so the constraints on where a character can wander when it comes to irregular parcel shapes, but VoidPointer is not making any promises on this.
He’s completely batty! – Voidpointer Linden’s avatar at the Server Beta UG meeting, Jan 3rd, 2013
There is no stated delivery time for this new feature, other than it is currently being worked on.
Server Object Rezzing Code
Baker Linden has been looking to improve how objects with large file sizes are handled by the simulator software when being rezzed. He describes the work thus, “What I’ve been working on is hopefully significantly decreasing lag spikes when rezzing large, complex objects. Large does not necessarily imply size, but size of the files being read. When an object is rezzing, we have to parse the object / mesh files and create our in-world objects with that data.”
Until now, reading and parsing of any files related to objects which require rezzing has been on the main thread. When several such objects requiring rezzing at the same time, the simulator stalls. Baker has been moving the reading / parsing operation to a background thread in the expectation that rezzing multiple “large” (again, in terms of file size, not the size of the object itself) objects will not choke the simulator.
The key point about this work is that it is specifically aimed at preventing the simulator processes from choking and a region stalling when there are a number of large object files being read / parsed, not at actually “speeding up” the physical rezzing process. As such, it is unlikely that objects will appear any faster in people’s in-world view as a result of this work. However, what it does mean is that the simulator code will be better able to handle rezzing multiple “large file” objects without the attendant region lagging which can occur as a result of the simulator being unable to process messages from viewers and other simulators, etc.
Inara Pey: Living in a Modemworld 