Update 28th September: Please also refer to an update post on some of the projects / news given here.
SL Viewer Status Updates
Linden Lab have been working hard on a range of viewer-related issues, notably crash rates and memory leaks, which have slowed the viewer update a release process up over the last few weeks. In terms of memory leaks, tcmalloc has been identified as the culprit, with Linden Lab deciding that dropping it is “probably a good idea”, according to Oz (tcmalloc has previously been implicated in crashes linked to the use of things like Microsoft’s Skydrive). There have also been an issues with LL’s statistics system which have meant that the viewer hasn’t necessarily been accurately tracked in terms of crash rates, etc.
Beta Releases
As it stands, LL hope to have the blocks on the various code merges removed during this week, which should see a rapid series of beta releases coming down the pipe. This work commenced with an initial 3.4.1 beta release (3.4.1.265134) emerging on September 24th. It will be followed by around three or so additional and rapid 3.4.1 build iterations aimed at confirming the viewer’s stability and at replacing various fixes which had previously been removed from the viewer code while trying to identify the causes for the viewer crashing / suffering memory leaks. It is expected that each of these iterations will be on the beta release build channel for a couple of days, prior to being replaced. Following these there will be a series of project updates, the first of which gatekeeper compliance project, which is also targeted for a 3.4.1 release build.
Project-related Releases
Once the stability of the beta viewer has been confirmed, it is anticipated that project-related code will be merged into the viewer, most likely starting with 3.4.2 builds. Among the releases planned for 3.4.2 is Monty Linden’s HTTP Library Services and Baker Linden’s Group Services code. These are currently targeted to reach the beta build channel in week 40 (week commencing Monday October 5th).
These releases will at some point include the Steam updates currently in a Development branch as well, which might in turn mean that Second Life could be ready to appear on Steam in the very near future, once these updates have reached a release version of the viewer.
Account creation prompt: heading for the beta viewer
Group Services Project
The Group Services project is an attempt to improve the management and editing of large SL groups by replacing the current UDP-based service (which has capacity issues with the size of group lists it can comfortably handle) with a new HTTP-based service. The project viewer for this is already available (for Windows, Linux and OSX.), however, as mentioned above, the current plan is to get this into the 3.4.2 build stream alongside the HTTP textures project, possibly in week 40.
Originally, the server code for this project was due to have been rolled to the RC channels during week 38, (week commencing September 17th), but the channel deploys were postponed after QA issues were found. As a consequence, the roll-out was due to take place on Wednesday 26th September, but has again been postponed.
There has been some confusion as to the aim of this project, with some people believing it is focused on fixing group chat issues such as lag and chat failing to start. This is not the case at all; as stated above, the project is aimed at improving the management and editing of large groups (10K+) through the use of a new HTTP service.
HTTP Library Services
As indicated above, the first phase of this work, covering a new texture fetch service, should be appearing in a 3.4.2 beta release of the viewer in the near future.
HTTP Libraries project viewer: improved texture loading and rezzing
Please use the page numbers below left to continue reading this article
There is a lot going on in terms of projects and development work in Second Life. The following is a further update on a number of key projects I’ve been following and reporting upon through the pages of this blog.
Group Services Project
The Group Services project is an attempt to improve the management and editing of large SL groups by replacing the current UDP-based service (which has capacity issues with the size of group lists it can comfortably handle) with a new HTTP-based service. The project viewer for this is already available (for Windows, Linux and OSX.), as I reported last week.
This week sees the server-side code rolled-out to all three RC channels (Magnum, LeTigre and BlueSteel), allowing the project viewer to be tested in handling very large groups (significantly larger than are available on Aditi). Note that those running viewers without the new code on any RC region will be unaffected, as they will continue to access the current UDP service.
There are still no timescales as to how long the testing of the service will last (“It’ll take as long as it takes,” Baker commented recently), or when the viewer code will progress beyond the project viewer. However, a number of things should be noted in reference to the eventual roll-out:
The viewer code is not being made back-compatible with V1 code by the Lab. Therefore, TPV developers using the V1 code base will have to backport the code themselves in order to use the new service
The initial HTTP service roll-out does not include any data compression. This means there will still be some delay in downloading member lists for very large groups with tens of thousands of members
Once the new service is rolled-out, which service is used is entirely transparent to the user. If a viewer with the new code is running on a region which has the server-side HTTP service, it will connect to that service. If it is on a region using the older UDP service, it will connect to that service instead
Once the HTTP service is fully deployed, viewers which do not implement the viewer-side code will still be able to access groups with member lists up to 10K in size via the UDP service until such time as it is switched off (which will not occur for some time after the HTTP service has been rolled-out). However, attempts to access groups with lists larger than 10K will fail.
To recap: when you enter a region at the moment, your viewer receives a huge amount of information on what requires updating, much of it relating to things you can’t even see from your position in the region. This information is received in no particular order, with the familiar result that things appear to rez in your view in a totally random order. Not only that, but the chances are that if you’ve previously visited the region, much of the information being sent to your viewer is already locally cached – but is being ignored. The focus of this project is to both optimise the data being sent to the viewer and the information already cached on the viewer with the aim of significantly improving object rezzing times in terms of speed and order (so objects closer to you rez before those further away, for examples).
Object caching and interest list changes: easing the pain of random rezzing
Andrew Linden had hoped the project would be going to QA this week ready for roll-out to one or more RC channels in the near future, but some last-minute problems popped-up and have delayed things until he can get them sorted out. In the meantime, the code has been deployed to a number of regions on Aditi, and Andrew plans to, “Try to throw a pile of test avatars at it to stress it out. Later this week.” No viewer-side changes are associated with this work.
Materials Processing
Work continues on the project to bring materials processing to Second Life. Last week, it appeared as if the new materials – normal and specular maps – would have their own rotation and positioning options independent of any texture (diffuse) map. This week, it appears that this is the hoped for situation, but the matter is still open to question – which goes to show how fluid the project is.
The new capabilities require changes to the rendering pipeline, and details have been released on some of what this entails.
In order to work, normal and specular maps require what is referred to as per-pixel lighting (as noted in the original blog post on the subject). As such, there has been a debate on whether it would be better to develop a per-pixel lighting framework within the viewer, or work to make the current deferred rendering system more accessible to per-pixel lighting capabilities. As the latter approach will allow getting materials processing working within SL sooner than would otherwise be the case, it has been decided to go that route. Thus work is focused on making the current lighting system more configurable and able to better handle a broad range of material types (metallic, matte, plastic, etc.), together with adding support for both per-object and per material shading differences.
However, a dedicated per-pixel lighting framework does offer advantages of its own, and as such is being considered as a possible future extension to the project, which may be implemented at some point down the road. One such advantage is that could potentially be run in a non-deferred mode, which might lightening the load on older graphics systems.
Please use the page numbers below to continue reading
On the 29th June, Linden Lab announced Project Shining, aimed at improving avatar and object streaming speeds. At the TPV/Developer meeting on Friday 13th July, the project was discussed in terms of how the various elements within it will affect Second Life viewers.
The following is a summary of that discussion, based on the recording of the meeting, and focused primarily on the viewer changes / updates that will be most directly seen / felt by the majority of users.
HTTP Library
Commencing at 22:30 into recording.
The aim of this project is to improve the underpinning HTTP messaging that is crucial to simulator / simulator and simulator / Viewer communications. Monty Linden is leading this project.
Key points:
LL will release a project viewer containing a new “wrapper” implemented around how data is handled and a new texture fetch library (see time frame comments at the end of this article)
Providing there are no major problems with the project viewer, the initial code release will move to a release version of the viewer
This will be followed by changes to group services and a “more ubiquitous” use of the library in the viewer – which is where Oz’s warning to TPV developers comes into play, as some services and the behaviours will start to change to improve throughput and reliability – and may even help improve the SL experience for those on older routers.
As a side note, some of this work has involved router testing aimed at determining what router hardware is compatible with Second Life. While it is hoped that work around the HTTP libraries will improve the SL experience for some using older router hardware as noted above, the tests have revealed that certain types of older router – Linksys WRT and Belkin G series routers were specifically named – are not compatible with running Second Life.
Avatar Baking
Bake fail: a familiar problem for many
Commencing at 32:38 into the recording.
The aim of this work (Project Sunshine) is to improve issues around avatar baking and to eliminate bake fail issues. It will primarily focus on moving the emphasis for the baking process from the viewer to a new Texture Compositing server. The viewer will retain some elements involved in avatar baking – the actual baking of the avatar shape (i.e. shape values and IDs) will still take place on the viewer side, for example.
Precisely how this new service will work on the server-side of things is yet to be fully determined by Linden Lab. However, work is progressing on the viewer side of the equation, with the current key points as follows:
The new service will use the Current Outfit folder to drive the new baking service
TPVs not currently supporting Current Outfits will have to implement it, otherwise they will effectively fail on avatar baking
The basic process will be that when it is time to send a rebake request (e.g. after a user has finished editing their appearance) the viewer must send a new message to the baking service which effectively says, “Look at the contents of my Current Outfit folder and give me back a new appearance based on that”
Viewers in general will have to support this new message that is sent to the service, and change how they perform the fetching of avatar textures; for the technically inclined, this will be HTTP without UDP fallback.
Currently, the plans is for LL to integrating the new way of doing avatar baking into their viewer code, which will be available for TPVs to integrate – although none of the Linden Lab 1.x code will be updated to support the new process, so this will effectively break their own Viewer 1.23.5, which currently is still in use within SL.
The viewer code will support both the “current” method of avatar baking (within the viewer itself) and the new baking service (using the Texture Compositing server) until the new service is fully rolled out across the grid. This means that if a user is in a region that does not make use of the new baking service, avatar baking will continue to be handled using the viewer-side mechanism we currently have. However, if the user is on a region that utilises the new baking service, avatar baking will be handled through that. The viewer will be able to recognise whether it is connected to a region supporting the “new” method through the region capabilities.
In order to ensure as smooth a transition to the new baking process as possible, LL are proposing a relatively long lead-in to the new service, making the code available well ahead of the new service being enabled, allowing TPVs to integrate it into experimental builds. The server-side changes will initially be implemented on a number of beta grid regions for testing with viewers there, prior to being scaled-up. The server changes will then be released onto the main grid in a controlled manner and then scaled up from there.
What Does This Mean for Users?
If all goes according to plan, and providing that you keep up-to-date with releases of your preferred viewer, this actually shouldn’t mean very much in real terms. There are however a number of things to be aware of:
If you use a viewer that is not updated to use the new code (i.e. the official viewer 1.23.5 or a viewer that is not updated to use Current Outfit folder and / or to support the new bake request message / HTTP texture fetch mechanism) OR you continue to use an old version of a viewer rather than updating, there will come a time when your avatar – and those around you – will not bake correctly
There are two issues that may occur during the transitional period when both the “current” and the “new” baking methods are in issue:
When teleporting or crossing between regions that use the different methodologies, users will experience their avatar rebaking, as the viewer will effectively be using two sets of data for the bake process
If there are two adjacent regions, one of which is uses the current avatar bake process and the other is using the “new” baking service viewers in one region will not be able to correctly resolve the textures of avatars in the other region
It is hoped that the transitional period where both methods of avatar baking are active will only last for about two weeks.
Object Caching and Interest Lists
Commencing at 57:25 into the recording.
When you enter a region at the moment, your viewer receives a huge amount of information on what requires updating, much of it relating to things you can’t even see from your position in the region. The data is received in no particular order, with the familiar result that things appear to rez in your view in a totally random order – quite often with the thing you actually want to see being one of the last to rez due to the mechanics of Sod’s Law. What’s more, if you have previously visited the region, the chances are that much of the information being sent to your viewer is already cached.
Object caching and interest list changes: easing the pain of random rezzing
The focus of this project is to optimise the data being sent to the viewer, information already cached on the viewer and the manner in which that data is used in order to ensure it is used more efficiently so that things rez both faster and in a more orderly manner than is currently the case.
At this point in time, this work is in a greater state of flux than the HTTP library and avatar bake projects. This is more a process of optimisation both on the server-side of things and within the viewer itself, rather than that of new functionality within the viewer per se. There are no general time frames for this work at present, but there will be updates once things become clearer as to how the optimisation is going to be addressed.
Time Frames
The precise timeframes for implementing these changes have yet to be properly defined. However, Oz Linden hopes that there will be at least a two month period between Linden Lab making the code for each of these project elements available for integration by TPV developers into their viewers and the point at which the Lab states the code must be in use.
At the moment it is likely that the HTTP library element of the project will but rolled-out first, although this is unlikely to be within the next two months, for the reason given above. Project Sunshine, dealing with avatar baking, will then follow after that – or although how soon after has yet to be determined; as described earlier in this article, this will be a very controlled roll-out. It is possible that the object caching / interest lists part of the project many not be rolled-out for another six months. However, timeframes are still in discussion within LL, so any of this may well change.
Expect updates on all three of these project elements as and when more information is supplied by Linden Lab.
Yesterday Linden Lab announced a major series of new initiatives aimed at improving the overall SL experience. The announcement came via a Tools and Technology blog post, which covers the initiatives in great detail. These focus on four main areas of activity, one of which is directly related to hardware and infrastructure, and the remaining three are focused on the platform itself and are grouped under the Shining project banner.
The hardware / infrastructure element of the work is described thus:
This year, Linden Lab is making the single largest capital investment in new server hardware upgrades in the history of the company. This new hardware will give residents better performance and more reliability. Additionally, we are converting from three co-locations to two co-locations. This will significantly reduce our inter-co-location latency and further enhance simulator performance.
The Shining project is something that is already known to many SL users – especially those who attend some of the User Group meetings. It is perhaps most famously associated with the Lab’s work on the Viewer rendering code, removing outdated functions and calls no longer supported in modern graphics systems (most notably Nvidia) and improving graphics handling overall. Shining has also been responsible for other incremental improvements to issues around streaming objects and avatars.
Under the new initiative, Shining is split into three core performance projects.
Bake fail: a familiar problem for many
Project Sunshine: One of the biggest complaints from users in SL is related to avatar rezzing. This can appear slow, and usually manifests in avatars remaining grey for periods of time, or in skin and system clothes remaining blurry (see right) – and at its worst, result in a user changing their avatar’s outfit – but others either seeing the avatar still dressed in the previous outfit or naked. Collectively, these issues are known as “bake fail” and are the result of the Viewer having to do all the compositing of avatar textures locally, then sending the results to the SL servers, which then send the information back to the simulator the avatar is in to be accessed by other Viewers in the same simulator.
Under Project Sunshine, to precis the blog post, much of this work is moved server-side, using a new, dedicated server, the Texture Compositing Server, which is separate to the simulator servers. This effectively allows all the “heavy” communications and calculations work relating to avatar texture calculations to performed within LL’s servers and across their own internal network, removing the reliance upon the Viewer and on Viewer / server communications which are outside of LL’s control.
Object Caching & Interest Lists: This is intended to directly address another common request from users: improving how the Viewer handles local object caching. This effectively means that once the Viewer has information relating to a specific region, and providing the information is still valid (i.e. there have been no changes to objects that the Viewer already has cached), then it will no longer need to re-obtain that information from the server. Only “new” or “changed” data needs to be streamed to the Viewer. This should mean that on entering a previously visited region, the Viewer should immediately be able to start rendering the scene (rather than requesting a download from the server), while simultaneously requesting any “updates” from the server through a comparison of UUID information and timestamps.
HTTP Library: The final aspect of Shining’s three-phase approach is to improve the underpinning HTTP messaging that are crucial to simulator / simulator and simulator / Viewer communications (and thus key to the other elements of Shining) through the implementation of “modern best practices in messaging, connection management, and error recovery”.
Overall, Shining will be tackling some of the major causes of Viewer-side lag and user frustration in dealing with avatar bake fail and the complexity and wastefulness of scene rendering that is encountered when moving around SL.
No definitive time frames for the improvements have been forthcoming with the announcements – and this is understandable; there’s a lot to be done and matters are complex enough that LL will want to proceed with minimal disruption to the grid and to users. Doubtless, more information will be made available as becomes known through the LL forums and (possibly more particularly) via the relevant User Groups.
Inara Pey: Living in a Modemworld 