Server Deployments – week 9
Server Life Server (Main) Channel
As noted in part 1 of this report, the Second Life Server (SLS / Main channel) received the server maintenance project which was on all three RC channels in week 8, comprising miscellaneous bug fixes, and the improved region restart notification. Deployed on Tuesday 26th February, the roll-out was followed by a number of issues being reported via the deployment forum thread.
Issues have included:
- Parcels / regions vanishing from search for a period of time following the restart (an issue WolfBaginski Bearsfoot gave some thought to in commenting on part 1 of this report)
- Further region crossing issues, with loss of control followed by “broken” camera positioning after recovery. This issue has been reported at some of the Simulator and Server Beta meetings previously, and Eric Gregan has produced a video demonstrating the problem as it occurs on some aircraft, although he’s also come up with a means of avoiding the issue which may or may not work for people.
- Problems with Dwarfins attachments.
Release Candidate Channels
On Wednesday February 27th, all three Release Candidate channels received a new server maintenance package which includes a fix for a crash mode (see the release notes (BlueSteel)). No significant issues have been reported for this deployment.
The Communications Hub User Interface finally made its debut in both the SL development and SL beta servers, arriving as release 22.214.171.1240826 for the development viewer on Tuesday 26th February, and 126.96.36.1990825 for the beta viewer on Wednesday 27th. I’ve provided a brief introduction to CHUI, and there is also a video from Torley Linden.
CHUI is liable to remain in beta for a “Good long run”, to quote Oz Linden. Hopefully, this may mean that materials processing will be the next significant viewer update to arrive as a project viewer.
The recent updates to Andrew Linden’s interest list work are apparently won’t reach a RC channel in week 10, but should see deployment in week 11 (week commencing Monday 11th March). This work includes fixing issues with avatar appearance teleporters which use llSetRegionPos(), as well as correcting problems where objects which change appearance behind the camera ‘snapping’ into place when you rotate the camera back.
On Friday March 1st, Linden Lab unexpectedly released an updated version of the SSB project viewer (release 188.8.131.521118), which includes their approach to overcoming the problems of the Z-offset capability common to many TPVs being broken as result of the SSB code. The new approach, which I was able to outline when it “launched”, introduces an additional shape slider into the Edit Shape floater; as such, the ease-of-use of the approach, particularly for those who may swap between different avatar shapes (e.g. “normal” and petite) is very questionable.
HTTP Project Update
The major part of the Server Beta meeting on Thursday February 28th was taken up by Monty Linden describing his work on the HTTP 1.1 project. The core details of this work can be found in the SL wiki, in the Server Beta group notes.
For the last several months Monty has been working on a number of aspects of improving / providing HTTP services within SL (an initial release of this work was made last year, with improved HTTP-based texture fetching). He has now reached a point where he’s ready to open-up this work to broader testing on Aditi using a number of (still to be established at the time of writing) test regions which will support different configurations of the new capabilities for more rigorous and broader user testing than LL can undertake. As such, Monty attended the Server Beta meeting to provide both an overview and some technical insight into the various aspects of the work.
The following is intended to provide a broad overview of the project without diving too deeply into the technical aspects. For those interested in reading his more technical comments on the work, please refer to the meeting transcript, which should be posted in the Server Beta meetings archive shortly.
Essentially, there are two aspects to the work Monty has been carrying out. The first is to provide better HTTP headers coming from the mesh and texture asset services; the second, much deeper effort, is to improve the overall robustness of user connections to the SL services through the implementation of HTTP persistent / keepalive connections capabilities.
Discussing the improvements this work should provide for activities such as texture fetching from the servers, Monty noted that in test simulations, the time required for a 3.4.3-based viewer in Boston to download 6500 textures “cold” (i.e. nothing pre-cached in the viewer) from the region server was reduced from 180s to 120s using the new capabilities. While he pointed out that not everyone will see such a dramatic improvement once the new services go live, it is hoped there will still be noticeable improvements for most users when downloading / rendering textures. It is also hoped the work might see an improvement to inventory downloads, although Monty did suffix this point with a “(maybe)”.
In terms of the keepalive work, the hope is that it will result in a system which is more robust than the current connection mechanisms, and which at the very least is “no worse” than the current connection capabilities.
One focus of the keepalive work is to improve managing the number of parallel requests made between the viewer and Second Life. During his ongoing testing of router hardware, which he’s been performing for a good while, Monty found that in certain routers – such as the Belkin wireless G series and Linksys WRT54G range, the firmware would lock the router for up to 5 minutes at a time if too many parallel HTTP requests were made. So the hope is that in providing improved request / connection management, those using less capable routers should see an improvement in their ability to stay connected to SL.
One limitation with improving connection management is that the new services are still built on libcurl, which Monty points out as not having a good model for connection management. As such, this may have some impact on plans the Lab has for further improvements to their HTTP services in the future, such as pipelining. As such, Monty is considering longer-term options for “fixing” libcurl, or developing a workaround solution of the problems or replacing it altogether.
Monty also noted that there are some limitations to the keepalive work at present. He specifically noted that it will not initially support mesh, which will continue using the existing server / viewer services. The reason for this, he explained, is that mesh tends to, “Shotgun the network badly” in terms of connections, so more work is required if it is to be supported effectively. Similarly, HTTP-In functions within LSL, such as llRequestURL, will continue using the current capabilities. He also noted that the Content Length header field is currently disabled in the code, as it triggers a bug in LL’s pre-3.4.3 viewer code.
In terms of the testing work, Monty’s plan is to have a number of regions set-up on Aditi with various configurations of the new capabilities, which can be used TPVS and scripters and those with an interest in the new capabilities, to both test them and provide him with further metrics. He’s particularly keen to have those who use external HTTP services to participate, noting that, “They’ll want to test robustness of communication between their scripts running HTTP servers in-world and external HTTP clients.” So, for example, those running combat systems using HTTP protocols should consider joining-in the testing once the Aditi regions are configured and running.
There is no timescale for the testing period at present – as mentioned, the various regions have yet to be set-up or if they will continue to use the project channel Monty has been using for internal testing (DRTSIM-203). Monty also sounded a note of caution as to the testing both on Aditi and going forward to a main grid RC at some point in the future. Aditi itself isn’t adequate for comprehensive load testing, which will have to be performed on a main grid RC channel, so there is a risk that this work, in whole or in part, may fail to adequately scale across the main grid and require much more work in the future.
Nevertheless, should the new capabilities both function as Monty hopes and prove they can scale across the main grid safely, they should lead to see some significant benefits. Not all of these may be readily apparent to users (as they are, as Monty puts it, “behind the walls”), but they should lead to an overall improvement in user experience.
As mentioned in the first part of this report, the good news is that LL have completed maintenance on Aditi which should help ease / resolve most inventory syncing issues people had been experiencing,
The bad news is that there is currently a hardware fault on the Aditi inventory hardware, which has resulted in around 40% of those with active Aditi accounts being unable to log-in to the beta grid. This problem is still being sorted out.
Even Linden Lab staff were not immune from the problem – Monty Linden was forced to attend the Server Beta meeting on Thursday 28th February using a test alt!