SL project update: week 9 (2): servers, HTTP, SSB

Server Deployments – week 9

Server Life Server (Main) Channel

As noted in part 1 of this report, the Second Life Server (SLS / Main channel) received the server maintenance project which was on all three RC channels in week 8, comprising miscellaneous bug fixes, and the improved region restart notification. Deployed on Tuesday 26th February, the roll-out was followed by a number of issues being reported via the deployment forum thread.

Issues have included:

  • Parcels / regions vanishing from search for a period of time following the restart (an issue WolfBaginski Bearsfoot gave some thought to in commenting on part 1 of this report)
  • Further region crossing issues, with loss of control followed by “broken” camera positioning after recovery. This issue has been reported at some of the Simulator and Server Beta meetings previously, and Eric Gregan has produced a video demonstrating the problem as it occurs on some aircraft, although he’s also come up with a means of avoiding the issue which may or may not work for people.

  • Problems with Dwarfins attachments.

Release Candidate Channels

On Wednesday February 27th, all three Release Candidate channels received a new server maintenance package which includes a fix for a crash mode (see the release notes (BlueSteel)). No significant issues have been reported for this deployment.

SL Viewer

The Communications Hub User Interface finally made its debut in both the SL development and SL beta servers, arriving as release for the development viewer on Tuesday 26th February, and for the beta viewer on Wednesday 27th. I’ve provided a brief introduction to CHUI, and there is also a video from Torley Linden.

CHUI is liable to remain in beta for a “Good long run”, to quote Oz Linden. Hopefully, this may mean that materials processing will be the next significant viewer update to arrive as a project viewer.

Interest List

The recent updates to Andrew Linden’s interest list work are apparently won’t reach a RC channel in week 10, but should see deployment in week 11 (week commencing Monday 11th March). This work includes fixing issues with avatar appearance teleporters which use llSetRegionPos(), as well as correcting problems where objects which change appearance behind the camera ‘snapping’ into place when you rotate the camera back.

Server-side Baking

On Friday March 1st, Linden Lab unexpectedly released an updated version of the SSB project viewer (release, which includes their approach to overcoming the problems of the Z-offset capability common to many TPVs being broken as result of the SSB code. The new approach, which I was able to outline when it “launched”, introduces an additional shape slider into the Edit Shape floater; as such, the ease-of-use of the approach, particularly for those who may swap between different avatar shapes (e.g. “normal” and petite) is very questionable.

The new "Hover" option in the Edit Shape panel for adjusting avatar height offsets in the Sunshine project viewer. Not the most elegant solution
The new “Hover” option in the Edit Shape panel for adjusting avatar height offsets in the Sunshine project viewer. Not the most elegant solution

HTTP Project Update

The major part of the Server Beta meeting on Thursday February 28th was taken up by Monty Linden describing his work on the HTTP 1.1 project. The core details of this work can be found in the SL wiki, in the Server Beta group notes.

Monty Linden in a more familiar guise, having been forced into using an alt for the Server Beta meeting
Monty Linden in a more familiar guise, having been forced into using an alt for the Server Beta meeting (see Aditi Update, below)

For the last several months Monty has been working on a number of aspects of improving / providing HTTP services within SL (an initial release of this work was made last year, with improved HTTP-based texture fetching). He has now reached a point where he’s ready to open-up this work to broader testing on Aditi using a number of (still to be established at the time of writing) test regions which will support different configurations of the new capabilities for more rigorous and broader user testing than LL can undertake. As such, Monty attended the Server Beta meeting to provide both an overview and some technical insight into the various aspects of the work.

The following is intended to provide a broad overview of the project without diving too deeply into the technical aspects. For those interested in reading his more technical comments on the work, please refer to the meeting transcript, which should be posted in the Server Beta meetings archive shortly.

Essentially, there are two aspects to the work Monty has been carrying out. The first is to provide better HTTP headers coming from the mesh and texture asset services; the second, much deeper effort, is to improve  the overall robustness of user connections to the SL services through the implementation of HTTP persistent / keepalive connections capabilities.

Discussing the improvements this work should provide for activities such as texture fetching from the servers, Monty noted that in test simulations, the time required for a 3.4.3-based viewer in Boston to download 6500 textures “cold” (i.e. nothing pre-cached in the viewer) from the region server was reduced from 180s to 120s using the new capabilities. While he pointed out that not everyone will see such a dramatic improvement once the new services go live, it is hoped there will still be noticeable improvements for most users when downloading / rendering textures. It is also hoped the work might see an improvement to inventory downloads, although Monty did suffix this point with a “(maybe)”.

In terms of the keepalive work, the hope is that it will result in a system which is more robust than the current connection mechanisms, and which at the very least is “no worse” than the current connection capabilities.

Keepalive capabilities for HTTP connections a focus of Monty Linden's work
Keepalive capabilities for HTTP connections a focus of Monty Linden’s work (click to enlarge)

One focus of the keepalive work is to improve managing the number of parallel requests made between the viewer and Second Life. During his ongoing testing of router hardware, which he’s been performing for a good while, Monty found that in certain routers – such as the Belkin wireless G series and Linksys WRT54G range, the firmware would lock the router for up to 5 minutes at a time if too many parallel HTTP requests were made. So the hope is that in providing improved request / connection management, those using less capable routers should see an improvement in their ability to stay connected to SL.

One limitation with improving connection management is that the new services are still built on libcurl, which Monty points out as not having a good model for connection management. As such, this may have some impact on plans the Lab has for further improvements to their HTTP services in the future, such as pipelining. As such, Monty is considering longer-term options for “fixing” libcurl, or developing a workaround solution of the problems or replacing it altogether.

Monty also noted that there are some limitations to the keepalive work at present. He specifically noted that it will not initially support mesh, which will continue using the existing server / viewer services. The reason for this, he explained, is that mesh tends to, “Shotgun the network badly” in terms of connections, so more work is required if it is to be supported effectively. Similarly, HTTP-In functions within LSL, such as llRequestURL, will continue using the current capabilities. He also noted that the Content Length header field is currently disabled in the code, as it triggers a bug in LL’s pre-3.4.3 viewer code.

In terms of the testing work, Monty’s plan is to have a number of regions set-up on Aditi with various configurations of the new capabilities, which can be used TPVS and scripters and those with an interest in the new capabilities, to both test them and provide him with further metrics. He’s particularly keen to have those who use external HTTP services to participate, noting that, “They’ll want to test robustness of communication between their scripts running HTTP servers in-world and external HTTP clients.” So, for example, those running combat systems using HTTP protocols should consider joining-in the testing once the Aditi regions are configured and running.

There is no timescale for the testing period at present – as mentioned, the various regions have yet to be set-up or if they will continue to use the project channel Monty has been using for internal testing (DRTSIM-203). Monty also sounded a note of caution as to the testing both on Aditi and going forward to a main grid RC at some point in the future. Aditi itself isn’t adequate for comprehensive load testing, which will have to be performed on a main grid RC channel, so there is a risk that this work, in whole or in part, may fail to adequately scale across the main grid and require much more work in the future.

Nevertheless, should the new capabilities both function as Monty hopes and prove they can scale across the main grid safely, they should lead to see some significant benefits. Not all of these may be readily apparent to users (as they are, as Monty puts it, “behind the walls”), but they should lead to an overall improvement in user experience.

Aditi Update

As mentioned in the first part of this report, the good news is that LL have completed maintenance on Aditi which should help ease / resolve most inventory syncing issues people had been experiencing,

More Aditi woes
More Aditi woes

The bad news is that there is currently a hardware fault on the Aditi inventory hardware, which has resulted in around 40% of those with active Aditi accounts being unable to log-in to the beta grid. This problem is still being sorted out.

Even Linden Lab staff were not immune from the problem – Monty Linden was forced to attend the Server Beta meeting on Thursday 28th February using a test alt!

Related Links

5 thoughts on “SL project update: week 9 (2): servers, HTTP, SSB

  1. You do such a good job explaining this stuff. I still don’t understand half of it but that is me, not you. I don’t know how people have enough time to stay current on everything…tech, events, blogs… and still have time to have fun inworld. I know I don’t stay current or spend enough time inworld. Anyway, thanks for your continued service in writing about these things.


    1. Thanks 🙂

      tbh, I’m far from a techie, and this post was on the late side as I had to do some reading-up to try and get my head around everything that was said (often the case with me), and still not 100% sure I’ve encapsulated everything – hence the little caveat at the start of the HHTP section. Hopefully, as the HTTP project develops and I read more / talk to others in more detail, I’ll be able to summarise more clearly. As to staying current – I miss a lot, but I’m lucky enough to have a lot of people forwarding event info, suggestions for places to visit, etc. Not all of it makes it into the blog (or some destinations can take a while to appear), but the support I get is greatly appreciated as well.


  2. The preoblem with height offsets is beginning to look like another example of the Open Source idea failing at Linden Labs. TPVs noticed a problem, and came up with a fix, involving a signal the viewer could send to the sim. We have to assume that somebody told the Labs about the problem, but if it’s in the JIRA we shall never know.
    Time passes.
    The Labs start the SSB thing to change how the texture displayed on an Avatar is generated and distributed. There’s no obvious reason why that should affect the height-offset, but it’s possible that it changes the signalling behaviour that the TPVs depended on. Isn’t there anyone at the Labs keeping track? And, lets be honest, who would expect the different baking method to have any effect on height offsets.
    There is an inevitable need to conform on SSB: the TPVs are going to have to implement it, or be useless in SL. And it’s not so hard to see this as a part of a process of building barriers between SL and OpenSim. But the key point here is that Linden Labs seems willing to break widely-used TPV features, and not to say anything about the change until somebody calls them on it. And then their own flawed fix appears with remarkable speed.
    It’s one thing not to accept the usual Open Source legal framework, and require specific contracts. One wonders how much their lawyers know about the software business, but it isn’t a killer. Affairs such as this one give the impression of corporate America flying false colours as they close on a hostile shore, ready to drop the Open Source flag as they run out their guns. And, ooh look, the halyards have jammed. Fire when ready, Mr, Gridley.


    1. Calm down. The new solution to add it into the shape isn’t bad at all. After all, it does mostly depend on the avatar shape, nothing else (changing shape and then changing slider would be awkward). Not working on nomod shapes may be annoying, though I don’t think that shapes should be nomod (honestly, I’d to adjust/personalize stuff all the time, esp. in the face (if human avatar)). While I don’t know, how the TPVs actually transmitted the offset information, I do suspect that it was a bit of a hack. The Lab is by no means required to support that feature in that incarnation (and neither to test their projects against it).. and now everyone is going to enjoy it.
      It will doubtlessly work as a barrier between SL and OpenSim, which already should be in place due to the havok licencing, iirc. Intentionally? Probably not.


  3. Cross sims on mainland using any type of vehicle are all screwed!
    Corsica, Sansara, Zindra, name a continent where we have homes and used to be able to cross more then 50 sims without a crash, now is impossible to do, you cross 1 sim and float in the air for 45 sec then camera is screwed, then you corss 1 more and again same thing!


Comments are closed.