SL project updates 2016 51: server, simulator OS update

DRD Arctic Express
DRD Arctic Expressblog post

Server Deployment

While the No Change window was supposed to have come into operation on Friday, December 16th, there was indeed a deployment to the Main (SLS) channel on Tuesday, December 20th. It comprised the same server maintenance package as deployed to the RC channel in week #50, comprising internal logging changes.

The deployment means that all of the server channels on the main grid (Agni) are now running the same simulator version. As there are no planned deployments to the RC channels, all four channels should remain on this release until deployments resume after the holiday period.

SL Viewer

We might see a 64-bit project viewer appear during the week. However, at the time of writing,  the list of viewers in the various pipelines remains as:

  • Current Release version:, dated December 1st, promoted December 5th – formerly the Project Bento RC viewer.
  • Release channel cohorts:
    • Maintenance RC viewer, version, dated December 9th – some 42 fixes and improvements + Bento support
  • Project viewers:
    • 360-degree snapshot viewer, version dated November 23rd – ability to take 360-degree panoramic images – hands-on review
  • Obsolete platform viewer version dated May 8, 2015 – provided for users on Windows XP and OS X versions below 10.7.

Bento Support

With the release of Kokua for Second Life 5.0.0 and RLV 2.9.21 (see my article here), the following currently maintained viewers all have Bento support: Black Dragon, Catznip, Cool VL, Firestorm, Kokua for Second Life, Restrained Love. Alchemy is expected to update to include Bento “soon” and UKanDo is still awaiting a complete overhaul.

In terms of clients with 3D viewing capabilities, Radegast has been updated to support Bento, but the update is pending release. Lumiya for Android will have Bento support added “soon” (Lumiya 3.3 with Voice support released on Sunday, December 18th, and my review is available here).

Simulator OS Update

The Lab is in the process of updating the operating system on the simulator servers. At the moment the new OS version is installed on a number of regions on Aditi (the beta grid), including: Fire Ants 1, Fire Ants 2, Grasmere, Oak Forest, and Twilight Shores, not all of which are open to the public.

“This is one of those ‘features’ that might perform a bit better, but otherwise is a success if it behaves exactly like our current servers,” Simon Linden said at the Simulator User Group meeting on Tuesday, December 20th.

“We’re in the starting process of testing things. We have a lot of testing to do. [We’re] updating from an old version of Linux to a not-so-old version of Linux,” April Linden added. “Just to keep things moving. All the usual reasons. Performance updates, security, etc.”

Second Life hits a lot of low-level server code hard, including networking, memory and multi-processing. The overall hope is to have the newer revisions of the operating system will improve these. Expect this work to reach Agni in early 2017.

Group Chat

Group chat was long a bane of Second Life. In 2014/15 however, the Lab put a considerable amount of work into improving things, although some issues remain. One of these is that if an individual role outside of the default “Everyone” has a very high number of group members assigned to it (e.g. several thousand), it can dramatically impact things like group chat performance and can prevent the members’ list loading. One solution might be to delete the role (converting those in it to Everyone).


SL project updates 2016 50/3: TPV Developer meeting

Mineral Ridge; Inara Pey, December 2016, on Flickr Mineral Ridgeblog post

The notes in this update are taken from the abbreviated TPV Developer meeting held on Friday, December 16th. The video of that meeting is embedded at the end of this update. My thanks as always to North for recording and providing it.

SL Viewer

Current Pipeline

Expect no further viewer promotions to release or RC status until 2017, as the No Change window is now in effect through until January 2nd, 2017. although this doesn’t necessarily prevent project viewers from appearing (see below).

64-bit Builds

[00:07] The Lab is progressing with the 64-bit builds (Project Alex Ivy – which I assume is a reference to 64 in Roman numerals: LXIV = aLeX IVy). Oz reports they are not quite there with the Mac builds as yet, with more work needed on the library builds. However, Whirly Fizzle has already uncovered  some issues with the Windows build, although she reports it as being largely stable through her own testing. Oz indicates there is still “some chance” a project viewer may appear before the holiday break takes full hold.

360-Snapshot Viewer

[05:59] The 360 snapshot viewer remains on hold while development is focused is on the 64-bit viewer builds. As soon as working versions of the latter are available for all the OS platforms, efforts will be switched back to 360-snapshots.

Viewer-side Voice Updates

[01:54] A new Voice update viewer should be available early in the New Year. This should fix a number of bugs and add improved diagnostics. There is currently no time line on when older versions of the Voice package will be blocked from connecting to the service, as “other things” are now seen as having a higher priority.

New Strategy for Rendering Fixes

[16:45] Up until now, fixing for rendering issues have been handled as a part of the Lab’s “standard” methodology for viewer updates: develop a fix for an issue and release it in a Maintenance release candidate viewer. This hasn’t always worked, with some fixes introducing problems of their own (the solid grey / black rendering of invisiprims perhaps being the most visible in recent times), which then prevent other fixes in the same Maintenance RC from progressing while the Lab works on the rendering fixes.

To try to avoid this, the Lab is going to experiment with separating out rendering fixes and moving them to their own release branch of the viewer. This should both prevent other Maintenance fixes and updates from being bottlenecked, but also allow for better QA testing of rendering system fixes / changes.

Sounds and Animations: HTTP and CDN Delivery

[02:22] In 2013 / 2014, the Lab made a huge change to how avatar appearance information and texture and mesh assets are delivered to users, shifting them away from UDP (User Datagram Protocol) delivery through the simulators, to HTTP via Content Delivery Networks (CDNs) – see my past reports on the HTTP updates. and CDN work.

For 2017, the Lab plan to move sound and animation assets (which may or may not include gestures) to delivery via HTTP and thence to CDN distribution and delivery. There is no precise time frame for this work, but once fully implemented (including by TPVs) the UDP  / simulator messaging and routing for these assets will be removed.

While some have experienced CDN related issues with textures and meshes, the hope is that the move will make the delivery of sounds and animations more robust and faster (sounds are reported as being “amazingly slow” to delivery over UDP via the simulator – 10K/second), and remove more of the heavy lifting of assets from the simulators.

Other Items

First TPV Developer Meeting for 2017

[24:26] The December 16th meeting marked the last TPV Developer meeting for 2016. The next meeting will not by until Friday, January 13th, 2017.  This is largely due to the fact that not a lot will have changed between now and the second week of January to make a meeting worthwhile.

Second Life and Oculus Rift

[33:28] In July 2016, Linden Lab suspended development on Oculus Rift support in the viewer, but left the door open a crack for the potential for the work to be picked-up at some point in the future. However, as things stand with the current generation of headsets, this is not going to happen in the foreseeable future.

The belief is that the rendering requirements – particular frame rates sit well above those which can be reasonably achieved in Second Life through the viewer (Oculus VR quotes a minimum of 60 fps and a preferred rate of 90 fps, which the Lab sees as being doubled to 120 and 180 fps when rendering an SL scene in stereo).

Firestorm 5.0.1 Download Issues

[20:16] Jessica Lyon gave further insight into the recent issues with people trying to obtain the Firestorm 5.0.1 Bento release (see my review here, and subsequent updates on the download situation here and here).

While there were some indications demand would be high – the Firestorm Preview group had expanded to over 8,000 people – it nevertheless outstripped all expectations, and the Firestorm download server almost came to a standstill. To try to correct this, the server was restarted, driving traffic back to the web server in the process, which then overloaded and crashed.

Firestorm hope that with Bento out the door, the viewer can resume a more “normal” QA / release cycle through the Preview and Beta groups. However, a mirror site for new release downloads will be maintained going forward. Options for hosting the Firestorm JIRA service, considered a major resource hog on the FS servers, are also being considered, although there are pro and cons involved in making changes.

Jess also took the opportunity to again thank Linden Lab for stepping up and providing a mirror for downloads.

SL project updates 2016 50/2: re-caps and Content Creation UG

Fris' Land, Liebe; Inara Pey, December 2016, on Flickr Fris’ Land, Liebeblog post

Server Deployments – Recap

As always, please refer to the server deployment thread for the latest updates and information.

  • On Tuesday, December 13th, the Main (SLS) channel received the same server maintenance package, as deployed to the RC channels in week #49. This includes the following feature requests: BUG-6377 – llGetObjectDetails(id,[OBJECT_ATTACHED_SLOTS_AVAILABLE])  and BUG-40871 – llGetEnv() constant “region_object_bonus”.
  • On Wednesday, December 14th, all three RC channels received the same new server maintenance package, comprising improved internal server logging.

SL Viewer

The official viewers list remains unchanged from earlier in the week:

  • Current release viewer, version, dated December 1, promoted December 5 – formerly the Project Bento RC viewer download page, release notes
  • Maintenance RC viewer, version, dated December 9th
  • 360-degree snapshot viewer, version, dated November 23rd.
  • Obsolete platform viewer, version, dated May 8th, 2015 – provided for users on Windows XP and OS X versions below 10.7.

No Change Window

A reminder that the end-of-year No Change window comes into effect from Friday, December 16th. There will be no further planned server deployments or official viewer promotions after that date, through until Monday, January 2nd, 2017.

Content Creation User Group

On Thursday, December 15th, the Project Bento User Group officially morphed into the Content Creators User Group (thus in a way  also now standing as a revamp of Nyx Linden’s old group).

Meetings will continue to take place at 13:00 SLT on Thursdays, at the Hippotropolis camp fire circle, with the exceptions of Thursday, December 29th and Thursday, January 5th. Anything related to Bento remains on topic for the group, but things of general interest to content creators are also open for discussion.

New Bento Issues

Bento is generating considerable interest among users – as witnessed by the release of Firestorm 5.0.1, when demand overwhelmed the Firestorm servers, and additional mirror services had to be set-up, with assistance from the Lab in doing so. However, some further issues have been identified as a result of Bento reaching a broader audience:

  • BUG-41063 – [Bento] Some Fitted Mesh No Longer Fits on Bento Avatars
  • BUG-41048 – [Bento] When wearing some Bento content, all walking animations freeze

The first of these bugs has been accepted by the Lab and is being investigated. The second appears to be most notably related to animations using the Bento wing bones, which appear to be overriding animation walks, causing an avatar to slide along the ground, rather than walking.

The issue particularly seems to impact walk animation using the llSetAnimationOverride capability, while older ZHAO systems (which tend to use the llPlayAnimation capability, appear to be unaffected. This may be due to the manner in which the latter handle start / stop animations calls, which is successfully preventing the issue manifesting with them.

The underlying cause is thought to be a possible animation priority setting / conflict or possibly the result of the wing bone animations having bone positions being set for all other, but not rotation data – but further investigation is required. Vir has taken samples of items which can cause the issue (again, notably, but not necessarily limited to, wings), and is going to have a poke at them to see if he can see what is going on.

Elsewhere, creators are waiting on Avastar to update, and Damien Fate has a rather neat HUD to help animate Bento hands and fingers. It’s unlisted, so I’m linking to it, rather than sharing it via embedding.

Fun Fact

The new Linden Lab portal parks have a LI capacity of 45,000 (with thanks to Cube Republic and Arton Rotaru for the pointers).

Portal Park LI capacity
Portal Park LI capacity




SL project updates 2016 50/1: Server, viewer, no change window

North Pole at Alki, Alki; Inara Pey, December 2016, on Flickr “Hand over your coal and carrots, and don’t try anything funny. This hair-dryer is plugged in, and I know how to use it!” North Pole at Alki, Alkiblog post

Server Deployments

As always, please refer to the server deployment thread for the latest updates and information.

  • On Tuesday, December 13th, the Main (SLS) channel received the same server maintenance package, as deployed to the RC channels in week #49. This includes the following feature requests:
    • BUG-6377 – llGetObjectDetails(id,[OBJECT_ATTACHED_SLOTS_AVAILABLE]) – Returns a value that is number of attachment slots allowed by the server minus the number of attachments worn by avatar. Returns 0 if avatar is not in the same region or if UUID is not an agent.
    • BUG-40871 – llGetEnv() constant “region_object_bonus” – returns the object bonus set for a region.
  • On Wednesday, December 14th, all three RC channels should receive the same new server maintenance package, comprising improved internal server logging.

SL Viewer

The Maintenance RC viewer updated to version, bringing it to parity with the current release viewer, incorporating the Bento updates. This leaves the remaining viewers in the pipelines unchanged from week #48:

  • 360-degree snapshot viewer, version, dated November 23rd.
  • Obsolete platform viewer, version, dated May 8th, 2015 – provided for users on Windows XP and OS X versions below 10.7.

No Change Window

The Christmas and New Year 2016/17 No Change window comes into effect from Friday, December 16th. This means there will be no further server deployments and no further official viewer promotions after that date, through until Monday, January 2nd, 2017.


Mesh and Texture Rezzing

As noted in my week #49 update, people have been noticing increased delays in object mesh and texture rezzing, with fingers being pointed at the Lab’s CDN supplier(s).  The issues are continuing for some, while for others they appear to have cleared up. It’s still not obvious if it is a potential LL / CDN issue or a network problem in general.

As Simon Linden said in the meeting, the problems have been seen by the Lab, but are proving to be intermittent and hard to pin down. The problem also seems to manifest differently for people: some report very slow texture rendering, other report textures load and render fine, but mesh items are prone to failing to fully render, others report a mix of the two.

Region Crossings

Region crossings have been widely reported as increasing again – including avatars being dumped at 0,0,0 (again). Some are reporting the issue as cumulative: the more regions they cross, the greater the likelihood they’ll encounter a serious issue (becoming unseated, return of vehicle, forced log-out). The usual advice is being circulated – reduce script load, handle crossings with caution, etc. However, the Lab are again aware of the uptick, but have not come to any specific conclusion on the cause.

The avatar appearing at a region’s 0,0,0 co-ordinates appears to be linked to connection problems and / or UDP packet loss (usually the first packet) resulting in messages arriving in the wrong order and the viewer and simulator falling out of synch with one another.

With both the rendering issues and the region crossing issues, fingers have been directed at the increased land impact allowances. This may be a case of post hoc, ergo propter hoc. In particular, it has been claimed that region crossings “became” bad after the mainland LI allowance increase, although people were reporting issues before the LI increase took place.

Appearance Issues / Bake Fails

Some are finding their avatar is failing to render in their view (bake fail) when logging-in. Changing outfits, camming away / back to your avatar may fix this, or a re-log. In extreme cases reverting to the default Character Test avatars may be required, or deleting and recreating the affected avatar appearance folder on your hard drive.

Rider Linden is looking into the problem, and believes it may be due to a missed inventory fetch at log-in, leaving the viewer thinking you’re missing a critical part of your appearance (e.g. your shape), but he is not 100% certain at this point in time.