2018 SL UG updates #14/1: Simulator User Group

Bay of Dreams; Inara Pey, February 2018, on FlickrBay of Dreamsblog post

Server Deployments

As always, please refer to the server deployment thread for the latest updates.

  • The Main (SLS) channel was updated on Tuesday, April 3rd  to server release 18#18.03.27.513831, containing internal fixes and updates the new off-line IM and Abuse Report capabilities (see below).
  • On Wednesday, April 4th, the BlueSteel and LeTigre release candidate channels should receive server maintenance package 18#18.03.29.513939, containing internal fixes; the Magnum RC has no planned deployment, and remains on 18#18.03.27.513831.

SL Viewer

There have been no changes to the viewer pipelines at the start of the week, leaving them as:

  • Current Release version 5.1.2.512803, dated February 23, promoted March 1 – formerly the Nalewka Maintenance RC – No change
  • Release channel cohorts:
  • Project viewers:
    • Bakes on Mesh project viewer, version 5.1.3.513936, March 30 (note this version of the viewer does not support applying bakes to mesh bodies / heads).
    • Animesh project viewer, version 5.1.3.513013, March 7 – project overview.
    • 360 snapshot viewer, version 5.1.3.513006,  March 6 (this version can have significant rendering issues, see my hand-on update).
  • Linux Spur viewer, version 5.0.9.329906, dated November 17, 2017 and promoted to release status 29 November – offered pending a Linux version of the Alex Ivy viewer code.
  • Obsolete platform viewer, version 3.7.28.300847, May 8, 2015 – provided for users on Windows XP and OS X versions below 10.7. This viewer will remain available for as long as reasonable, but will not be updated with new features or bug fixes.

Region Crossings

As I’ve reported in recent Simulator User Group meetings, Joe Magarac (animats) working on a scripted workaround to improve region crossings (“partial unsits, etc.) – see BUG-214653. The Lab is also working on crossings on the simulator side. While no time frames for the work were given, Oz Linen did observe, “How successful we’ll be remains to be seen, but we’re optimistic that we can make significant improvements.”

2018 SL UG updates #13/1: Simulator User Group

Soul2Soul River; Inara Pey, February 2018, on FlickrSoul2Soul Riverblog post

Server Deployments

As always, please refer to the server deployment thread for the latest updates.

  • The Main (SLS) channel was updated on Tuesday, March 27th, to server release 18#18.03.14.513292, containing the new server capabilities (see below).
  • At the time of writing, the Release Candidate channels were all TBD regarding potential deployments. This report will be updated if the deployment thread provides further information on the RC channels.

New Capabilities

The new capabilities in 18#18.03.14.513292 for the Main (SLS) channel is the first part of a set of server and viewer updates.

  • The new IM cap is to overcome of off-line IMs failing to be delivered when a user logs in. Currently, these are delivered via UDP, whether or not the viewer is ready to receive them. With the new capability (once grid-wide and implemented within the viewer), the viewer will request off-line IMs, which the server will package and deliver to the viewer via HTTP.
  • The new abuse report cap will replace the need for the viewer to have AR categories hard-coded into it. Once fully deployed, and a viewer update released, it will mean the view will request the current list of AR categories from the server when starting up, making the management of the list easier, and hopefully reducing the number of ARs filed under outdated categories.

Updates to the viewer incorporating these changes will be made available by the lab in the near future.

SL Viewer

  • The Maintenance RC viewer updated to version 5.1.3.513630 on Friday, March 23rd.
  • The Media Update RC viewer updated to version 5.1.3.513644, on Tuesday, March 27th.

The remainder of the pipeline remains as:

  • Current Release version 5.1.2.512803, dated February 23, promoted March 1 – formerly the Nalewka Maintenance RC – No change
  • Release channel cohorts:
  • Project viewers:
  • Linux Spur viewer, version 5.0.9.329906, dated November 17, 2017 and promoted to release status 29 November – offered pending a Linux version of the Alex Ivy viewer code.
  • Obsolete platform viewer, version 3.7.28.300847, May 8, 2015 – provided for users on Windows XP and OS X versions below 10.7.

In Brief

First Name / Last Name Changes

This is still a long way off from being implemented, however, Oz Linden confirmed llDetectedName() will return the current name for an avatar, no matter what the change. However, it may take some time for it to change everywhere due to caches.

More on the return of last names and name changes, please refer to The return of Second Life Last Names – update with audio.

SL Messaging Layer

Simon Linden is looking into the Second Life messaging layer, which may be the problem behind a lot of “lag” issues. “There’s actually a number of small improvements I want to make, but I’m being careful to do them one at a time and have real data showing it gets better,” he said in providing an update on the work.

Friendship Offers Failing

Some are experiencing Friendship offers failing, even when the offer is accepted – see BUG-215977. According to Simon Linden, this might require a server-side update to fully correct.

Region Crossings

Simon Linden has been looking at vehicle region crossings alongside of  Joe Magarac (animats) testing with the viewer (See Firestorm JIRA FIRE-21915, BUG-214653, this SL Forum thread, this Google document, and my update here for more).

Part of the issue, a previously noted, is viewer / simulator communications. If these are suffering latency or packet loss, then things can get rough with vehicle region crossing very quickly. This is something Joe has been trying to compensate for by introducing a script that turns off physics and freezes the vehicle when received by a new region until it can confirm the associated avatar data has arrived.

Unfortunately, excising the viewer from region crossing data handling would be difficult, as it has to be involved to move and change its primary connection for an avatar. It would take a major protocol change to remove the viewer from the region crossing loop and separate connection hand-off from crossings. Further, if such a protocol change were to be made, it would require more work to support both new and old until enough viewers get updated.

Mainland Price Restructuring

While the Lab does not issue numbers, Oz Linden indicated at the Simulator User Group meeting that since the Mainland Price Restructuring, “mainland ownership is up quite significantly.”

2018 SL UG updates #12/1: Simulator User Group

Sanctuary; Inara Pey, February 2018, on FlickrSanctuaryblog post

Server Deployments

As always, please refer to the server deployment thread for the latest updates.

  • There was no deployment / restart to the Main (SLS) channel on Tuesday, March 6th, leaving it on server maintenance package 18#18.02.12.512536.
  • The main RC channel should be updated as follows on Wednesday, March 21st:
    • The BlueSteel RC should receive a new server maintenance package. 18#18.03.17.513365 is said to comprise “internal fixes”.
    • The LeTigre and Magnum regions should receive a new server maintenance package. 18#18.03.14.513292, comprising internal fixes, additional logging to help diagnose an issue with in-world HTTP servers returning HTTP 503 (BUG-214702); add two new capabilities: one to request IMs that were delivered to the requesting agent while the agent was off-line; one to request the most up to date list of abuse report categories.

New Capabilities

The new capabilities in 18#18.03.14.513292 for Magnum and LeTigre are the first part of a set of server an viewer updates.

  • The new IM cap is to overcome of off-line IMs failing to be delivered when a user logs in. Currently, these are delivered via UDP, whether or not the viewer is ready to receive them. With the new capability (once grid-wide and implemented within the viewer), the viewer will request off-line IMs, which the server will package and deliver to the viewer via HTTP.
  • The new abuse report cap will replace the need for the viewer to have AR categories hard-coded into it. Once fully deployed, and a viewer update released, it will mean the view will request the current list of AR categories from the server when starting up, making the management of the list easier, and hopefully reducing the number of ARs filed under outdated categories.

SL Viewer Updates

There have been no updates to the current selection of official viewer at the start of the week, leaving the pipelines as follows:

  • Current Release version 5.1.2.512803, dated February 23rd, promoted March 1st – formerly the Nalewka Maintenance RC.
  • Release channel cohorts
  • Project viewers:
  • Linux Spur viewer, version 5.0.9.329906, dated November 17th, 2017 and promoted to release status 29 November – offered pending a Linux version of the Alex Ivy viewer code.
  • Obsolete platform viewer, version 3.7.28.300847, May 8th, 2015 – provided for users on Windows XP and OS X versions below 10.7.

Region Crossings

As noted over the last few weeks, user Joe Magarac (animats) has been digging into region crossings, particularly looking at the issue of avatars getting unseated from vehicles (See Firestorm JIRA FIRE-21915, this SL Forum thread, and this Google document, plus my last update here for more).

His latest approach involves adding a script to a vehicle effectively turns off physics and freezes the vehicle when received by a new region until it can confirm the associated avatar data has arrived (using llGetObjectDetails (avatar, [OBJECT_ROOT]) to check for the presence of avatars). On confirming the avatars are present, the script check the vehicle is allowed to resume.

“The typical freeze time is 40-120ms in addition to the regular region crossing delay, so you barely notice.” Joe told me in discussing the approach, which he has now made available to those wishing to test it using a simple motorbike build. “For double region crossings, the vehicle should slow down, although not by too much. I’ve also tried the same script in aircraft, and it seems to work.”

Joe continued, “Right now, I want to see if it ever fails. Issues are logged to an external server, so I can see what’s going on. I’d also love to get some of the SL motorcycle clubs involved in testing , so I’m giving out these test bikes. However, if anyone wants to test the script with their own vehicles, they can contact me for a copy, and I’ll be happy to work with them.”

Again, this workaround isn’t intended to solve all region crossing issues (e.g. it won’t stop “rubber banding” or address motion prediction – which the Lab sees as probably too aggressive), but it should help with full or partial avatar unsits; so those interested in given it a go are welcome to drop Joe a line.

Other Items

SL Messaging Layer

“I’m looking at the messaging layer; that seems to be the problem behind a lot of laggy things,” Simon Linden stated during the SUG meeting, revealing some of his recent work.

“There’s been a lot of learning. That code is complicated and there’s actually a lot of wisdom written into it, but I’ve found it doesn’t really do the best things when packet loss increases. The most basic thing I’ve found is that the estimated times based on pings used for time-outs can grow too fast: like on a 100ms connection, it can jump up to using 5-10 seconds. There’s a balancing act between resending data, and thinking it needs tons of time to get there.

“It really goes bad if a ping packet or the reply gets lost. SL just uses the same time-out for each retry – that’s another part that could be smarter.”

Defining this as an “interesting” part of the code, he’s reasonably certain it could be doing things better – and in doing so, could also help with region crossings and overall updates.

He also noted that also the SL bandwidth code is broken. This appears largely due to the volume of traffic going over the HTTP caps and via the CDN(s), which isn’t counted. “So I’m pretty sure we clog up a narrow pipeline more than we should,” he added.

2018 SL UG updates #11/1: Simulator User Group

Kamigami, Pandora Resort Town; Inara Pey, February 2018, on Flickr Kamigami, Pandora Resort blog post

Server Deployments

As usual, please refer to the server deployment thread for the latest updates.

  • There was no deployment to the Main (SLS) channel on Tuesday, March 13th, leaving it on server version 18#18.02.12.512536. According to the deployment notes, the channel should have been restarted.
  • On Wednesday, March 14th, the main RC channels should be updated as follows:
    • TBC: no deployment to BlueSteel, leaving it on server version 18#18.03.05.513046.
    • LeTigre and Magnum should both receive a new server maintenance package, 18#18.03.12.513258, comprising internal fixes and further simulator logging improvements.

SL Viewer

Updates

  • The Media Update RC viewer updated to version 5.1.3.513038, on Monday, March 12th.
  • A new Maintenance RC viewer was release on Tuesday, March 13th. Version 5.1.3.513234, code-named Ouzo, after the anise-flavoured aperitif. This viewer includes over 50 bug fixes and improvements.

Remaining Pipeline

Besides these two updates the official viewer pipeline remains as follows:

  • Current Release version 5.1.2.512803, dated February 23rd, promoted March 1st – formerly the Nalewka Maintenance RC.
  • Release channel cohorts:
    • Love Me Render RC viewer, version 5.1.3.513005, dated March 2nd.
  • Project viewers:
  • Linux Spur viewer, version 5.0.9.329906, dated November 17th, 2017 and promoted to release status 29 November – offered pending a Linux version of the Alex Ivy viewer code.
  • Obsolete platform viewer, version 3.7.28.300847, May 8th, 2015 – provided for users on Windows XP and OS X versions below 10.7.

Other Items

Meeting Reminders

  • There is NO Content Creation User Group meeting this week – the next CCUG meeting will be on Thursday, March 22nd at 13:00 SLT.
  • There is no Web User Group meeting this week – the next WUG meeting will be on  Wednesday, March 28th, at 14:00 SLT.

2018 SLUG updates #10/1: Simulator User Group

Meraki Islands; Inara Pey, February 2018, on Flickr Meraki Islandsblog post

Server Deployments

As usual, please refer to the server deployment thread for the latest updates.

  • There was no deployment / restart to the Main (SLS) channel on Tuesday, March 6th, leaving it on server maintenance package 18#18.02.12.512536.
  • No deployments are scheduled to either the Magnum or LeTigre RCs on Wednesday, March 7th, leaving them on server maintenance package 18#18.02.12.512536. However, given it is now two weeks since these channels were last restarted, they should undergo a restart this week.
  • The BlueSteel RC should receive a new server maintenance package on Wednesday, March 7th. 18#18.03.05.513046 is said to comprise “internal fixes”.

SL Viewer

360-Snapshot Viewer 5.1.3.513006

The 360-snapshot project viewer updated to 5.1.3.513006, intended to offer improved image resolutions and clarity. However, the update includes some significant issues of its own, including:

  • A failure to render in-world objects / Linden Water in 360-snapshots if Object-Object Occlusion (CTRL-SHIFT-O) is active.
  • A failure to render Linden Water (and possibly in-world objects) when saving snapshots to disk when using the Custom drop-down and leaving Constrain Proportions unchecked.
  • Include HUDs option in the snapshot viewer is inverted: when unchecked, HUDs are displayed in snapshots (and might appear in 360-snaps), when checked, HUDs are not displayed.
  • UI placement issues when the UI is displayed in the snapshot floater preview window – these issues do not translate to captured images, which display the UI correctly.

JIRAs have been raised against these issues.

Other Viewers

Apart from the 360-snapshot project viewer update, all viewers currently in LL’s pipelines remain unchanged from the end of week #9:

  • Current Release version 5.1.2.512803, dated February 23, promoted March 1 – formerly the Nalewka Maintenance RC – NEW
  • Release channel cohorts:
  • Project viewers:
  • Linux Spur viewer, version 5.0.9.329906, dated November 17, 2017 and promoted to release status 29 November – offered pending a Linux version of the Alex Ivy viewer code.
  • Obsolete platform viewer, version 3.7.28.300847, May 8, 2015 – provided for users on Windows XP and OS X versions below 10.7.

Region Crossing Issues Investigation

As noted over the last few weeks, user Joe Magarac (animats) has been digging into the viewer code handling region crossings (See Firestorm JIRA FIRE-21915),SL Forum thread, and this Google document, plus my last update here for more).

He has now come up with s further approach, using  a scripted means to turn off physics and freeze a vehicle on hand-off to the next region and then wait for the avatar data to arrive, using llGetObjectDetails (avatar, [OBJECT_ROOT]) to check for the presence of avatars. On confirming they are present, the vehicle resumes. Overall, the delay is a few milliseconds, and seems to work. However, as Rider Linden notes, Linden Lab still needs to address the crossing logic, although where such work might fit in the current schedule is unclear.

Environment Enhancement Project (EEP)

Project Summary

A set of environmental enhancements, including:

  • The ability to define the environment (sky, sun, moon, clouds, water settings) at the parcel level.
  • New environment asset types (Sky, Water, Days – the latter comprising multiple Sky and Water) that can be stored in inventory and traded through the Marketplace / exchanged with others.
  • Experience-based environment functions
  • An extended day cycle (e.g a 24/7 cycle) and extended environmental parameters.

This work involves simulator and viewer changes, and includes some infrastructure updates.

Current Status

“I got my first inventory objects on the test grid yesterday,” Rider noted at the Simulator User Group meeting. “I’m cleaning up the inventory code today and will wire the old editors in so you can create assets. I expect to have a viewer that people can play with very, very soon.”

2018 SL UG updates #9/1: SUG, and DDOS

Les Reves Perdus; Inara Pey, January 2018, on Flickr Les Reves Perdusblog post

Server Deployments

As usual, please refer to the server deployment thread for the latest updates.

On Tuesday, February 27th, the Main (SLS) channel was updated with server maintenance package 18#18.02.12.512536, previously deployed to the three RC channels. This update was to directly address an odd viewer crash situation some users have experienced. Speaking at the week #8 Simulator User Group meeting, and reproduced here for completeness, Simon Linden said of the issue:

The server is doing some better checking on update data it sends to the viewer. We saw a very odd situation a week or two ago where the region was sending odd data and viewers would crash immediately. It went away after we restarted the region, and we think it was some memory corruption … FWIW, the server was sending a value of zero for a prim-code … which is totally invalid … There were also some other invalid data (like a zero’ed UUID) so my theory was memory corruption.

We didn’t have any other smoking guns. That region was fine after restarting, or when we tried our own copy. It was one of those mystery bugs, which we sometimes get since SL is so big and complex. We don’t know why it got that way, or how to make it happen again. we ended up making both the region and the viewer more robust. The underlying problem is still there and, assuming it happens again, will still cause problems.

(See also: BUG-214564.)

On Wednesday, February 28th, 2018 the three main RC channels should be updated as follows:

  • LeTigre and Magnum; no deployment, no restart, leaving them on server maintenance package 18#18.02.12.512536.
  • BlueSteel should received a new server maintenance package, 18#18.02.23.512831, containing further simulator logging improvements and internal fixes. This is dependent upon what happens with the ongoing global DDoS attack over the course of the next 12-18 hours.

SL  Viewer

There have been no official viewer updates at the start of the week, leaving the various pipelines as per the end of week #8:

  • Current Release version  5.1.1.512121, dated January 26, promoted February 7 – formerly the Voice Maintenance RC.
  • Release channel cohorts:
    • Nalewka Maintenance viewer, version 5.1.2.512803, February 23.
    • Love Me Render RC viewer, version 5.1.2.512751, February 21.
    • Media Update RC viewer, version 5.1.2.512574, February 15.
  • Project viewers:
  • Linux Spur viewer, version 5.0.9.329906, dated November 17, 2017 and promoted to release status 29 November – offered pending a Linux version of the Alex Ivy viewer code.
  • Obsolete platform viewer, version 3.7.28.300847, May 8, 2015 – provided for users on Windows XP and OS X versions below 10.7.

Log-in Issues

On Monday, February 26th and continuing through Tuesday, February 27th, large numbers of SL users experienced significant issues in trying to log-in as a result of a widespread UDP-based Distributed Denial of Service (DDoS) attack on the Internet as a whole (rather than specific to SL).

On Tuesday, February 27th, whilst still dealing with the situation, April Linden at the SL Ops team took time out to post on the situation in the official blog:

Hi everyone.

As I’m sure most of y’all have noticed, Second Life has had a rough 24 hours. We’re experiencing outages unlike any in recent history, and I wanted to take a moment and explain what’s going on.

The grid is currently undergoing a large DDoS (Distributed Denial of Service) attack. Second Life being hit with a DDoS attack is pretty routine. It happens quite a bit, and we’re good at handling it without a large number of Residents noticing. However, the current DDoS attacks are at a level that we rarely see, and are impacting the entire grid at once.

My team (the Second Life Operations Team) is working as hard as we can to mitigate these attacks. We’ve had people working round-the-clock since they started, and will continue to do so until they settle down. (I had a very late night, myself!)

Second Life is not the only Internet service that’s been targeted today. My sister and brother opsen at other companies across the country are fighting the same battle we are. It’s been a rough few days on much of the Internet.

We’re really sorry that access to Second Life has been so sporadic over the last day. Trying to combat these attacks has the full attention of my team, and we’re working as hard as we can on it. We’ll keep posting on the Second Life Status Blog as we have new updates.

See you inworld!
April Linden
Second Life Operations Team Lead

The attack led to a wide number of issues for SL users, from an inability to log-in to SL, through to disconnects or other problems. As Simon Linden explained in the Simulator User Group meeting on Tuesday, February 27th:

It’s not just logins … and often there is cascading problems. If one system gets attacked, it might actually be functioning fine but if the rack or network switch it’s on is overloaded, anything else connected there also has problems.

As it stands, the worst of the attacks appear to be over, at least where SL is concerned, but users should keep an eye on the official blogs for further updates, as well as grid status updates.