2019 Simulator User Group week #49 summary

Butter, October 2019 – blog post

Update, December 5th: a deployment was made to an RC channel. Server update version 2019-12-04T20:29:26.533447 comprises:

  • A build of release 2019-11-15T21:13:13.532828 using new build technology.
  • Addresses some cases of scripts erroneously stopping.
  • Fixes a crash.

Hopefully, this addresses the recent issues with scripts failing until a region is restarted (see BUG-227864).

Simulator Deployments

Please refer to the server deployment thread for news and updates:

  • On Tuesday, 3rd December, the planned Main SLS channel deployment was cancelled. Had it been deployed, it would have comprised server maintenance release 2019-11-19T22:26:38.532992, built with the updated server build tools, and which includes:
    • Improved crash detection during shut-down.
    • Fixes for some race conditions with LSL scripts that could cause them to handle the same event more than once during a roll.
    • Updates to improve simulator security.
  • The planned RC deployment for Wednesday, December 4th, comprising server update has also been cancelled.

The reasons for the cancellation are related to the return of recent issues with scripts failing until a region is restarted (see BUG-227864) and which has been an intermittent issue recently, with Simon Linden noting:

We originally had planned to update the main channel regions with the version that’s on the RC channels, but held off as we are looking into issues about scripts not running. It appears to be an old issue that’s always been there but is worth investigating … we’re trying to get an update together for tomorrow morning [Wednesday, December 4th]. It was a very small percentage [of scripts affected] but that’s enough to be painful when there are millions running.

– Simon Linden

So again, keep an eye on the server deployment thread / technology → second life server sub-forum for updates.

Simulator Performance Issues

There is a forum thread concerning simulator overload issues that occurred over the Thanksgiving weekend (see: Huge intermittent sim overload). These issues were further confirmed at the SUG meeting by others seeing similar problems on other regions. Linden Lab do not currently have an explanation for the problems, but Maxidox Linden, one of the Second Life QA team members, did take a moment to address claims in the thread that LL “not caring” about issues like this:

I’m going to let the devs here handle this topic but I just want to chime in that I’ve read that thread and we absolutely do care about issues like this, and don’t put our development time solely into new feature work (I’m the one who tests a bunch of the stability and performance change.

– Mazidox Linden

The discussion on this topic raised questions on the simulator statistic Simulation Time and what it actually records, given it is often tied to simulator issues – as is the case with this issue. Exactly what it records has been a subject of discussion for around the last decade, and came up again in terms of whether it might point to a possible cause. After taking a look, Simon Linden noted:

So I did some digging. “Simulation Time” seems to be a catch-all category that includes odds and ends like dealing with terrain layers, some physics synchronization, parcel info. So not a lot to use tracking this issue down.

SL Viewer

There have been no official viewer updates to mark the start of the week.

2019 Simulator User Group week #48 summary

Fox Road, October 2019 – blog post

Simulator Deployments

Please refer to the server deployment thread for news and updates:

  • On Tuesday, 26th November, servers on the Main SLS channel were updated with server maintenance release 2019-11-15T21:13:13.532828, previously deployed to a release candidate channel. It comprises:
    • Improves crash detection during shut-down.
    • Fixes some race conditions with LSL scripts that could cause them to handle the same event more than once during a roll.
    • Includes updates to improve simulator security.
  • On Wednesday, 27th November, there should be a single RC deployment 2019-11-19T22:26:38.532992, which is functionally the same build as 532828, but compiled using updated server build tools.
    • Given the issues with Tuesday deployment (see below) the status of the deployment was in a state of flux at the time of writing.

Tuesday Deployment Issues

The Tuesday server updates required an extended deployment window, together with more restarts of the updated simulators than is usually the case. Given the simulator version had been deployed to RC servers in week #47, the issues encountered with the deployment took LL by surprise:

This is one of those situations where we are left trying to figure out what went so wrong; we don’t know of changes that would have caused this much churn. That is the big difference between our Tuesday and Wednesday updates. Tuesday is more than 2x bigger, and divided up differently. Seeing something happen and knowing why it happened are two completely different things. Given that last week was OK, I suspect it is a scaling problem however.

The real frustration is that there isn’t a good way to test or simulate the scale where the issues happen. Updating one or two servers can be perfectly fine; but then do a few hundred and something goes up in smoke.

– Simon Linden, commenting on the November 26th deployment issues

Currently, the Lab is digging into what may have gone wrong (e.g. by checking server logs, etc.). They have already tracked down what might be a contributing factor, but the overall root cause analysis will take time. However, as the issues appears to be with the deployment process itself rather than the updated simulator code, at this point in time it is unlikely the simulator update will be rolled back.

SL Viewer

The following viewers were updated during the latter half of week #47:

  • November 22nd:
    • The Wassail Maintenance RC viewer updated to version 6.3.5.533043.
    • The Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999.
  • November 20th

The remainder of the viewers in the current pipelines are as follow:

  • Current Release version 6.3.4.532299, formerly the Ordered Shutdown RC viewer, dated November 4th, – No Change.
  • Release channel cohorts:
  • Project viewers:
    • Legacy Profiles viewer, version 6.3.2.530836, September 17th. Covers the re-integration of Viewer Profiles.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16th.

Name Changes

  • There are concerns over how long names might be cached within services, how changes might be tracked by external databases (e.g. customer lists for things like redelivery terminals), etc.
    • To help ensure user-developed tools, etc., can keep up with Name Changes, Linden Lab plan to offer a REST API (REpresentational State Transfer) that can be used to perform a name<-> avatar key translations and that will always return the same key for any name an account has ever had, and the key will always return the current name.
    • Creators are again reminded that their products and tools should utilised the avatar agent key, no avatar names.
  • LSL enquiries on agent key will return the current name for an avatar, not any past name(s).
  • There was concern that Name Changes could lead to issues in tracking griefers. However, as was pointed out in the meeting:
    • There are fees associated with Names Changes – signing-up to Premium and then for each change. These are unlikely to find appeal with griefers.
    • Griefers already create multiple accounts, and this will remain likely remain their preferred means of causing upset.
  • BUG-216397 “llName2Key, llRequestUserKey both accept a lone “R” as a surname for “Resident” agents” has been raised with the Lab again, and has been marked for inclusion in the next internal maintenance fix for Name Changes.
  • Once again, a “fee” figure was mentioned at the Simulator User Group meeting (US $20) – but it is important to note this is only a guess on the part of a user, and not any form of statement from Linden Lab. So If you hear it anywhere in the next month or so, ignore it.

2019 Simulator User Group week #47 summary

Gentle Breezes, October 2019 – blog post

Updates, November 20th, 2019:

  • Regarding BUG-227179 “All off-line inventory offers from scripted objects are STILL lost”. As per the notes below, this was believed fixed in the Wassail RC viewer. Testing has confirmed the issue hasn’t been fixed, and can be more confusing  – see BUG-227901.
  • Following the discussion about the error message, “Wire Chewed By Rabbit”, April Linden offered the following explanation via the forums:

/me raises her paw!

Soooo, uh, I was working on some server code a few years ago, and figured “no one’s gonna see this anyhow, so I’m gonna have some fun with it!” hehehe. 😝

You never know where the GridBuns might be lurking…!

Although she admits why the error surfaced with regards to Singularity is a bit of a mystery.

Simulator Deployments

Please refer to the server deployment thread for news and updates:

  • There was no update to the Main SLS channel on Tuesday, November 19th, leaving it on server release 2019-11-01T18:02:37.532376, previously deployed to an RC and comprising: internal script improvements. internal logging changes and improvements to simulator state saves, which should make rolls smoother.
  • On Wednesday, November 20th, there will be a maintenance update to RC servers: 2019-11-15T21:13:13.532828, comprising:
    • Improves crash detection during shut-down
    • Fixes some race conditions with LSL scripts that could cause them to handle the same event more than once during a roll.
    • Includes updates to improve simulator security.

An SLS deployment had been planned for the Main channel, however, at the meeting, Simon Linden noted:

We didn’t have an update this morning as we found an issue preventing the RC code from going forward. It isn’t anything visible or buggy from your point of view, but was about some internal data routing. To be on the safe side, we decided to get that sorted out first.

SL Viewer

There have been no updates to the current SL viewer pipelines, leaving them as follows:

  • Current Release version 6.3.4.532299, formerly the Ordered Shutdown RC viewer, dated November 4, – NEW.
  • Release channel cohorts:
  • Project viewers:
    • Copy / Paste viewer, version 6.3.5.532780, November 15.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.531949, October 28.
    • Legacy Profiles viewer, version 6.3.2.530836, September 17. Covers the re-integration of Viewer Profiles.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16.

In Brief

  • Simulator Deployments Causing Script Issues? -There have been reports that recent simulator deployments are causing running scripts to be reset when regions come back up, rather than  being restored to their previous state. Potential issues linked to this – impacting breedables – are indicated as being BUG-227688 and BUG-227897. As far as the  Lab is aware, there have been no back-end changes that would cause such script issues.
  • BUG-227179 “All off-line inventory offers from scripted objects are STILL lost” – it is believed that a fix for this is in the Wassail release candidate viewer (if not recorded in the release notes) – tests are being made to verify whether this is the case or not.
  • Error message: “Wire Chewed By Rabbit” in log files following log-in with missing textures / time-outs – this appears to be an issue specific to the Singularity viewer, and believed fixed in the most recent nightly / alpha / beta builds. If you are on Singularity and encounter it when examining your viewer log files as a result of texture load issues, please file a report with the Singularity team.

2019 Simulator User Group week #46 summary

Hazardous, September 2019 – blog post

Simulator Deployments

Please refer to the server deployment thread for news and updates:

  • The SLS (main) channel on Tuesday, November 12th, leaving it on server release 2019-11-01T18:02:37.532376, previously deployed to an RC and comprising: internal script improvements. internal logging changes and improvements to simulator state saves, which should make rolls smoother.
  • On Wednesday, November 13th, there will be two RC deployments

SL Viewer

The Ordered Shutdown RC viewer, version 6.3.4.532299 and dated Monday, November 4th, has been promoted to the de facto release viewer.

The Maintenance RC viewer updated to version 6.3.5.532631 on Tuesday, November 12th.

The rest of the current viewer pipelines remain as follows:

  • Release channel cohorts:
  • Project viewers:
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.531949, October 28.
    • Copy / Paste viewer, version 6.3.3.531844, October 21.
    • Legacy Profiles viewer, version 6.3.2.530836, September 17. Covers the re-integration of Viewer Profiles.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16.

In Brief

  • There is a relatively new member of the Lab’s QA team for Second Life: Johngalt Linden.

2019 Simulator User Group week #45 summary

Whimsies and Dreamscapes, September 2019 – blog post

Simulator Deployments

Please refer to the server deployment thread for news and updates:

  • The SLS (main) channel on Tuesday, November 5th, leaving it on server release 2019-10-26T00:06:48.532192, previously released on October 30th and comprising:
    • A previously released hotfix to fix teleports being 5%-7% less reliable.
    • A change that makes the simulator take a little bit longer to report as “Up” to the Lab’s internal tools to more accurately reflect when residents can actually access a region.
  • There is a single RC deployment planned for Wednesday, November 6th: version 2019-11-01T18:02:37.532376, which is essentially a rebuild of 532143, originally deployed on Wednesday, October 30th and containing the above updates as well. No release had been released at the time of writing.

Note: a second RC deployment, 532377, comprising an internal update and a script to monitor additional region data during rolls was cancelled after it was discovered in QA it broke script uploading. 

Linden Lab request that if anyone sees messages about “Unable to upload 12345678-90ab-cdef-1234567890ab due to the following reason” on a version that isn’t 532377, could they please file a bug report and provide full information of when, where and how they encountered the message. 

SL Viewer

The Ordered Shutdown RC viewer was updated to version 6.3.4.532299 on Monday, November 4th. At the time of writing, the rest of the viewer pipelines remain as follows:

  • Current Release version 6.3.3.531811, formerly the Voice RC viewer, dated October 18, promoted October 31 – NEW.
  • Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
  • Project viewers:
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.531949, October 28.
    • Copy / Paste viewer, version 6.3.3.531844, October 21.
    • Legacy Profiles viewer, version 6.3.2.530836, September 17. Covers the re-integration of Viewer Profiles.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16.

Simulator Performance

There are continued reports of simulator performance issues – notably with Mainland regions. The most common description of the problem is that over time, script performance on Full region simulators declines to the point where only around 20% or so of scripts are running per cycle, and the simulator has no spare time, forcing the region holder to submit a ticket requesting a move to a different server. This tends to resolve the problem – at least until the next rolling restart, when the performance degradation starts again.

There appears to be no discernible reason for this. Rider Linden has been making improvements on how scripts – particularly idle scripts – are handled, and at the meeting, Simon Linden noted:

We’ve been looking at scripts and performance and there are some changes in the works that will hopefully help … those are on about 15% of the regions now, and will expand to about 20% tomorrow.

…Rider’s work looked at a bunch of the event distribution code like that, and he made it faster with re-writing it and getting rid of brute-force searching and so on. Of course, that changed some script timing and killed some kittens and pot plants. Sometimes it really hits me what a strange job this is.

The issues have led to speculation that the Lab might be stacking more regions on their servers or are reducing the server resources available to each simulator. Responding to an enquiry on this, Simon indicated that the Lab have not changed either for years. However, he did note the fact that the servers the Lab uses do change, so something may have been introduced to impact performance.

2019 Simulator User Group week #44

Abrahamstrup, September 2019 – blog post

Simulator Deployments

At the time of writing, no deployment notes had been published. However:

  • There was no deployment to the SLS (main) channel on Tuesday, October 29th, leaving it on server release 2019-10-03T01:12:11.531528.
  • There are two RC deployments planned for Wednesday, October 30th:
    • 2019-10-24T19:07:13.532143, comprising further internal script improvements, internal logging changes and improvements to simulator state saves.
    • 2019-10-26T00:06:48.532192, comprising a previously released hotfix to fix teleports being 5%-7% less reliable and makes the simulator take a little bit longer to report as “Up” to the Lab’s internal tools to more accurately reflect when residents can actually access a region.

SL Viewer

The Project Muscadine (Animesh follow-on) project viewer updated to version 6.4.0.531949 on October 28th. The update brings it to parity with the release viewer, but contains not project updates.

. The rest of the viewer pipelines remain as follows:

  • Current Release version 6.3.2.530962, formerly the Vinsanto Maintenance RC viewer, dated September 17th, promoted October 15th – NEW.
  • Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
  • Project viewers:
    • Copy / Paste viewer, version 6.3.3.531844, October 21st.
    • Legacy Profiles viewer, version 6.3.2.530836, September 17th. Covers the re-integration of Viewer Profiles.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16th.

Script Event Order

It was asked if the script updates would affect the order in which script events are handled, Rider Linden stated:

Some events have always had priority just by virtue of the order in which they were collected. The order of collection has changed. For instance, sensor events were collected and posted before chat events and then touch events. Chat events are now posted immediately upon processing in the simulator. It should still be FIFO… just don’t bet on what event gets collected when.

In addition, it was noted in regards to event messages:

  • Generally, event handling should not to be counted on in any sort of coding since it may change again in the future.
  • Link messages:
    • If multiple link messages are sent from a single source to a single receiver script, the ordering should be preserved. Similarly, when using llLinkMessage to send a message from script A to script B in the same prim, they are posted immediately, and the order is maintained.
    • If the same message is sent to link A and then to link B, the order the links get them is not always the same. Similarly, if script A and script C are using llLinkMessage to post to B, all bets are off which gets there first.