There were no deployments to the main SLS channel on Tuesday, January 21st, although there are reports some regions were restarted.
An RC deployment is planned for Wednesday, January 22nd, but no details were available at the time of writing.
Simulator Issues
Some region holders are still reporting script runtime issues. There may be some work in the RC deployment to help with these issues, but again, this wasn’t clear at the SUG meeting, due to the lack of release notes.
There is some work to help manage scripted attachments and vehicles during region crossings, but no specifics given.
There have been reports of object_rez() events randomly failing. It’s not clear how widespread the issue is, but it is a known issue with the Lab, but a reliable reproduction of the problem has yet to be reported / found.
SL Viewer
There have been no viewer updates to marks the start of the week, leaving the official pipelines as follows:
Current Release version 6.3.5.533275, formerly the Wassail RC viewer, dated December 4, promoted December 12. – No change.
Release channel cohorts:
Xanté Maintenance RC viewer, version 6.3.6.533748, January 13.
Love Me Render RC viewer, version 6.3.6.534234, January 9.
There are no server deployments planned for week #3 due to the next batch of updates being ready for deployment.
Simulator Issues
There are still reports of recent simulator updates causing issues for certain types of breedables. The Lab is aware of these problems, and while fixes are in the works, they may not be in the next set of simulator updates.
LL has continued a post-mortem into the region restart issues from week #2, with Simon Linden noting they have hopefully learned enough such that future restarts such run a lot better (and hopefully more smoothly!).
“A couple of hosts” that were not handling teleports as expected at the end of week #2, which should have been corrected.
Additional data that the Lab has been gathering on simulator performance does show that long-running simulators incur increased host resource utilisation, but for reasons yet to be identified.
Scripted avatar loads might in part be responsible, depending on what their scripts are doing, but LL would prefer to gather further data rather than speculate on possible causes.
However, there are regions where issues have been recorded, but which do not have large avatars loads.
LL have tried multiple approaches to making similar slow-downs occur on the beta grid (Aditi), but the problem doesn’t appear to occur “on demand”, making possible causes harder to identify.
SL Viewer
The Xanté Maintenance RC viewer updated to version 6.3.6.533748 on Monday, January 13th.
The remainder of the current official viewer pipelines remain unchanged for the end of week #2:
Current Release version 6.3.5.533275, formerly the Wassail RC viewer, dated December 4, promoted December 12 – No change.
Release channel cohorts:
Love Me Render RC viewer, version 6.3.6.534234, January 9th.
Copy / Paste viewer, version 6.3.5.533365, December 9th, 2019.
Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22nd , 2019.
Legacy Profiles viewer, version 6.3.2.530836, September 17th, 2019. Covers the re-integration of Viewer Profiles.
360 Snapshot project viewer, version 6.2.4.529111, July 16th, 2019.
In Brief
Rider Linden is looking at region crossings, and hopes to have some code that should help improve things when it is ready for deployment. However, he notes that the improvements will be “nowhere near what I would ultimately like.
The thread refers to the SLS Main channel being on simulator release 533895. However:
Following restarts, it appears that simulators on the Main channel (and some on RC channels) have remained on 2019-12-04T20:29:26.533447, which according to previous deployment threads, was the last formal deployment to the Main channel.
According to the release note page, 533895 was deployed on December 19th, but has channel names are now obfuscated, it is not clear which RC received the update.
The thread refer to restarts on the RC channels on Wednesday, January 8th, some RC servers should be restarted, but again without any deployment – although the 533895 version number is given for some when it should perhaps be 53447.
Simulator Issues
Restart Issues
The Tuesday, January 7th restarts were repeated a number of times, for reasons Simon Linden explained:
So we haven’t had any server updates since mid-December. Today we restarted a bunch because that can help performance, and in the middle of that we uncovered a bug that’s been lurking there for months, so it got a bit exciting.
Mazidox Linden added that – as of the time of writing this update – further restarts could not be ruled out.
The bug itself was related to at least one directory required by the Mono compiler not being created correctly, preventing scripts from being saved. It is apparently a bug that has been around for some time, but only surfaced as a result of changes to the way simulator restarts are run. Ironically, the changes were intended to make simulator restarts faster and smoother
Holiday Issues
Numerous region holders reported significant performance issues over the holiday period. According to Grumpity Linden, part of the issue was a failure with an automated tool, itself something of a workaround for dealing with simulator performance issues:
Hi all! I was really hoping my first post of the new year would be more jolly, but here we are. Happy New Year, though!
For a couple of years now, we’ve had automated tools, aptly named “Grid Poking Bot” (GPB for short) responsible for doing region restarts, and this has been working quite well – most of the time. Very unfortunately, there was a problem with the GPB over the holidays, and due to a combination of events, it took us much too long to notice – and we finally caught it in part thanks to this very forum thread and a certain vigilant “Spray Can”. We’re now actively pursuing the least disruptive ways to address this problem as quickly as possible. We’ll have a more detailed post-mortem blog in a couple of days as well.
We’re very sorry about souring your holidays.
It’s possible there is more than one issue causing problems, and investigations are still in progress, including the distribution of poorly-performing regions (e.g. are they on their own on the grid, do they have neighbours) as well as looking at distribution by server.
SL Viewer
There have been no viewer updates to mark the start of the week, leaving the current pipelines as follows:
Current Release version 6.3.5.533275, formerly the Wassail RC viewer, dated December 4th, promoted December 12th – No change.
Xanté Maintenance RC viewer, version 6.3.6.533748, December 19th.
Love Me Render RC viewer, version 6.3.5.533347, December 5th.
EEP RC viewer, version 6.4.0.532771, November 20th.
Project viewers:
Copy / Paste viewer, version 6.3.5.533365, December 9th.
Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22nd.
Legacy Profiles viewer, version 6.3.2.530836, September 17th. Covers the re-integration of Viewer Profiles.
360 Snapshot project viewer, version 6.2.4.529111, July 16th.
Destination Guide
Not strictly part of the engineering team’s remit, but it has been reported the Destination Guide is giving errors when trying to submit new destinations. This has been seen by the Web Team, and is apparently being looked at.
There was no deployment on Tuesday, December 17th, leaving the SLS Main channel and a portion of the RC channel servers on simulator version 2019-12-04T20:29:26.533447, originally deployed on Thursday, December 5th, and comprising:
A build of release 2019-11-15T21:13:13.532828 using new build technology.
Addresses some cases of scripts erroneously stopping.
Fixes a crash.
On Wednesday, December 18th, some RC servers should be updated with server release 2019-12-06T21:03:45.533558, comprising internal fixes. This is a re-deployment of the release originally made on December 9th, 2019, containing “a difference under the hood”.
No Change Window
These deployments are the last scheduled for 2020. Due to the end of year No Change Window (December 20th through to January 2nd, 2020, the next simulator updates will not likely be deployed before Tuesday, January 7th, 2020.
SL Viewer
There have been no viewer updates to mark the start of the week, leaving the current pipelines as follows:
Current Release version 6.3.5.533275, formerly the Wassail RC viewer, dated December 4th, promoted December 1th2 – NEW.
On Tuesday, December 10th, the SLS Main channel was updated to simulator version 2019-12-04T20:29:26.533447, originally deployed on Thursday, December 5th, and comprising:
A build of release 2019-11-15T21:13:13.532828 using new build technology.
Addresses some cases of scripts erroneously stopping.
Fixes a crash.
On Wednesday, December 11th, there should be an RC deployment to the BlueSteel RC. Simulator version 2019-12-06T21:03:45.533558 comprises internal fixes.
Deployment Notes
The Tuesday deployment started a little later than usual.
Further, as some of the processes managing deployments are being modified to increase stability, some deployments may run longer than usual. Once this work has finished, it is hoped deployments will be faster and more stable.
SL Viewer
At the end of week #49, the following viewers were updated:
On December 4th, the Maintenance RC viewer updated to version 6.3.5.533275.
On December 5th, the Love Me Render RC viewer updated to version 6.3.5.533347.
On Monday, December 9th the Copy / Paste project viewer updated to version 6.3.5.533365. The rest of the official viewer pipelines are as follows:
Current Release version 6.3.4.532299, formerly the Ordered Shutdown RC viewer, dated November 4th, – No change.
Release channel cohorts:
EEP RC viewer, version 6.4.0.532771, November 20th.
Project viewers:
Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22nd.
Legacy Profiles viewer, version 6.3.2.530836, September 17th. Covers the re-integration of Viewer Profiles.
360 Snapshot project viewer, version 6.2.4.529111, July 16th.
On Tuesday, 3rd December, the planned Main SLS channel deployment was cancelled. Had it been deployed, it would have comprised server maintenance release 2019-11-19T22:26:38.532992, built with the updated server build tools, and which includes:
Improved crash detection during shut-down.
Fixes for some race conditions with LSL scripts that could cause them to handle the same event more than once during a roll.
Updates to improve simulator security.
The planned RC deployment for Wednesday, December 4th, comprising server update has also been cancelled.
The reasons for the cancellation are related to the return of recent issues with scripts failing until a region is restarted (see BUG-227864) and which has been an intermittent issue recently, with Simon Linden noting:
We originally had planned to update the main channel regions with the version that’s on the RC channels, but held off as we are looking into issues about scripts not running. It appears to be an old issue that’s always been there but is worth investigating … we’re trying to get an update together for tomorrow morning [Wednesday, December 4th]. It was a very small percentage [of scripts affected] but that’s enough to be painful when there are millions running.
There is a forum thread concerning simulator overload issues that occurred over the Thanksgiving weekend (see: Huge intermittent sim overload). These issues were further confirmed at the SUG meeting by others seeing similar problems on other regions. Linden Lab do not currently have an explanation for the problems, but Maxidox Linden, one of the Second Life QA team members, did take a moment to address claims in the thread that LL “not caring” about issues like this:
I’m going to let the devs here handle this topic but I just want to chime in that I’ve read that thread and we absolutely do care about issues like this, and don’t put our development time solely into new feature work (I’m the one who tests a bunch of the stability and performance change.
– Mazidox Linden
The discussion on this topic raised questions on the simulator statistic Simulation Time and what it actually records, given it is often tied to simulator issues – as is the case with this issue. Exactly what it records has been a subject of discussion for around the last decade, and came up again in terms of whether it might point to a possible cause. After taking a look, Simon Linden noted:
So I did some digging. “Simulation Time” seems to be a catch-all category that includes odds and ends like dealing with terrain layers, some physics synchronization, parcel info. So not a lot to use tracking this issue down.
SL Viewer
There have been no official viewer updates to mark the start of the week.
