At the time of writing. the server deployment thread appears at odds with reality:
- The thread refers to the SLS Main channel being on simulator release 533895. However:
- Following restarts, it appears that simulators on the Main channel (and some on RC channels) have remained on 2019-12-04T20:29:26.533447, which according to previous deployment threads, was the last formal deployment to the Main channel.
- According to the release note page, 533895 was deployed on December 19th, but has channel names are now obfuscated, it is not clear which RC received the update.
- The thread refer to restarts on the RC channels on Wednesday, January 8th, some RC servers should be restarted, but again without any deployment – although the 533895 version number is given for some when it should perhaps be 53447.
The Tuesday, January 7th restarts were repeated a number of times, for reasons Simon Linden explained:
So we haven’t had any server updates since mid-December. Today we restarted a bunch because that can help performance, and in the middle of that we uncovered a bug that’s been lurking there for months, so it got a bit exciting.
Mazidox Linden added that – as of the time of writing this update – further restarts could not be ruled out.
The bug itself was related to at least one directory required by the Mono compiler not being created correctly, preventing scripts from being saved. It is apparently a bug that has been around for some time, but only surfaced as a result of changes to the way simulator restarts are run. Ironically, the changes were intended to make simulator restarts faster and smoother
Numerous region holders reported significant performance issues over the holiday period. According to Grumpity Linden, part of the issue was a failure with an automated tool, itself something of a workaround for dealing with simulator performance issues:
Hi all! I was really hoping my first post of the new year would be more jolly, but here we are. Happy New Year, though!
For a couple of years now, we’ve had automated tools, aptly named “Grid Poking Bot” (GPB for short) responsible for doing region restarts, and this has been working quite well – most of the time. Very unfortunately, there was a problem with the GPB over the holidays, and due to a combination of events, it took us much too long to notice – and we finally caught it in part thanks to this very forum thread and a certain vigilant “Spray Can”. We’re now actively pursuing the least disruptive ways to address this problem as quickly as possible. We’ll have a more detailed post-mortem blog in a couple of days as well.
We’re very sorry about souring your holidays.
It’s possible there is more than one issue causing problems, and investigations are still in progress, including the distribution of poorly-performing regions (e.g. are they on their own on the grid, do they have neighbours) as well as looking at distribution by server.
There have been no viewer updates to mark the start of the week, leaving the current pipelines as follows:
- Current Release version 18.104.22.1683275, formerly the Wassail RC viewer, dated December 4th, promoted December 12th – No change.
- Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
- Project viewers:
- Copy / Paste viewer, version 22.214.171.1243365, December 9th.
- Project Muscadine (Animesh follow-on) project viewer, version 126.96.36.1992999, November 22nd.
- Legacy Profiles viewer, version 188.8.131.520836, September 17th. Covers the re-integration of Viewer Profiles.
- 360 Snapshot project viewer, version 184.108.40.2069111, July 16th.
Not strictly part of the engineering team’s remit, but it has been reported the Destination Guide is giving errors when trying to submit new destinations. This has been seen by the Web Team, and is apparently being looked at.