On Tuesday, October 14th the SLS (main) channel was updated with server release 2019-10-03T01:12:11.531528, previously deployed to an RC channel and comprising:
Fixes: BUG-227645 EEP issue; windlight no longer rendering properly.
Internal logging changes.
Improvements to simulator state saves, which should make rolls smoother.
On Wednesday, October 16th, a new server update, 2019-10-11T18:12:36.531693, should be deployed. This comprises all of the above updates plus the internal script improvements previously documented in these updates. This deployment will expand these updates (originally deployed to one RC on Wednesday, October 9th in release .531529) to all of the primary RC channels.
The Vinsanto Maintenance RC viewer, dated September 17th, 2019 was promoted to de facto release status on Tuesday, October 15th. The remainder of the pipelines remained unchanged at the time of writing:
An important point to note with this is that when release 2019-10-03T01:23:43.531529 has been deployed, any scripts that still exhibit the kind of communication issues indicated by the blog post will likely need to be altered by their creator to match the example scripts supplied in the blog post, or at least follow the communications process defined within it.
We’ve also learned a bit more about esoteric scripting behaviour; for example, if an event happens and it’s going to get picked up by multiple handlers, there is NO promises about the order they get it. And with communication or transfers between prims and objects, the big lesson is to make sure everything is ready with “hello” exchanges and confirmations that both sides are ready. It’s like passing a ball – make sure the other side is ready to catch it.
– Simon Linden
The long-awaited Voice RC, version 184.108.40.2061587, was issued on Tuesday, October 8th. Primarily intended to improve voice detection when you’re speaking, this voice includes the following fixes (non-public Jira reports):
BUG-227356 [Win] ‘SLVoice.exe’ starts an unexpected cmd window
VOICE-56 Voice is cutting out – seems like a threshold is too low
SL-11958 viewer-manifest should treat missing files as errors
The remainder of the official viewer pipelines remains as follows
Current Release version 220.127.116.110559, formerly the Umeshu Maintenance RC viewer, dated, September 5th – No Change.
Release channel cohorts:
Love Me Render viewer, version 18.104.22.1681296, September 30th.
Ordered Shutdown RC viewer, version 22.214.171.1240972, September 24th.
Many of us are familiar with the Lab’s approach to viewer and simulator releases – but equally, many only have a passing understanding of what goes on. This was something reinforced to me as a result of in-world conversations I’ve had recently, so I thought I’d reach back to 2013, when I provided a guide to the viewer release process (see: New viewer release process implemented), and use that and some additional notes on simulator releases to try to provide and easy-to-follow overview of how the Lab manages official viewer releases and simulator updates.
The Viewer Release Process
Note: just to avoid any confusion, please remember these notes only apply to the official Second Life viewer supplied by Linden Lab (and which from the last Lab-derived comment on numbers (late 2016) I have could account for approximately 15%-20% of user usage).
The current viewer release process was introduced in July 2013 as a result of a number of issues occurring in 2012 that combined to produce a severe bottleneck in the Lab’s ability to make timely viewer releases and deploy everything from bug fixes to major new releases.
With it, the Lab can produce multiple versions of the viewer in parallel with one another, some of which may initially follow their own development / testing path independently of other versions, but which can, when they are ready, be tested for their suitability for promotion as the next de facto release viewer through direct monitoring of their performance and through comparison of that performance, one to the next.
Types of Official Viewer
The process achieves this by allowing viewers to be developed on a rolling basis, defined by project internally, and which eventually appear for public use in one of two “pre-release” versions, as it were: Project viewers and release candidate viewers.
Project viewers are generally viewers dedicated to a single new feature, capability or function within the viewer. They are essentially “first look” / experimental viewers designed to expose new features and capabilities (and any new viewer UI that might come with them) to users interested in them, who can then test and provide feedback (including bug reports) to the Lab, allowing the feature or capability to be refined and improved.
Release candidate (RC) viewers are viewers considered to be close to the point where they can be promoted as the de facto release viewer.
They might be former project viewers that have progress to a point where the Lab is considering formally releasing them, OR they might start as RC viewers in their own right without ever having been a project viewer.
It is these RC versions that allow the Lab to gather statistics on the behaviour of individual viewers in order to help determine their suitability for promotion to full release status.
Broadly speaking, whether a viewer starts its public life as a project viewer or a release candidate viewer depends on what it contains. A viewer containing a major new feature – such as Animesh, Bakes on Mesh or EEP, for example – will generally make an initial public appearance as a project viewer for the reasons noted above. Maintenance releases, hot fixes, and things like updates to the viewer rendering system will – in general – tend to appear directly as release candidate viewers.
No viewer ever goes directly from project status to a full release – all project viewers will go by way of progressing first to being a release candidate, then being judged as ready for promotion to full release status.
Where to Find Them and How They are Handled
Both types of viewer appear on the Alternate Viewers Page, but how they are handled is somewhat different – and this is one of the important aspects in understanding them.
Project viewers are largely “independent” viewer versions.
Users must opt to visit the Alternate Viewers Page and select, download and install one.
Each project viewer installs into its own dedicated location (although they share the same settings files and cache locations as the release viewer), so they can be run alongside the release version of the official viewer, if installed.
Release candidate viewers are considered as “alternative release viewers”.
Release candidates are assigned a “cohort number” the Lab believes will present a reasonable cross-section of users.
When a release candidate viewer is made available, the system automatically triggers the viewer update process among randomly selected users on the current official release viewer, moving them to the release candidate.
When the cohort number for a release candidate viewer is reached, it is no longer made available for automatic download / installation.
Once a user has been selected to receive a release candidate version of a viewer, they will continue to receive updates for that particular RC on a mandatory basis until it is promoted to release status – they will not be selected to receive other RCs (or updates to others RCs) until the RC they have been using in promoted to de facto release status.
The reason for doing this is to allow the Lab to monitor the performance of individual viewer release candidates and capture data on things like performance, stability, crash rates, etc. This data, together with bug reports, etc., filed by users is then used to determine an individual RC’s suitability for promotion to release status.
Alternatively, users can opt to manually install any RC viewer that interests them directly via the Alternate Viewers Page. Again, by default, any RC viewer installed in this way will overwrite any existing installation of the official release viewer (unless an alternative installation location is provided by the user), and the user will thereafter receive updates for that RC.
Note that users who do not wish to have RC viewers installed on their system can, if they wish opt out of the release viewer update loop from within their viewer, as shown below.
How are Viewers Progressed to Release?
As noted above, project viewers follow defined path: they initially appear as a public project viewer, and then may go through multiple iterations of improvement, updates, added capabilities, bug fixes, etc., before they reach a point where the Lab determines they are ready for upgrade to release candidate status.
Release candidate viewers are more closely monitored by the Lab through their various cohorts, whilst similarly being subject to multiple iterations designed to remove bugs, help with performance, address further perceived shortfalls in functionality, etc., based on things like bug reports and feature requests from users.
While this approach means that multiple viewers can be developed, tested and readied for promotion to de facto release status, it often means that at any one time, there are several RC viewers vying for release. When this happens:
The Lab will select a viewer or promotion based on a number of factors, including stability, performance, number of remaining bugs / issues, the potential impact of said bugs issues, the urgency with which an RC needs to be released (e.g. an “late breaking” RC with a hot fix could well be promoted ahead of other RCs that have been available for longer) and so on.
Generally speaking, the Lab tries to promote no more than one RC to de facto release status every two weeks. However, depending on the overall state of individual RC viewers, the period between promotions can be longer.
Once a release candidate has been promoted to release status, the first order of business is to merge the code it contains into all over available RC viewers and then monitor them to see how they behave when built using the “new” release code, a process that also feeds back into determining which of them might next be promoted.
What this all Means in Summary
Simply, put, that official viewer and viewer updates can be produced on a rolling basis, with some starting as project viewers, others directly as release candidates, with the latter being objectively monitored both individually and in comparison with one another to determine which is best suited to become the next de facto release viewer.
A rollback was performed across the grid on September 27th/28th, which apparently moved all regions back to server release 2019-09-06T22:03:53.530715, first deployed to an RC channel on September 10th. This was due to widespread issues being reported across the grid in relation to the script timing / performance fixes that were deployed – and which revealed a further underpinning issue. See the conversation in this forum thread and this status update for more.
Commenting on the situation, Simon Linden stated:
We had some chaos last week after our main channel roll exposed some “interesting” issues with the server update. That was all reverted early Saturday morning. We’re on track to have another update tomorrow morning [Wednesday, Oct 2nd] which should bring back the script performance work as well as fixing the issues we discovered.
[The Issue] was an ugly timing issue involving rezzing and starting up scripts … and unfortunately would work in some circumstances, not in others, or fail once and then work fine the next time you tried it. So it was tough to catch as well as sort out.
Mazidox Linden added thanks from LL to all those hitting script-related issues who took time to dig into matters, try to identify causes and raise bug reports, allowing the Lab to get a fix in the works.
While the offending code had only been deployed to the SLS (Main) channel on Tuesday, September 24th, it had already seen the light of day on an RC channel in previous weeks (server deployment 2019-09-13T19%3A08%3A35.530941, September 11th); so in order to completely remove it from the grid, a full rollback was performed and place the grid on the same simulator version.
It had been hoped that the fix mentioned above would be ready for an RC deployment on Wednesday, October 3rd. However, a late-breaking issue with the fix meant that the deployment of the update had to be cancelled,
This topic – including what might be done to avoid it in future, what was and wasn’t affected, what might or might not have been contributing factors – e.g. heavy region loads (pointed to as a possibly contributing reason), etc.
A new RC version of the Love Me Render viewer was released on Monday, September 30th, version 126.96.36.1991296, containing a handful of reported fixes:
SL-12025 (non-public) – “Animated mesh objects are disappeared when avatar rendering parameter is off”.
SL-11656 (non-public) – “Alpha textures with Alpha mask cut-off of 255 look glitchy while ALM is off”.
SL-11614 (non-public) – “Rotating objects flicker if Render type Avatar is disabled”.
BUG-2635 Objects rotating with llTargetOmega now “vibrate” on spin axis when the camera is focused on them.
Not a lot to report – the meeting was largely a solstice party with live music. Simon Linden did, however, use the larger-than-usual gathering to monitor animation performance, lag and streaming performance.
Update, September 28th: a rollback was performed across the grid on September 27th/28th, which apparently moved all regions back to server release 2019-09-06T22:03:53.530715, first deployed on September 10th, this was due to widespread issues being reported across the grid in relation to the script timing / performance fixes that were deployed – and which revealed a further underpinning issue. See this status update for more.
On Tuesday, September 24th, the SLS (Main) channel was updated with server release 2019-09-13T20:04:44.530946, comprising minor improvements to starting and stopping regions and EEP updates and fixes, and which was originally deployed to the Magnum RC channel.
On Wednesday, September 25th, the RC channels are to be updated with two deployments (no channel details provided):
2019-09-20T23:03:22.531148, comprising internal logging changes and improvements to simulator state saves intended to make deployments smoother.
On Tuesday, September 17th, the SLS (Main) channel was updated with server release 2019-09-06T22:03:53.530715. Originally deployed to the Magnum RC on September 11th, it contains the fix to address most cases of experience-enabled scripts losing association with their experience – see this blog post.
On Wednesday, September 18th, the RC channels are to be updated as follows:
The Legacy Profile project viewer was updated to version 188.8.131.520836.
On Monday, September 16th, the Ordered Shutdown RC viewer, version 184.108.40.2060901, was released. This viewer has changes intended to make crashes on shut-down less likely, but does not have any changes to existing features.
At the time of writing, the rest of the current official viewer pipelines remain as follows:
Project Muscadine (Animesh follow-on) project viewer, version 220.127.116.110100, August 19th.
360 Snapshot project viewer, version 18.104.22.1689111, July 16th.
Linux Spur viewer, version 22.214.171.1249906, dated November 17, 2017 and promoted to release status 29 November 2017 – offered pending a Linux version of the Alex Ivy viewer code.
Obsolete platform viewer, version 126.96.36.1990847, May 8, 2015 – provided for users on Windows XP and OS X versions below 10.7.
The Lab is “very focused” on the problem of avatars teleporting into or out of a region overpowering local performance (scripts, etc.).
It’s been widely assumed that the performance is due to things like overall complexity and / or script load, etc.
However, while both script load and avatar complexity do have a general impact on performance, LL does not believe they are responsible for the issues seen when avatars enter / leave a region.
Data has been gathered on the problem, and Rider Linden indicated that LL feel they have a reasonable handle on the problem and are in a position to start experimenting to verify their findings in the near future.
There is period of voice maintenance due on Thursday, September 19th. This involves back-end updates to the voice system.
It is not clear if these updates will assist those users who, when activating voice, appear to be in their own channel with just one or two other users and must relog to join the main channel with all the others on voice.
This is a problem LL has noted, but Vivox have been unable to determine the cause.
There is a voice viewer update in the works that includes additional debugging capabilities that might help with determining the problem.