A closer look: Second Life viewer & simulator release processes

Part of the official viewer release process (see below for the full diagram). Credit: Linden Lab

Many of us are familiar with the Lab’s approach to viewer and simulator releases – but equally, many only have a passing understanding of what goes on. This was something reinforced to me as a result of in-world conversations I’ve had recently, so I thought I’d reach back to 2013, when I provided a guide to the viewer release process (see: New viewer release process implemented), and use that and some additional notes on simulator releases to try to provide and easy-to-follow overview of how the Lab manages official viewer releases and simulator updates.

The Viewer Release Process

Overview

Note: just to avoid any confusion, please remember these notes only apply to the official Second Life viewer supplied by Linden Lab (and which from the last Lab-derived comment on numbers (late 2016) I have could account for approximately 15%-20% of user usage).

The current viewer release process was introduced in July 2013 as a result of a number of issues occurring in 2012 that combined to produce a severe bottleneck in the Lab’s ability to make timely viewer releases and deploy everything from bug fixes to major new releases.

With it, the Lab can produce multiple versions of the viewer in parallel with one another, some of which may initially follow their own development / testing path independently of other versions, but which can, when they are ready, be tested for their suitability for promotion as the next de facto release viewer through direct monitoring of their performance and through comparison of that performance, one to the next.

The Viewer Release and Integration Process is intended to allow viewers to be better developed tested and prepared for release in parallel. Image credit: Linden Lab

Types of Official Viewer

The process achieves this by allowing viewers to be developed on a rolling basis, defined by project internally, and which eventually appear for public use in one of two “pre-release” versions, as it were: Project viewers and release candidate viewers.

  • Project viewers are generally viewers dedicated to a single new feature, capability or function within the viewer. They are essentially “first look” / experimental viewers designed to expose new features and capabilities (and any new viewer UI that might come with them) to users interested in them, who can then test and provide feedback (including bug reports) to the Lab, allowing the feature or capability to be refined and improved.
  • Release candidate (RC) viewers are viewers considered to be close to the point where they can be promoted as the de facto release viewer.
    • They might be former project viewers that have progress to a point where the Lab is considering formally releasing them, OR they might start as RC viewers in their own right without ever having been a project viewer.
    • It is these RC versions that allow the Lab to gather statistics on the behaviour of individual viewers in order to help determine their suitability for promotion to full release status.

Broadly speaking, whether a viewer starts its public life as a project viewer or a release candidate viewer depends on what it contains. A viewer containing a major new feature – such as Animesh, Bakes on Mesh or EEP, for example – will generally make an initial public appearance as a project viewer for the reasons noted above. Maintenance releases, hot fixes, and things like updates to the viewer rendering system will – in general – tend to appear directly as release candidate viewers.

No viewer ever goes directly from project status to a full release – all project viewers will go by way of progressing first to being a release candidate, then being judged as ready for promotion to full release status.

Where to Find Them and How They are Handled

Both types of viewer appear on the Alternate Viewers Page, but how they are handled is somewhat different – and this is one of the important aspects in understanding them.

  • Project viewers are largely “independent” viewer versions.
    • Users must opt to visit the Alternate Viewers Page and select, download and install one.
    • Each project viewer installs into its own dedicated location (although they share the same settings files and cache locations as the release viewer), so they can be run alongside the release version of the official viewer, if installed.
  • Release candidate viewers are considered as “alternative release viewers”.
    • Release candidates are assigned a “cohort number” the Lab believes will present a reasonable cross-section of users.
      • When a release candidate viewer is made available, the system automatically triggers the viewer update process among randomly selected users on the current official release viewer, moving them to the release candidate.
      • When the cohort number for a release candidate viewer is reached, it is no longer made available for automatic download / installation.
      • Once a user has been selected to receive a release candidate version of a viewer, they will continue to receive updates for that particular RC on a mandatory basis until it is promoted to release status – they will not be selected to receive other RCs (or updates to others RCs) until the RC they have been using in promoted to de facto release status.
      • The reason for doing this is to allow the Lab to monitor the performance of individual viewer release candidates and capture data on things like performance, stability, crash rates, etc. This data, together with bug reports, etc., filed by users is then used to determine an individual RC’s suitability for promotion to release status.
    • Alternatively, users can opt to manually install any RC viewer that interests them directly via the Alternate Viewers Page. Again, by default, any RC viewer installed in this way will overwrite any existing installation of the official release viewer (unless an alternative installation location is provided by the user), and the user will thereafter receive updates for that RC.
    • Note that users who do not wish to have RC viewers installed on their system can, if they wish opt out of the release viewer update loop from within their viewer, as shown below.
Users of the official viewer who do not wish to be involved in testing any release candidate viewers can opt out of the selection / download process by unchecking the Willing to Update to Release Candidates option in Preference. Note that having the option checked does not mean you will be subject to RC testing: users for each RC issued by LL are selected entirely at random from the poll of official viewer users.

How are Viewers Progressed to Release?

As noted above, project viewers follow defined path: they initially appear as a public project viewer, and then may go through multiple iterations of improvement, updates, added capabilities, bug fixes, etc., before they reach a point where the Lab determines they are ready for upgrade to release candidate status.

Release candidate viewers are more closely monitored by the Lab through their various cohorts, whilst similarly being subject to multiple iterations designed to remove bugs, help with performance, address further perceived shortfalls in functionality, etc., based on things like bug reports and feature requests from users.

While this approach means that multiple viewers can be developed, tested and readied for promotion to de facto release status, it often means that at any one time, there are several RC viewers vying for release. When this happens:

  • The Lab will select a viewer or promotion based on a number of factors, including stability, performance, number of remaining bugs / issues, the potential impact of said bugs issues, the urgency with which an RC needs to be released (e.g. an “late breaking” RC with a hot fix could well be promoted ahead of other RCs that have been available for longer) and so on.
  • Generally speaking, the Lab tries to promote no more than one RC to de facto release status every two weeks. However, depending on the overall state of individual RC viewers, the period between promotions can be longer.
  • Once a release candidate has been promoted to release status, the first order of business is to merge the code it contains into all over available RC viewers and then monitor them to see how they behave when built using the “new” release code, a process that also feeds back into determining which of them might next be promoted.

What this all Means in Summary

Simply, put, that official viewer and viewer updates can be produced on a rolling basis, with some starting as project viewers, others directly as release candidates, with the latter being objectively monitored both individually and in comparison with one another to determine which is best suited to become the next de facto release viewer.

Simulator Release Process

Simulator releases are also handled via a release candidate mechanism that functions along similar lines to the viewer.

Essentially the grid is split into a number of channels:

  • The core Second Life Server (SLS) channel, also sometimes referred to as the “Main” channel. The accounts for the majority of regions on the grid, and might be regarded as the “release” channel.
  • A number of release candidate channels.
    • As a rule of thumb, there are generally three “primary” RC channels (often referred to as BlueSteel, Magnum and LeTigre), each accounting for – in rounded terms – about 10% of the grid.
    • Depending on requirements, there may also be very small RC channels (Cake and Snack perhaps being the most famous).
      • These can be subsets of regions pulled from SLS and / or one or more of the “primary” RC channels.
      • They are used for initial testing of very specific simulator updates / changes, and the code in them may go on to appear in larger RC deployments.

Simulator updates commence life internally at Linden Lab, where the undergo development and initial testing on small, internal grid environments. They can then progress to the beta grid – called Aditi – where they are referred to as “DRTSIM” updates, and can be set to run on specific regions to allow more general testing, including user testing. This happens with – but is not limited to – things like new SL capabilities that require server-side support, such as the original Mesh support deployment, Bento, Animesh, and EEP.

When deemed ready for main grid – Agni – release, simulator updates will initially be deployed to one or more release candidate channels. How many RC channels receive the update depends on a number of factors, such as:

  • What else the engineering team are working on for the simulators that may also need to be deployed at the same time.
  • What is already on the RC channels and available for promotion to SLS.
  • The kind of update being deployed:
    • Certain bug fixes and smaller updates may be rolled to all of the RC channels, if available, for example, as they are less likely to cause mass disruption if they have any adverse effect.
    • New features, significant updates, etc., may only initially be deployed to a single RC channel, or perhaps 2 of the 3 primary RC channels because of the potential for unintended impact.

These factors, together with things like performance, stability and reported / identified issues also weigh in on how long a release might remain within its RC channel(s).

  • For example, if a release includes significant updates and / or new functionality, it might initially only be deployed to one RC channel,  then after a time be deployed to an second RC alongside the first, exposing it to a larger number of regions / users, before it then gets deployed to the third RC channel alongside of the other two, exposing it to 30% of the total grid and allowing it to be further assessed before being deployed across the entire grid.
  • Another release, however, might initially be deployed to one or two RCs, where it remains for a time for monitoring purposes before it gets promoted to SLS (and absorbed into the version currently on the remaining RC(s) when it is updated).

Thus, through the use of the RC channels, it is possible for the Lab to have simulator updates running in parallel and being tested and monitored for suitability for deployment to the SLS channel. Once a release has been promoted to the SLS channel, the RC channel(s) it was on are available to receive further new updates.

Deployment Days and Promotion

Generally speaking:

  • RC channel deployments are made on Wednesdays. Every RC deployment will spend at least one week on RC (many often spend more than that) prior to being deployed to the SLS channel. The exact channel a release candidate is being deployed to is not always specified by the Lab.
  • SLS deployments are made on Tuesdays, and will comprise the RC deployment deemed by the Lab as being ready for promotion based on things like performance, stability, bug fixes / reported issues, etc.
  • Deployments to the very small RCs like Cake and Snack might eventually move on to being part of larger deployments or may not move beyond their initial testing, and the regions within them re-absorbed into the other channels.

Thus, simulator deployments can be seen s rolling – RC first, then a week (minimum) later, promotion to SLS.

Promotion to SLS can be dependent on a number of factors, including:

  • What updates are currently in the RC channels, how they are performing in terms of stability etc.
  • Whether there is a particularly urgent RC update that needs grid-wide deployment.
  • Whether the update has other dependencies. For example, is it a feature that has corresponding viewer updates that are substantially behind the simulator code such that the simulator code can be held on an RC for longer.

It is entirely possible for no promotion to be made to SLS on a Tuesday as none of the RC releases are deemed as being ready for promotion. Similarly, there might not always be a deployment to the RCs, as the simulator updates under development internally at the Lab might not be ready for a main grid release.

Issues and Roll Backs

Given this approach, people often wonder why the grid can still be hit with significant issues only when a simulator release reached the SLS channel, sometimes prompting a roll back to a previous simulator version, rather than problems being picked up when a release is on one or more RCs.

The answer to this is the sheer diversity of ways in which people use Second Life, including how tools are used, scripts are written and so on.

It is entirely possible that an issue with a specific simulator update escapes notice while on an RC because the sampling of regions on that RC aren’t used in a manner that will highlight the issue sufficiently well enough for it to be recognised as a potential problem; it’s not until the release is in use across the vast majority of the grid (80-100%) that sufficient users or scripts or whatever are exposed to it, that the full impact of the problem is identified.

Release Notes and Release Numbers

Details of upcoming simulator deployments are generally made available through the Technology Forum → Second Life Server. Individual simulator package release notes are available via the Simulator Release Notes Index.

  • Note that the date appearing on the release note pages refers to the initial RC deployment date, and is not updated when an RC deployment is promoted to the SLS channel.
  • Server release numbers are in the format: yyyy-mm-ddThr:mn:sc.XXXXXX where:
    • yyyy-mm-ddThr:mn:sc is the date / time stamp associated with the release generation.
    • XXXXX is the actual release number (e.g. 531148).
    • It is the six digits of the release number that are generally the most important when referencing a release.

What This All Means in Summary

Simulator deployments generally follow a set pattern, thus:

  • Deployments usually occur each week on Tuesday (SLS deployments) and Wednesday (RC deployments).
  • New simulator versions are initially deployed to one or more RC channels on a Wednesday.
    • A RC version will generally spend at least a week on its RC channel(s) after initial deployment (Wednesday through Monday), prior to being considered for deployment to the SLS channel on the following Tuesday.
    • However a RC deployment may go through several iterations in its assigned RC channel(s) and / or be expanded to additional RC channels (and iterated upon) over two or more weeks, prior to being promoted to the SLS channel.
  • When a RC version is deemed ready for promotion to the SLS channel, the deployment to SLS will be made on the next available Tuesday.

Related Links

Advertisements

2 thoughts on “A closer look: Second Life viewer & simulator release processes

  1. Good explanation … one very minor correction: When a user is initially selected to be offered a release candidate version that upgrade is optional; any subsequent upgrades to that candidate will be mandatory.

    Like

Have any thoughts?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.