Lab updates on unscheduled deployments and other issues

secondlifeAs noted in my recent SL project update, there was an unscheduled deployment to the three Agni (main) grid release candidate (RC) channels of Bluesteel, LeTigre and Magnum on Thursday, February 18th, which saw regions on these channels undergo a rolling restart. This was followed on Friday, February 19th by rolling restarts across the Main (SLS) channel.

During the Server Beta User Group (SBUG) meeting on Thursday, April 18th, Steven Linden provided some information on why a deployment was made to the RC channels, and indicated that a similar deployment would be forthcoming on the Main (SLS) channel, and promising further information would be provided once that deployment had been made:

We had an unscheduled RC deploy earlier today. It’s for a security vulnerability that was released, and we discovered that Second Life regions were vulnerable. A full public post-mortem will be coming after we deploy to the rest of the main grid. I can’t say until it goes out to the rest of Agni; I can say that it was related to region availability only…. I honestly can’t say a great deal, other than we have a fix, and that it’s coming very soon to the rest of Agni.

True to this promise, following the Main channel roll on Friday, February 19th, April Linden blogged Why the Friday Grid Roll?

The reason essentially boiled down to a vulnerability in the GNU version of Linux used to run the grid servers. The vulnerability lay within the GNU C library, commonly referred to as glibc, which if exploited could allow remote access to a devices – be it a computer, internet router, or other connected piece of equipment. It was initially discovered by Google on Tuesday, 16th February, and was labelled CVE-2015-7547.

April’s blog post provides a concise explanation of just what went into the Lab’s security and operations teams’ efforts in ascertaining SL’s exposure to the vulnerability and developing an update to secure their servers against the vulnerability.

All of this took time – but all things considered, it was still a remarkably fast effort. The Lab went from hearing about the risk on Tuesday 16th February through to understanding the full extent of the possible exposure SL faced, to having an update coded, tested and ready for release by Thursday, which as April explained, then left them with another decision:

Do we want to roll the code to the full grid at once? We decided that since the updates were to one of the most core libraries, we should be extra careful, and decided to roll the updates to the Release Candidate (RC) channels first. That happened on Thursday morning.

Given the Lab wanted to monitor how things progressed on the RC channels (which between them represent roughly 30% of the total grid), and ensure the update itself didn’t introduce anything unexpected. So it was that the deployment to the rest of the grid couldn’t be made until Friday, February 19th.

April emphasises that at no point during the known period of exposure or before, was there any attempt to use the vulnerability against the SL servers.  At the time of the Thursday roll, there was some criticism directed at the Lab for the lack of warning. April also explains why this was the case:

The reason there was little notice for the roll on Thursday is two-fold. First, we were moving very quickly, and second because the roll was to mitigate a security issue, we didn’t want to tip our hand and show what was going on until after the issue had been fully resolved.

When things like unscheduled rolls are disruptive, leaving us prone to grumbling and pointing the finger, it’s perhaps worthwhile taking this incident as an example that sometimes, there are reasons why the Lab does announced things first.

April’s post is actually one of three published recently by the operations / engineering teams which provide interesting insight into what goes on behind the scenes in keeping Second Life running.

In Recent Issues with the Nightly Biller, Steven Linden provides and explanation on why some Premium members recently experienced billing issues, up to and including inadvertently receiving delinquent balance notices. Once again, the explanation of what happened and what has been done to try to ensure a similar problem doesn’t occur in the future makes for a worthwhile read.

In Tale of the Missing ACK, Chris Linden describes another unusual and challenging incident the Lab’s engineering team had to deal with when testing a new API endpoint hosted in Amazon. This again illustrates the overall complexity of the Second Life services and infrastructure, which extends far beyond the simulator servers we some often take for granted as being “the” SL service, and the complexities involved in tracking issues down when things don’t go as expected  / planned.

Thanks again to April, Steven and Chris for providing the explanations and the insight into SL’s services.

Advertisements

Within the Shadows of a Dream in Second Life

Shadows of a Dream - Nitroglobus Hall
Shadows of a Dream – Nitroglobus Hall

Shadows of a Dream is the title of the latest exhibition at Nitroglobus Hall, curated by Dido Haas,  which has a soft opening on Friday, February 19th, and a formal opening on Sunday, February 21st at 12:30 SLT.

The exhibition features the work of two artists, Angelika Corral and SheldonBr, who together present a series of mostly nude, black-and-white avatar studies. These are offered in two contrasting styles, charcoal sketches (by Sheldonbr) and digital images (by Angelika), all of which are presented the very large format familiar to Nitroglobus exhibitions.

Shadows of a Dream
Shadows of a Dream – Nitroglobus Hall

“Charcoal drawings can be seen in the earliest primitive caves of early humans. Digital images are a clear sign of the evolution of the species – the use of computers,” the artists note of the exhibition. They continue, “However, art remains the same; an idea, an attitude; the vital expression of oneself and art isn’t dependent on a specific medium.”

The result is a stunning sequence of images around the walls of the hall which invite the observer, “to question the value of the arts in its more pure essence; the art of representation, using only light and dark,” as they embark on “a poetic journey through the depiction of the figure gesture. It is a representation of the duality between real life and the digital world (the artists prefer this term instead of second life), and the contrast between light and shadow.”

Shadows of a Dream - Nitroglobus Hall
Shadows of a Dream – Nitroglobus Hall

To try to describe the images with dry words is something of a futile effort; they all speak eloquently and clearly for themselves. Each one, whether charcoal or digital, has its own vital essence which is best experienced first-hand.

The gallery space itself also adds significantly to the pieces, further encouraging the need to see them first-hand. Divided in to two distinct spaces by a translucent awash in an animated mist or smoke, the gallery space allows the images on display to be “reflected” in the floor by the use of duplicate images mounted in the space below. This, coupled with the presence of sculptures by the late Nitro Fireguard which are not similarly “reflected” in the floor, adds to the dream-like aspect of the exhibit in keeping with its title.

Shadows of a Dream - Nitroglobus Hall
Shadows of a Dream – Nitroglobus Hall

Shadows of a Dream is a superb offering from two talented artists, and an exhibition not to be missed. Should you wish to attend the formal opening, please be advised that attendees are asked to wear something black and/or white, in keeping with the theme of the exhibition.

SLurl Details

SL project updates 16 7/2: Unscheduled server deployments; SL viewer

Goatswood; Inara Pey, June 2015, on Flickr Recalling  Goatswood) – blog post

Unscheduled Server Deployments

Update, February 19th, 22:40 GMT: the SLS channel restarts have been completed and the Lab has issued a blog post on why they were required, which I’ve also blogged about

Update, February 19th: the deployment of the update referred to below will commence at 15:00 SLT

On Thursday, February 18th, there was an unscheduled server deployment to all three RC channels, which at the time of deployment was described as an, “Update on the simhosts. Nothing is changing Second Life functionality wise.”

Speaking at the Server Beta User Group meeting following the deployment, Steven Linden had this to say:

We had an unscheduled RC deploy earlier today. It’s for a security vulnerability that was released, and we discovered that Second Life regions were vulnerable. A full public post-mortem will be coming after we deploy to the rest of the main grid. I can’t say until it goes out to the rest of Agni; I can say that it was related to region availability only…. I honestly can’t say a great deal, other than we have a fix, and that it’s coming very soon to the rest of Agni.

All Steven could say about the issue was that a) it was related to region availability; b) it could only be exploited from within Second Life; c) there has been no evidence the issue  is being actively exploited on Agni.

However, given the apparent urgency of the situation, it is likely that the update deployed to the RC channels will be also be rolled to the Main (SLS) channel well ahead of Tuesday, the normal day for Main channel deployments and restarts.

I’ll have more on this following the post-mortem release from the Lab.

Scheduled Updates

Details are scant at the moment, but Wednesday, February 24th should see a new server maintenance package which includes some code clean up around the area of parcel bans. There’s no new functionality being added, and the changes shouldn’t break anything. More details when the update notes are published.

SL Viewer

The Quick Graphics RC viewer updated on Wednesday, February 17th to version 4.0.2.311103. This sees the addition of the following resolved issues:

  • MAINT-1945: Outgoing packets logging always says 0 bytes
  • MAINT-5613:  Complexity readings vary greatly for each avatar using the QuickGraphics viewer
  • MAINT-5620: Clicking on Graphics Preset title triggers favourite
  • MAINT-5681: Particles still render when complexity threshold is reached
  • MAINT-5682: Some avatars are invisible
  • MAINT-5685: Light still renders when complexity threshold is reached
  • MAINT-5690: Viewer crash when zooming out
  • MAINT-6070: Add detailed logging for how Avatar Rendering Complexity is computed.

The updates also sees the removal of SL-217: Document Avatar Complexity, from the list of resolved issues, presumably because the documentation is still a work-in-progress.

Other Items

Aditi Intellectual Property Tutorial

As mesh content creators are aware, in order to be able to upload mesh content to Second Life, you must a) have payment information on file, and b) complete the Intellectual Property Tutorial.  The same is also true for Aditi; however, a problem with the Aditi services has meant that some people have been unable to complete the tutorial there (accessed when you log-in to your Aditi dashboard), due to the test page failing to load / failing to display all the questions.

If you wish to use Aditi to upload test models of your mesh content, but have encountered issues in trying to complete the tutorial, the interim workaround is to try refreshing the page to force it to load, as there appears to be a load balancing issue in the Aditi back-end services. However, the issue is expected to be resolved for next week.