Lab updates on unscheduled deployments and other issues

secondlifeAs noted in my recent SL project update, there was an unscheduled deployment to the three Agni (main) grid release candidate (RC) channels of Bluesteel, LeTigre and Magnum on Thursday, February 18th, which saw regions on these channels undergo a rolling restart. This was followed on Friday, February 19th by rolling restarts across the Main (SLS) channel.

During the Server Beta User Group (SBUG) meeting on Thursday, April 18th, Steven Linden provided some information on why a deployment was made to the RC channels, and indicated that a similar deployment would be forthcoming on the Main (SLS) channel, and promising further information would be provided once that deployment had been made:

We had an unscheduled RC deploy earlier today. It’s for a security vulnerability that was released, and we discovered that Second Life regions were vulnerable. A full public post-mortem will be coming after we deploy to the rest of the main grid. I can’t say until it goes out to the rest of Agni; I can say that it was related to region availability only…. I honestly can’t say a great deal, other than we have a fix, and that it’s coming very soon to the rest of Agni.

True to this promise, following the Main channel roll on Friday, February 19th, April Linden blogged Why the Friday Grid Roll?

The reason essentially boiled down to a vulnerability in the GNU version of Linux used to run the grid servers. The vulnerability lay within the GNU C library, commonly referred to as glibc, which if exploited could allow remote access to a devices – be it a computer, internet router, or other connected piece of equipment. It was initially discovered by Google on Tuesday, 16th February, and was labelled CVE-2015-7547.

April’s blog post provides a concise explanation of just what went into the Lab’s security and operations teams’ efforts in ascertaining SL’s exposure to the vulnerability and developing an update to secure their servers against the vulnerability.

All of this took time – but all things considered, it was still a remarkably fast effort. The Lab went from hearing about the risk on Tuesday 16th February through to understanding the full extent of the possible exposure SL faced, to having an update coded, tested and ready for release by Thursday, which as April explained, then left them with another decision:

Do we want to roll the code to the full grid at once? We decided that since the updates were to one of the most core libraries, we should be extra careful, and decided to roll the updates to the Release Candidate (RC) channels first. That happened on Thursday morning.

Given the Lab wanted to monitor how things progressed on the RC channels (which between them represent roughly 30% of the total grid), and ensure the update itself didn’t introduce anything unexpected. So it was that the deployment to the rest of the grid couldn’t be made until Friday, February 19th.

April emphasises that at no point during the known period of exposure or before, was there any attempt to use the vulnerability against the SL servers.  At the time of the Thursday roll, there was some criticism directed at the Lab for the lack of warning. April also explains why this was the case:

The reason there was little notice for the roll on Thursday is two-fold. First, we were moving very quickly, and second because the roll was to mitigate a security issue, we didn’t want to tip our hand and show what was going on until after the issue had been fully resolved.

When things like unscheduled rolls are disruptive, leaving us prone to grumbling and pointing the finger, it’s perhaps worthwhile taking this incident as an example that sometimes, there are reasons why the Lab does announced things first.

April’s post is actually one of three published recently by the operations / engineering teams which provide interesting insight into what goes on behind the scenes in keeping Second Life running.

In Recent Issues with the Nightly Biller, Steven Linden provides and explanation on why some Premium members recently experienced billing issues, up to and including inadvertently receiving delinquent balance notices. Once again, the explanation of what happened and what has been done to try to ensure a similar problem doesn’t occur in the future makes for a worthwhile read.

In Tale of the Missing ACK, Chris Linden describes another unusual and challenging incident the Lab’s engineering team had to deal with when testing a new API endpoint hosted in Amazon. This again illustrates the overall complexity of the Second Life services and infrastructure, which extends far beyond the simulator servers we some often take for granted as being “the” SL service, and the complexities involved in tracking issues down when things don’t go as expected  / planned.

Thanks again to April, Steven and Chris for providing the explanations and the insight into SL’s services.

Lab Chat: January 21st, 2016: 10:30 SLT

Lab Chat LogoDon’t forget that recording for the second instalment of Lab Chat takes place on Thursday, January 21st, at 10:30 (am) SLT , at the LEA Threatre in Second Life.

The guest will once again be Linden Lab CEO Ebbe Altberg,through his alter ego, Ebbe Linden. He’ll be answering questions selected from the 80+ submitted to the Lab Chat forum. The selected questions include the following topics:

  • Project Bento
  • Notifying users in-world about technical issues occurring with Second Life
  • SL credit processing
  • How is “Sansar” to be regarded – a place to live, or a place to visit?
  • Sansar avatars
  • Sansar in-world and out-world building tools.

In addition, and time allowing, there may be questions on things as diverse as land, art, sailing and aviation in Sansar, permissions systems and (inevitably!) Oculus Rift.

Ebbe Linden takes questions from jo Yardley and Saffia Widdershins during the first Lab Chat discussion in November 2015
Ebbe Linden takes questions from jo Yardley and Saffia Widdershins during the first Lab Chat discussion in November 2015

So, make sure you get a seat during the live recording. You can access the LEA Theatre via any of the following SLurls:

If you can’t make the recording, videos of the show will be available in due course, and I’ll have a transcript of the Q&A both here and on the Lab Chat website iASAP after the show.

Lab: share a Valentine’s vision and help promote SL

With Valentine’s Day roughly a month away, the Lab is inviting residents to share what that special day means to them by means of a photograph – and gain the opportunity to be a part of the Lab’s advertising campaigns for Second Life.

The post reads in part:

Valentine’s Day is just on the horizon and will be here before you can get through a handful of heart candies! We’re looking for some great Second Life pictures from the community to feature in upcoming email and banner campaigns – with credit to the photographer – of course!

In a snapshot, let us know what Valentine’s Day means to you in Second Life.

Share your story in a picture and on our Official Flickr Page with the tag “SLVday2016” so that we can see all of your amazing works. You may submit as many as you like between now and January 29, 2016.

We’ll showcase the chosen image(s) in an email to Residents, as well as in some banner campaigns. We’ll let the chosen image creators know via Flickr and ask for an avatar name for the credits.

The submission guidelines are fairly straightforward:

  • Images must be at least 2048×1207 in size
  • Avatars must be shown in-world,and suitably lit so they can be seen
  • All images must be free of additional text or logos, and must be appropriate for all audiences.
  • Images that tell a story are going to make an impression.

As noted in the Lab’s blog post text, the closing date for submissions is Friday, January 29th, 2016.

 

Lab explains Second Life’s weekend woes

We’re all used to Second Life misbehaving itself at the weekend, but it with rezzing or rendering or region crossings and so on. However, Saturday, January 9th, and Sunday January 10th proved to be a lot rougher than most weekend in recent memory, with Sunday in particular affecting a lot of SL users.

When situations like this arise, it’s easy to shake a verbal fist at “the Lab” and bemoan the situation whilst forgetting we’re not the only one being impacted. Issues and outages bring disruption to the Lab as well, and often aren’t as easy to resolve as we might think. Hence why it is always good to hear back from the Lab when things do go topsy-turvy – and such is the case with the weekend of the 9th / 10th January.

Posting to the Tools and Technology blog on Monday, January 11th, April Linden, a member of the Operations Team (although she calls herself a “gridbun” on account of her purple bunny avatar), offered a concise explanation as to what happened from the perspective of someone at the sharp end of things.

April starts her account with a description of the first issue to hit the platform:

Shortly after midnight Pacific time on January 9th (Saturday) we had the master node of one of the central databases crash. The central database that happened to go down was one the most  used databases in Second Life. Without it Residents are unable to log in, or do, well, a lot of important things.

While the Lab is prepared for such issues, it does take time to deal with them (in this case around 90 minutes), with services having to be shut-down and then restarted in a controlled manner so as not to overwhelm the affected database. Hence why, when things like this do happen, we often see notices on the Grid Status Page warning us then log-ins may be suspended and /  or to avoid carrying out certain activities.

Sadly, this wasn’t the end of matters; on Sunday an issue with one of the Lab’s providers had a major impact on in-world asset loading (while April doesn’t specifically point at which provider, I’m assuming from her description it may have been one of the CDN providers). While the Lab is versed in working with their providers to analyse the root cause of problems and rectify them, this particular issue appears to have had a knock-on effect in a quite unexpected way, impacting the avatar baking service.

This is the mechanism by which avatar appearances are managed and shared (and is also known as Sever-Side Appearance and / or Server-Side Baking). Designed to overcome limitations with using the viewer / simulator to handle the process, it was cautiously deployed in 2013 after very extensive testing, and it has largely operated pretty reliably since its introduction. As such, the fact that it was so negatively impacted at the weekend appears to have caught the Lab off-guard, with April noting:

One of the things I like about my job is that Second Life is a totally unique and fun environment! (The infrastructure of a virtual world is amazing to me!) This is both good and bad. It’s good because we’re often challenged to come up with a solution to a problem that’s new and unique, but the flip side of this is that sometimes things can break in unexpected ways because we’re doing things that no one else does.

Taking this to be the case, it doubtless took the Lab a while to figure-out how best to deal with the situation, which likely also contributed to the time taken for things to be rectified to the point where people weren’t being so massively impacted. Hopefully, what did occur at the weekend will help the Lab better assess circumstances where such problems – unique as they may be – occur, and determine courses of action to mitigate them in the future.

In the meantime, April’s post, like Landon Linden’s update on the extended issues of May 2014, help remind us of just what a hugely complex beast of systems and services Second Life is, and that how even after 13 years of operations, it can still go wrong in ways that not only frustrate users, but also take the Lab by surprise, despite their best efforts. Kudos to April for presenting the explanation and for apologising for the situation. I hope she, together with all involved, have had time to catch-up on your sleep!

Related Links

Lab Chat #2 announced

Lab Chat LogoLab Chat is the name of the new series aimed at providing Second Life users with the opportunity to have their questions put to Lab management and personnel.

The first such event, which was recorded and made available as a three-part video, took place on Thursday, November 19th, with guest Ebbe Altberg, CEO of Linden Lab. I have a full transcript of that event available, complete with audio of both the questions asked and Ebbe’s replies.

The second on the series have now been announced through the Lab Chat blog and via an official blog post from the Lab. It will take place on Thursday, January 21st, again at the Linden Endowment for the Arts Theatre, starting at 10:30 SLT.

Ebbe Altberg, in his alter ego of Ebbe Linden, will again be facing questions on Linden Lab, Second Life and "Project Sansar" as put forward by SL users
Ebbe Altberg, in his alter ego of Ebbe Linden, will again be facing questions on Linden Lab, Second Life and “Project Sansar” as put forward by SL users

Once again, the guest under the spotlight will be the Lab’s CEO, Ebbe Altberg.

As with the first event, people are being invited to submit questions via a Lab Chat forum thread they’d like to see asked during the show for consideration. The deadline for such questions is Friday, January 15th

Questions to be asked will be selected by members of the Lab Chat production team (of which I’m listed, although my role is focused on producing written transcripts of recordings, rather than being hands-on with the actual production). Those submitting selected questions will be invited to attend the recording of the session, and put their question directly to Ebbe in voice if they wish; otherwise questions will be asked the Lab Chat hosts, Saffia Widdershins and Jo Yardley.

As per the first Lab Chat, I will (hopefully) be recording the event, and will certainly be providing a full transcript a few days after it  has taken place.

In the Press: a virtual world without Parkinson’s

Tom Boellstorff and his digital alter ego, Tom Bukowski (image: Steve Zylius / UCI)
Tom Boellstorff and his digital alter ego, Tom Bukowski (image: Steve Zylius / UCI)

There are probably few Second Life residents who keep an eye on blogs and the more well researched media reports on Second Life who are unaware of Fran Seranade’s story.  Now 88 years old, Fran is a keen Second Life resident, who spends time in-world with her daughter and son, enjoying much of what the platform can offer: dancing, Tai Chi, swimming, horseback riding, walking, exploring. All far removed from the physical world, where she suffers from Parkinson’s Disease, which deeply affects her ability to do many of the things we take for granted: such as simply getting out of a chair and walking to the next room.

Fran’s story, and that of Creations for Parkinson’s and their support of Team Fox, the fund-raising arm of the Michael J. Fox Foundation for Parkinson’s Research (MJFF)  first came to my attention in September 2013, thanks to an excellent article in the San Diego Union-Tribune by Pam Kragen. Fran’s story has also been related by Draxtor Despres in his World Makers video series, as well as being featured in numerous other SL-related blogs.

Fran Swenson (Fran serenade in Second Life), seen with her avatar and friends in Second Life
Fran Swenson (Fran Seranade in Second Life), seen with her avatar and friends in Second Life

It is now also the story used to introduce an article appearing in UCI News, the on-line newsletter of the University of California, Irvine, which explores the work of Tom Boellstorff, a professor of anthropology at the university.

In A (virtual) world without Parkinson’s, Heather Ashbach talks to Tom about his ongoing work with Fran and other Parkinson’s Disease sufferers using Second Life, which is now part of a US $276,900 National Science Foundation study that began in July 2015, and will run through until June 2018, examining how on-line environments affect social interaction and self-understanding as well as physical-world experiences of disability.

Tom's work has also been more broadly covered through The Drax Files World Makers #31, which I covered here
Tom’s work has also been more broadly covered through The Drax Files World Makers episode #31, which I covered here

Tom’s work as an anthropologist studying our digital interactions with ourselves and others has already given rise to two books, Coming of Age in Second Life: An Anthropologist Explores the Virtually Human, and Ethnography and Virtual Worlds: A Handbook of Method. With this study, he hopes to build on the knowledge he’s acquired over the last several yeas of study, and also learn which aspects of on-line social interaction are linked to specific disabilities and how differing platforms and devices are utilised.

“Billions of people now participate in on-line environments – social networking sites like Facebook, games, virtual worlds,” he notes in the article. “Since some early human first picked up a stick to use as a cane, persons with disabilities have been at the forefront of technology innovation, so it’s natural that they would be doing creative things in virtual worlds too.”

His examination of Parkinson’s and the effect Second Life has had on those suffering from it is not limited to Fran; the article specifically touches on the lives of Solas NaGealai, who founded the original Team Fox presence in Second Life (and whose story I covered in August 2014), and architect DB Bailey, and they are two among many Tom has talked to within Second Life.

Second Life has provided Solas wearing one of her own gowns
Second Life has provided Solas NaGealai to continue her love of fashion and design, and provided her with the means to directly support MJFF through Team Fox SL.

Solas was diagnosed with a particular form of Parkinson’s Disease known as Young Onset Parkinson’s Disease (YOPD) – the same form of the disease which struck Michael J. Fox.

When first diagnosed, Solas was a full-time fashion designer. However, as the illness progressed, she was forced to leave that career behind. Fortunately, her discovery of Second Life allowed her a way to re-engage in her passion for design, and she founded her own fashion label at Blue Moon Enterprise.

Prior to his diagnosis, DB Bailey was a noted architect who, among other projects, collaborated on the design of Los Angeles’ Walt Disney Concert Hall. Since the disease has forced him to step back from that career, he’s been able to use Second Life as his creative outlet as an architect and artist.

Tom’s work, which has involved in-world and face-to-face interviews and studies, has also touched upon the possible physical benefits some of those with disabilities report as a result of their use of Second Life. In Fran’s case, for example, she his indicated an improvement in her sense of balance and co-ordination as a result of using the platform. This is something Tom has suggested might be down to what is called “mirror neurons“, with Fran’s brain responding positively to the visual stimulus she receives as a result of her in-world activities. However, examining such potential benefits does not form a part of the current study, primarily because Tom is not a medical doctor; but he is of the opinion that they are worth  future examination.

The psychological and sociological benefits of using platforms like Second Life in matters of healthcare are clear, and have been the subject of many studies. With the promise of more immersive VR experiences and opportunities now on the horizon, a better understanding of how different platforms and mediums assist those with disabilities enjoy a richer, fuller life is liable to be of enormous benefit, and also offers an opportunity for broader public discussion on matters of disability and the use of technology.

As Tom himself notes in closing the interview with Heather Ashbach, “disability is a fascinating category of human experience because anyone can enter it at any moment – we’re all just a car accident away. And even if we avoid it, if we live long enough, age will present us with obstacles that limit our ability to function as we currently do. Studying how people adapt through the use of technology helps advance the public conversation on disability and digital technology.”

Altogether, an excellent read, and one thoroughly recommended.