Tag: SL Issues

Tuesday, August 23rd was not a particularly good day for Second Life, with an extended period of unscheduled maintenance with log-ins suspended and those in-world advised to refraining from rezzing No Copy objects, or making any LindeX related transactions, etc.

At the time of the problem, there was speculation that it might be due to further issues with the central database node (and kudos for Caitlyn for suggesting this 🙂 ). Writing in a Tools and Technology blog post on August 24th, Operations Team lead April Linden confirmed this was in fact the case:

Shortly after 10:30am [PDT], the master node of one of the central databases crashed. This is the same type of crash we’ve experienced before, and we handled it in the same way. We shut down a lot of services (including logins) so we could bring services back up in an orderly manner, and then promptly selected a new master and promoted it up the chain. This took roughly an hour, as it usually does.

Given this has happened in the relatively recent past (see here and here), the Ops Team are getting pretty good with handling these situations. Except this time there was a slight wrinkle in the proceedings. The previous failures had occurred when concurrency was relatively low due to the times they occurred. This time, however, the problem hit when rather a lot of people were trying to get into SL, so as April notes:

A few minutes before 11:30am [PDT] we started the process of restoring all services to the Grid. When we enabled logins, we did it in our usual method – turning on about half of the servers at once. Normally this works out as a throttle pretty well, but in this case, we were well into a very busy part of the day. Demand to login was very high, and the number of Residents trying to log in at once was more than the new master database node could handle.

Around noon we made the call to close off logins again and allow the system to cool off. While we were waiting for things to settle down we did some digging to try to figure out what was unique about this failure, and what we’ll need to do to prevent it next time.

It wasn’t actually until a third attempt was made to bring up the login hosts one at time that things ran smoothly, with services being fully restored at around 2:30pm PDT.

Now, as April notes, she and her team have a new challenge to deal with: understanding why they had to turn the login servers back on much more slowly than in the past. There is, however, a bright spot in all this: the work put into making the Grid Status feed more resilient paid off, with the service appearing to cope with the load placed on it by several thousand people trying to discover what was going on.

None of us like it when the go wrong, but it’s impossible for SL to be all plain sailing. What is always useful is not only being kept informed about what is going on when things do get messed up (and don’t forget, if you’re on Twitter you can also get grid status updates there as well), but in also being given the opportunity to understand why things went wrong after the fact.

In this respect, April’s blog posts are always most welcome, and continue to be an informative read, helping anyone who does so just what a complicated beast Second life is, and how hard the Lab actually does work to try to keep it running smoothly for all of us – and to get on top of this as quickly as they can when things do go wrong.

I normally keep a close eye on outgoing communications from the Lab, but this week I’ve had other things distracting me, and so haven’t been keeping an eye on the official blog for posts and updates. My thanks therefore to reader BazdeSantis for pointing me to April Linden’s Tools and Technology update, The Story Behind Last Week’s Unexpected Downtime.

April has very much become the voice of the Lab’s Operations team, and has provided us with some excellent insights to Why Things Sometimes Went Wrong – a valuable exercise as it increases both our understanding of the complexities inherent in Second Life, and also what is likely to be going on behind the scenes when things do go drastically sideways.

April’s post refers to the issues experienced on Friday May 6th, when a primary node of a central database failed, with April noting:

The database node that crashed holds some of the most core data to Second Life, and a whole lot of things stop working when it’s inaccessible, as a lot of Residents saw.

When the primary node in this database is off-line we turn off a bunch of services, so that we can bring the grid back up in a controlled manner by turning them back on one at a time.

There’s an interesting point to note here. This is the same – or very similar – issue to that which occurred in January 2016, which again goes to show that given the constant usage it sees, Second Life is a volatile service  – and that the Operations team are capable of turning major issues around in a remarkably short time; around 90 minutes in January, and less than an hour this last time.

Both events were also coupled with unexpected parallel issues as well: in January,  the database issue was followed by issues with one of the Lab’s service providers – which did take a while to sort out. This time it was the Grid Status service. As I’ve recently reported, the Grid Status web pages have recently moved to a new provider. A couple of changes resulting from this have been with the RSS Feed, and integrating the Grid Status reporting pages with the rest of the Lab’s blog / forum Lithium service. However, as April also notes:

It can be really hard to tune a system for something like a status blog, because the traffic will go from its normal amount to many, many times that very suddenly. We see we now have some additional tuning we need to do with the status blog now that it’s in its new home.

She also points out that people with Twitter can also track the situation with Second Life by following the Grid Status Twitter account.

April’s posts are always welcome and well worth reading, and this one is no exception. We obviously don’t like things when the go wrong, but it’s impossible for SL to be all plain sailing. So, as I’ve said before (and above), hearing just what goes on behind the scenes to fix things when the do go wrong helps remind and inform us just how hard the Lab actually doe work to keep the complexities of a 13-year-old platform running for us to enjoy.

As noted in my recent SL project update, there was an unscheduled deployment to the three Agni (main) grid release candidate (RC) channels of Bluesteel, LeTigre and Magnum on Thursday, February 18th, which saw regions on these channels undergo a rolling restart. This was followed on Friday, February 19th by rolling restarts across the Main (SLS) channel.

During the Server Beta User Group (SBUG) meeting on Thursday, April 18th, Steven Linden provided some information on why a deployment was made to the RC channels, and indicated that a similar deployment would be forthcoming on the Main (SLS) channel, and promising further information would be provided once that deployment had been made:

We had an unscheduled RC deploy earlier today. It’s for a security vulnerability that was released, and we discovered that Second Life regions were vulnerable. A full public post-mortem will be coming after we deploy to the rest of the main grid. I can’t say until it goes out to the rest of Agni; I can say that it was related to region availability only…. I honestly can’t say a great deal, other than we have a fix, and that it’s coming very soon to the rest of Agni.

True to this promise, following the Main channel roll on Friday, February 19th, April Linden blogged Why the Friday Grid Roll?

The reason essentially boiled down to a vulnerability in the GNU version of Linux used to run the grid servers. The vulnerability lay within the GNU C library, commonly referred to as glibc, which if exploited could allow remote access to a devices – be it a computer, internet router, or other connected piece of equipment. It was initially discovered by Google on Tuesday, 16th February, and was labelled CVE-2015-7547.

April’s blog post provides a concise explanation of just what went into the Lab’s security and operations teams’ efforts in ascertaining SL’s exposure to the vulnerability and developing an update to secure their servers against the vulnerability.

All of this took time – but all things considered, it was still a remarkably fast effort. The Lab went from hearing about the risk on Tuesday 16th February through to understanding the full extent of the possible exposure SL faced, to having an update coded, tested and ready for release by Thursday, which as April explained, then left them with another decision:

Do we want to roll the code to the full grid at once? We decided that since the updates were to one of the most core libraries, we should be extra careful, and decided to roll the updates to the Release Candidate (RC) channels first. That happened on Thursday morning.

Given the Lab wanted to monitor how things progressed on the RC channels (which between them represent roughly 30% of the total grid), and ensure the update itself didn’t introduce anything unexpected. So it was that the deployment to the rest of the grid couldn’t be made until Friday, February 19th.

April emphasises that at no point during the known period of exposure or before, was there any attempt to use the vulnerability against the SL servers.  At the time of the Thursday roll, there was some criticism directed at the Lab for the lack of warning. April also explains why this was the case:

The reason there was little notice for the roll on Thursday is two-fold. First, we were moving very quickly, and second because the roll was to mitigate a security issue, we didn’t want to tip our hand and show what was going on until after the issue had been fully resolved.

When things like unscheduled rolls are disruptive, leaving us prone to grumbling and pointing the finger, it’s perhaps worthwhile taking this incident as an example that sometimes, there are reasons why the Lab does announced things first.

April’s post is actually one of three published recently by the operations / engineering teams which provide interesting insight into what goes on behind the scenes in keeping Second Life running.

In Recent Issues with the Nightly Biller, Steven Linden provides and explanation on why some Premium members recently experienced billing issues, up to and including inadvertently receiving delinquent balance notices. Once again, the explanation of what happened and what has been done to try to ensure a similar problem doesn’t occur in the future makes for a worthwhile read.

In Tale of the Missing ACK, Chris Linden describes another unusual and challenging incident the Lab’s engineering team had to deal with when testing a new API endpoint hosted in Amazon. This again illustrates the overall complexity of the Second Life services and infrastructure, which extends far beyond the simulator servers we some often take for granted as being “the” SL service, and the complexities involved in tracking issues down when things don’t go as expected  / planned.

Thanks again to April, Steven and Chris for providing the explanations and the insight into SL’s services.