Tag: SL Issues

The last full week of January 2022 saw Second Life experience a number of hiccups that causes no small amount of gnashing of teeth and rumblings from those of us affected.

With April Linden now departed, I wondered if we’d get any formal notification as to what what wrong – and on Monday January 31st, we indeed did, when the Lab blogged Recent Outages.

In short:

• The week-long, if intermittent (for some) inventory issues: appear to have boiled down to an infrastructure overload, with an initial fix put in place followed by a back-end deployment it is hoped will prevent any recurrence.
• The SLS Main channel restart freeze: for around 40% of the grid: a change in the simulator restart code resulted in some 12,000 simulators on the main grid attempting to re-start simultaneously, overloading the system, with many then hanging mid-restart. Steps have been taken to prevent this happening again.
• Wednesday January 26th roll-out / rollback: an attempt to deploy (and then rollback) tools intended to help gather information on group chat performance hit problems, resulting in issues with login, group chat, and presence information, requiring Operations to intervene and crank the rollback.

The official post isn’t quite to April’s level of detail although it is chattier than the above, and provides all the info for who may need further info should it be required. 🙂 .

Sunday, August 30th saw some hiccups in people’s ability to connect to Second Life, with users either unable to log-in or, if already logged, abruptly found themselves abruptly disconnected and unable to log back in.

For some, the issues were relatively transitory (I was logged out and unable to log back in  for about 20 minutes) whilst others were subjected to longer periods of frustration being expressed at the lack of any immediate status feed updates.

On Monday, August 31st, April Linden blogged as to why this was the case.

In short, the issue wasn’t with Second Life; rather, US-based CenturyLink/Level(3), a global supplier of Internet bandwidth providing Internet services via their Tier 1 network to Internet carriers in Europe, Asia, and North America, suffered a significant outage. As a result of this many services and users around the world suffered issues in network connectivity / their ability to connect to the Internet. However, from the Lab’s operational perspective, nothing initially appeared to be wrong: all services were running, no alerts were received, and no alarms triggered. However, as April notes in her blog post:

Of course, from the Resident point of view, Second Life was effectively down in some parts of the world, and that’s really what matters.

To help us react quicker in the future we’ve made a few changes.

Yesterday evening we added a new monitoring service that checks on some of Second Life’s core systems from all around the globe. It’s a service that a lot of other companies use too, so we’ll get alerted better in the future. When Internet-scale events like yesterday happen there’s not a lot we can do about it, but we can post on the status page quicker to let our Residents know we’re aware things aren’t right.

We’re sorry for the lack of communication yesterday. We know how important Second Life is to our Residents, and we’re taking steps to increase our visibility into issues outside of our servers. It’s our hope that these steps will enable us to communicate better with y’all in the future.

See you inworld!

April Linden
Second Life Operations Manager

I’ve long appreciated April’s blog posts, as not only do they help explain the complexities of Second Life and when things can go wrong as and when they do, they also help to remind us that using Second Life isn’t simply a matter of the viewer and the simulator it is connected to. There are a lot of intermediary services and steps that can also cause problems for users, and which lie well outside of Linden Lab’s sphere of influence and ability to rectify. In this particularly case, April’s post also shows that even when the latter is the case, it doesn’t stop her team from trying to tweak / improve things so they can be better informed about potential issues in the future.

So thank you again, April, for keeping us informed and educated!

On November 4th, some users on the Marketplace who accessed their account page ended up seeing some account details for another user currently logged-in to the Marketplace at the same time.

The user account page gives a user’s SL account name, L$ balance, a small portion of their activity on the  Marketplace activity, their wish lists, received gifts list, and the obfuscated version of their e-mail address (e.g. i****@g****.com, designed to provide the user with enough information to identify their own e-mail address without revealing it to others).

Multiple bug reports on the issue were raised with Linden Lab, and at least one forum thread was raised on the subject, with some pointing to the Marketplace maintenance that was in progress as a possible cause – and they were right, as the Lab’s Second Life Operations Manager has revealed in a blog post (Report on the Recent Marketplace Issue), that reads in part:

We’ve been working to make the Second Life Marketplace more robust and handle higher numbers of page views at once. Due to a change made this morning, the user account page got cached when we didn’t mean for it to be. Once we realised what had happened, we rolled back the changes immediately and deleted all of our caches. No other parts of Second Life were impacted.

Our engineering teams are now working with our QA (quality assurance) team to make sure we develop better testing for this in the future. We want to make sure we catch something like this long before it makes it out into the hands of Residents.

We’d like to extend a really big thank you to everyone who reported the issue to us the moment they saw it! Because of your vigilance we were able to react really quickly and limit the time that this misconfiguration was live. (Seriously, y’all rock! 💜)

We’re sorry this issue happened this morning. We’re working to make sure it never happens again, and developing better test procedures for use in the future.

While the error was unfortunate, and might have been a little discomforting for some who encountered it, the Lab estimates that no more than 500 users visited the account page during the time the issue could occur, and not all of them were given the wrong page to view.

Where the issue did occur, April notes that it did so at random, and randomly selected the incorrect page to be displayed, so it was impossible for a user to “pick” another user’s information and intentionally view it. She also notes that it was not possible to either make purchases via an incorrect account page, or to make any changes to the page.

As always, details in full in April’s blog post – and many thanks to her again for providing an explanation of the issue and what is being done to hopefully avoid future repetitions.

The period of Thursday, October 24th through Sunday 27th October, 2019 saw Second Life encounter a rolling set of issues which finally came to a head on Sunday, October 27th. The issues affected many Second Life users and services from logging-in through to inventory / asset handling.

As has become the case with these matters, April Linden, the Second Life Operations Manager, has provided a post-mortem blog post on the issue and her team’s work in addressing the problems. And as always, her post provides insight into the complexities in keeping a platform such as Second Life running.

In short, the root cause of the weekend’s upsets lay not with and of the Second Life services but with one of the Lab’s network providers – and was exacerbated by the fact the first couple of times it happened – Thursday and Friday – it appeared to correct itself on both occasions before the Lab could fully identify the root cause.

On Sunday, the problems started up again, but fortunately April’s team were able to pin down the issue and commence work with their provider – which obviously meant getting Second Life back on an even keel was pretty much in the hands of a third-party rather than being fully under the Lab’s control.

Our stuff was (and still is) working just fine, but we were getting intermittent errors and delays on traffic that was routed through one of our providers. We quickly opened a ticket with the network provider and started engaging with them. That’s never a fun thing to do because these are times when we’re waiting on hold on the phone with a vendor while Second Life isn’t running as well as it usually does.

After several hours trying to troubleshoot with the vendor, we decided to swing a bigger hammer and adjust our Internet routing. It took a few attempts, but we finally got it, and we were able to route around the problematic network. We’re still trying to troubleshoot with the vendor, but Second Life is back to normal again.

– Extract from April Linden’s blog post

As a result of the problems April’s team is working on moving some of the Lab’s services to make Second Life more resilient to similar incidents.

During the issues, some speculated if the problems were a result of the power outages being experienced in California at the time. As April notes, this was not the case – while Linden Lab’s head office is in San Francisco, the core servers and services are located in Arizona. However, resolving the issues from California were affected by the outages, again as April notes in her post.

It’s something I’ve noted before, and will likely state again: feedback like this from April, laying out what happened when SL encounters problems are always an educational  / invaluable read, not only explaining the issue itself, but in also providing worthwhile insight into the complexities of Second Life.