2018 SL UG updates #9/1: SUG, and DDOS

Les Reves Perdus; Inara Pey, January 2018, on Flickr Les Reves Perdusblog post

Server Deployments

As usual, please refer to the server deployment thread for the latest updates.

On Tuesday, February 27th, the Main (SLS) channel was updated with server maintenance package 18#, previously deployed to the three RC channels. This update was to directly address an odd viewer crash situation some users have experienced. Speaking at the week #8 Simulator User Group meeting, and reproduced here for completeness, Simon Linden said of the issue:

The server is doing some better checking on update data it sends to the viewer. We saw a very odd situation a week or two ago where the region was sending odd data and viewers would crash immediately. It went away after we restarted the region, and we think it was some memory corruption … FWIW, the server was sending a value of zero for a prim-code … which is totally invalid … There were also some other invalid data (like a zero’ed UUID) so my theory was memory corruption.

We didn’t have any other smoking guns. That region was fine after restarting, or when we tried our own copy. It was one of those mystery bugs, which we sometimes get since SL is so big and complex. We don’t know why it got that way, or how to make it happen again. we ended up making both the region and the viewer more robust. The underlying problem is still there and, assuming it happens again, will still cause problems.

(See also: BUG-214564.)

On Wednesday, February 28th, 2018 the three main RC channels should be updated as follows:

  • LeTigre and Magnum; no deployment, no restart, leaving them on server maintenance package 18#
  • BlueSteel should received a new server maintenance package, 18#, containing further simulator logging improvements and internal fixes. This is dependent upon what happens with the ongoing global DDoS attack over the course of the next 12-18 hours.

SL  Viewer

There have been no official viewer updates at the start of the week, leaving the various pipelines as per the end of week #8:

  • Current Release version, dated January 26, promoted February 7 – formerly the Voice Maintenance RC.
  • Release channel cohorts:
    • Nalewka Maintenance viewer, version, February 23.
    • Love Me Render RC viewer, version, February 21.
    • Media Update RC viewer, version, February 15.
  • Project viewers:
  • Linux Spur viewer, version, dated November 17, 2017 and promoted to release status 29 November – offered pending a Linux version of the Alex Ivy viewer code.
  • Obsolete platform viewer, version, May 8, 2015 – provided for users on Windows XP and OS X versions below 10.7.

Log-in Issues

On Monday, February 26th and continuing through Tuesday, February 27th, large numbers of SL users experienced significant issues in trying to log-in as a result of a widespread UDP-based Distributed Denial of Service (DDoS) attack on the Internet as a whole (rather than specific to SL).

On Tuesday, February 27th, whilst still dealing with the situation, April Linden at the SL Ops team took time out to post on the situation in the official blog:

Hi everyone.

As I’m sure most of y’all have noticed, Second Life has had a rough 24 hours. We’re experiencing outages unlike any in recent history, and I wanted to take a moment and explain what’s going on.

The grid is currently undergoing a large DDoS (Distributed Denial of Service) attack. Second Life being hit with a DDoS attack is pretty routine. It happens quite a bit, and we’re good at handling it without a large number of Residents noticing. However, the current DDoS attacks are at a level that we rarely see, and are impacting the entire grid at once.

My team (the Second Life Operations Team) is working as hard as we can to mitigate these attacks. We’ve had people working round-the-clock since they started, and will continue to do so until they settle down. (I had a very late night, myself!)

Second Life is not the only Internet service that’s been targeted today. My sister and brother opsen at other companies across the country are fighting the same battle we are. It’s been a rough few days on much of the Internet.

We’re really sorry that access to Second Life has been so sporadic over the last day. Trying to combat these attacks has the full attention of my team, and we’re working as hard as we can on it. We’ll keep posting on the Second Life Status Blog as we have new updates.

See you inworld!
April Linden
Second Life Operations Team Lead

The attack led to a wide number of issues for SL users, from an inability to log-in to SL, through to disconnects or other problems. As Simon Linden explained in the Simulator User Group meeting on Tuesday, February 27th:

It’s not just logins … and often there is cascading problems. If one system gets attacked, it might actually be functioning fine but if the rack or network switch it’s on is overloaded, anything else connected there also has problems.

As it stands, the worst of the attacks appear to be over, at least where SL is concerned, but users should keep an eye on the official blogs for further updates, as well as grid status updates.