Tag: SL Issues

April Linden blogs the weekend’s DDOS

Posted on by Inara Pey
Graph showing “normal” log-ins over the course of a day compared with Sunday, October 28th. Credit: April Linden

In my week #44/1 User Group update, I noted that April Linden had indicated the issues Second Life users experienced with the platform on Sunday, October 28th through Monday October 29th, 2018 were the result of a Distributed Denial of Service (DDOS) attack.

April has now issued a blog post expanding on her original forum comments, with the full text of her post reading:

Hello amazing Residents of Second Life!

A few days ago (on Sunday, October 28th, 2018) we had a really rough day on the grid. For a few hours it was nearly impossible to be connected to Second Life at all, and this repeated several times during the day.

The reason this happened is that Second Life was being DDoSed.

Attacks of this type are pretty common. We’re able to handle nearly all of them without any Resident-visible impact to the grid, but the attacks on Sunday were particularly severe. The folks who were on call this weekend did their best to keep the grid stable during this time, and I’m grateful they did.

Sunday is our busiest day in Second Life each week, and we know there’s lot of events folks plan during it. We’re sorry those plans got interrupted. Like most of y’all, I too have an active life as a Resident, and my group had to work around the downtime as well. It was super frustrating.

As always, the place to stay informed of what’s going on is the Second Life Grid Status Blog. We do our best to keep it updated during periods of trouble on the grid.

Thanks for listening. I’ll see you in-world!

April Linden
Second Life Operations Team Lead

Shug Maitland kept an eye on the ups and downs of log-ins during the DDOS attack via https://etitsup.com/slstats/ through Sunday, October 28th, 2018 and into the early hours of Monday, October 29th, sending me the above screen capture

There not a lot more that can be added – DDOS attacks are an unfortunate fact of life, and while the Lab has learned to try to deal with them without impacting the normal flow of activities for Second Life users, it’s also unfortunate that at time this cannot always be the case.

Thanks once again to April for the update on the situation.

April Linden explains August 22nd’s Second Life woes

Posted on by Inara Pey

Tuesday, August 22nd was not a particularly good day for Second Life, with an extended period of unscheduled maintenance with log-ins suspended and those in-world advised to refraining from rezzing No Copy objects, or making any LindeX related transactions, etc.

If these words sound familiar (except the date), it’s because I wrote them a year ago to the day, on August 23rd, 2016, when Second Life experienced some significant issues.

Back then, the problem was the core database. The initial problems on August 22nd, 2017 weren’t software related, nor were they related to the Main (SLS) channel deployment taking place at the time. Instead, they lay with a piece of hardware, as April Linden, writing in the Tools and Technology blog, explained in another concise explanation of the problem, which started:

Early this morning (during the grid roll, but it was just a coincidence) we had a piece of hardware die on our internal network. When this piece of hardware died, it made it very difficult for the servers on the grid to figure out how to convert a human-readable domain name, like www.secondlife.com, into IP addresses, like 216.82.8.56.

Everything was still up and running, but none of the computers could actually find each other on our network, so activity on the grid ground to a halt. The Second Life grid is a huge collection of computers, and if they can’t find other, things like switching regions, teleports, accessing your inventory, changing outfits, and even chatting fail. This caused a lot of Residents to try to relog.

We quickly rushed to get the hardware that died replaced, but hardware takes time – and in this case, it was a couple of hours. It was very eerie watching our grid monitors. At one point the “Logins Per Minute” metric was reading “1,” and the “Percentage of Successful Teleports” was reading “2%.” I hope to never see numbers like this again.

Unfortunately, as April went on to explain, the problems didn’t end there, as the log-in service got into something of a mismatch once the hardware issue had been resolved. Whilst telling viewers attempting to log-in to the grid their attempts were unsuccessful, the service was telling the simulators the log-ins had been successful. Things didn’t start returning to normal once this issue had been corrected.

There is some good news coming out of this latter situation however, as April goes on to note in the blog post:

We are currently in the middle of testing our next generation login servers, which have been specifically designed to better withstand this type of failure. We’ve had a few of the next generation login servers in the pool for the last few days just to see how they handle actual Resident traffic, and they held up really well! In fact, we think the only reason Residents were able to log in at all during this outage was because they happened to get really lucky and got randomly assigned to one of the next generation login servers that we’re testing.

Testing of the new log-in servers has yet to be completed, but April notes that the hope is they be ready for deployment soon.

Thanks once again to April for the update on the situation.

SL project updates 2017-7/1: server, viewer, “blue world” bug fix

Posted on by Inara Pey

Anduril, Anduril; Inara Pey, February 2017, on FlickrAnduril – blog post

Server Deployments

In short, there are no deployments scheduled for this week. The Main (SLS) channel will remain on release 17#17.01.27.323172.

While there had been an RC release planned, it apparently didn’t clear QA in time, so all three RC channels will remain on 17#17.01.27.323172 as  well.  However, all three channels will be restarted on Wednesday, February 15th, in keeping with the Lab’s policy or restarting channels every two weeks, whether or not there is an associated deployment.

SL Viewer

The Maintenance RC viewer updated to version 5.0.2.323567 on Tuesday, February 14th.  As reviewed in this blog, this viewer includes a number of updates and new features, including the ability to select your own preferred folders for uploading image, animations, sounds and mesh models.

Outside of this update, the viewer pipelines remain as per the end of week #6:

  • Current Release version: 5.0.1.323027, dated January 25, promoted February 3 – formerly the Maintenance RC viewer.
  • RC viewers:
    • Love Me Render RC viewer version Version 5.0.2.323361, dated February 9th – rendering pipeline fixes and improvements
  • Project viewers:
    • Project Alex Ivy (LXIV), 64-bit project viewer, version 5.1.0.501863 for Windows and Mac, released on January 10
    • 360-degree snapshot viewer updated to version 4.1.3.321712 on November 23, 2016 – ability to take 360-degree panoramic images – hands-on review.
  • Obsolete platform viewer version 3.7.28.300847 dated May 8, 2015 – provided for users on Windows XP and OS X versions below 10.7.

Nvidia Driver 64-bit Viewer “Blue World” Bug

As I reported in week #4, Nvidia’s release of their 378.49 driver on January 24th resulted in many 64-bit viewer users (TPVs and the Lab’s own Alex Ivy 64-bit project viewer) seeing their Second Life world view turn decidedly blue when running with Advanced Lighting Model (ALM) disabled.

The Nvidia 378.66 driver should fix the
The Nvidia 378.66 driver should fix the “blue world” issue for those using 64-bit viewers with ALM disabled

On February 14th, Nvidia release the 378.66 driver package, and this reportedly fixes the SL issues.

SL project updates 2017-4/1: Server, camera pre-sets, Nvidia issue

Posted on by Inara Pey

Devin, Devin; Inara Pey, January 2017, on FlickrDevin – blog post

Server Deployments

As always, please refer to the server deployment thread for the latest updates.

On Tuesday, January 24th, the Main (SLS) channel was updates with the same server maintenance package deployed to the RC channels during week #3. This includes a partial fix for (non-public) BUG-3286, “Can’t move object” fail notifications (fixes for regions/objects with longer names are pending), together with enhanced server logging and minor internal server enhancements.

There will be no RC deployment on Wednesday, January 25th – but the RC region will be restarted in keeping with the Lab’s new policy of restarting the channels every 2 weeks, regardless of whether or not there is an associated deployment.

The next RC deployment is expected to be week #5 (commencing Monday, 30th January, 2017).

SL Viewer

No changes since my last update. The status of viewers in the pipeline remains thus:

  • Current Release version: 5.0.0.321958, dated December 1st, promoted December 5th, 2016 – formerly the Project Bento RC viewer
  • Release channel cohorts:
    • Maintenance RC viewer, version 5.0.1.322791, dated January 12th
  • Project viewers:
    • Project Alex Ivy (LXIV), 64-bit project viewer, version 5.1.0.501863 for Windows and Mac, dated January 10th
    • 360-degree snapshot viewer, version 4.1.3.321712, dated November 23, 2016 – ability to take 360-degree panoramic images – hands-on review
  • Obsolete platform viewer, version 3.7.28.300847, dated May 8, 2015 – provided for users on Windows XP and OS X versions below 10.7.

Camera Presets

As I noted in a recent TPVD meeting update, Jonathan Yap is working on a code contribution for the official viewer which will allow users to set and save their own preferred camera presets in the viewer.

The idea is that, like the graphics presets functionality Jonathan contributed to the viewer in 2016, users will be able to define their own placements for the SL camera around their avatar (e.g. an over-the-should view, a view from overhead, etc.), which can then be saved and selected / used as required. Jonathan has only recently started on the work – which has an associated feature JIRA, STORM-2145 – but that should hopefully change once various decisions have been made by the Lab.

Nvidia Driver 378.49 + 64-bit Viewer Bug

Nvidia release their 378.49 driver on Tuesday, January 24th, and it can cause an unusual bug / issue with 64-bit viewers. The problem was first noted on Firestorm 5.0.1 (see: FIRE-20774), but I have repro’d it on the Lab’s own 64-bit project viewer (version 5.1.0.501863 at the time of writing) and  on Alchemy 4.0.0 (a crash issue had prevented comprehensive testing on Alchemy 5.0.0 at the time of writing).

The Nvidia 378.49 driver bug which can occur with 64-bit viewers when ALM is disabled, as seen on a 64-bit version of Windows)
The Nvidia 378.49 driver bug which can occur with 64-bit viewers when ALM is disabled

The issue only manifests when Advanced Lighting Model (ALM) is disabled in a 64-bit viewer, and renders the in-world view with an odd blue tinge which almost looks like the blue colour channel is impinging on the red channel. As noted in the Firestorm JIRA, enabling ALM can prevent the issues, as can toggling Glow off when ALM is disabled. See the Firestorm JIRA for workarounds, should you encounter the problem.

How the same scene looks in the same viewer (SL Alex Ivy 64-bit project viewer for Windows, version 5.1.0.501863 at the time of writing)
How the same scene looks in the same viewer (SL Alex Ivy 64-bit project viewer for Windows, version 5.1.0.501863 at the time of writing)

The issue was raised at the Simulator User Group meeting on Tuesday, January 24th, a JIRA for the issue on the Lab’s 64-bit project viewer is available on BUG-41294.

 

SL project updates 2017-2/1: 64-bit viewer and Monday Blues

Posted on by Inara Pey

Nagare no Shimajima, Restless Times; Inara Pey, January 2017, on FlickrNagare no Shimajima, Restless Times – blog post

Server Deployments

There are no planned deployments for the week. However, all servers on the three RC regions will be subject to a rolling restart. This is in accordance with the Lab’s new policy of restarting channels every fortnight, whether or not there is an associated deployment. As the Main (SLS) channel underwent a restart on Tuesday, January third, server on this channel were not restarted this week.

SL Viewer

Project Alex Ivy

The 64-bit versions of the official viewer arrived in project viewer form on Tuesday, January 10th, under the code name Project Alex Ivy – which I take to be a reference to 64 (LXIV being 64 in Roman numerals, hence aLeX IVy).

The viewer, version 5.1.0.501863, has been built using the newly updated and upgraded libraries and build process the Lab has been putting together, which will also be used for 32-bit Windows builds. Thus, the project viewer is available in three flavours:

  • 64-bit Mac
  • 64-bit Windows
  • 32-bit Windows.

There is no Linux viewer as yet, but the Lab has indicated it is their intention to provide one, although TPVs and open-source contributors are likely to still be asked to help with its ongoing support.

Additionally, the following points, as specified in the release notes, should be underlined (although please ensure you read the release notes in full if you intend to try this viewer:

  • The Mac build has several known limitations:
    • There is currently no Mac Havok build,so pathfinding paths cannot be visualised, and it may not be possible to upload mesh assets.
    • Video media using QuickTime does not play.
  • The 64-bit version will not run on Windows 10 systems with Intel HD 2000/3000 GPUs and may not run on other systems that do not have GPUs explicitly supporting Windows 10.

These shortfalls will be addressed as the viewer progresses through the project and release candidate phases to release status in the next weeks / months. Once released, it will signal the end of the 32-bit MAC version of the viewer (and possibly the 32-bit Linux version). The Windows version will continue to be available as a 32-bit build as well as having the new 64-build available.

Also, note that this viewer doesn’t include any functional updates / changes to the existing viewer.

Remaining Viewers Pipelines

Outside of the 64-bit project viewer, the various viewer pipelines remain as my last SL project update:

  • Current Release version: 5.0.0.321958, dated December 1st, promoted December 5th – formerly the Project Bento RC viewer
  • Maintenance RC viewer, version 5.0.1.322513, dated December 21st – some 42 fixes and improvements + Bento support
  • 360-degree snapshot project viewer, version 4.1.3.321712, dated November 23rd – ability to take 360-degree panoramic images – hands-on review
  • Obsolete platform viewer version 3.7.28.300847, dated May 8, 2015 – provided for users on Windows XP and OS X versions below 10.7.

Monday Outage

On Monday, January 9th, many users were hit with significant issues, with many finding themselves unable to log-in, or being disconnected from the simulators and unable to log back in. On Tuesday, January 10th, April Linden from the Ops team posted another of her excellent post-mortem blog posts on what happened, and I recommend it as a worthwhile and informative read.

In essence a failure within a third-party provider used by the Lab failed to trigger the expected automatic switch-over of connections for all users accessing Second Life through that provider. As a result, those users were disconnected from the service, and due to the volume of people trying to re-connect, couple (I assume with those simply trying to log in, unaware of problems) generated a backlog, forcing the Lab to bring additional log-in servers on-line.

Once again, April does an excellent job in explaining things – revealing more of the complexities of SL in the process (which, as I’ve oft said in the past, goes well beyond just the simulator servers), and also offers apologies for the Monday problems.

SL project updates 2016 50/1: Server, viewer, no change window

Posted on by Inara Pey

North Pole at Alki, Alki; Inara Pey, December 2016, on Flickr “Hand over your coal and carrots, and don’t try anything funny. This hair-dryer is plugged in, and I know how to use it!” North Pole at Alki, Alki – blog post

Server Deployments

As always, please refer to the server deployment thread for the latest updates and information.

  • On Tuesday, December 13th, the Main (SLS) channel received the same server maintenance package, as deployed to the RC channels in week #49. This includes the following feature requests:
    • BUG-6377 – llGetObjectDetails(id,[OBJECT_ATTACHED_SLOTS_AVAILABLE]) – Returns a value that is number of attachment slots allowed by the server minus the number of attachments worn by avatar. Returns 0 if avatar is not in the same region or if UUID is not an agent.
    • BUG-40871 – llGetEnv() constant “region_object_bonus” – returns the object bonus set for a region.
  • On Wednesday, December 14th, all three RC channels should receive the same new server maintenance package, comprising improved internal server logging.

SL Viewer

The Maintenance RC viewer updated to version 5.0.1.322219, bringing it to parity with the current release viewer, incorporating the Bento updates. This leaves the remaining viewers in the pipelines unchanged from week #48:

  • 360-degree snapshot viewer, version 4.1.3.321712, dated November 23rd.
  • Obsolete platform viewer, version 3.7.28.300847, dated May 8th, 2015 – provided for users on Windows XP and OS X versions below 10.7.

No Change Window

The Christmas and New Year 2016/17 No Change window comes into effect from Friday, December 16th. This means there will be no further server deployments and no further official viewer promotions after that date, through until Monday, January 2nd, 2017.

Issues

Mesh and Texture Rezzing

As noted in my week #49 update, people have been noticing increased delays in object mesh and texture rezzing, with fingers being pointed at the Lab’s CDN supplier(s).  The issues are continuing for some, while for others they appear to have cleared up. It’s still not obvious if it is a potential LL / CDN issue or a network problem in general.

As Simon Linden said in the meeting, the problems have been seen by the Lab, but are proving to be intermittent and hard to pin down. The problem also seems to manifest differently for people: some report very slow texture rendering, other report textures load and render fine, but mesh items are prone to failing to fully render, others report a mix of the two.

Region Crossings

Region crossings have been widely reported as increasing again – including avatars being dumped at 0,0,0 (again). Some are reporting the issue as cumulative: the more regions they cross, the greater the likelihood they’ll encounter a serious issue (becoming unseated, return of vehicle, forced log-out). The usual advice is being circulated – reduce script load, handle crossings with caution, etc. However, the Lab are again aware of the uptick, but have not come to any specific conclusion on the cause.

The avatar appearing at a region’s 0,0,0 co-ordinates appears to be linked to connection problems and / or UDP packet loss (usually the first packet) resulting in messages arriving in the wrong order and the viewer and simulator falling out of synch with one another.

With both the rendering issues and the region crossing issues, fingers have been directed at the increased land impact allowances. This may be a case of post hoc, ergo propter hoc. In particular, it has been claimed that region crossings “became” bad after the mainland LI allowance increase, although people were reporting issues before the LI increase took place.

Appearance Issues / Bake Fails

Some are finding their avatar is failing to render in their view (bake fail) when logging-in. Changing outfits, camming away / back to your avatar may fix this, or a re-log. In extreme cases reverting to the default Character Test avatars may be required, or deleting and recreating the affected avatar appearance folder on your hard drive.

Rider Linden is looking into the problem, and believes it may be due to a missed inventory fetch at log-in, leaving the viewer thinking you’re missing a critical part of your appearance (e.g. your shape), but he is not 100% certain at this point in time.