SL project updates week 10/1: server, general news

Leka, Nordan om Jorden; Inara Pey, March 2015, on Flickr Leka, Nordan om Jorden (Flickr) – blog post

Server Deployments

Tuesday, March 3rd, saw the Main (SLS) channel receive the server maintenance package deployed to the RC channels in week #9. This includes:

  • A server-side fix for BUG-8297, “Unable to teleport anywhere using SLGO”
  • Improvements to server logging.

There were no scheduled deployments to the RC channels on Wednesday, March 4th.

Group Chat

Following the last deployment of back-end group chat changed during week #9, some large groups with active group chat have reported an increase in issues of message failures, although they appear to do so somewhat randomly, with some people seeing them and others simply not receiving them at all.

Commenting on the problem at the Simulator User Group meeting on Tuesday, March 3rd, Simon Linden summarised the situation thus:

In short, yes, it’s cranky, and yes, we’re (as in I am) looking at it … the chat server itself is actually running better than before, believe it or not. A back-end service it relies on, what we call “agent presence” [used to help locate someone on the grid], seems to be having new problems, so the changes may have added load to those servers and is causing problems, or something else unexpectedly changed … [So] some people don’t get the messages when chat is failing … it’s dropping sending some updates and messages when it times out with some other internal requests.

Further updates will be provided as the Lab / Simon continues to look at the problems.

CDN Notes

There have been recent reports of people experiencing slow texture and mesh load issues, leading to questions concerning the CDN service (although some of the issues that have been mentioned might be related to local caching more than the CDN). In particular questions have been asked as to how long a CDN server retained its cache of data relating to regions prior to going “cold” and requiring a “reload” from the SL services. Commenting on this at the Open-source Developer meeting on Monday, March 2nd, Oz Linden said that some CDN caches do age out more quickly than others.

The Lab has also been experimenting with more than one CDN provider, and are continuing with different CDN configurations as well to further tune things, as well as continuing to measure results; so we may yet see further changes  / improvements, and a possible decrease in instances that may be related to “cold” CDN loads.

Other Items

Rigged Mesh Crashers

The Server Beta Meeting on Thursday, February 26th saw the issue of a “new” mesh crasher being used on the grid. This is essentially a deliberately corrupted rigged mesh attachment which, when worn will cause viewers around it to immediately crash, with no warning or ability to take preventative action, such as muting the offending avatar.

Just over a year ago, some advice was given on how to counter graphics crashers by adjusting the viewer’s debug settings, and some people many be getting pointed towards it again in order to avoid being affected by the “new” crasher.

However, changing the specified debug settings can lead to a failure to render much of what you actually want to see, as noted in  this comment following the article. At the time the advice was given, the Firestorm team tracked many of the problems their users were experiencing directly to the settings having been changed. Ergo, if you are pointed to this particular article as a means of combating graphics crashers, please keep in mind you may gain undesirable results, and keep a note of the original settings so you can switch back to them should this be the case.

During the discussion on this matter at the SBUG meeting, speculation was raised on whether or not the forthcoming new viewer rendering controls (see: STORM-2082). opinion is divided, as the viewer downloads the data which may cause a graphic crash and starts processing some of it in order to determine what to render or not, and even this initial processing could be enough to crash it.

SL Feed Issues

There has been an uptick in the number of snapshot uploads to the SL feeds failing over the course of the last week, with some additionally reporting issues of comments failure to appear / “loves” failing to stick. Some users also reported issues over the weekend with web profiles failing to load, and a JIRA (see BUG-8677) was logged on this issue on March 3rd.

The last several days have seen people again encounter issues with snapshots failing to process / display in their feeds
The last several days have seen people again encounter issues with snapshots failing to process / display in their feeds

Whether the two issues had a common cause isn’t clear, but as the latter has been resolved, and you are one of those continuing to experience snapshot upload failures, please file a JIRA providing as much information as possible (links to any feed post with a missing snapshot, date / time of upload, number of failures, etc.).


6 thoughts on “SL project updates week 10/1: server, general news

  1. I have been noticing the problems with mesh and textures upload, growing on last days.
    Still if some i can assure is that on last week i didn’t had a single crash due to a cross sim, even yesterday i did travel along route 8 and 14 on Satori on a truck for over 40 regions without a single issue but…
    My soulmate did crash 4 times as a passenger and after the 1st crash a old issue long forgotten show to me, she did log in, she joined, we start travelling and after a cross sim her sculpted hair would be showing as being left behind in the air, while she had it still in his skull.
    Strange to do a trip and saw your loved’s one hair being left behind the truck,
    Worse is the crash rate of my soul mate, compared to mine, while i leve in Europe and she is on east coast of Usa and we both use the same hardware, some software and even some viewer.
    But we did not had an encounter with the Mesh crasher, lucky, even if i did think that the box on LL viewer graphics tab would prevent a hardware issue.
    And yesterday all was working pretty flawless on the side of profile feed, at least for me. all snapshots are in fact being processed much faster then they used to, i could see them all uploaded asap and did not had issues while seeing others profiles.
    I still have a question, to what size one should raise the cache?
    I always make it the way up (9.894gb) and i do not clear my local cache for a few months.
    As i do in fact cross a lot of sims, that cache reaches the limit pretty fast as a lot of diff textures are being uploaded.
    Should i lower it or even after reaching the limit the cache will work as expected?


    1. From the article: “This is essentially a deliberately corrupted mesh attachment which, when worn will cause viewers around it to immediately crash, with no warning or ability to take preventative action, such as muting the offending avatar.”

      Shutting off Volume rendering is no more effective than the other methods if one has no time to change the setting before the crash hits.


      1. You are correct, if the word “immediately” means “less than one second,” because that’s all the time it takes to hit Ctrl + Alt + Shift + 9. I have not seen a handheld mesh graphics crasher, but I suspect that “immediately” means a rapid but not instantaneous decrease in speed, which can be countered by using that keyboard shortcut. The result is a rapid but not instantaneous increase in speed, and the avoidance of a crash. I created a video to demonstrate the effect, which is linked on the blog post. The crasher in the video may still be sitting in Sandbox Island. Give that method a try and see if it works for you too.

        I had no idea that people were using Firestorm Support to recover from the settings in that original blog post. I apologize if I caused problems for anyone. The original post has been updated to point to the new article written today.


  2. Ctrl + Alt + Shift + 9 won’t derender worn *rigged* mesh (unrigged yes).
    Some of these mesh crashers are rigged mesh attachments.

    The “new” mesh crashers I’ve seen arn’t anything special really – just meshes with a disgustingly high vertex count rather then a “corrupted” mesh that triggers a “real” viewer crash – by “real” crash I mean an actual crash in the viewer code because of a bug, not just a crash from blowing memory or graphics drivers.

    There have been some of those “corrupted” meshy crashers in the past but those crashes generally get fixed pretty fast and can be fixed fairly easily in the viewer code. There is one *rigged* mesh crash still unfixed on the LL viewer (possibly other viewers too, its fixed on Firestorm) that is instant death as soon as you render one of those meshes – no lag, no FPS drop, no warning, instant crash to desktop. The crash is caused by a viewer bug – meshes that trigger this crash are not laggy to render at all and can easily be accidently created by anyone using any rigged mesh thats mod.
    Hint: disable Advanced Lighting Model and you wont be affected by this one.

    With the standard lagger mesh crashers it’s a toss up how you crash – either blowing memory in the viewer or a graphics driver crash. 32 bit systems especially will likely die from out memory before the graphics driver has a chance to crash and these crashes really can be pretty instant with no time to react if you were getting low on available memory anyway.

    Chalice Yao has a nice feature in her NaCl viewer to prevent high vert *rigged* meshes from crashing the viewer –
    I’m hoping we can get this into Firestorm before the next release (I’m testing a patch of it now).
    Unfortunately this feature would probably need to be disabled by default because it would cause some legit (though I grit my teeth calling very high vert mesh legit…) content to not render in the viewer.

    Hal, your old blog post at has been causing some broken users yes. Firestorm support hates you just a little bit for that 😉
    Unfortunatelt those setting changes will cause a fair bit of tame legit content to never render in the viewer (any viewer).
    Personally I find typing “dd 0” into chat (Firestorm comand line feature to set draw distance to zero) is the fastest and easiest way to save a crash when you feel that lag start from a suspected graphics crasher.

    Liked by 1 person

Have any thoughts?

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s