Category Archives: News

Second Life’s August 23rd woes by April Linden

secondlifeTuesday, August 23rd was not a particularly good day for Second Life, with an extended period of unscheduled maintenance with log-ins suspended and those in-world advised to refraining from rezzing No Copy objects, or making any LindeX related transactions, etc.

At the time of the problem, there was speculation that it might be due to further issues with the central database node (and kudos for Caitlyn for suggesting this🙂 ). Writing in a Tools and Technology blog post on August 24th, Operations Team lead April Linden confirmed this was in fact the case:

Shortly after 10:30am [PDT], the master node of one of the central databases crashed. This is the same type of crash we’ve experienced before, and we handled it in the same way. We shut down a lot of services (including logins) so we could bring services back up in an orderly manner, and then promptly selected a new master and promoted it up the chain. This took roughly an hour, as it usually does.

Given this has happened in the relatively recent past (see here and here), the Ops Team are getting pretty good with handling these situations. Except this time there was a slight wrinkle in the proceedings. The previous failures had occurred when concurrency was relatively low due to the times they occurred. This time, however, the problem hit when rather a lot of people were trying to get into SL, so as April notes:

A few minutes before 11:30am [PDT] we started the process of restoring all services to the Grid. When we enabled logins, we did it in our usual method – turning on about half of the servers at once. Normally this works out as a throttle pretty well, but in this case, we were well into a very busy part of the day. Demand to login was very high, and the number of Residents trying to log in at once was more than the new master database node could handle.

Around noon we made the call to close off logins again and allow the system to cool off. While we were waiting for things to settle down we did some digging to try to figure out what was unique about this failure, and what we’ll need to do to prevent it next time.

It wasn’t actually until a third attempt was made to bring up the login hosts one at time that things ran smoothly, with services being fully restored at around 2:30pm PDT.

Now, as April notes, she and her team have a new challenge to deal with: understanding why they had to turn the login servers back on much more slowly than in the past. There is, however, a bright spot in all this: the work put into making the Grid Status feed more resilient paid off, with the service appearing to cope with the load placed on it by several thousand people trying to discover what was going on.

None of us like it when the go wrong, but it’s impossible for SL to be all plain sailing. What is always useful is not only being kept informed about what is going on when things do get messed up (and don’t forget, if you’re on Twitter you can also get grid status updates there as well), but in also being given the opportunity to understand why things went wrong after the fact.

In this respect, April’s blog posts are always most welcome, and continue to be an informative read, helping anyone who does so just what a complicated beast Second life is, and how hard the Lab actually does work to try to keep it running smoothly for all of us – and to get on top of this as quickly as they can when things do go wrong.

Second Life wiki: editing locked for now

secondlifeCrap Mariner has been working on a new Arts wiki page on the SL wiki – on which I’ll have more in a future article – when he contacted me earlier in August to let me know that SL wiki editing had been locked to a limited number of permitted editors / users.

The locking has since become more widely seen, prompting Soft Linden to issue an e-mail update as a result of enquiries on what is happening, made via the open-source dev e-mail list. In the e-mail, Soft states:

There is a ridiculously persistent party who uses numerous wikis to promote fake Microsoft support phone numbers. Given the fraud or malware potential, we couldn’t let that keep returning.

There’s a less intrusive long-term solution in the queue, currently blocked by other work.

Hopefully, whatever is blocking the long-term solution will be cleared soon, allowing full edit access to the wiki to be restored.

In the meantime, those requiring urgent access to wiki pages to maintain data  / information, are asked to create a support ticket (I’d suggest include which pages need to be accessed and why, rather than just raising a generic request to gain access). Third-party viewer developers who require wiki edit rights can also contact Oz Linden to request access to their wiki pages.  If you only require access to update you profile or add a discussion point to a page, etc., it might be advisable to wait until after the wiki has been re-opened to all, rather than filing a support ticket.

This isn’t the first time the wiki had to be placed in lock-down. In October 2014, a lock-down was initiated while the wiki underwent an extensive security update, resulted in limited edit access for users through until January 22nd, 2015. Hopefully, this current period of lock-down, which started in early August, will not be for as long.

Lab announces new SL Marketplace search now live

secondlifeUpdate: August 16th: Merchants have been reporting a number of reported issues with the new search tool. The Lab has set-up a forum thread for general discussions on issues and problems. If you are experiencing issues with the new MP search, please join the discussion on the forum thread.

If you are noting specific issues which appear to be bugs, the Lab requests you add details to  BUG-37670.When doing so, please provide the following: Affected Field:  (Demo linking, Related Items, Quick Fill, etc.);  Name of Product you are editing:  (What is the name of, or link to, the item listing that you are trying to add a demo or related item); Search Term:  (What is the exact search term you have entered to try to locate the Demo, Related, or Quick Fill Template);  and if possible, provide a screenshot showing the empty search return showing the search text and the lack of results. The exception to this is “Adult rated items that contain profanity”.

On Monday, August 15th, Linden Lab announced the “new” SL Marketplace search is now live.

First issued as a beta test in November 2015, the updated search has been undergoing refinement and improvement (and bug fixing!) since then, hence why it has taken some time to reach a release status.

The new search mechanism is designed to address some of the shortfalls inherent in searching for listings and products on the Marketplace, which can often by a frustrating and time-consuming, as anyone who has used it will be able to attest.

To achieve this, it is said to be a complete step away from the older infrastructure supporting Marketplace search, with the new system hopefully being faster and more robust. In particular, the Lab point to it having a learning algorithm intended to improve search results over time – essentially, the more search is used, the better it performs. The new search also supports the use of boolean operators – AND, OR, NOT – so users can hopefully set-up more focused search criteria.

During testing, the Lab took feedback from merchants and users into account, using it to further refine and improve the new system. This obviously doesn’t mean it is flawless, but the hope is the overall experience will be improved, and will continue to improve over time. Should anyone encounter significant bugs or issues, they are asked to file a bug report. There is also a pinned Commerce Forum thread where general discussions might be continued.

SL project updates 16 32 (2): TPVD meeting

Eclectica – Mysticablog post

The majority of the notes in this update are taken from the TPV Developer meeting held on Friday, August 12th. The video of that meeting is embedded at the end of this update, and references to it are indicated through the use of time stamps in the paragraphs below. My thanks as always to North for recording and providing it.

Server Deployment – Recap

There was only a Main (SLS) channel deployment this week. This saw the roll of the server maintenance package previously deployed to the three RC’s in week #31, on Tuesday, August 9th. This comprised internal fixes and an update to prevent BUG-37573.

SL Viewer

Release Viewer – Texture Handling Fixes

[0:35] The Maintenance RC viewer release on Monday, August 8th gained a rapid promotion to the de facto release viewer on Thursday, August 11th. Version (dated August 8th).

This viewer includes a number of fixes in the image pipeline (e.g. fixes for “bad” textures – those texture files which have invalid data in them). Also included is a fix to prevent the viewer crashing when you system runs out of main (not GPU) memory while  attempting to load a texture file. Instead, the viewer will substitute a plain grey texture. So, when you start to see grey everywhere instead of expected textures, it’s time to restart the viewer. This may be a pain to look at, but it is considered preferable to having the viewer crash at a potentially inconvenient moment.

[1:35] The Lab intends to move along similar lines for other issues within the viewer which can result in a hard crash, and also go through cleaning-up how exceptions are generated and caught by the viewer, and this work should be appearing in the next but one Maintenance RC update. The overall goal is to improve the image pipeline and some other points in the viewer where a relatively low-level thing results in the viewer crashing. Some of this work might also help prevent attempts to deliberately crash other viewers using textures.

In the meantime, issues have emerged affecting attachments and the Current Outfit Folder with this release – see BUG-37646 “Attachments get ghosted at login on”; and BUG-37653 “Every time I delete Cache and Relog, my Saved Appearances do not load and I am left as a White Cloud in Second Life Viewer”, for details.

Remaining Viewers in the Release Channel

[4:20] The VLC Media Plugin RC viewer, version dated July 28th at the time of writing, which contains the LibVLC-based replacement for QuickTime for Windows, is liable to be the next RC that will be promoted to release status.  A new RC version of this viewer, merged-up to the code, should appear in the release pipeline in week #33 (commencing Monday, August 15th).

[4:32] The plan remains to update the Mac version of the viewer to use VLC as a part of the 64-bit viewer development.

[6:10] The Visual Outfits Browser RC viewer, version dated August 1st at the time of writing, which allows users to preview images of outfits in the Appearance floater should be updated in week #33 following a merge with the code. This update will also include a further round of bug fixes for this project.

Project Bento

[7:35] A new project viewer is being readied, which includes bug fixes and which has been merged with the code. This should hopefully appear in week #33. See my Bento update 22 for more on the project.

Upcoming Viewers

[6:43] A new Maintenance viewer should appear in week #33. This will contain further fixes and improvements, although not the exception handling improvements referred to above.

[7:00] Work is expected to resume on the 64-bit versions of the official viewer in week #33.

SL Voice

[9:45] Work is progressing on Voice, with a further SL Voice plugin update expected from Vivox soon. Oz has been debugging an upcoming project / RC viewer with more Voice fixes – although this isn’t yet ready to be issued.

Avatar Complexity

It has been noted that Avatar Complexity values can fluctuate when seen from different systems, on average by around 5%. This is because it is next to impossible to come up with a single figure that s accurate across all systems, as the calculations have a degree of hardware dependency (GPU, rendering capabilities, etc),

However, the Lab will continue to tweak the calculations to try to make them as consistent as possible, but this will be a gradual process for reasons Oz discussed in the meeting, and which I’ve extracted in the audio file below

A couple of particular issues which have been reported for avatar complexity calculations are BUG-37631 “Rigged mesh with partially transparent texture on it have 4 times higher complexity”, and BUG-37642 “ACI randomly changes (often at login or following a TP)”.

Other Items

Memory Bloat Crashers

[8:50] With the arrival of Avatar Complexity, which provides protection against worn graphics crashers (just don’t set your Maximum Complexity slider to No Limit), it appears that inconsiderates in the virtual world are swapping to use attachments which cause memory bloat in order to crash viewers. There are, for example, attachments which can raise viewer memory to 4 Gb which immediately crasher 32-bit viewers, even if the offending avataris “Jelly Dolled”.

Oz has requested the Lab be supplied with examples, so they can start looking into the matter and hopefully come up with a fix.

Abuse Report Categories

[10:31] One of the possible issues for some Abuse Reports (ARs) appearing to go unanswered is that there are still viewers using the “old” AR categories, rather than the newer categories (as found in the official viewer). This is particularly true where users are still on versions of the viewer which do not have the revised list of AR categories.

To prevent this is the future, the Lab plan to make Abuse Report categories a capability handled by the simulator and downloaded to the viewer. This removes storing the categories in the viewer & having older viewer fail to reflect more recent category updates. It will also make it easier for the Lab to update AR categories to better meet users’ needs. A project viewer will be appearing at some point in the future supporting this new capability.

Also, within the official viewer, appending a snapshot to an AR is to become mandatory, rather than optional, to further help support identify issues and deal with them. Having a picture may not be relevant for all ARs, but for those where it could help in identifying issues, it ensures the picture is provided, rather than ignored.