April Linden blogs on the May 13th/14th downtime

The week of May 13th-17th saw a planned period of Second Life network maintenance work, as announced in the Grid Status updates.

The first tranche of this work – Monday, May 13th through Tuesday May 14th – appeared to go well, until there was a completely unexpected 4(ish) hours of downtime, which at the time caused significant upset.

On May 17th, April Linden, the Second Life Operations Manager, has provided an insightful blog post on both the work being carried out and the cause of the downtime.

This week we were doing much needed maintenance on the network that powers Second Life. The core routers that connect our data centre to the Internet were nearing their end-of-life, and needed to be upgraded to make our cloud migration more robust.

Replacing the core routers on a production system that’s in very active use is really tricky to get right. We were determined to do it correctly, so we spent over a month planning all of the things we were going to do, and in what order, including full roll-back plans at each step. We even hired a very experienced network consultant to work with us to make sure we had a really good plan in place, all with the goal of interrupting Second Life as little as we could while improving it …

Everything started out great. We got the first new core router in place and taking traffic without any impact at all to the grid. When we started working on the second core router, however, it all went wrong.

– Extract from April Linden’s blog post

In essence, a cable had to be relocated, which was expected to cause a very brief period of impact. However, things didn’t recover as anticipated, and April resumes her explanation:

After the shock had worn off we quickly decided to roll back the step that failed, but it was too late. Everyone that was logged into Second Life at the time had been logged out all at once. Concurrency across the grid fell almost instantly to zero. We decided to disable logins grid-wide and restore network connectivity to Second Life as quickly as we could.
At this point we had a quick meeting with the various stakeholders, and agreed that since we were down already, the right thing to do was to press on and figure out what happened so that we could avoid it happening again…
This is why logins were disabled for several hours. We were determined to figure out what had happened and fix the issue, because we very much did not want it to happen again. We’ve engineered our network in a way that any piece can fail without any loss of connectivity, so we needed to dig into this failure to understand exactly what happened.

– Extract from April Linden’s blog post

April Linden

In other words, while it may have been painful for those who were unceremoniously dumped from Second Life and found they could not get back in, the Lab were working with the best of intentions: trying to find out exactly why connectivity was lost within a network where such an event should not cause such a drastic breakage – and its worth noting that as per April’s blog post, even the engineers from the manufacturer of the Lab’s network equipment were perplexed by what happened.

As always, April’s blog post makes for an invaluable read in understanding some of the complexities of Second Life, and goes so far as to answer a question raised on the forums in the wake of the week’s problems: Why didn’t LL tell us exactly when this maintenance was going to happen? – in short there are bad actors in the world who could make use of publicly available announcements that give them precise information on when a network might be exposed.

If you’ve not read April’s blog posts on operational issues like this, I really cannot recommend them enough – and thanks are again offered April for providing this post. And while things might have hurt at the time, there is a silver lining to things, as she notes:

Second Life is now up and running with new core routers that are much more powerful than anything we’ve had before, and we’ve had a chance to do a lot of failure testing. It’s been a rough week, but the grid is in better shape as a result.

Caravanserai: a Silk Road celebration in Second Life

Saturday, May 18th, 2019 will see the latest iteration of Caravanserai, a celebration of music, poetry and the spoken word from around the world, which this year is again aimed at raising money for Feed A Smile.

From Singapore to Second Life: come to the Caravanserai for a one-night stop over to witness a grand live concert, streamed on-line as Singapore wordsmiths and improv musicians team up with accomplished artists, writers, musicians, comics, film-makers and poets from Germany, UK, Holland, Portugal, Japan, Mexico, Costa Rica, Hawaii, Canada and the USA.

– From the Caravanserai Press Release

I first wrote about Caravanserai, which has been developed and coordinated by Dr Chris Moonie-Singh (Singh Albatros in SL) of The Writers Centre, Singapore and Scott Grant (Kaylee West/ Xilin Yifu in SL) of Monash University in Melbourne Australia ahead of the first event in 2017 (see here for more).

With that event, Caravanserai sought to celebrate the sharing of different cultures by transporting guests to an earlier time when the Silk Road contributed hugely to artistic and cultural understanding and world culture. Travellers going both directions would seek shelter in oasis’ or a caravanserai: an inn with a central courtyard for wayfarers in the desert regions of Asia or North Africa.

And so it is with 2019 as well, with events taking place within the walls of a desert caravanserai, featuring the main stage and various attractions to entertain, with more attractions beyond the caravanserai’s immediate walls – including the opportunity to race on a magic flying carpet.

Commencing at 05:00  through until 08:00SLT on Saturday, May 18th, the event features live streaming from the Writers Centre, Singapore, as well as in-world performances. The latter include:

  • Some Teardrop (singer-songwriter); USA.
  • Yadleen (Clara Mayef) Germany – ambient synth composer/performer.
  • Karima Hoisan and Natascha Randt, poet and film-makers (Costa Rica/Germany).
  • Dansy Insight (Traditional Chinese songs).
  • Rosedrop Rust (Oregon)
  • Poets: Persephone (NYC), Shyla (MiD-West), Klannex Northmead (London), Adele Ward (London), Key Key Underwood (Cleveland), Frederique McMIllan (Quebec, Canada).
  • Storyteller: Caledonia Skytower: Seattle area, USA
  • Esteban Voljeti – comedy.
  • Singh Albatros – original spoken word and music.

For those unable to make it in-world to the event, but who can watch it on-line, it will be streamed via Twitch.

As the Singaporeans share South-East Asian culture, world artists likewise share their poems, songs and stories this unique programme will be streamed live from Second Life from their on-line locations in the spirit of bygone days along the digital Silk Road.

So much of world history is built on the dialogue and cultural exchange represented by the land and sea Silk Roads.

We are very happy to start the Silk Road Series and share the heritage of East-West arts.

– Kaylee West and Singh Albatros

Caravanserai the crossroads of the arts

About Feed A Smile

Feed A Smile is a programme run by Live and Learn Kenya (LLK) to provide nutritious warm lunches for over 400 children every day, paid for entirely from donations to the project. It is part of a broader programme managed by LLK, which includes finding sponsors to finance the education of children in Kenya and helping to provide them with everything they need: uniforms, shoes, text books, school supplies, etc., and the building of the Nakuru school, Kenya.

In addition, the organisation also provides medical and dental care for children, including check-ups and vaccinations. 100% of the donations received by LLK are transferred directly to Kenya to care for children, provide education, medicine, food, shelter and foster care. Nothing is lost to salaries, fees or administrative costs at LLK.

SLurl Details

New SL viewer release notes pages: an overview

Update, May 21st: The Alternate Viewers wiki page has been retired and replaced by a new Alternate Viewer page, which follows the same broad format as the Release Notes page (making the two slightly confusing, as they both reference recent RC viewers. However, this new page also draws a distinction between RC and project viewers, thus overcoming some of the concerns voiced in the second half of this article.

As noted in my week #20 SL Content Creation summary, Linden Lab have unveiled their new viewer release notes pages.

At the time of writing, the new pages are focused on the release candidate (RC) viewers that are in development and currently available as download cohorts in place of the de facto release viewer. It is not currently clear if project viewers will be included in the new format or not.

As Steven Linden from the viewer team notes in a Tools and Technology blog post on the subject, these new pages are part of a new website for viewer release information. This website comprises a dedicated home page with an introduction to viewer release notes. together with links on the left side to:

  • “Recent viewer releases”: a clickable list of the most recent RC viewer updates, provided as viewer version numbers. These are provided in release date order, with the most recent updates at the top.
  • Additional links to viewer-related support information:
The new Release Notes home page – access specific viewer release notes via the version numbers outlined in red on the left

Individual viewer release notes can be accessed by clicking one of the the listed version numbers, which will open a page specific to that viewer. These pages comprise:

  • Icon links to the available OS versions (Windows 32/64-bit, Mac OS).
  • The general release notes (description, etc.).
  • A list of resolved issues.

A significant change in these pages is that, where relevant, Jira links in the Resolved Issues section now, wherever possible, reference “public” bug reports (e.g. BUG-XXXXXX), rather than the Lab’s internally cloned versions of such bugs (e.g. MAIT-XXXXXX).

Currently, the new pages can also be accessed from the existing Alternate Viewers wiki page, (click the Release Notes link for an RC viewer on that page). However, whether this page will remain relevant if the release notes for project viewers are also converted to the new format, remains to be seen.

An example of the revised release notes for a viewer (in this case the Teranino Maintenance RC) – note the use of “public” Jira numbers where relevant, in the known issues list

The new pages are a lot easier on the eye, although I have a number of reservations at this time.

  • While I understand understand why version numbers are used to reference individual viewers (they are URLs and so can be dropped into the pages without necessarily requiring human intervention), they are less user friendly to those wishing to quickly look-up the specifics on a viewer.
  • The “recent Viewer Releases” lists can include links to multiple versions of a given viewer (at the time of writing, two versions of the EEP and Teranino RC viewers are listed, for example). This might cause a degree of confusion for some users, who may mist he “most recent at the top” arrangement of the list.
  • If project viewers are to be added to these pages, I would hope there will be some form of clearer distinction between them and any listed RC viewers, other than just a top-down list of version numbers, again for ease of user reference.

Related Links

2019 SL User Groups 20/2: Content Creation summary

Grauland; Inara Pey, March 2019, on FlickrGrauland – blog post

The following notes are taken from the Content Creation User Group (CCUG) meeting, held on Thursday, May 16th 2019 at 13:00 SLT. These meetings are chaired by Vir Linden, and agenda notes, meeting SLurl, etc, are usually available on the Content Creation User Group wiki page.

New Viewer Release Notes Pages

The Lab’s RC viewer releases / updates all have shiny new viewer release notes pages on the web. These move the notes away from the old wiki pages, and into a new format that provides:

  • The release notes can be accessed via a new home page, with links to recent SL viewer releases – that is, the current RC viewers.
  • The release notes for a specific viewer, with new icon links to its respective download versions (Windows 64 / 32-bit and Mac OS).
  • There are also links to support information: a new Repositories overview page; an explanation of the viewer version numbering system; a link to the Viewer Support Policy.
  • A significant change is that many of the Jira links in a release page reference the public Jira bug reports, rather than the Lab’s internal MAINT clones. This should make specific bugs addressed in an update more visible to interested users.
  • These pages can also (for the time being at least) be accessed from the existing Alternate Viewers wiki page, when clicking on the release notes link for a specific RC viewer.


The new format viewer release notes pages for SL release candidate viewers (using the EEP viewer as an example)

Environment Enhancement Project

Project Summary

A set of environmental enhancements allowing the environment (sky, sun, moon, clouds, water settings) to be set region or parcel level, with support for up to 7 days per cycle and sky environments set by altitude. It uses a new set of inventory assets (Sky, Water, Day),  and includes the ability to use custom Sun, Moon and cloud textures. The assets can be stored in inventory and traded through the Marketplace / exchanged with others, and can additionally be used in experiences.

Due to performance issues, the initial implementation of EEP will not include certain atmospherics such as crepuscular rays (“God rays”).


Current Status

  • The EEP viewer updated to version on Thursday, May 16th. This update should include a number of bug fixes and shader updates that will hopefully improve things “considerably”.
  • Graham Linden continues to work on the remaining shader / graphics issues.

Animesh Follow-On

  • Vir continues to work on adding visual parameter support to allow shape adjustments to be made to Animesh.
  • This work has new reached a point of being able to store visual parameter information relating to Animesh objects, and to be able to send it to viewers.
  • The next stage of the work is to get to get the viewer to understand what it is supposed to do with these messages.
  • Vir hopes to have an internal prototype for this running in the next few days. If successful, this should pave the way towards a project viewer being made available down the road.
    • It will initially focus on using list-based LSL input to set individual shape parameters.
      • These will probably be throttled to prevent over-use (e.g. to prevent the capability being used as a high performance cost alternative to animation).
    • Once this is working that shapes can potentially be added for manipulation.
    • Obviously, this will require simulator updates to be able to support the LSL commands, as well as the updates required for the new message types, etc.

Animesh as NPCs

  • There has been a lot of forum discussion on using Animesh as non-player characters (NPCs).
  • A limitation here is that a decent-looking NPC, suitably clothed and ready for use can too often have a huge LI (into the 100s).
    • Part of the problem is that until Animesh,  is there was no real incentive to optimise rigged mesh, as being worn by an avatar, it never really had a “Land Impact” per se, and a lot of clothing has an “insane” triangle count (e.g. 21,000 for a tank top).
    • As Animesh calculations do try to allow  for the cost of rigged mesh, such high triangle counts are (simply put) converted in LI, driving up the LI for any Animesh character on which it is used.
  • It is hoped that as the ARCTan project comes to pass, it will offer some form of incentive that will encourage those clothing makers who may not consider optimisations to do so, which will help improve things for both Animesh using wearables and also help reduce some of the performance overhead overly complex avatars can cause.
  • Another alternative would be to (at some point) extend Bakes on Mesh to include Animesh objects, allowing them to be clothed using system layers.
  • In the meantime, the suggestions is that those wishing to create NPCs perhaps consider doing so entirely in Blender (or similar), and not rely on using clothing and attachments that might be available in-world as a means of clothing / equipping them.

Bakes On Mesh

Project Summary

Extending the current avatar baking service to allow wearable textures (skins, tattoos, clothing) to be applied directly to mesh bodies as well as system avatars. This involves viewer and server-side changes, including updating the baking service to support 1024×1024 textures, but does not include normal or specular map support, as these are not part of the existing Bake Service, nor are they recognised as system wearables. Adding materials support may be considered in the future.


Current Status

  • There is a simulator change pending. This includes a means of accessing BOM UUIDs.
    • These were changed in the last back-end update as a result of underlying asset property issues. If there is BOM content using the old UUIDs, this will have to be updated.
    • The simulator update is intended to allow access to the texture UUIDs without having to do so numerically, as is currently the case. This should re-enable the ability to access them via their name abbreviations.
  • There is also a further Appearance Service change pending, designed to correctly handle tattoo layer with partial transparency (currently, if a tattoo with partial transparency is sent for baking via the new BOM channels without any underlying opaque layer, then the alphas are not correctly resolved).