Second Life Cloud Uplift: April Linden updates

Logos ©, ™ and ®Linden Lab and Amazon Inc

Things are moving apace with the project to transition all of the Second Life services and simulators to running on Amazon AWS cloud services.

  • On Friday, October 16th, Oz Linden offered a blog update (which I also blogged about) – when there were 100 regions on the main grid running via AWS.
  • On  Tuesday October 20th, I then blogged about the move to expand the number of regions running on AWS.
  • On the same day, I was able to give an update with news that the number of regions on AWS had increased to around 300.

On Wednesday, October 21st, April Linden gave a further update on the project’s status – A Light in the Cloud: A Migration Updatenoting that around 1,000 regions are now based on AWS. However, her blog post also comes with a request:

The favour I’ve come to ask you for is your patience.
We’re doing our very best to fix things that come up as we go. This means that we might need to restart regions more often than you’re used to, and things may break just a little more often than we’ve all been accustomed to.
In order to get this project done as fast as possible and minimize the time (and resulting bugs) we have to spend with one foot in our datacenter and the other in the cloud, we don’t want to limit ourselves to restarting regions just once a week. We’re ready to get this project done! We’ve seen how much better Second Life runs in the cloud, and we’re ready to have everyone on the grid experience it.
I’m sorry that things might be a little rough over the next few weeks. It’s our goal to finish the cloud migration by the holidays, so that everyone, Resident and Linden alike, can have a nice quiet holiday with our friends and families.
We can’t promise we’ll make it by then, but we’re sure giving it all we’ve got. The mood around the Lab is really positive right now, and we’re all working hard together to make it happen. I’m really proud to be a part of the team that’s transforming Second Life as we know it.

April Linden

There’s obviously a good way to go before everything is settled, 1,000 regions is a small number compared to the the total on the grid, and there are still various back-end systems to be migrated.

But when you consider the sheer volume of work involved in migrating all of the Lab’s systems and properties related to Second Life, the entire project has, from a user perspective, been remarkably smooth thus far – and hopefully things will continue to be so.

Read all of April’s post here.

Lab expanding number of regions on AWS

Logos ©, ™ and ® Linden Lab and Amazon Inc

On Friday, October 16th, Oz Linden posted about the work in migrating Second Life services to running on Amazon AWS cloud servers and infrastructure – see: Uplift Update, which I expanded upon in Oz Linden posts on Second Life cloud uplift status.

At the time of that update, it was indicated that about 100 regions on Agni, the main grid, had been transitioned to running on AWS, comprising a mix of Linden-held, Mainland and private regions.

Commencing on Tuesday, October 20th, the Lab will be expanding on the number by “a few hundred” regions during week #43. The announcement came via a forum post by Bugsly Linden, which reads in full (re-formatted for ease of reading):

This week marks the beginning of medium-scale migration of production regions to the cloud (AWS). We will be transferring a few hundred regions from all existing channels of Second Life based on the Engineering team’s need for additional data and performance metrics. Bulk region migration will take place this week between 6:00 AM and 12:00 PM (Noon) Pacific Daylight Time.
If you are interested in having a region you own (or are the alternate payer for) migrated to the cloud, please submit a Support Ticket. Regions that are migrated to the cloud may experience degraded performance or behave incorrectly; if you are in a cloud based region (you can check by clicking Help->About, and if you see the URL on line 3 end in “amazonaws.com” you’re in the cloud) and observe behaviour you believe is newly incorrect, please file a BUG at jira.secondlife.com.
Due to the rapid speed of our Uplift efforts, we are unable to guarantee that regions in the cloud that are behaving incorrectly will be moved immediately back to our existing hosts.

Bugsly Linden

Spotting a region hosted in the cloud via Help About. Top: a region hosted at the Lab’s co-location facility (note the agni.lindenlab.com in the address). Bottom: and a region running on a simulator in the cloud

There have been concerns about aspects of performance with regions operating in the AWS environment – particularly with regards to communications with the KVP database associated with experiences, which has yet to be migrated), and there are on-going teleport issues that might be related to the Uplift work, although as Oz indicated in his blog post, this has yet to be confirmed.

Elsewhere, some of those with regions already uplifted – such as London City (see London City Uplifted) – have reported generally good performance, if with a one or two caveats such as legacy profiles being slow to load in viewers that support them (which again may or may not be Uplift related). Similarly,those testing region crossings between those based in the AWS environment and those still within the Lab’s co-lo facility have reported them to be generally “smooth”.

However, as Bugsly’s forum post implies, there could be a period of degraded performance within regions that are transitioned to AWS, so keep an eye on where you are in SL.

Byeline

The Fourmilab Cloud Halo detecting an AWS region. Credit: Fourmilab

For those who are curious about whether or not they are in a region that has been uplifted to the cloud or not, and who don’t like opening menus and panels to find out, Fourmilab may have the answer: a freebie full permission halo that attaches to your avatar.

By default, it is invisible, but should you move from a region hosted by the Lab in their co-lo facility to one running on AWS, it will sparkle briefly into life to the faint accompaniment of harps being played (well, you’re in the cloud, after all) before vanishing.

Similarly, when you leave a region running in the cloud and return to one still based in the Lab’s co-lo facility, the halo will again briefly appear, this time a dull grey and without the golden sparkles, while a trombone plays a couple of sad notes.

Note that it will only play when moving between regions hosted in the two facilities, not when moving between regions in the same facility (so you won’t repeatedly get the trombone when moving between regions that are all hosted at the Lab’s co-lo for example).

You can grab the halo here.

 

2020 TPVD meeting week #42: summary

Isolation’s Passengers – blog post

The following notes are taken from the TPV Developer meeting held on Friday, October 16th, 2020. These meetings are generally held every other week, unless otherwise noted in any given summary. The embedded video is provided to Pantera – my thanks to her for recording and providing it. Time stamps are included with the notes will open the video at the point(s) where a specific topic is discussed. Note these summaries are not intended to be a full reporting on all topics discussed, but focus on those items that are more directly user-facing.

Another exceptionally brief meeting, with some discussion in chat, so please refer to the video as well.

SL Viewer News

[0:15-2:18]

As per my CCUG meeting notes, the Mesh Uploader RC was promoted this week to de facto release status – refer to the release notes (link below) for details of the cosmetic and informational changes made to the uploader (which are in part based on contributions by Beq Janus of the Firestorm team). This leaves the current official viewer release list as follows:

  • Current release viewer version 6.4.10.549686, formerly the Mesh Uploader RC released October t and promoted on October 14 – NEW.
  • Release channel cohorts:
    • Cachaça Maintenance RC viewer, version 6.4.10.549752, issued October 1.
  • Project viewers:
    • Project Jelly project viewer (Jellydoll updates), version 6.4.10.549690, October 1.
    • Custom Key Mappings project viewer, version 6.4.5.544079, June 30.
    • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
    • Legacy Profiles viewer, version 6.3.2.530836, September 17, 2019. Covers the re-integration of Viewer Profiles.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

General Viewer Notes

  • As per this week’s CCUG meeting, Vir is hopeful that the Legacy Profiles project viewer may be able to move forward soon using a workaround to get it past the current blocker of needing some web-side updates.
  • The Custom Key Mappings project viewer is being considered from promotion to RC status with its next update.
  • There is an upcoming RC viewer featuring some UI changes – possibly including the update to give HUD items in inventory their own inventory icon, as discussed by Steeltoe Linden in the CCUG week #40 meeting. Thus might be appearing in the next week or so.
The proposal UI change to distinguish HUD items from other objects in inventory.Left: how things are now, with HUDs using the same cube icon as other objects. Right: the proposed new HUD cog icon.
  • [5:24-6:54] There have been some issues with viewer documentation not appearing on the relevant web pages, leading to concerns that such documentation might be “going away”.
    • The problem has actually been caused by the mechanism used to generate the information not having been updated for some time (and also not even using current information), so when it was migrated to AWS services, it “fell apart”.
    • Rather than try to fix the issue, the decision has been made so that once uplift has been completed, the mechanism will be replaced with an easier to maintain process. However, it does mean that for the time being, some viewer related documentation may not be available.

In Brief

  • [9:20-10:00] BUG-228679: Media-on-a-Prim doesn’t register keystrokes correctly on Mac – unfortunately, no progress to report.
  • [12:12-13:40] A couple of TPVs have been holding off making a formal release of the Environment Enhancement Project (EEP) updates beyond their current beta releases in case LL make further EEP update releases like the recent Love Me Render #4 viewer release. However, the advice from the Lab to these TPVs is:
    • To consider the high-priority EEP fixes as being “done”, as the Graphics team is now focused on priority long-term architecture work (including moving the viewer beyond OpenGL).
    • That future EEP fixes, when made, will form a part of the regular Maintenance RC viewer releases, rather than having any dedicated RC viewer updates.
  • [13:49-16:58] EEP performance issues:
    • Some TPVs have reported significant performance / FPS drop-off with EEP (around 15%). However, the Lab is not seeing anything close to this with the official viewer.
    • Unless there are detailed reports filed on the issue – using the official release viewer – that provide information on where / when / how the issue is being witnessed (e.g. the kind of test carried out, where in-world, the hardware configuration and viewer settings used, the results obtained, etc.), that allow LL to reproduce the problems, they do not see what can be done by way of investigation / fixes.
      • It was pointed out that comparable testing on the official viewer is difficult due to the limited amount of video memory available (512 MB compared to a minimum of 1 GB on most TPVs), which also affects performance, and so might skew results / how results are seen (e.g. are they EEP or are they due to texture load issues?).
      • Vir indicated that the amount of assigned texture memory is something the Lab will be looking at in the future.

Oz Linden posts on Second Life cloud uplift status

Logos ©, ™ and ® Linden Lab and Amazon Inc

On Friday,  October 16th, Oz Linden posted on the status of the cloud uplift work – see: Uplift Update -, the article coming as regions on the main (Agini) grid are gradually starting to be migrated to AWS services. For those possibly unaware of this project, Oz provides an opening explanation:

We’ve been working hard on the Uplift of Second Life. If you have not been following this project, that’s what we’re calling the migration of our Second Life simulators, services, and websites from a private data centre to hosting in The Cloud (Amazon Web Services). It’s a massive, complicated project that I’ve previously compared to converting a steam-driven railroad to a maglev monorail — without ever stopping the train. This undertaking has at times been smooth sailing, at other times a very bumpy ride. We wanted to share some more of the story with you.

The uplift project was first announced in August 2017, and formed a part of the Lab’s presentation at the AWS Reinvent conference that same year – which if nothing else points to the amount of planning and testing that has been going on both before and during the gradual migration of services, which has been going on for somewhere between 12 and 18 months, recently reaching the point were the aforementioned main grid region migrations could commence.

As Oz notes, the work has been very incremental in nature, and always with a the aim of transitioning services in such a way that users generally have not been aware of which services have moved and when. This has certainly been true for many of the back-end services (no-one noticed when the log-in services and the inventory  / asset services moved to AWS, for example). However, as Oz notes in his post, there have been a few bumps on the road.

Some of these problems were initially manifested on Aditi, the Bet grid, which saw batches of regions cloned from the main grid and transitioned to the cloud. Region crossings were one such problem which, thanks to extensive testing by users on Aditi, allowed the Lab to make changes to region crossing that have generally improved things even sans the uplift – although as the Lab readily notes, there is still future work to be done on region crossings once the uplift work has been completed.

Work related to the uplift project allowed the lab to make improvements to region crossings that have benefited Second Life even before regions on Agni (the main grid) commenced a slow migration to AWS

However,  some problems unfortunately only manifested once some back-end services had been uplifted and were so bedded-in to running on AWS, reverting to running them out of the Lab’s co-lo is no longer an option. Again, as Oz notes, the recent group service issues being a case in point. Other issues – such as the recent bout of avatar bake (appearance) failures – have been the result not of moving that service to the cloud (the Bake Service has also been AWS based for a while without most users noticing), but in making subsequent changes to a related service – again pointing to the complexities involved in moving multiple systems and services from an established operating environment to an entirely new operating environment.

Elsewhere, there has been a need to revert the Marketplace to running via the lab’s co-lo (albeit it temporarily), whilst some known issues  – such as teleport failures – may or may not be linked to migration issues, with the Lab engaged in trying to get to the bottom of things. So if you do see a problem, don’t automatically assume it is uplift related; even without the current migration work, SL can be temperamental!

Currently, around 100 regions on the main grid have been uplifted, and Oz confirms that, barring the unforeseen, the end-of-2020 for uplift completion is still very much the goal.

For more information, please take a read of his post.

2020 TPVD meeting week #40: summary (more cloud uplift)

Eterea, August, 2020  – blog post

The following notes are taken from the TPV Developer meeting held on Friday, October 2nd, 2020. These meetings are generally held every other week, unless otherwise noted in any given summary. The embedded video is provided to Pantera – my thanks to her for recording and providing it. Time stamps are included with the notes will open the video at the point(s) where a specific topic is discussed. Note these summaries are not intended to be a full reporting on all topics discussed, but focus on those items that are more directly user-facing.

This was an exceptionally brief meeting, with some discussion in chat, so please refer to the video as well.

Cloud Uplift – Agni

[0:06-3:39]

In my CCUG meeting summary, I noted that regions running on AWS are starting to appear on the main grid (Agni).

Commenting on progress of the Uplift project at the start of the meeting, Oz Linden indicated the Testylvania region (a restricted access region intended for testing purposes) is also running via AWS. It is regarded as “feature complete”, and the region is specifically available to TPV viewer developer for compatibility testing.

Any TPV developers who cannot access the region should contact one of Mazidox, Maestro or Kyle Linden.

It addition, Oz noted:

We currently have several dozen regions [on Agni] running uplifted, some of them user-accessible and some not. We will be ramping that up over the next few weeks. … At some point we will be moving the regular RC [deployments]  there as well, but we’re not quite ready for that yet due to back-end considerations that shouldn’t affect users directly … But one of the ways you can ask for trouble when dealing with computer programmes is by saying it “shouldn’t” do something.  

Oz Linden, TPV Developer Meeting, October 2nd

There may be a issue with attachments ghosting more frequently when teleporting between two regions running on AWS, but this has yet to be confirmed / a bug report raised.

SL Viewer News

[2:45-6:08]

As per may SUG and CCUG meeting updates, the current official viewer pipelines are as follows:

  • Current release viewer :Love Me Render #4 (EEP fixes), version 6.4.9.549455, released September 24, promoted September 28th.
  • Release channel cohorts:
    • Cachaça Maintenance RC viewer, version 6.4.10.549752, issued October 1st.
    • Mesh uploader RC viewer, version 6.4.10.549686, October 1st.
  • Project viewers:
    • Project Jelly project viewer (Jellydoll updates), version 6.4.10.549690, October 1st.
    • Custom Key Mappings project viewer, version 6.4.5.544079, June 30th.
    • Copy / Paste viewer, version 6.3.5.533365, December 9th, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22nd, 2019.
    • Legacy Profiles viewer, version 6.3.2.530836, September 17th, 2019. Covers the re-integration of Viewer Profiles.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16th, 2019.

General Viewer Notes

  • With the promotion of the LMR #4 viewer, the graphics team are turning to other graphics / rendering projects.
    • Note that LMR #4  appears to resolve the issue of the large performance hit linden Water has on EEP viewers.
  • The next viewer liable for promotion is likely to be the Mesh Uploader RC viewer, most likely in week #42 (commencing Monday, October 12th, 2020).
  • An upcoming series of viewers will be focused on UI improvements. /one of the aims of this work will be to overcome some of the long-standing viewer issues, as well as adding some new features.
  • For the project viewers:
    • The Legacy Profiles viewer is still awaiting the back-end changes.
    • The Custom Key Mapping viewer has not had a lot of feedback, and as a result is being considered for update to RC status.
    • The remaining project viewer are unlikely to change in the short-term.

In  Brief

  • [8:10-8:45] There are concerns among some Mac users on older hardware over their upgrade path with Apple’s announcement of a move to using ARM-based CPUs. LL is trying to get hold of an ARM-based test system directly from Apple to enable them to carry out in-depth investigation / testing.

 

2020 Simulator User Group week #39 summary

Dim Sum Gardens, July 2020 – blog post

The following notes were taken from the September 22nd Simulator User Group meeting – although there is not a lot in practical terms to report.

Simulator Deployments

Please refer to the server deployment thread for news and updates.

  • On Tuesday, September 22nd, the grid’s main SLS channel was updated to simulator maintenance release 548903, containing updates related to the cloud uplift work which contain no user-visible changes.
  • There are no RC deployments planned for the week.

SL Viewer

The Bormotukha Maintenance RC viewer, version 6.4.8.548890 dated September 18th, was promoted to de facto release status on Tuesday, September 22nd.

The rest of the official viewer pipelines remain as follows:

  • Current release viewer version 6.4.7.546539, dated August 11, promoted August 17, formerly the Arrack Maintenance RC viewer – No Change.
  • Release channel cohorts:
    • Mesh uploader RC viewer, version 6.4.8.548061, September 8.
    • Love Me Render RC viewer, version 6.4.8.547427, August 21.
  • Project viewers:
    • Project Jelly project viewer (Jellydoll updates), version 6.4.8.547487, issued August 26.
    • Custom Key Mappings project viewer, version 6.4.5.544079, June 30.
    • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
    • Legacy Profiles viewer, version 6.3.2.530836, September 17, 2019. Covers the re-integration of Viewer Profiles.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

Cloud Uplift

  • There is a known issue with Navmesh retrieval on cloud-hosted regions on Aditi.
  • The Lab believe they have a solution for the outward e-mail issue with cloud-based regions. However, due to the way in which Aditi regions are set-up, testing of the fix is being carried out elsewhere, but the hope is the fix will be usable once there are user-accessible regions on Agni that are running on AWS services.

In Brief

  • There is no update on the Group chat / IM failure issue (see BUG-229219).
  • The majority of the meeting focused on a general discussion on region crossings – notably with vehicles – and the assorted problems that can arise, including the inconsistencies in some things like object entry and group access are handled (the former can prevent vehicle and avatar entry, whereas the latter can allow a vehicle into a region / parcel, but then eject the driver / passenger, obviously resulting in problems).