Tag: SL in the Cloud

2020 SUG meeting week #43: further uplift update

Posted on by Inara Pey
Bungenäs, Binemust – blog post

The following notes were taken from the October 20th Simulator User Group meeting.

Cloud Uplift Update

There are no formal simulator deployments for week #43. This is to make way for selected regions (both RC and Main channel) to be migrated to AWS services (the cloud). This means those regions that are transitioned will be restarted.

For details of the initial announce on the uplift, please refer to Lab expanding number of regions on AWS.

Speaking at the simulator User Group meeting on October 20th, Maxidox Linden provided the following update:

This morning we started our medium-size (at least by the size of a world) movement of regions to cloud based simhosts. Specifically we now have over 300 simulators running in the cloud. We’re looking at how quickly we can move more, as well.
For this round we went with a mixture of extremely high impact regions and extremely low impact “filler” regions, to stress various parts of our systems like the region allocation process and how we pack regions into a host.
In future rounds it’s likely we’ll be focusing on specific Release Candidate channels.
Basically: My *goal* (I’m not saying it’s what we’re doing, but I’m saying it’s what I’m trying to do) is to make this the worst it gets, and it only gets better from here.

– Mazidox Linden

Bullet Notes on Uplift

  • As of the October 20th SUG meeting, some 300 regions have been transitioned to the cloud, representing a mix of region in terms of use.
  • Performance issues have been warned about because there are a number of back-end systems that have yet to be uplifted (and presumably need the simulators in place before they can be), which may have some impact.
  • The hope is that some of these additional services will be transitioned when they can, rather than being “left until last”.
    • One of those earmarked for transition that has been causing some angst are the servers managing the KVP database for experiences.  There have already bee reports of degraded experience performance on AWS-hosted regions, so the hope is to get the KVP database moved, with “quiet a bit of effort” going into it.
  • Those who have had regions uplifted and feel they are suffering adversely from the move can request a roll back to being hosted in the Lab’s co-lo. However:
    • As all regions will be running on AWS services by year’s end, LL would rather region holders bear with any issues they may encounter whilst things are in a state of flux, and if they are encountering specific issues, to work with the Lab to try and identity and rectify matters.
    • Those who feel they need a roll-back to the lab’s co-lo facility should file a support ticket, providing a clear indication of why they would like their region(s) rolled back, and the issues they are encountering.
During the transition (as now) either the uplifted or data centre regions will have some penalty … we expect that will mostly disappear by the time we’re done

– Oz Linden on possible simulator (and other) performance impacts
during region migration

  • The Lab’s aim is to continue to make the transitioning of services from their co-lo to AWS services as invisible to users as possible (that is, you shouldn’t really be able to tell the difference between a service now running on AWS and when it was running via the Lab’s co-lo).

SL Viewer

There have been no updates to the current list of official viewers, leaving the pipelines as follows:

  • Current release viewer version 6.4.10.549686, formerly the Mesh Uploader RC released October t and promoted on October 14 – No Change.
  • Release channel cohorts:
    • Cachaça Maintenance RC viewer, version 6.4.10.549752, issued October 1.
  • Project viewers:
    • Project Jelly project viewer (Jellydoll updates), version 6.4.10.549690, October 1.
    • Custom Key Mappings project viewer, version 6.4.5.544079, June 30.
    • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
    • Legacy Profiles viewer, version 6.3.2.530836, September 17, 2019. Covers the re-integration of Viewer Profiles.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

In Brief

  • Group chat issues are being worked on, but is pending the uplift work.
  • The long-promised increase in sound loop support from 10 seconds to 30seconds, first promised a fair while ago now, is … pending the uplift work – although it is rumoured (and subject to confirmation) to be a Premium Plus benefit.

Lab expanding number of regions on AWS

Posted on by Inara Pey
Logos ©, ™ and ® Linden Lab and Amazon Inc

On Friday, October 16th, Oz Linden posted about the work in migrating Second Life services to running on Amazon AWS cloud servers and infrastructure – see: Uplift Update, which I expanded upon in Oz Linden posts on Second Life cloud uplift status.

At the time of that update, it was indicated that about 100 regions on Agni, the main grid, had been transitioned to running on AWS, comprising a mix of Linden-held, Mainland and private regions.

Commencing on Tuesday, October 20th, the Lab will be expanding on the number by “a few hundred” regions during week #43. The announcement came via a forum post by Bugsly Linden, which reads in full (re-formatted for ease of reading):

This week marks the beginning of medium-scale migration of production regions to the cloud (AWS). We will be transferring a few hundred regions from all existing channels of Second Life based on the Engineering team’s need for additional data and performance metrics. Bulk region migration will take place this week between 6:00 AM and 12:00 PM (Noon) Pacific Daylight Time.
If you are interested in having a region you own (or are the alternate payer for) migrated to the cloud, please submit a Support Ticket. Regions that are migrated to the cloud may experience degraded performance or behave incorrectly; if you are in a cloud based region (you can check by clicking Help->About, and if you see the URL on line 3 end in “amazonaws.com” you’re in the cloud) and observe behaviour you believe is newly incorrect, please file a BUG at jira.secondlife.com.
Due to the rapid speed of our Uplift efforts, we are unable to guarantee that regions in the cloud that are behaving incorrectly will be moved immediately back to our existing hosts.

Bugsly Linden

Spotting a region hosted in the cloud via Help About. Top: a region hosted at the Lab’s co-location facility (note the agni.lindenlab.com in the address). Bottom: and a region running on a simulator in the cloud

There have been concerns about aspects of performance with regions operating in the AWS environment – particularly with regards to communications with the KVP database associated with experiences, which has yet to be migrated), and there are on-going teleport issues that might be related to the Uplift work, although as Oz indicated in his blog post, this has yet to be confirmed.

Elsewhere, some of those with regions already uplifted – such as London City (see London City Uplifted) – have reported generally good performance, if with a one or two caveats such as legacy profiles being slow to load in viewers that support them (which again may or may not be Uplift related). Similarly,those testing region crossings between those based in the AWS environment and those still within the Lab’s co-lo facility have reported them to be generally “smooth”.

However, as Bugsly’s forum post implies, there could be a period of degraded performance within regions that are transitioned to AWS, so keep an eye on where you are in SL.

Byeline

The Fourmilab Cloud Halo detecting an AWS region. Credit: Fourmilab

For those who are curious about whether or not they are in a region that has been uplifted to the cloud or not, and who don’t like opening menus and panels to find out, Fourmilab may have the answer: a freebie full permission halo that attaches to your avatar.

By default, it is invisible, but should you move from a region hosted by the Lab in their co-lo facility to one running on AWS, it will sparkle briefly into life to the faint accompaniment of harps being played (well, you’re in the cloud, after all) before vanishing.

Similarly, when you leave a region running in the cloud and return to one still based in the Lab’s co-lo facility, the halo will again briefly appear, this time a dull grey and without the golden sparkles, while a trombone plays a couple of sad notes.

Note that it will only play when moving between regions hosted in the two facilities, not when moving between regions in the same facility (so you won’t repeatedly get the trombone when moving between regions that are all hosted at the Lab’s co-lo for example).

You can grab the halo here.

 

Oz Linden posts on Second Life cloud uplift status

Posted on by Inara Pey
Logos ©, ™ and ® Linden Lab and Amazon Inc

On Friday,  October 16th, Oz Linden posted on the status of the cloud uplift work – see: Uplift Update -, the article coming as regions on the main (Agini) grid are gradually starting to be migrated to AWS services. For those possibly unaware of this project, Oz provides an opening explanation:

We’ve been working hard on the Uplift of Second Life. If you have not been following this project, that’s what we’re calling the migration of our Second Life simulators, services, and websites from a private data centre to hosting in The Cloud (Amazon Web Services). It’s a massive, complicated project that I’ve previously compared to converting a steam-driven railroad to a maglev monorail — without ever stopping the train. This undertaking has at times been smooth sailing, at other times a very bumpy ride. We wanted to share some more of the story with you.

The uplift project was first announced in August 2017, and formed a part of the Lab’s presentation at the AWS Reinvent conference that same year – which if nothing else points to the amount of planning and testing that has been going on both before and during the gradual migration of services, which has been going on for somewhere between 12 and 18 months, recently reaching the point were the aforementioned main grid region migrations could commence.

As Oz notes, the work has been very incremental in nature, and always with a the aim of transitioning services in such a way that users generally have not been aware of which services have moved and when. This has certainly been true for many of the back-end services (no-one noticed when the log-in services and the inventory  / asset services moved to AWS, for example). However, as Oz notes in his post, there have been a few bumps on the road.

Some of these problems were initially manifested on Aditi, the Bet grid, which saw batches of regions cloned from the main grid and transitioned to the cloud. Region crossings were one such problem which, thanks to extensive testing by users on Aditi, allowed the Lab to make changes to region crossing that have generally improved things even sans the uplift – although as the Lab readily notes, there is still future work to be done on region crossings once the uplift work has been completed.

Work related to the uplift project allowed the lab to make improvements to region crossings that have benefited Second Life even before regions on Agni (the main grid) commenced a slow migration to AWS

However,  some problems unfortunately only manifested once some back-end services had been uplifted and were so bedded-in to running on AWS, reverting to running them out of the Lab’s co-lo is no longer an option. Again, as Oz notes, the recent group service issues being a case in point. Other issues – such as the recent bout of avatar bake (appearance) failures – have been the result not of moving that service to the cloud (the Bake Service has also been AWS based for a while without most users noticing), but in making subsequent changes to a related service – again pointing to the complexities involved in moving multiple systems and services from an established operating environment to an entirely new operating environment.

Elsewhere, there has been a need to revert the Marketplace to running via the lab’s co-lo (albeit it temporarily), whilst some known issues  – such as teleport failures – may or may not be linked to migration issues, with the Lab engaged in trying to get to the bottom of things. So if you do see a problem, don’t automatically assume it is uplift related; even without the current migration work, SL can be temperamental!

Currently, around 100 regions on the main grid have been uplifted, and Oz confirms that, barring the unforeseen, the end-of-2020 for uplift completion is still very much the goal.

For more information, please take a read of his post.

2020 Simulator User Group week #41 summary (w/uplift)

Posted on by Inara Pey
Broken, August 2020 – blog post

The following notes were taken from the October 6th Simulator User Group meeting.

Simulator Deployments

Please refer to the server deployment thread for news and updates.

  • There are currently no planned deployments for week #41. An RC update that had been planned had to be pulled after QA found an issue when attempting to uploading items that had been previously uploaded and edited.
  • This means that Some RC regions should be started during the Wednesday, October 7th deployment window, but they will not receive any updates.

SL Viewer

There have been no updates to the current list of official viewers, leaving the pipelines as follows:

  • Current release viewer :Love Me Render #4 (EEP fixes), version 6.4.9.549455, released September 24th, promoted September 28th – No Change.
  • Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
    • Cachaça Maintenance RC viewer, version 6.4.10.549752, issued October 1st.
    • Mesh uploader RC viewer, version 6.4.10.549686, October 1st.
  • Project viewers:
    • Project Jelly project viewer (Jellydoll updates), version 6.4.10.549690, October 1st.
    • Custom Key Mappings project viewer, version 6.4.5.544079, June 30th.
    • Copy / Paste viewer, version 6.3.5.533365, December 9th, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22nd, 2019.
    • Legacy Profiles viewer, version 6.3.2.530836, September 17th, 2019. Covers the re-integration of Viewer Profiles.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16th, 2019.

Cloud Uplift

More regions on the main (Agni) grid are being noted as being on AWS (see my previous TPV Developer meeting and CCUG meeting notes). As a result some are (prematurely) proclaiming this to be a sign that “the end of the uplift work is in sight”; however, Oz Linden sobered such statements, commenting:

I’m not sure that we’re at the beginning of the end … but we’re past the end of the beginning.

To which Mazidox Linden added:

We’re currently running some tests with production data on AWS regions, yep. … There are definitely some outstanding issues we’re aware of, and more we’re probably not (yet) that we’re trying to shake out.

One aspect of the current situation is that while there may be some regions with associated experiences running via AWS, the core data handling for experiences still lies within the Lab’s co-lo facility; this may affect the experience performance in those regions, but should be rectified as the uplift work continues.

In terms of when specific aspects of the uplift will be “completed” (remembering that LL is looking to have a majority of the work done by year-end), Oz Linden further commented:

We’re lifting as quickly as we can, consistent with checking for problems. So far it’s going better than expected, but making any prediction would jinx it.
For the curious – spotting a region hosted in the cloud via Help About. Top: a region hosted at the Lab’s co-location facility (note the agni.lindenlab.com in the address). Bottom: and a region running on a simulator in the cloud, using an AWS address.

As per K.T. Kingsley’s comment below, scripters confused about obtaining host names via scripted means should refer to this forum thread.

In Brief

  • The Jejina region (old-style Linden Homes) has been reported as having “weird” EEP settings that are out of sync with the surrounding regions, and also has a odd Map tile. The exact cause is unknown. Anyone seeing similar oddities in Mainland EEP settings / Map tiles should contact support.
  • A large swath of Satori suffered significant issues with regions staying up during a 24-hour period over the weekend. Exact cause is unknown, but the issue now appears resolved.

2020 TPVD meeting week #40: summary (more cloud uplift)

Posted on by Inara Pey
Eterea, August, 2020  – blog post

The following notes are taken from the TPV Developer meeting held on Friday, October 2nd, 2020. These meetings are generally held every other week, unless otherwise noted in any given summary. The embedded video is provided to Pantera – my thanks to her for recording and providing it. Time stamps are included with the notes will open the video at the point(s) where a specific topic is discussed. Note these summaries are not intended to be a full reporting on all topics discussed, but focus on those items that are more directly user-facing.

This was an exceptionally brief meeting, with some discussion in chat, so please refer to the video as well.

Cloud Uplift – Agni

[0:06-3:39]

In my CCUG meeting summary, I noted that regions running on AWS are starting to appear on the main grid (Agni).

Commenting on progress of the Uplift project at the start of the meeting, Oz Linden indicated the Testylvania region (a restricted access region intended for testing purposes) is also running via AWS. It is regarded as “feature complete”, and the region is specifically available to TPV viewer developer for compatibility testing.

Any TPV developers who cannot access the region should contact one of Mazidox, Maestro or Kyle Linden.

It addition, Oz noted:

We currently have several dozen regions [on Agni] running uplifted, some of them user-accessible and some not. We will be ramping that up over the next few weeks. … At some point we will be moving the regular RC [deployments]  there as well, but we’re not quite ready for that yet due to back-end considerations that shouldn’t affect users directly … But one of the ways you can ask for trouble when dealing with computer programmes is by saying it “shouldn’t” do something.  

Oz Linden, TPV Developer Meeting, October 2nd

There may be a issue with attachments ghosting more frequently when teleporting between two regions running on AWS, but this has yet to be confirmed / a bug report raised.

SL Viewer News

[2:45-6:08]

As per may SUG and CCUG meeting updates, the current official viewer pipelines are as follows:

  • Current release viewer :Love Me Render #4 (EEP fixes), version 6.4.9.549455, released September 24, promoted September 28th.
  • Release channel cohorts:
    • Cachaça Maintenance RC viewer, version 6.4.10.549752, issued October 1st.
    • Mesh uploader RC viewer, version 6.4.10.549686, October 1st.
  • Project viewers:
    • Project Jelly project viewer (Jellydoll updates), version 6.4.10.549690, October 1st.
    • Custom Key Mappings project viewer, version 6.4.5.544079, June 30th.
    • Copy / Paste viewer, version 6.3.5.533365, December 9th, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22nd, 2019.
    • Legacy Profiles viewer, version 6.3.2.530836, September 17th, 2019. Covers the re-integration of Viewer Profiles.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16th, 2019.

General Viewer Notes

  • With the promotion of the LMR #4 viewer, the graphics team are turning to other graphics / rendering projects.
    • Note that LMR #4  appears to resolve the issue of the large performance hit linden Water has on EEP viewers.
  • The next viewer liable for promotion is likely to be the Mesh Uploader RC viewer, most likely in week #42 (commencing Monday, October 12th, 2020).
  • An upcoming series of viewers will be focused on UI improvements. /one of the aims of this work will be to overcome some of the long-standing viewer issues, as well as adding some new features.
  • For the project viewers:
    • The Legacy Profiles viewer is still awaiting the back-end changes.
    • The Custom Key Mapping viewer has not had a lot of feedback, and as a result is being considered for update to RC status.
    • The remaining project viewer are unlikely to change in the short-term.

In  Brief

  • [8:10-8:45] There are concerns among some Mac users on older hardware over their upgrade path with Apple’s announcement of a move to using ARM-based CPUs. LL is trying to get hold of an ARM-based test system directly from Apple to enable them to carry out in-depth investigation / testing.

 

Play Linden Realms in the cloud and help the Lab

Posted on by Inara Pey
Linden Realms is now running in the cloud – on the beta grid. Image courtesy of Linden Lab

Linden Realms was the first Linden Lab developed game using experience tools. First introduced in 2011 (see: The Linden Realms game: Rock on!), the game has been updated numerous times over the years, the last time in 2018 (see: The further revamp of Linden Realms in Second Life).

It is now the next major element of Second Life (after the core Blake Sea regions) to be cloned to Aditi, the beta grid, and uploaded to the cloud – and Linden Lab are asking users to give it a go, so they can further test region  / simulator operations under load when running on AWS services.

The request for assistance came via a blog post on Monday, September 15th, which reads in full:

As you may know, we are in the process of moving Second Life to the cloud! Our first ever cloud simulators, on the beta grid, have been uplifted, and we can use your assistance. Here’s your opportunity to be among the first Residents to test the performance of uplift.
Log in to the beta grid (click here for instructions) and start at the Aditi Portal Park to try out Linden Realms in the cloud. Bring your friends and spend some time engaging in the virtual experience produced and provided by Linden Lab. You may even run into the often unseen technical Lindens working away. Don’t forget to ask them for their Linden bear, and beware of the rock monster!
If you find any issues with Linden Realms on the beta grid, please file a BUG jira at https://jira.secondlife.com, and make sure to include the time, date, region you were in when you found the issue, and a description of what happened, as well as what you would expect to happen in a similar situation on the Main Grid today.

– We Need Your Help Testing Performance on Uplifted Simulator – Linden Lab, September 15th, 2020

Linden Realms. Image courtesy of Linden Lab

So, if you want to help speed the cloud uplift process, why not follow the instructions contained in the links in the Lab’s blog post, log-in to Aditi and hop over to the beta grid and spending a little time playing Linden Realms – even if you don’t find anything to report, issue-wise, your time playing the game is still helping the Lab gather data on region and simulator performance.