Tag: SL in the Cloud

Don’t forget: Lab Gab, November 6th: cloud update

Posted on by Inara Pey
via Linden Lab

Lab Gab returns on Friday, November 6th, 2020, with a cloud migration update.

As most are aware, the work to transition Second Life to operating via Amazon Web Services (AWS) has now progressed to a point where regions on the main grid (called Agni) are starting to be transitioned. In fact, by the time the Lab Gab show live streams, approximately one-third of all Agni regions will be operating via AWS services.

At the same time, as as per my November 2020 Web User Group summary, the Web teams are hopeful that all web properties will be running via AWS by early December, placing the Lab on course to achieve its target of completing the migration (referred to as Project Uplift) by the end of 2020 (although there will likely be more work related to it to follow in early 2021).

This being the case, the Lab Gab segment will feature Oz Linden, the Lab’s Vice President of Engineering (and the man pretty much in overall charge of the engineering / technical aspect of the work) and Mazidox Linden, the Lab’s senior QA Engineer who has been particularly involved in the migration work, testing the simulator code in reference to the migration work, and who describes the project as “the largest change to the simulator [software] ever.”

“The Bugspray” Mazidox Linden (l) and Oz Linden will be joining Strawberry Linden on the Friday, November 6th segment of Lab Gab to discuss the cloud migration work

As usual, the programme will be streamed via YouTube, Facebook, or Periscope, at 10:00 SLT, and if all goes according to plan, I’ll have a summary of the video (and the video itself) available soon after the the broadcast, for those unable to watch live.

For those who may have questions on the migration work, there is still time to submit them via the Lab Gab Google form, in addition, and if there is time, questions may also be taken from the chat feeds associated with the live stream channels.

2020 SUG meeting week #45: further uplift update

Posted on by Inara Pey
A Thousand Windows, September 2020 – blog post

The following notes were taken from the November 3rd Simulator User Group meeting.

Server Updates and Cloud Uplift

Please reference to the server deployment thread for the latest updates.

  • There are no planned deployments to the simulators running on the Lab’s core SLS channel.
  • RC deployments are planned as follows:
    • On Wednesday, 4th November all simulators on the LeTigre and BlueSteel RC channels should become AWS hosted.
    • On Thursday, 5th November all simulators on the Magnum RC channel should also become AWS hosted.
    • However, at the time of the meeting, it was not clear if all of the RC channels would be running the same version of simulator software.

The current work on migration is such that as per Private Regions Available in Limited Quantity (via Linden Lab), private regions are once more being made available.

SL Viewer

The Start of the week saw the following viewer updates on Monday, November 2nd:

  • The Jellydoll project viewer updated to version 6.4.11.551213.
  • Custom Key Mappings project viewer updated to version 6.4.10.549685.

The rest of the official viewers in the pipelines remain as follows:

  • Current release viewer version 6.4.10.549686, formerly the Mesh Uploader RC promoted on October 14 – No Change.
  • Release channel cohorts:
    • Cachaça Maintenance RC viewer, version 6.4.11.551139, issued October 27.
  • Project viewers:
    • Legacy Profiles viewer, version 6.4.11.550519, October 26.
    • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

In Brief

  • Group Chat: LL deployed updates to the group chat service in an attempt to relieve at least some of the issues that groups have been experiencing over the last several weeks. Testing has suggested the group chat sessions should be faster and more reliable than has been experienced within some groups (notably those with large memberships). However, the issue remains open pending further observation / feedback.
  • TP failures continue, and are being noted by the Lab, although not at the rates at which users appear to be experiencing them.
    • However, the nature of the beast means that at present, correlation of all the logs involved in a teleport has to be done manually, and this is impacting the Lab’s ability to arrive at a potential root cause (or causes).
    • Once the majority of cloud migration work has been completed, and if the matters hasn’t been resolved, Simon Linden may set-up another round of TP testing by users as we’ve seen in past issues of teleport issues.

Lab resumes offering private regions in Second Life

Posted on by Inara Pey
via Linden Lab

In May, as the Lab started gearing-up to move Second Life regions to being hosted on AWS cloud services, an announcement was made that private regions would be subject to limited availability (see Limited Availability of New Second Life Region), prior to further sales of private regions being effectively frozen. 

However, the work in migrating regions from the Lab’s co-lo facility to AWS services progressing well (up to approximately 30% of all main grid regions should be migrated by the end of week #45), so much so that on Tuesday, November 3rd, the Lab announced that private regions are to be made available once more, initially on a limited quantity (per order) basis.

The blog post making the announcement reads in part:

As Oz recently discussed in a post, we are hard at work on uplifting Second Life to the cloud. It’s an incredibly exciting time, and we’re already seeing some significant improvements to Second Life as a service from this process. 
Additionally, many regions on the grid are now running on AWS simulators, with more being added every day!
In light of more regions being moved to the cloud, we are pleased to announce that we are ready to begin offering new private region purchases in limited quantities. As we continue to uplift more of the grid, more regions will become available over time, eventually leading to the Land Store being reopened. 
If you wish to purchase a private region, please submit a support ticket through our Support Portal. The ticket can be submitted under the Land & Region > Order Private Region category. Please include the region name, which must conform to region naming guidelines. Following your ticket submission, we will either process the region purchase, or add your request to a wait list, which will be handled in the order the submission is received. In the event that your request is put on the waiting list, we will not charge your account until the point when the region is delivered.

– Linden lab blog post

Current pricing and requirements for obtaining private regions can be found here, with further information on private regions is available via available here.

2020 TPVD meeting week #44: summary with uplift news

Posted on by Inara Pey
The Muse, September 2020 – blog post

The following notes are taken from the TPV Developer meeting held on Friday, October 30th, 2020. These meetings are generally held every other week, unless otherwise noted in any given summary. The embedded video is provided to Pantera – my thanks to her for recording and providing it. Time stamps are included with the notes will open the video at the point(s) where a specific topic is discussed. Note these summaries are not intended to be a full reporting on all topics discussed, but focus on those items that are more directly user-facing.

Another exceptionally brief meeting, with some discussion in chat, so please refer to the video as well.

Cloud Uplift

[0:00-1:01]

  • The transitioning of regions to running on AWS services in progressing “extremely well”.
  • So far, LL has been able to deal with those issues that have arisen.
  • At the time of writing, just under 15% of the the main (Agni) regions are now running on AWS.
  • This amount is set to increase “significantly” in week #45 (commencing Monday, November 2nd).
  • If there are issues users are encountering with regions running on AWS that haven’t been reported, now is the time to test them (preferably with the official viewer) to confirm they can be reproduced and then report them via the Second Life Jira.

SL Viewer News

[1:06-2:48]

  • Current release viewer version 6.4.10.549686, formerly the Mesh Uploader RC promoted on October 14 – No Change.
  • Release channel cohorts:
    • Cachaça Maintenance RC viewer, version 6.4.11.551139, issued October 27.
  • Project viewers:
    • Project Jelly project viewer (Jellydoll updates), version 6.4.10.549690, October 1.
    • Custom Key Mappings project viewer, version 6.4.5.544079, June 30.
    • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.
    • legacy Profiles project viewer, version 6.3.2.530836 – but see below.

General Viewer Notes

  • The Legacy Profiles project viewer appears to be in a state of flux:
    • Updates to the viewer were dependent on on back-end changes which were in turn dependent on completion of the cloud migration work.
    • It had been hoped that an interim workaround could be made to allow the viewer to progress without the back-end changes.
    • On Monday, October 26th, the viewer was apparently updated to version 6.4.11.550519 (including on the Alternative viewers web page), presumably to bring it to par with the release viewer code base.
    • By Friday, October 30th, this appears to have been rolled-back to version 6.3.2.530836 (noted above), dated September 2019.
    • [10:07-10:35] When released, this viewer will see the deprecation of all aspects of the current web profiles, other than the Feed.
    • [11:02-11:36] The Feed will be made available through the viewer “as is”, but may be enhanced or possibly retired in the future.
  • An upcoming project / RC viewer will feature a replacement for the VFS (Virtual File System) cache.
    • There may be some benefits from the initial release of this viewer, however its primary aim is to get a new cache framework in place for upcoming cache-related projects (e.g. texture fetching and caching).

In Brief

  • [2:54-3:37] Firestorm currently have a pre-release of their EEP Beta viewer in testing, and are moving to promote that viewer to full release status, and is moving towards a code freeze so that it can progress to a release.
  • [4:19-5:13] BUG-229555 “[CEF 2020] AltGr key doesn’t work within internal web browser” – this issues also apparently affects special characters on windows as well. It has been accepted by LL as an internal Jira, and a fix should be in an upcoming Maintenance viewer / update.

2020 SUG meeting week #44: further uplift update

Posted on by Inara Pey
Boulder, September 2020 – blog post

The following notes were taken from the October 20th Simulator User Group meeting.

Server Updates and Cloud Uplift

  • There are no planned deployments to the regions running within the Lab’s co-lo facilities.
  • Starting on Tuesday, October 27th, all regions on the “AWS channel” will be updated with a simulator release  – version 551155 – that includes extended logging capabilities and a range of fixes, as indicated in the release notes.

For details of the deployment of main grid regions running via AWS cloud services, please see the following:

Issues

The grid has seen a range of issues coincident to the uplift work that may or may not be related to that work (as they are occurring across regions that are both now running on AWS and those still running at the Lab’s co-lo facility, although some appear to happen more frequently within AWS region.) The latest of these is object failing to rez when pulled from inventory. However, Mazidox Linden indicated that LL believe they have an handle on this issue, and the AWS region simulator version may have address it.

The most significant  issues (again which may or may not be related to the cloud uplift) being experienced comprise:

  • Group chat failures.
  • Teleport failures.
  • Rezzing issues.
  • Slow opening of legacy profiles.
  • Slow opening on the World Map (hopefully fixed with the 551155 release).

A further issue that has been noted, but yet to be confirmed, is that idle scripts appear to be taking up to to between 16% and 26% more script timing on regions running on AWS when compared to regions still operating out of the Labs co-lo facility.

Uplift Status

  • As per April Linden’s blog post, around 1,000 region on the main grid that are running on AWS.
    • Thus far, LL have only seen minor issues from their perspective, with those regions are running with “only minor hiccups”.
  • The deployment this week will see the total number of regions on AWS increase to between 2,000-2,500.
  • The updates required to allow the simulator to run on AWS systems represents – in Mazidox Linden’s words – “”the largest change to the simulator [software]  ever.”
  • Maxidox also confirmed there is an upcoming change that will see simulators running on AWS given a synthetic hostname rather than the viewer just giving the IP address of the AWS compute node in URI form.  The synthetic hostname will map to how LL describes simhosts internally.

SL Viewer

The Start of the week has seen the following viewer updates:

  • The Cachaça Maintenance RC viewer updated to version 6.4.11.551139 on Tuesday, October 27th, 2020.
  • The Legacy Profiles Viewer updated to version 6.4.11.550519 on Monday, October 26th, 2020.

The rest of the official viewers in the pipelines remain as follows:

  • Current release viewer version 6.4.10.549686, formerly the Mesh Uploader RC released October t and promoted on October 14 – No Change.
  • Project viewers:
    • Project Jelly project viewer (Jellydoll updates), version 6.4.10.549690, October 1.
    • Custom Key Mappings project viewer, version 6.4.5.544079, June 30.
    • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

In Brief

  • Some users (notably Firestorm users) are claiming that uplift is “breaking” region crossings, and that Firestorm should offer a higher default bandwidth allowance. However:
    •  The bandwidth setting applies to UDP only – the majority of data related to avatars, attachments and objects are handled via HTTP through the CDN, over which the UDP bandwidth has no influence.
    • While UDP messaging is involved in region crossings, changing the bandwidth default is viewed as potentially unwise whilst the uplift for is in progress, as it would largely be impossible to objectively tell if the change has improved matters.

Second Life Cloud Uplift: April Linden updates

Posted on by Inara Pey
Logos ©, ™ and ®Linden Lab and Amazon Inc

Things are moving apace with the project to transition all of the Second Life services and simulators to running on Amazon AWS cloud services.

  • On Friday, October 16th, Oz Linden offered a blog update (which I also blogged about) – when there were 100 regions on the main grid running via AWS.
  • On  Tuesday October 20th, I then blogged about the move to expand the number of regions running on AWS.
  • On the same day, I was able to give an update with news that the number of regions on AWS had increased to around 300.

On Wednesday, October 21st, April Linden gave a further update on the project’s status – A Light in the Cloud: A Migration Update– noting that around 1,000 regions are now based on AWS. However, her blog post also comes with a request:

The favour I’ve come to ask you for is your patience.
We’re doing our very best to fix things that come up as we go. This means that we might need to restart regions more often than you’re used to, and things may break just a little more often than we’ve all been accustomed to.
In order to get this project done as fast as possible and minimize the time (and resulting bugs) we have to spend with one foot in our datacenter and the other in the cloud, we don’t want to limit ourselves to restarting regions just once a week. We’re ready to get this project done! We’ve seen how much better Second Life runs in the cloud, and we’re ready to have everyone on the grid experience it.
I’m sorry that things might be a little rough over the next few weeks. It’s our goal to finish the cloud migration by the holidays, so that everyone, Resident and Linden alike, can have a nice quiet holiday with our friends and families.
We can’t promise we’ll make it by then, but we’re sure giving it all we’ve got. The mood around the Lab is really positive right now, and we’re all working hard together to make it happen. I’m really proud to be a part of the team that’s transforming Second Life as we know it.

April Linden

There’s obviously a good way to go before everything is settled, 1,000 regions is a small number compared to the the total on the grid, and there are still various back-end systems to be migrated.

But when you consider the sheer volume of work involved in migrating all of the Lab’s systems and properties related to Second Life, the entire project has, from a user perspective, been remarkably smooth thus far – and hopefully things will continue to be so.

Read all of April’s post here.