Lab Gab returns on Friday, November 6th, 2020, with a cloud migration update.
As most are aware, the work to transition Second Life to operating via Amazon Web Services (AWS) has now progressed to a point where regions on the main grid (called Agni) are starting to be transitioned. In fact, by the time the Lab Gab show live streams, approximately one-third of all Agni regions will be operating via AWS services.
At the same time, as as per my November 2020 Web User Group summary, the Web teams are hopeful that all web properties will be running via AWS by early December, placing the Lab on course to achieve its target of completing the migration (referred to as Project Uplift) by the end of 2020 (although there will likely be more work related to it to follow in early 2021).
This being the case, the Lab Gab segment will feature Oz Linden, the Lab’s Vice President of Engineering (and the man pretty much in overall charge of the engineering / technical aspect of the work) and Mazidox Linden, the Lab’s senior QA Engineer who has been particularly involved in the migration work, testing the simulator code in reference to the migration work, and who describes the project as “the largest change to the simulator [software] ever.”
“The Bugspray” Mazidox Linden (l) and Oz Linden will be joining Strawberry Linden on the Friday, November 6th segment of Lab Gab to discuss the cloud migration work
As usual, the programme will be streamed via YouTube, Facebook, or Periscope, at 10:00 SLT, and if all goes according to plan, I’ll have a summary of the video (and the video itself) available soon after the the broadcast, for those unable to watch live.
For those who may have questions on the migration work, there is still time to submit them via the Lab Gab Google form, in addition, and if there is time, questions may also be taken from the chat feeds associated with the live stream channels.
In May, as the Lab started gearing-up to move Second Life regions to being hosted on AWS cloud services, an announcement was made that private regions would be subject to limited availability (see Limited Availability of New Second Life Region), prior to further sales of private regions being effectively frozen.
However, the work in migrating regions from the Lab’s co-lo facility to AWS services progressing well (up to approximately 30% of all main grid regions should be migrated by the end of week #45), so much so that on Tuesday, November 3rd, the Lab announced that private regions are to be made available once more, initially on a limited quantity (per order) basis.
As Oz recently discussed in a post, we are hard at work on uplifting Second Life to the cloud. It’s an incredibly exciting time, and we’re already seeing some significant improvements to Second Life as a service from this process.
Additionally, many regions on the grid are now running on AWS simulators, with more being added every day!
In light of more regions being moved to the cloud, we are pleased to announce that we are ready to begin offering new private region purchases in limited quantities. As we continue to uplift more of the grid, more regions will become available over time, eventually leading to the Land Store being reopened.
If you wish to purchase a private region, please submit a support ticket through our Support Portal. The ticket can be submitted under the Land & Region > Order Private Region category. Please include the region name, which must conform to region naming guidelines. Following your ticket submission, we will either process the region purchase, or add your request to a wait list, which will be handled in the order the submission is received. In the event that your request is put on the waiting list, we will not charge your account until the point when the region is delivered.
Current pricing and requirements for obtaining private regions can be found here, with further information on private regions is available via available here.
Things are moving apace with the project to transition all of the Second Life services and simulators to running on Amazon AWS cloud services.
On Friday, October 16th, Oz Linden offered a blog update (which I also blogged about) – when there were 100 regions on the main grid running via AWS.
On Tuesday October 20th, I then blogged about the move to expand the number of regions running on AWS.
On the same day, I was able to give an update with news that the number of regions on AWS had increased to around 300.
On Wednesday, October 21st, April Linden gave a further update on the project’s status – A Light in the Cloud: A Migration Update– noting that around 1,000 regions are now based on AWS. However, her blog post also comes with a request:
The favour I’ve come to ask you for is your patience.
We’re doing our very best to fix things that come up as we go. This means that we might need to restart regions more often than you’re used to, and things may break just a little more often than we’ve all been accustomed to.
In order to get this project done as fast as possible and minimize the time (and resulting bugs) we have to spend with one foot in our datacenter and the other in the cloud, we don’t want to limit ourselves to restarting regions just once a week. We’re ready to get this project done! We’ve seen how much better Second Life runs in the cloud, and we’re ready to have everyone on the grid experience it.
I’m sorry that things might be a little rough over the next few weeks. It’s our goal to finish the cloud migration by the holidays, so that everyone, Resident and Linden alike, can have a nice quiet holiday with our friends and families.
We can’t promise we’ll make it by then, but we’re sure giving it all we’ve got. The mood around the Lab is really positive right now, and we’re all working hard together to make it happen. I’m really proud to be a part of the team that’s transforming Second Life as we know it.
April Linden
There’s obviously a good way to go before everything is settled, 1,000 regions is a small number compared to the the total on the grid, and there are still various back-end systems to be migrated.
But when you consider the sheer volume of work involved in migrating all of the Lab’s systems and properties related to Second Life, the entire project has, from a user perspective, been remarkably smooth thus far – and hopefully things will continue to be so.
At the time of that update, it was indicated that about 100 regions on Agni, the main grid, had been transitioned to running on AWS, comprising a mix of Linden-held, Mainland and private regions.
Commencing on Tuesday, October 20th, the Lab will be expanding on the number by “a few hundred” regions during week #43. The announcement came via a forum post by Bugsly Linden, which reads in full (re-formatted for ease of reading):
This week marks the beginning of medium-scale migration of production regions to the cloud (AWS). We will be transferring a few hundred regions from all existing channels of Second Life based on the Engineering team’s need for additional data and performance metrics. Bulk region migration will take place this week between 6:00 AM and 12:00 PM (Noon) Pacific Daylight Time.
If you are interested in having a region you own (or are the alternate payer for) migrated to the cloud, please submit a Support Ticket. Regions that are migrated to the cloud may experience degraded performance or behave incorrectly; if you are in a cloud based region (you can check by clicking Help->About, and if you see the URL on line 3 end in “amazonaws.com” you’re in the cloud) and observe behaviour you believe is newly incorrect, please file a BUG at jira.secondlife.com.
Due to the rapid speed of our Uplift efforts, we are unable to guarantee that regions in the cloud that are behaving incorrectly will be moved immediately back to our existing hosts.
Spotting a region hosted in the cloud via Help About. Top: a region hosted at the Lab’s co-location facility (note the agni.lindenlab.com in the address). Bottom: and a region running on a simulator in the cloud
There have been concerns about aspects of performance with regions operating in the AWS environment – particularly with regards to communications with the KVP database associated with experiences, which has yet to be migrated), and there are on-going teleport issues that might be related to the Uplift work, although as Oz indicated in his blog post, this has yet to be confirmed.
Elsewhere, some of those with regions already uplifted – such as London City (see London City Uplifted) – have reported generally good performance, if with a one or two caveats such as legacy profiles being slow to load in viewers that support them (which again may or may not be Uplift related). Similarly,those testing region crossings between those based in the AWS environment and those still within the Lab’s co-lo facility have reported them to be generally “smooth”.
However, as Bugsly’s forum post implies, there could be a period of degraded performance within regions that are transitioned to AWS, so keep an eye on where you are in SL.
Byeline
The Fourmilab Cloud Halo detecting an AWS region. Credit: Fourmilab
For those who are curious about whether or not they are in a region that has been uplifted to the cloud or not, and who don’t like opening menus and panels to find out, Fourmilab may have the answer: a freebie full permission halo that attaches to your avatar.
By default, it is invisible, but should you move from a region hosted by the Lab in their co-lo facility to one running on AWS, it will sparkle briefly into life to the faint accompaniment of harps being played (well, you’re in the cloud, after all) before vanishing.
Similarly, when you leave a region running in the cloud and return to one still based in the Lab’s co-lo facility, the halo will again briefly appear, this time a dull grey and without the golden sparkles, while a trombone plays a couple of sad notes.
Note that it will only play when moving between regions hosted in the two facilities, not when moving between regions in the same facility (so you won’t repeatedly get the trombone when moving between regions that are all hosted at the Lab’s co-lo for example).
On Friday, October 16th, Oz Linden posted on the status of the cloud uplift work – see: Uplift Update -, the article coming as regions on the main (Agini) grid are gradually starting to be migrated to AWS services. For those possibly unaware of this project, Oz provides an opening explanation:
We’ve been working hard on the Uplift of Second Life. If you have not been following this project, that’s what we’re calling the migration of our Second Life simulators, services, and websites from a private data centre to hosting in The Cloud (Amazon Web Services). It’s a massive, complicated project that I’ve previously compared to converting a steam-driven railroad to a maglev monorail — without ever stopping the train. This undertaking has at times been smooth sailing, at other times a very bumpy ride. We wanted to share some more of the story with you.
The uplift project was first announced in August 2017, and formed a part of the Lab’s presentation at the AWS Reinvent conference that same year – which if nothing else points to the amount of planning and testing that has been going on both before and during the gradual migration of services, which has been going on for somewhere between 12 and 18 months, recently reaching the point were the aforementioned main grid region migrations could commence.
As Oz notes, the work has been very incremental in nature, and always with a the aim of transitioning services in such a way that users generally have not been aware of which services have moved and when. This has certainly been true for many of the back-end services (no-one noticed when the log-in services and the inventory / asset services moved to AWS, for example). However, as Oz notes in his post, there have been a few bumps on the road.
Some of these problems were initially manifested on Aditi, the Bet grid, which saw batches of regions cloned from the main grid and transitioned to the cloud. Region crossings were one such problem which, thanks to extensive testing by users on Aditi, allowed the Lab to make changes to region crossing that have generally improved things even sans the uplift – although as the Lab readily notes, there is still future work to be done on region crossings once the uplift work has been completed.
Work related to the uplift project allowed the lab to make improvements to region crossings that have benefited Second Life even before regions on Agni (the main grid) commenced a slow migration to AWS
However, some problems unfortunately only manifested once some back-end services had been uplifted and were so bedded-in to running on AWS, reverting to running them out of the Lab’s co-lo is no longer an option. Again, as Oz notes, the recent group service issues being a case in point. Other issues – such as the recent bout of avatar bake (appearance) failures – have been the result not of moving that service to the cloud (the Bake Service has also been AWS based for a while without most users noticing), but in making subsequent changes to a related service – again pointing to the complexities involved in moving multiple systems and services from an established operating environment to an entirely new operating environment.
Elsewhere, there has been a need to revert the Marketplace to running via the lab’s co-lo (albeit it temporarily), whilst some known issues – such as teleport failures – may or may not be linked to migration issues, with the Lab engaged in trying to get to the bottom of things. So if you do see a problem, don’t automatically assume it is uplift related; even without the current migration work, SL can be temperamental!
Currently, around 100 regions on the main grid have been uplifted, and Oz confirms that, barring the unforeseen, the end-of-2020 for uplift completion is still very much the goal.
STÖMOL, the feature-length science-fiction move produced entire in Second Life, has been awarded second place in the 2020 SUPERNOVA Digital Film Festival.
The film, written, directed, produced and starring Huckleberry Hax, together with Caitlin Tobias, Ylva, Boudicca Amat, Anthony Wesburn and Mich Michabo in leading roles, was one of a number of entries to the festival that were wholly filmed in Second Life and which were open to viewing on-line throughout the festival’s almost month-long run.
The film gained numerous plaudits from those at the festival for its style, content, story arc and composition, including founder and Creative Director Ivar Zeile. It was runner-up to #21XOXO, “a stylized window into the zeitgeist of youth and pop culture,” by Sine Ozblige, and earned a prize of US $750 for Huckleberry and his team.
I was stunned when I watched STÖMOL for the first time. I’ve seen some Second Life animations – we’ve really supported artists that are working in that platform over the years – and I didn’t believe I was going to be taken by a feature length film made in Second Life, but Huckleberry absolutely delivered the goods.
Ivar Zeile, Supernova founder and Creative Director
Very special and deserved congratulations to Huckleberry and his team for STÖMOL’s runner-up status at the festival, it was well deserved. I hope that – as nerve racking as entering this festival was to him – this award will spur Huckleberry to continue filming, and we will get to see the sequel and the answers to the question posed by the film’s end credits (nope, not going to spoil it for those who haven’t seen it! – but if you are interested, my own review of the film can be found in Second Life’s STÖMOL: a review).
Other pieces filmed in Second Life and featured in the festival were machinima by Erik Mondrian, and a piece by Tizzy Canncci.
Erik had a total of three pieces accepted by the festival, comprising his CalArts Master of Fine Arts thesis For the Light of Other Shores, a marvellous 10-part series, selected for the Director’s Choice programme of the festival; together with Tripping through Skyscrapers, his video journey through Gem Preiz’s Skycrapers installation ,and Cetatea Poenari, recording a visit to the region of the same name, and both of which featured in the festival’s Everything Abstract-Sonic programme.
For her entry, Tizzy submitted Out of Isolation Came Forth Light, a recording of a performance by artist SaveMe Oh that took place in CapCat Ragu’s installation entitled Isolation, displayed at Ribong Gallery’s Artspace 1789.
I’d also like to offer my congratulations to Tizzy and Erik for entering their work for the festival. I know that Erik had a certain amount of trepidation in making his submissions, but his work – which I have the privilege to be able to discuss with him and write about (see Erik Mondrian: master of fine arts in and beyond Second Life) as well as follow via You Tube – full deserves recognition through a wider audience.
Inara Pey: Living in a Modemworld 