Tag: Lab Gab

VWBPE 2021: Patch Linden – the board, Second Life, and more

Posted on by Inara Pey
VWBPE 2021

On Thursday, March 18th, 2021 Patch Linden, the Lab’s Vice President of Product Operations and a member of the company’s management team, attended the 2021 Virtual Worlds Best Practice in Education (VWBPE) conference in the first of three special events featuring representative from Linden Lab.

The following is a summary of the session covering the core topics raised. The notes provided have been taken directly from the official video of the session, which is embedded at the end of this article. Time stamps to the video are also provided to the relevant points in the video for those who wish to listen to specific comments.

Notes:

  • This is a summary, not a full transcript, and items have been grouped by topic, so may not be presented chronologically when compared to the video.
  • In places, information that is supplementary to Patch’s comments is provided in square braces (i.e. [ and ]) are used in the body text below to indicate where this is the case.

Linden Lab’s New Board

[Video: 4:04-10:55]

[For additional information on the new board members, please also see: Meet Linden Lab’s new board of directors (January 9th, 2021) and Linden Lab’s board of directors: snippets of news (February 4th, 2021).]

Linden Lab’s board of directors (l to r): Brad Oberwager, J. Randall Waterfield and Raj Date
  • New ownership team is a “joy to work with”.
  • Brad Oberwager is particularly active, and has the avatar name Oberwolf Linden  and is described as “a lot of fun” to be around and to work with. [He is both on the board and serves as Executive Chairman on the management team.]
Brad Oberwager has joined the Lab’s management team as Executive Chairman, and his long-time colleague, Cammy Bergren serving as Chief of Staff
  • As Executive Chairman Brad Oberwager’s aim is to see Second life set as the “largest and best” virtual world,  and has a genuine love for the platform.
  • Both J. Randal Waterfield and Raj Date (particularly) appear to lean more towards the Tilia Pay side of things, with Brad Oberwager more “in the middle”. However, this doesn’t mean there is a dichotomy. Tilia is a key component of Second Life (it runs the entire Linden Dollar eosystem), and Tilia’s own success and growth will benefit SL.
    • [Tilia is a wholly-owned subsidiary of Linden Research (Linden Lab). It’s board comprises two members of the Linden Research Board: Brad Oberwager and Raj Date), together with Aston Waldman, the Chief Financial Officer (CFO) at Linden Lab. The management team comprises: members of the the Lab’s management team: Aston Waldman, David Kim, Ray Johnson, Emily Stonehouse and Brett Attwood.]
    • The two entities enjoy a symbiotic relationship: Tilia is owned by Linden Research with Linden Research also a primary customer. However, day-to-day operations are carried out by two separate  teams.
  • [48:48-49:59] The new owners are bringing a tremendous new energy to Linden Lab, and are “super enthusiastic” about growing Second Life, including its educational use. What gets to be invested in the platform will only benefit everyone.
    • The key question Brad Oberwager asks and prompts people to ask is, “How will it benefit the residents, and how will it benefit Second Life?”

SL Short-Term and Longer Term

Priorities for the Second Life Team in the Next 12 Months

[Video: 10:57-16:02]

  • Immediate priority is to increase the Second Life active user base. This is very much being driven as a goal by Brad Oberwager, and includes:
    • “Drilling down into” the new user experience.
    • Refactoring the on-boarding process and orientation islands.
    • The work will include viewer-side changes that are intended to “smooth out a lot of the bumps in the road”.
    • The will will be built on two years of active study and A/B testing to try to determine what the on-boarding path should look like, together with learning from users returning to Second life as a a result of the SARS-CoV-2 impact.
    • It is hoped this work will both help LL improve user retention and also feed through to the community gateways, particularly with regards to the upcoming changes which will be made to the viewer.
    • No specifics provided, but the viewer changes are described as:
      • “New UIs”
      • Refreshed looks.
      • Easier to find information.
  • There is also the need to complete the work of transitioning to AWS – fixing the current issues directly related to the move and also on-going work to properly leverage the AWS environment for the benefit of the platform.
    • [29:24-30:00] This work includes a  lot of under-the-hood simulator performance improvements that will be continuing throughout the year.

Second Life in Five Years Time

[Video: 16:04-19:33]

  • The company would like to at least double the active user population over the next 3-5 years.
  • This is seen as a realistic goal in light of the shift in emphasis seen within business, education, etc., from purely physical world interactions towards more digitally-based interactions / hybrid opportunities that mix various formats [e.g. digital + virtual + remote working / learning].
  • AWS offers the potential for regions to be geographically located around the world, potentially bringing them closer to their core audience.
    • This could allow educational regions, for example, to be hosted much closer to the schools / colleges / students they serve, making them more responsive.
    • This approach could potentially start to be used towards the end of 2021.
  • Further out, geolocating regions could potentially offer the ability for the Lab to offer white label grids to specific customers / groups.
  • [24:00-28:25] White label grids present the opportunity for the Lab to better meet specific client requests to remove features and capabilities from the viewer – and also take features an capabilities required for a specific environment and potentially make them available across the entire Second Life product.
    • Two examples of the latter already exists: the new extended chat range feature available to region owners, and the Chrome Embedded Framework updates that allow video to be streamed into Second Life, as originally demonstrated in the Adult Swim streaming of episodes from The Shivering Truth in May 2020.

What Lessons has LL Learned due to the Pandemic?

[Video: 20:31-24:00]

  • The pandemic, particularly as a result of attempts to leverage the platform for education, business and similar use by organisations and groups, reinforced the fact that the new user experience needs to the overhauled.
  • It has also underlined the fact that people’s usage habits have changed.
  • The Land Team in particular has learned a lot about business, etc., needs of clients – the team deals directly with such requests as they come in to the Lab, and so have been dealing first-hand with understanding client requirements, determining the best for of assistance (e.g. providing one of the Lab’s turn-key solutions or brokering contact between the client and a solution provider who can meet their requirements.

Pricing and Options

[Video: 30:27-34:17]

  • Nothing on the roadmap related to pricing; land costs should remain untouched through the rest of the year.
  • There is the potential for AWS to allow the Lab to develop new region products; this is something that may start to be looked at 12-24 months hence.
  • AWS might also allow for on-demand spin-up of regions, initially building on the idea of Homestead holders being able to take a temporary upgrade to a Full region to run a specific event, then downsizing back to a Homestead.

Competition and Experimentation

[Video: 34:39-45:58]

  • Competition helps drive innovation.
  • There is no Lab-based group specifically tasked with investigation competitive platforms, but staff tend to try them out through their own interest.
  • Attention is paid to how other platforms adopt newer technologies and the challenges encountered in such adoptions.
  • There is still no real, direct competitor to Second Life in terms of size, flexibility of use, or in having an in-built content creation tool set.
  • LL don’t regard users as beta testers per se. However, major new features do require trialling / testing, which can involve selected users / tried at scale to determine feasibility / performance, etc. Sometimes the result is a capability has to be withdrawn as it is not performant enough (e.g. the VR headset viewer) and / or negatively impacts the user experience.

Oz Linden’s Departure

[Video: 46:09-46:56]

  • Oz was a fabulous colleague to work with. His retirement leaves a “gaping chasm” at the Lab.

Patch Linden talks Linden Homes and more

Posted on by Inara Pey
Patch linden appeared at a Home and Garden presentation session,on Wednesday, March 10th, 2021

On Wednesday, March 10th, 2021 Patch Linden attended the home and Garden Expo to talk about his role, Linden homes and other aspects of SL  and Linden Lab, and to answer questions from the audience.

The following is a summary of the session covering the core topics raised. The notes provided have been taken directly from the official video of the session, which is embedded at the end of this article. Time stamps to the video are also provided for ease of reference.

Notes:

  • This is a summary, not a full transcript, and items have been grouped by topic, so may not be presented chronologically when compared to the video.
  • The last 20+ minutes of the session is a general Q&A session where Patch was addressing questions and comments put into local chat, which is not visible in the video. Some of these are highly specific questions based on an individual’s experience, other more generic – please refer to the video directly for this part of the session (commencing at 56:50).
  • In places, information that is supplementary to Patch’s comments is provided in square braces (.i.e. [ and ]) are used in the body text below to indicate where this is the case.

About Patch

  • Originally a Second Life resident and business owner who joined the platform in 2004, and became a Linden in 2007.
  • Initially worked as a support agent and then as a support liaison. From there he moved to the Concierge team, eventually becoming that team’s manager. From there he took on the role of Operations Support Manager for a year, then moved to the Product group, the team responsible for defining the features, etc., found within Second Life.
  • In 2019 he was promoted to Vice President, Product Operations, and joined the Lab’s management team (see: Linden Lab’s management team expands: congrats to Grumpity, Patch and Oz).
  • In this role, he has two major departments reporting into him: those of Support and Product Operations, the latter of which comprises the Lab’s internal content creation team (which includes the Moles of the Linden Department of Public Works) and the Land Operations team, which he originally established whilst working within the Product group. Together, these make up the largest teams at Linden Lab.
  • Together with Grumpity Linden, who is Vice President of Product and Acting Vice President of Engineering since Oz Linden retired, he oversees Second Life’s continued development.
  • He is based in Atlanta, Georgia, and was responsible for establishing the Lab’s support office there.

Linden Homes And Bellisseria

Linden Homes

[Video: 6:38-23:06]

  • The “new” Linden Homes, as launched in 2019, came about in part because of the Premium “free” tier being upgraded from 512 sq m or 1024 sq m. They were also driven by the “ageing” design of the original Homes themselves (i.e. using older capabilities, being non-mesh, etc), together with their relatively high density tending to make them less appealing over time.
  • Developing the new Homes has been both a learning process and an exercise in incorporating additional capabilities within the different themes.
  • The release of the Silt Homes (see: Linden Stilt Homes released in Second Life), the first release to be made following the move to AWS services, did bring with it a number of issues (e.g. the infamous “blue window” issue), but most of these have now been resolved or have fixes in the works.
  • The release of the “Alpine” – or fachwerkhaus, as Patch would prefer them to be called – theme is now “just around the corner”, and may be occurring in the “next couple of weeks”.
    • Preparations for the release have seen an large extensive in Linden Home regions to the north-east of Bellisseria beyond the Silt Home / Houseboat regions, such that the estate now almost reaches Satori.
    • While the regions containing the fachwerkhaus theme will have hilly / mountainous areas, they will not be “snowy”. The Lab has learned through various channels that predominantly snow-covered regions tend to be polarising: people either love them or hate them, so prefer keep it seasonal.
    • However, were there a demand to make such regions “snowy” in theme, it may be something the Lab could look at doing in the future.
The fachwerkhaus theme of Linden Homes could be released within the next 2-3 weeks
  • Right now the overall drive with Linden Homes is to get all the planned themes – fachwerkhaus and beyond – released by the end of 2021.
  • Once this has been achieved, it is likely that the retirement of the “old” Linden Homes will commence.
  • There are currently no plans to directly replicated the themes found in the “old” Linden Homes beyond what has been seen (e.g. the Log Homes offering a similar environment to the Tahoe theme).
    • This is something that might be contemplated some time after all the currently planned themes have been released, but no guarantees.
    • In terms of the “old” styles, only one – the Meadowbrook (the “suburban” style 1- and 2-storey homes) – proved to be particularly popular, beating the other three styles by “leaps and bounds”.
  • There have been requests to allow groups of people to select Linden Homes that are located close to one another, so that they might establish little communities of friends, etc. This is actually difficult to achieve, but might be something that could be looked at some time after the roll-out of Home themes has been completed.
  • [33:20-34:55] The next NEW theme for Linden Homes (to follow the fachwerkhaus theme) will be previewed at SL18B in June.
    • It is promised that it will “Blow your socks off” and be the “most impressive theme released to date”.
    • It will have “unique” capabilities not previously seen in Linden Homes
    • It is unlikely the theme will actually be released during the SLB event, but will likely be available some time afterwards.

Bellisseria and Community

[Video: 23:15-32:49]

  • It was anticipated that some form of community would develop around Bellisseria,  if only going by lessons learned from Bay City. However, the speed with which it developed and grew has been surprising to Patch.
  • Has been watching the growth of the many sub-communities with the continent, which now cover all interests and social aspects – boating, flying, merfolk, LGBTQ+ – even Adult.
  • While it was not with a specific aim of building “community”, having public spaces to visit and explore within Bellisseria and venues that could be used was key part of planning the estate.
  • These remain a focal point of effort through seasonal events and activities – such as those in Millbank such as the Halloween build there.
  • There is a dynamic between the community and the Lab – the latter listen to the former, may adopt ideas from the former, and the former may take ideas and facilities from the Lab and run with them (as with the use of the Bellisseria Fairgrounds), etc.
    • An example of this is the adoption by the Lab of the house number system introduced by the community.
    • Also, the Bellisseria Bureau of Bureaucracy “passport” system will be adopted, with the Lab providing their own kiosks within the various locations within Bellisseria they maintain, allowing visitors to have their passports uniquely stamped.
  • There are no plans to name roads in Bellisseria – it is hard enough to come up with fresh region names [although I admit to having an amused groan over the likes of Salmon and Gillfunkel, OccupenSea (together the neighbouring xxxSea regions), Lone Shark, Miniature Gulf and so on!].

AWS Migration

[Video: 37:07-48:22]

[Note: details on several of the issues relating to the AWS migration and the on-going follow-up work on it can be found in my weekly Simulator User Group meeting summaries and on my notes from the February Lab Gab AWS update.]

  • The move to using AWS services that was completed at the end of 2020 was just that: relocating services to AWS without making significant changes to them, unless absolutely essentially to their smooth running [what Oz Linden and his teams referred to as “lift and shift”].
  • Unfortunately, this met that certain services (e.g. the Map tile generation and the Land Store) did break.
    • [The major cause of these breakages was down to the code having certain assumptions about the operating environment “baked in”, which are no longer true within the AWS environment.]
    • The Land store issue is now fixed, and the Map tile issues are well on their way to being fixed [there are still issues around the “stitching” and rendering process when zooming out from the Map].
  • With the migration work completed, the emphasis is not on performance tuning and on bug fixing.
  • Broader issues – such as Search  (notably People search) are also problematic, and these issues are also on a priority list for fixing.
  • As always, if people come across a specific technical issue (particularly if it can be reproduced using the official viewer, please raise a bug report.

SL, the Lab and the Pandemic

[Video: 46:28-56:50]

  • SL has seen numerous examples of increased activity during the SARS-CoV-2 pandemic. In particular, merchants across markets are seeing increased sales; charity events are seeing increased fund-raising, etc.
  • The Lab remains sensitive to the issue, particularly around issues of people who may have lost loved ones or are struggling with increased financial hardship as a result of the pandemic.
  • Obviously, the Lab is pleased that Second Life has provided a positive outlet for people, and they have noted upticks in “old” users returning to the platform an in “new” new user accounts being created.
    • The latter is particularly being noted through the use of the platform by the likes of the education, business and non-profit sectors, which has in turns spurred renewed media interest in the platform.
  • Patch [and others at the Lab?] appreciate the greater freedom using Second Life gives them over other other business tools they use – Zoom, Google Meet,  etc. – simply because it offers the chance to have a change of surroundings and relax into their avatars.

Lab Gab Feb 26 summary: AWS update and a farewell to Oz

Posted on by Inara Pey
via Linden Lab

On Friday February 26th, Lab Gab, the live streamed chat show hosted by Strawberry Linden on all things Second Life, returned with a show of two halves.

Featuring guests Grumpity Linden, the Lab’s Vice President of Product and Oz Linden, the Lab’s Vice President of Engineering, the first part of the show took a look at the latest of the work to migrate Second Life and all its services to running on Amazon Web Services (AWS) hardware and infrastructure and attempted to address questions forwarded to the Lab by Second life users.

 Table of Contents

The show was also an opportunity to say “farewell” to one of the leading lights at the Lab – Oz himself, who is retiring from the company and from full-time work as a whole – after more than 11 years with the company.

The official video of the segment is available via You Tube, and is embedded at the end of this article. The following is a summary of the key topics discussed and responses to questions asked.

Please be aware that as some topics were touched on more than once during the conversation, the notes blow collect references together, rather than presenting them in chronological order. However, where relevant, time-stamps are provided.

Strawberry linden (l), Oz Linden and Grumpity linden (wearing an Oz ‘tache and goatee in his honour)

On the Status of the AWS Migration and the Future

Current Status

[1:43-2:50]

  • All of the services related to Second Life were transitioned to running on AWS hardware and infrastructure by the end of December 2020.
  • The last aspect of the core work was the removal of all of the Lab’s own hardware and equipment from the Arizona co-location facility that had been hosting Second Life, which included the shredding of 10,588 hard and solid state drives to ensure data security.
  • The majority of the work went a lot more smoothly than had been thought might be the case, however, there are some services that have given rise to some problems that are still being resolved.
  • Chief among the latter is the Land Store, which was once again turned on ready for use on Thursday, February 24th.
  • Map title generation has also been a issue sine the migration, but work is progressing on fixing this.
    • [9:09-11:34] A core issue with the Map tile generation failure lay in the fact that the code had not been touched in a “long, long time” – so long, in fact, that the code isn’t geared to rendering mesh objects, hence why they can look so abstracted on a map tile.
    • In terms of the current problems, the code made a lot of assumptions about the architectural environment in what it was running, assumptions that are no longer true with the move to AWS.
    • The current work is focused purely on getting the service to generate Map tiles one more, without making any additional changes to the code to account for things like rendering mesh objects correctly  or addressing other bugs.
    • Most of this work is now thought to be complete and Map tiles are now being generated as they should. however, there is some work to be completed on stitching tile images together when a user zooms out on the Map.
    • There is a project to improve the overall appearance of Map tiles, but this was put aside in the focus of migrating to AWS, but will hopefully be picked up again at some point in the future.

What is Next?

[2:54-5:45]

  • While the physical migration of Second Life services from a proprietary environment to AWS is complete, the Uplift Project work is not, and so will continue to be a focus of engineering efforts.
  • In  particular, the immediate focus is on optimisation work, which encompasses:
    • Optimising the performance of the various series on the new hardware / infrastructure.
    • Optimising (for the Lab) the cost involved in running within an AWS environment.
    • Fine tuning systems and operations within the new environment.
    • Working to leverage the new hardware options and infrastructure presented by AWS to favour Second Life as a product running in that environment.
  • In this it should be remembered at the initial migration work of getting Second Life transitioned to AWS was devoted purely to taking all of the SL services – front-end simulators, back-end services, middleware, web properties and services, supporting tools, etc., – from the proprietary environment in which they had always run and just getting them running on AWS in what was called a “lift and shift” operation, whilst making as few changes to any of the services as possible.
  • With the “lift and shift” aspect of the work completed, the engineering team has turned its attention to gathering data on exactly how the various services are running in the new environment and understanding where opportunities for making the improvements noted above may lie, and how they might best achieve them.
  • In this, the Lab now has much improved service monitoring tools at their disposal, and these are now allowing the initial work on tuning performance on key services to be made.
  • Two practical benefits of the move are:
    • Regions running on AWS can run more scripts / script events in the same amount of time than can be achieved on non-AWS regions.
    • The way in which simulators are now managed mean that LL can more directly obtain logs for a specific region, filter logs by criteria to find information, etc., and the entire process is far less manually intensive.

Will the Migration Mean lower Prices for Users?

[5:46-9:02]

  • Sort answer for the foreseeable future: no.
  • There has been an idea circulating among users that running SL on AWS is “dramatically cheaper” for Linden Lab; but this is actually not accurate.
  • Prior to the migration, all of SL and its services had been running on LL’s own hardware  for which there had been no capital expenditure for years, and which had completely depreciated.
  • The move to AWS represents something of a new capex spend, increasing the Lab’s costs [although it is not unfair to say that the capex involved is liable to be significantly less over time than repeatedly buying-in new server clusters to allow SL to run on more modern systems].
  • Rather than presenting LL with reduced costs, the move to AWS is designed to:
    • Present the company with far broader options for delivering a more performant and capable service to users – although as noted above, it will take time for all of this to be delivered.
    • Improve the overall longevity of the Second Life service through the noted performance improvements and access to better hardware and infrastructure services.

Second Life Mobile App  Status

[19:20-20:39]

  • Mobile has taken longer than expected to bring forth, for two primary reasons:
  • The first is that while the initial release will be more of a communications tool, considerable foundational work has been put into ensuring the app can be encompass a lot more functionality than that in the future.
  • The second has been that as a result of testing by Apple, the Lab has been forced to make changes to the way in which chat works.
    • These changes will, in time, be filtering through into the viewer as well.
    • They should actually make chat more reliable in the future.
  • No commitment as to when the app may be more widely available.

Other Technical  Questions

  • [11:38-17:47] There have been numerous niggling issues of late: further issues with search (e.g. avatars failing to show in search), profile issues, etc). When are these likely to be addressed? Should users report bugs then find?
    • Whilst the majority of the migration process did go smoothly, there have been glitches, and the Lab is working to address them alongside of working on the performance, etc., work mentioned above.
    • There are a lot of aspects of SL built on old technology, so there is an expectation that, over time, and as things can be looked at, not only will niggles go away, but software and capabilities as a  whole can be made a lot more stable and resilient.
    • Bugs should most definitely be reported using the SL Jira. Information on how to file bug reports (and feature requests) can be found here:
  • [17:55-19:18] Will capabilities that were being worked on some time ago (e.g. 360 snapshot viewer) ever be completed?
    • The migration work has demonstrated what can be achieved with a tightly defined set of goals and teams focused solely on those goals.
    • This is an approach Grumpity would like to carry forward, with a commitment to review current and past projects to determine what might be required to bring them to completion (input, time, resources, etc), and then make decisions from there.

About the Lab’s New Owners

[20:50-24:06]

Looking Back at Oz’s Time at the Lab

[24:20-46:20]

The latter half of the programme looks back over Oz’s time at the Lab and provides him with the opportunity to discuss what attracted him to Linden Lab, the nature of his work, why he regards his time with LL as potentially the best job he’s ever had, and to discuss his post-retirement plans and answers various questions.

Rather than offer a summary of this part of the show, I encourage people to listen to it in full, as it really is informative and enlightening, particularly if you’re not familiar with Oz’s work, his teams, or the Lab as a whole.

Lindens say “farewell” to Oz

[46:23-end]

The end of the show sees Strawberry teleport Oz to s special in-world retirement party, where the teams reporting into him and other LL staff have gathered to wish him well. This again a touching and moving tribute that says so much about Oz and the high regard in which he is rightfully held, and should be seen without input from the likes of myself.

For my part – and because Oz has been both a direct and indirect influence in my SL time – I’d like to just repeat what I wrote a few days ago on reading of his upcoming retirement:

For my part, I cannot claim to know Oz as well as I would like to – but I’ve always found find his enthusiasm for Second Life never to be anything less than totally honest and infectious, and his high regard for users utterly genuine and sincere.
As such … I’d like to take this opportunity to offer him a personal and public “thank you” for all the times he’s provided me with insight and / or encouraged me to get involved in various projects, all of it has been greatly appreciated. I am, and will be, genuinely saddened to see him leave the Lab; we are all losing something in his departure, and the void left will not be easy for the management team to fill.

Don’t forget: Friday Feb 26th – Lab Gab AWS update and a farewell

Posted on by Inara Pey
via Linden Lab

Just a quick reminder to folks who may not have caught the official announcement at the start of the week.

The latest edition of the Lab’s chat show series hosted by Strawberry Linden, Lab Gab, streams at 11:00 am on Friday, November 26th. And it’s a special show.

As most are aware, the work to transition Second Life to operating via Amazon Web Services (AWS) was completed at the end of December 2020, and the Lab has completely moved out of its former co-location facility in Arizona.

Since then work has been continuing to make tweaks and updates to both help get some services that didn’t make the transition as smoothly as hoped (perhaps most notably to most people, Map tile updates) once again running as they should, with work also progressing on fine-tuning things, with the Lab looking to better optimise their services to take full advantage of the the hardware and infrastructure provided AWS.

Given all this, the show will feature Grumpity and Oz Linden, respectively the Lab’s Vice President of Product and Vice President of Engineering, will be providing a update on how things are going.

Oz and Grumpity Linden, with Strawberry Linden between them, will be appearing on Lab Gab on Friday, February 26th, at 11:00am SLT. Image courtesy of linden Lab

In addition, Friday, February 26th marks the end of an era. As he recently announced, Oz  Linden is retiring from the Lab as from today, as so the show marks once of his last public appearances as a member of the Lab’s management team – and indeed as a Linden.

In his time at the Lab – which amounts to something over 10 years -, he has achieved and overseen a lot, and has been responsible both directly and indirectly for making Second Life a much more capable platform, and in building a solid and fruitful relationship with both third-party viewer developers and the open-source community in Second Life; he also makes no secret of the fact that he has enjoyed his time at the Lab immensely.

To mark the fact that this is potentially the last time the users will get to hear from Oz, the show will also look back over his time at  the Lab – so be sure not to miss it and hear from him on a personal level.

You can catch it through the Lab’s streaming outlets on YouTube, Facebook, or Periscope, and I’ll more than likely have a summary of the show out within 24 hours of it airing.

Lab Gab November 6th: Cloud Uplift update

Posted on by Inara Pey
via Linden Lab

On Friday November 6th, 2020 Lab Gab, the live streamed chat show hosted by Strawberry Linden on all things Second Life returned to the the subject of the work to transition all Second Life services to Amazon Web Services (AWS) and away from running on the Labs’ proprietary hardware and infrastructure.

The session came some 7 months after the last Lab Gab to focus on this work in April 2020 with Oz Linden and April Linden (see Lab Gab 20 summary: Second Life cloud uplift & more), and this time, Oz Linden sat in the hot seat alongside Mazidox Linden.

The official video of the segment is available via You Tube, and is embedded at the end of this article. The following is a summary of the key topics discussed and responses to questions asked.

Mazidox Linden is a relative newcomer to the Linden Lab team, having joined the company in 2017 – although like many Lab staff, he’s been a Second Life resident for considerably longer, having first signed-up in 2005.

 Table of Contents

He is the lead QA engineer for everything simulator related, which means his work not only encompasses the simulator and simhost code itself, but also touches on almost all of the back-end services the simulator software communicates with. For the last year he has been specifically focused on QA work related to transitioning the simulator code to AWS services. He  took his name from the Mazidox pesticide and combined it with the idea of a bug spray to create is avatar, to visualise the idea of QA work being about finding and removing bugs.

Oz Linden joined the company in 2010 specifically to take on the role of managing the open-source aspects of the Second Life viewer and managing the relationship with third-party viewers, a role that fully engaged him during the first two years of his time at the Lab. His role then started expanding to encompass more and more of the engineering side of Second Life, leading to his currently senior position within the company.

“The Bugspray” Mazidox Linden (l) and Oz Linden joined Strawberry Linden for the Friday, November 6th segment of Lab Gab to discuss the cloud migration work

What is the “Cloud Uplift”?

[3:25-5:55]

  • Cloud Uplift is the term Linden Lab use for transitioning all of Second Life’s server-based operations and services from their own proprietary systems and services housed within a single co-location data centre to  commercial cloud services.
  • The work involves not only the visible aspects of SL – the simulators and web pages, etc., but also all the many back-end services operated as a part of the overall Second Life product,  not all of which may be known to users.
  • The process of moving individual services to the cloud is called “lift and shift” – take each element of software, making the required adjustments so it can run within a cloud computing environment, then relocate it to AWS infrastructure and hardware in a manner that allows it to keep running exactly as it did prior to the transfer, while avoiding disruptions that may impact users.
  • The current plan is to have all of the transitional work completed before the end of 2020.
  • However, this doe not mean all the the work related to operating SL in the cloud will have been completed: there will be further work on things like optimising how the various services run on AWS, etc.,

Why is it Important?

[5:56-12:12]

  • It allows Second Life to run on hardware that is a lot more recent than the servers the Lab operates, and allows the Lab to evolve SL to run on newer and newer hardware as it becomes available a lot faster than is currently the case.
    • In particular, up until now, the route to upgrading hardware has involved the Lab reviewing, testing and selecting hardware options, then making a large capital expenditure to procure  the hardware, implement it, test it, then port their services over to the hardware and test, then implement – all of which could take up to 18 months to achieve.
    • By leveraging AWS services, all of the initial heavy lifting of reviewing, testing, selecting and implementing new server types is managed entirely by Amazon, leaving the Lab with just the software testing / implementation work.
  • A further benefit is that when SL was built, the capabilities to manage large-scale distributed systems at scale didn’t exist, so LL had to create their own. Today, such tools and services are a core part of product offerings alike AWS, allowing the Lab to leverage them and move away from having to run (and manage / update) dedicated software.
  • Two practical benefits of the move are:
    • Regions running on AWS can run more scripts / script events in the same amount of time than can be achieved on non-AWS regions.
    • The way in which simulators are now managed mean that LL can more directly obtain logs for a specific region, filter logs by criteria to find information, etc., and the entire process is far less manually intensive.

How Secure is SL User Data on AWS?

[12:20-15:43]

  • It has always been LL’s policy when dealing with third-party vendors (which is what AWS is) not to expose SL user data to those vendors, beyond what is absolutely necessary for the Lab to make use of the vendor’s service(s).
  • This means that while SL user data is stored on AWS machines,it it not stored in a manner Amazon could read, and is further safeguarded by strict contractual requirements that deny a company like Amazon the right to use any of the information, even if they were to be able to read it.
  • In fact, in most cases, user-sensitive data is effectively “hidden” from Amazon.
  • LL  is, and always has been, very sensitive to the need to protect user data,even from internal prying.
  • In terms of the simulators, a core part of testing by Mazidox’s team is to ensure that where user data is being handled (e.g. account / payment information, etc.), it cannot even be reached internally by the lab, as certainly not through things like scripted enquiries, malicious intent or prying on the part of third-party vendors.
  • [54:30-55:18] Taken as a whole, SL on AWS will be more secure, as Amazon provide additional protection against hacking, and these have been combined with significant changes LL have made to their services in the interest of security.

Why is Uplift Taking So Long?

[15:48-19:20]

  • The biggest challenge has been continuing to offer SL as a 24/7 service to users without taking it down, or at least with minimal impact on users.
    • This generally requires a lot of internal testing beforehand to reach a point of confidence to transition a service, then make the transition and then step back and wait to see if anything goes dramatically wrong, or users perceive a degraded service, etc.
    • An example of this is extensive study, testing, etc., allowed LL to switch over inventory management from their own systems to being provisioned via AWS relatively early on in the process, and with no announcement it had been done – and users never noticed the difference.
  • Another major challenge has been to investigate the AWS service offerings and determine how they might best be leveraged by SL services.
  • As many of the SL services are overlapping one another (e.g. simulators utilise the inventory service, the group services, the IM services, etc.), a further element has been determining a methodical manner in which services can be transitioned without impacts users or interrupting dependencies on them that may exist elsewhere.
  • The technology underpinning Second Life is a lot more advanced and recent within the AWS environment, and this means LL have a had to change how they go about certain aspects of managing SL. This has in turn required experimentation, perhaps the deployment of new tools and / or the update / replacement of code, etc..

Will Running on AWS Lower Operating Costs?

[19:33-23:00]

  • During the transitional period it has been “significantly” more expensive to operate SL, inasmuch as LL is paying to continue to operate its proprietary systems and services within their co-lo facility and pay for running services via AWS.
  • Even after the need to continue paying for operating the co-lo facility has ended, it is unlikely that the shift to AWS will start to immediately reduce costs.
  • However, the belief is that moving to AWS will,  in the longer term, reduce operating costs.
  • Whether reduced operating costs lead to reduced costs to users, or whether the savings will be re-invested in making further improvements to the service lay outside of this discussion.
  • Right now the focus is not on driving down costs or making service significantly better, but is solely the work of getting everything transitioned. Lowering costs, making more efficient use of the underpinning capabilities provided by AWS will come after the migration work has been completed.

What Happens to the Old Hardware / Facility, Post-Uplift?

[23:09-25:15]

  • Several years ago, LL consolidated all of their hardware and infrastructure into a single co-location data centre in Arizona.
  • Most of the hardware in that facility is now so old it has depreciated in value to a point where it is pretty much worthless.
  • A specialist company has therefore been contracted to clear-out the Lab’s cage(s) at the co-lo facility and dispose of the hardware.
    • As a demonstration of LL’s drive to protect user data, all drives on the servers will be removed under inspection and physically destroyed via grinding them up on-site.

Continue reading “Lab Gab November 6th: Cloud Uplift update” →

Don’t forget: Lab Gab, November 6th: cloud update

Posted on by Inara Pey
via Linden Lab

Lab Gab returns on Friday, November 6th, 2020, with a cloud migration update.

As most are aware, the work to transition Second Life to operating via Amazon Web Services (AWS) has now progressed to a point where regions on the main grid (called Agni) are starting to be transitioned. In fact, by the time the Lab Gab show live streams, approximately one-third of all Agni regions will be operating via AWS services.

At the same time, as as per my November 2020 Web User Group summary, the Web teams are hopeful that all web properties will be running via AWS by early December, placing the Lab on course to achieve its target of completing the migration (referred to as Project Uplift) by the end of 2020 (although there will likely be more work related to it to follow in early 2021).

This being the case, the Lab Gab segment will feature Oz Linden, the Lab’s Vice President of Engineering (and the man pretty much in overall charge of the engineering / technical aspect of the work) and Mazidox Linden, the Lab’s senior QA Engineer who has been particularly involved in the migration work, testing the simulator code in reference to the migration work, and who describes the project as “the largest change to the simulator [software] ever.”

“The Bugspray” Mazidox Linden (l) and Oz Linden will be joining Strawberry Linden on the Friday, November 6th segment of Lab Gab to discuss the cloud migration work

As usual, the programme will be streamed via YouTube, Facebook, or Periscope, at 10:00 SLT, and if all goes according to plan, I’ll have a summary of the video (and the video itself) available soon after the the broadcast, for those unable to watch live.

For those who may have questions on the migration work, there is still time to submit them via the Lab Gab Google form, in addition, and if there is time, questions may also be taken from the chat feeds associated with the live stream channels.