SL17B Meet Oz Linden – a summary with video and audio

via Linden Lab
On Wednesday, June 24th, 2020 at the SL17B celebrations, the third of five Meet the Lindens sessions was held, featuring Oz Linden, the Lab’s Vice President of Engineering.

The following is a summary of the session covering the core topics raised, with selected audio extracts. The notes provided have been taken directly from the official video of the session, which is embedded at the end of this article. Time stamps to the video are also provided for ease of reference.

Note that this is a summary, not a full transcript, and items have been grouped by topic, so may not be presented chronologically when compared to the video.

Table of Contents

Audio extracts, where included, have been cleaned-up and balanced to remove pauses, repetitions, etc.

In places, information that is supplementary to Oz’s comments is provided in square braces (.i.e. [ and ]) are used in the body text below to indicate where this is the case.

About Oz

  • Oz Linden

    Joined the company in 2010 specifically to take on the role of managing the open-source aspects of the Second Life viewer and managing the relationship with third-party viewers.

  • He came to Linden Lab out of a desire to do something “fun” after working in the telecommunication arena, notably with voice over IP systems (VOIP), which he defines as being “really interesting technology with some really fascinating challenge”, but in terms of it being fun, it really didn’t do what I wanted it to do.”
  • For the first two years of his time at the Lab, he was primarily focused on the open-source viewer work and in refining the overall viewer maintenance process, before his role started expanding to encompass more and more of the engineering side of Second Life.
  • When work on Sansar started in earnest, he pro-actively campaigned within the Lab for the role of  Technical Director for Second Life, working to build a team of technical staff around him who all shared a passion for Second Life.
  • In 2019 he was promoted to Vice President, Second Life Engineering (Vice President of Engineering following the sale of Sansar in early 2020), and joined the Lab’s management team alongside Grumpity and Patch Linden (see: Linden Lab’s management team expands: congrats to Grumpity, Patch and Oz).
  • Together with Grumpity and Patch, he forms what Grumpity calls the “troika” overseeing Second Life’s continued development.
  • Classifies his attraction to working with Second Life as perhaps falling into three core areas:
    • The open-source nature of the viewer and being directly involved with how SL users are using the viewer and what they do with it – which can often times take the Lab entirely by surprise.
    • The challenge of trying to implement new technologies alongside of (rather than simply replacing) older technologies.
    • Working with the operations team and others to ensure SL constantly evolves without (as far as is possible) breaking anything – a process he refers to and rebuilding the railway from a moving train.
  • Note that his avatar appears bald in the Meet the Lindens publicity shot at the top of this article, as he and his team participated in the 2020 Bid a Linden Bald event to raise money for RFL of SL, and has the team raising the least, that had to spend a month in-world sans hair.

Back to top

His Team

Has it Expanded Since the Sale of Sansar?

[Video: 4:47-5:48]

  • He did persuade a number of people to move back from Sansar to Second Life [those known to have moved back at the time were Runitai Linden (graphics) and Maestro Linden and Monty Linden (engineering), although they obviously may not be the only people to move / move back to work on Second Life].
  • Hiring of new staff has also continued [notable within this are Ptolemy Linden and Euclid Linden (graphics) and at least one Android development specialist].
  • At the time of the event, also looking to hire a further systems engineer working on the back-end Linux systems.

What Impact has the Pandemic Had?

[Video: 6:02-8:07]

  • “Pretty minimal”
  • The Engineering and Operations teams and his developers were already “pretty distributed”, with some of the teams working out of three of the Lab’s offices – Seattle, San Francisco and Boston – but around one-third to half of the total staff reporting to him (Oz included) have generally worked from home as “Moon Labbers” [the “Moon Lab” being LL’s term for remote working].
  • So teams already very familiar with remote working, operating across time zones and holding meetings in SL, as well as tools like video chat, and the transition for the rest has been “pretty much” seamless.
  • Probably the biggest impact is that the team isn’t getting together for their summer meet-up where they socialise and lay plans for future work on SL.

Back to top

Cloud Uplift

Why Is It Being Done?

[Video: 9:00-13:01]

  • Historically, Linden Lab has operated its systems and services the “traditional” way: dedicated hardware, and infrastructure running in dedicated facilities [at one time three data centres, but for the last several years a single co-location (co-lo) centre in Arizona].
  • Actually had to develop a lot of the methodologies the company now uses to manage all of the SL services simply as a result of the speed at which the platform initially grew, building capabilities for which there were no “standard” solutions.
  • Time has moved on, and Amazon and others have developed the means for systems and services to be run / provisioned through the cloud. These services allow Linden Lab to leverage a range of options and capabilities in a number of ways.
  • A particular aspect of the move is that LL no longer has to invest time, effort and money into hardware and infrastructure, but can essentially hand these off to AWS, allowing them to concentrate on SL’s operations and development.
  • With hardware in particular, it has been a number of years since the Lab upgraded their servers, so transitioning to the cloud avoids an expensive capital expenditure in new hardware, and similar expenditures in the future.  For example:
    • In the current environment, if the Bake Service [a collection of servers use by the Lab to generate and manage avatar appearances and ensure they are consistent across viewers] needed upgrading to more powerful servers, LL would have to acquire, test and implement that hardware, and then transition the Bake Service to it.
    • Running via the cloud means picking the required hardware from a catalogue provided by Amazon, who then take care of the heavy lifting to ensure the Bake Service works as required on the selected hardware.
  • Overall, the priority of the work is such that the three goals Oz has set himself : Uplift, Uplift, Uplift.

How is the Uplift Progressing?

[Video: 16:46-19:05]

  • It’s stressful but going well.
  • All of the inventory databases were successfully moved several months ago – twice, in fact: first to the cloud, then to a different type of cloud server. This work was completed so successfully, users were not even aware of any change.
  • The intermediary service sitting between the inventory database and the viewer was also successfully transitioned to AWS. It has also been running for “some time now”, again without users noting any difference.
  • A lot of the back-end services that users never directly interact with have also been successful transitioned
  • There is still a lot of work to do, but the plan is to have Second Life “out of the co-lo by the end of the year”.

Will Uplift Allow for Different Land Products? Will Land be Decoupled from LI, and Prices Based on Storage / Bandwidth?

[Video: 15:07-16:41]

  • Land impact is a complicated subject of itself, and could use and entire hour of discussion time [hint: LI isn’t just about the simulator, it’s also about the rendering impact the viewer / client computer has to deal with.]
  • Additional comments on Land Impact from 1:20:01-1:23:21:

  • In terms of region sizes, LL has experimented with large region sizes, but is not sure if smaller sizes have been tried.
  • Currently, there are no firm plans for the introduction of new land products, but in theory, post-Uplift, there is no reason why the Lab potentially shouldn’t be able to in time offer other land products.

Will the Uplift Result in Performance Improvements?

[Video: 19:45-25:08]

  • The co-lo based servers are around 8 years old, so the move to current hardware available through Amazon should yield benefits.
  • However, testing of the simulator code on cloud servers has not yet progressed to a point where the Lab can generate meaningful performance data to share, even internally.
  • In terms of performance and with regards to products, one of the possible directions the Lab might take is to offer land products on the basis of the hardware on which it is running / time it is running. So, as examples:
    • Instead of having a fixed number of simulators running on a server, they could offer events, etc., the option of paying to have their regions run on a server with only X% the number of simulators, thus providing additional sever resources to help manage the event.
    • If there is a demand for regions only running at certain times (e.g. Monday to Friday), then potentially they could be only spun-up and charged for during the specified usage periods.
  • However, it’s important to remember, defining product options are not something Engineering has to consider, but is the responsibility of the Product teams to determine in terms of demand, viability, etc.
  • In terms of region crossings, the hope is that performance will be at least the same as now, but will hopefully see some improvement. Again, no definitive data on this has been produced during testing.
  • Region crossings will not just “go away” following the uplift, because it is impossible to make Second Life a single, seamless world without significant changes to the platform’s fundamental architecture.

Back to top

SL Viewer

Rendering Improvements

[Video: 25:17-26:57]

  • “Serious” reworking of the rendering pipeline.
  • In part being driven by Apple’s 2019 announcement of their intention to deprecate their support for OpenGL, the API used for viewer rendering. with a team at LL looking at options such as Vulkan to place of OpenGL on all supported operating systems.
  • At the same time, the graphics team is attempting to improve both the fidelity of the graphics and their performance.
  • These graphics updates are not a short-term deliverable, but a long-term project.

Usability Improvements

[Video: 26:58-27:36]

  • These include contributions from third-party developers [e.g. the Mesh Uploader improvement currently in a project viewer]
  • There is a project to overhaul texture caching and use in the viewer, which will extend to object fetching / caching as well.
  • [General note: viewer projects can be followed through my Content Creation User Group and TPV Developer Meeting summaries.]

VR Headset Support

[Video: 37:29-15]

  • LL unlikely to have time to return to VR headset support work until after the major rendering pipeline rebuild has been completed.
  • It will also be dependent on the level of rendering performance that comes out of that work – currently the rendering system just is not up to consistently handling the frame speed required by VR headsets for a smooth experience.

360º Snapshot Viewer Status

[Video: 38:25-41:33]

  • Work is held up by a couple of server-side changes that need to be made.
  • One of these is related to Project Interesting, introduced several years ago.
    • Part of Interesting was an attempt to improve viewer performance by not sending it data on items “behind” the avatar.
    • Given that the 360º Snapshot viewer tried to take pictures of the world all around you – interesting interferes with it being able to “see” items “behind” you.
    • So, adjustments need to be made so the viewer can tell the simulator it’s about to take a 360º snapshot, and have the required data delivered to it, and these must be done in a way that doesn’t impact the simulator, if the 360º capability is abused.
    • This work is unlikely to be high on the priority list until after the Uplift work is completed.
  • EEP did include a 360º Snapshot feature: the ability to freeze the cloud motion – making stitching the images together easier.
  • Even with the work left to be done, LL are not about to abandon this viewer.

Back to top

Second Life Mobile

[Video: 36:19-37:27]

  • [Please see my May 2020 mini-update on the SL mobile app]
  • Has a good team working on the mobile app for both iOS and (a little behind) Android.
  • Closed alpha using Lab staff and a small set of other people for the iOS version has been in progress.
  • There’s a small number of functions that need to be added to the initial version before it is ready for more extensive testing.

Back to top

Relationship with Third-Party Viewer Developers

[Video: 52:34-55:11]

  • The relationship is very good.
  • LL regularly receive code contributions for bug fixes from TPV developers and open-source developers.
  • Also obtain / request contributions for feature capabilities. Examples of these are:
    • The Graphics Presets and Camera Presets, courtesy of Jonathan Yap.
    • The current Mesh Uploader Project viewer, which includes contributions from Beq Janus.
    • Upcoming support for emojis in the viewer, from Kitty Barnett.

Back to top

General Audience Questions

  • [Video: 14:20-14:57] What will happen to private regions rented from SL land companies after the Uplift? Will users have to rent them direct from Linden Lab?
    • More a business question, not engineering; but really no reason to suspect / expect anything will change in this regard.
  • [Video: 28:21-30:56] Will the viewer be updated to make better used of multi-core processors?
    • Where feasible, yes. However, one of the major challenges faced by LL is the number of users using the viewer on really old hardware. While LL have always tried to ensure that Second Life will continue to work with these older systems, there are limits.
      • To assist in the decision making process on updates, etc., the Lab has incorporated code to find out more about what users’ systems can / cannot support.
      • Advice is always: upgrade if you can. Windows 10 64-bit will give significantly fewer issues than other versions of Windows.

  • [Video: 44:03-46:07] What is the best way to get long-standing LSL issues on the Jira taken notice of?
    • Try attending one of the following user group meetings:
    • Issues might not always be immediately addressed, depending on how they fit into the workflow.
  • [Video: 48:19-49:42] Would it be possible to have point lighting that doesn’t shine through walls?
    • Could be done, but would be really expensive in terms of performance.
    • Rather than point lights, where feasible, try using lighting projectors, as these may resolve the issue. [For information on lighting projectors, see: Lighting projectors: adding depth to SL]
  • [Video: 55:54-58:36] Will Linux be supported by the official viewer?
    • LL does not currently support Linux, as it makes up a very small percentage of the user base [approx. 1-1.5%], and Linux builds of the viewer require a lot of extra effort to test and build.
    • This is compounded by the number of Linux packages available, which means building to a single package isn’t necessarily a “one size fits all” like Windows / Mac.
    • In 2015, LL did put out a call for contributions to help them keep a Linux flavour of the viewer going, but there was little support for the initiative [see: Lab confirms open source support sought for Linux viewer development].
    • Several TPVs support Linux and use their own approaches to doing so. Therefore, these are always available as an alternative.
  • [Video: 1:00:03-1:01:06] Will there be an app for browsing the SL Marketplace from mobile devices?
    • Supporting shopping on mobile devices is on the roadmap.
    • This would preferably be done natively through the mobile app, although there are some hurdles in the way of achieving this.
    • Also work is planned to update the SL Marketplace to make it mobile device friendly.
  • [Video: 1:03:07-1:03:50] Will there be improvements to Group chat?
    • Right now, focus is not on improvements, but on uplifting the Group chat services along with the rest of SL.
    • It’s possible that simply uplifting the services to AWS could result in improvements. However, LL acutely aware that Group chat has some limitations that they would like to address over time.
  • [Video: 1:09:43-1:11:30] Why was Voice disabled in welcome / safe hubs?
    • Primarily because new users found it off-putting. Also because data showed new users more likely to remain engaged in SL if sent to a hub with Voice disabled than if deposited in a hub with Voice enabled.
  • [Video: 1:12:00-1:15:17] Could the viewer ever be made modular, so floaters and panels could be moved outside of the world view window (e.g. to a second monitor)?
    • Potentially, but a really big task to try to achieve, given the current viewer architecture.
    • One way to achieve this could be for LL to focus entirely on a new viewer that could present this kind of capability and eventually replace the current viewer with in. But the last time this was tried [2010 with viewer 2], the reception was that positive. It would also likely mean stalling all other viewer development work.
    • Would personally like to see the ability to move some functionality out of the main viewer window, but again with the current UI building tool kit, it’s a difficult goal to achieve.
    • [Note: Firestorm attempted to create a “dynamic user interface” viewer using the current viewer code in 2014 – see Firestorm Dynamic User Interface (DUI): it’s a real prototype – but the work never got beyond basic prototyping due to issues in actually getting panels outside of the viewer to function with the viewer, and lack of broader support for the initiative from developers.
  • [Video: 1:17:11-1:18:03] Are the bots that hop in and out of regions anything to do with LL and performance monitoring?
    • No. LL carries out all of its monitoring off the back end, and from within the viewer. A number of users do run bots for various purposes, however [e.g. Tyche Shepherd uses a bot to gather data for her Grid Survey website].
  • [Video: 1:23:37-1:24:43] Can inventory be made so it can be managed whilst off-line, say through the Second Life website?
    • It’s something LL has discussed, but it is a major project requiring a lot of investment [time, effort, etc].
    • It could be done, but when it might be done is not Oz’s call.

Back to top