Sansar: why LL are building their own engine w/audio

A frequent Sansar question asked of the Lab is why did they opt to build their own platform engine, rather than choosing to use an off-the-shelf engine such as Unity or Unreal. To try to address these questions, as a part of the Sansar Product Meeting on Thursday, March 7th, 2019 Richard Linden (Sansar’s Chief Architect) and Jeff Petersen (aka Bagman Linden), Linden Lab’s Chief Technology Officer gave a 25 minute overview of the Lab’s thinking on the matter and what they see as the advantages / disadvantages in either building their own engine or using an existing product.

The following is a summary of their discussion, with audio extracts, the full version of which can be found in the official video of the product meeting. Note that in presenting them, I have attempted to group comments made by topic, as such the audio extracts below do not necessarily chronologically follow the meeting video. However, care has been taken to not lose the overall context of the comments made by either Jeff or Richard.

The presentation was followed by a more general Q&A session, which also involved Landon McDowell (Linden Lab’s chief Product Officer) and Nyx Linden. Some of the questions relevant to the core subject of the platform and engine and roadmap are also included. Please refer the video for all questions ask at the meeting.

Jeff Petersen – Introduction

• 

  The decision to create an engine from the ground up was made some 5 years ago.

• There is a degree of envy towards the likes of Unity and Unreal, particularly because of their maturity and cross-platform (operating system) support, plug-in support, etc.
• However, the Lab’s aim is to build a platform that can compete with other platforms in the same arena, which includes both Unreal and Unity. Ergo, using one of those platforms as the foundation for something like Sansar made (and makes) little sense, simply because doing so would deny a product the kind of low-level control required to differentiate from those  platforms.
• In addition, there are directions in which Linden Lab would like to take Sansar and user-generated content (UGC) which perhaps do not sit well with the use of an off-the-shelf engine.
• Further, in opting to build their own engine for Sansar, LL has been able to make use of a number of third-party applications and tool sets (perhaps the most notable being Speech Graphics facial animation software and Marvelous Designer), which allow LL to leverage capabilities and integrate them into the platform to provide the necessary flexibility of approach / use.  This kind of approach isn’t so easy with platforms such as Unity and Unreal, which are seen as more “full stack” solutions: once you start using them, you are pretty much locked into the technology they support and the integration they want to provide.
• A further point of note in the decision to build Sansar’s engine is that Linden Lab obviously has an enormous amount of experience with development systems designed to support and enable user-generated content and user creativity.
  • In this respect, Richard Linden, Sansar’s Chief Architect, is one of the longest-serving members of the Lab’s staff, having been with the company since before the development of Second Life, in which he played a significant role.

Richard Linden – Constraints

• An important thing to remember with Sansar is it is not just a virtual world, it is a platform creation engine intended to allow users to design and implement compelling content they can use to attract an audience.
• So again, Sansar is in competition with the Unitys and Unreals that are out there, and thus needs to be differentiated from those platforms. This is done through the constraints / requirements LL apply, their own unique experience in running Second Life and in handling UGC on a scale and of a type normal games do not have to deal with.
• In terms of constraints, LL recognised a number of performance related constraints that informed their decision to develop their own engine:
  • Rendering: UGC comes in many flavours from the optimised to the unoptimised. From the managed poly count to the outrageous poly count. Sansar has to provide a rendering system that can handle all of this, and ensure that it can deliver experiences to uses that offer both a smooth experience in VR and do not cripple a user’s computer in doing so.
  • Physics: again, the physics engine must be robust enough to handle all kinds of UGC, optimised and unoptimised. In this, LL have 15 years in using the Havok physics engine in Second Life, so it made sense to leverage that experience in Sansar.
  • Scripting: experiences and (in time) avatars will be liable to have many, many scripts running in them / associated with them, scripts which (again) might not be optimised for streamlined execution, so the platform needs a scripting engine that can scale to the demands being placed upon it as experiences become more complex in their capabilities and avatars evolve (and appear in Sansar in (hopefully) greater numbers over time).
    • As noted by Bagman, this includes managing malicious (deliberately or otherwise) scripts whilst keeping the scripting environment open enough for creators to be able to do what they want with their scripts. This is something not easily achieved within a “full stack” engine architecture without requiring substantial changes to its scripting system.
  • Audio and UI capabilities: again providing the necessary flexibility for support of audio content from creators (FMod) and a UI tool (NoesisGUI) that is flexible enough to meet the needs of creators and of consumer users.

Richard and Jeff – Asset Management

• A further constraint is the sheer volume of UGC assets.
  • Second Life has an estimated 24 billion UGC assets associated with it.
  • Linden Lab hope that in time, Sansar will be at least an order of magnitude bigger than this.
  • To avoid issues of having to reprocess data associated with assets, SL and Sansar are founded on the idea of the immutability of assets. Linden Lab promise that so far as is possible, updates to the platform will not break existing content.
• The majority of games built on other engines don’t have to deal with any of this.
  • They have comparatively low number of assets they have to deal with.
  • When they update (or the engine they use updates) they can do one of two things:
    • Reprocess their assets, then burn and ship a new version of their game.
    • Remain on the current version of the engine and use the newer version for their next project.
• Given the above, engines like Unreal and Unity aren’t geared towards dealing with massive amounts of asset data or in maintaining the immutability of assets, as that is not expected of them.
• Using such engines for an environment like Sansar, where assets could be expected to be relevant for years (as is the case with Second Life now), and continue to work “as is”, without having to be reprocessed by the Lab each time the engine is updated, is therefore a non-starter.

Richard – Aims in Building an Engine

• LL ultimately want to make Sansar an environment where anyone can create and share, whether or not they are “hard-core” content creators. This means Sansar needs to:
  • Support users who may not create original content, but can use that content (as provided by others) to express themselves and present environments they and their friends can enjoy.
  • That has the ability to take on a lot of the heavy lifting involved in content optimisation, etc., and which doesn’t necessarily require those creating environments to have in-depth / professional level knowledge on scene optimisation, content development, etc., that might be required in other platforms.
  • That can (in time) offer a collaborative content creation environment, so people can work together to design and build as well as visit experiences together.
  • Collaborative editing does not only mean being in a shared editing space, it means also having access to all of your chat and communications tools to be able to stay connected to friends who are in Sansar but not in your edit environment – again, these types of capabilities aren’t necessarily provided in other engines.
• Not all engines have all of these types of capabilities built-in. And even where third-party plug-ins are available to achieve aspects of the functionality, they may not actually be as flexible to use or in meeting the constraints particular to Sansar as might initially seem to be the case.

Jeff  – IP Protection

• IP protection has and remains a major consideration, and was looked upon as a show-stopper for using other engines.
• Sansar is designed to provide a supply chain economy, with individual rights respected in the use of component elements (.e.g whether an item can be used just within a scene, how it can be used as a component in someone else’s model, how royalties are safeguards and paid in respect of the latter, etc.
• The use of personal inventory and the Sansar Store is also viewed as potentially being seamless (e.g. scene creators can use items they upload to their inventory and / or items available on the store, up to and including the potential for “try before you buy” from the Store, with all rights again respected.
• This kind of protection isn’t seen as being offered by engines like Unreal or Unity without a considerable amount of code re-writing which, as it is part of the overall engine “stack”, runs the risk of having to be re-implemented each time there is an update to an off-the-shelf engine each time that particular aspect of / tool within the engine gets updated by the provider.

Richard and Jeff – Broader Pros and Cons

A major disadvantage with using an off-the-shelf engine is seen as the back-end support.

• There tends to be very little out-of-the-box support to meet the requirements a platform like Sansar has.
• Trying to engineer around this using such a product can be difficult, particularly given the amount of information sharing that goes on between the Sansar client and the back-end.
• Most likely, it would have meant working on the engine’s code, it would have effectively been a fork of the original Unity / Unreal / whatever code base, which itself opens up all sorts of headaches.
  • The code won’t really be supported by the engine provider
  • How is the code maintained; how are major updates to the engine handled and merged without potential breakage to the forked code, etc?
  • This is already a problem for LL with Havok.
• As mentioned above, there is the issue of longevity. Game built using engines like Unity tend of have a finite requirement on the engine, when they are shipped, that’s largely it, and no need to necessarily maintain full backwards compatibility; the next version can be built on the latest version of the engine. Sansar doesn’t have that luxury, and most engine providers don’t see it as a need.
• The case against a dedicated engine is that, obviously, it takes a lot longer to build out all of the necessary functionality that an off-the-shelf product might provide and that can be used.
• LL is a small company with limit resources, ergo, building their own engine is a long-term task.
  • However, LL is uniquely positioned to be able to afford to take on the work, and has a fully supportive board who recognise the effort.

Q&A Session

Why isn’t Sanar Open-Source? Will it be? How about Plug-Ins? – Jeff, Landon

• Open-source can be hugely beneficial – look at Second Life. However it brings potential issues when trying to deploy new features, maintaining back compatibility, etc.
• That said, there are elements of Sansar that could be open-sourced, and the Lab is discussing this with a possible view to doing so down the road.
  • Doing so would allow greater development fire power to be used within the platform.
  • However, the Lab will not be open-sourcing the entire Sansar platform: just those elements that could both benefit from open-sourcing and provide benefit to the games community.
  • This will not happen in the near-term however, as there is still much work still to be done.

• Plug-ins the directly affect the run-time environment (r.g. that require the user to download and install some third-party plug-in in order to see / use content in Sansar is something the Lab do not want to have happen.
• However, having plug-in tools purely for the edit environment that allow creators to work more efficiently when developing their scenes (e.g. tools that automate editing activities) is something the Lab may support in the future.

How Easy is it to Support New Tech  – Headsets, etc?

• Potentially easier when writing your own engine, and depending on the technology itself.
• LL are currently working to support OpenVR as well as the Oculus and Vive.
• Touch controllers, etc., are potentially easier to integrate than waiting on support from an off-the-shelf engine developer to integrate.
• That said, it is highly variable: it would be relatively easy to integrate a successor to the Rift, whereas the Oculus Quest requires more engineering development.

Will it be Possible for Sansar to Run on an iPad or MS Surface?

• Having Sansar run on portable / mobile systems and devices  – and perhaps even consoles – is something LL would like to achieve. Ultimately this does come down to performance. If Sansar cannot run at reasonable frame rates on such systems, then it’s not worth trying to push the platform on to them.
• LL potentially have a unique constraint on this. Whereas other engines rely upon content that has been specifically targeted for individual platforms, LL would prefer to allow creators (particularly non-professionals) produce content that can then be delivered to all operating system platforms are supported, rather than having to be specifically created for various hardware platforms.
• This approach requires a lot of engineering effort to develop capabilities such as auto-simplification of content which can be used alongside other capabilities in development or being introduced (draw distance constraints, a LOD system, etc.), will eventually allow the same content to be delivered to different hardware platforms, which in turn will help lower Sansar’s minimum hardware specifications.
• However, the system will have to be bounded by realistic expectations: if a scene is built that pushes the latest generation of Nvidia graphics systems to their limits, all of the auto-simplifications, LOD support, etc., isn’t going to see the same scene running decently on an iPad.
• Thus the approach is not so much to enable anything built using Sansar to automatically run on any hardware platform, but to allow those scenes that are not overly complex to run across different platforms. Thus, the choice of where and how an experience will run is down to the creator to determine as they build their scenes.

Will Sansar Be offered as a Streaming Service?

• This was tried as a prototype 12-18 months ago, streaming Sansar to iOS devices and it worked.
• The problem is the cost: streaming a service like Sansar isn’t cheap, nor is it helped by the fact that costs associated with cloud services have been rising.
• Second Life had, and has, streaming services: SL Go from OnLive (now defunct) and Bright Canopy via Frame, and neither has enjoyed a high level of uptake among users. Ergo, streaming and costs associated with streaming is something LL are keeping an eye on, but it is not something for the near future.