Day: June 23, 2017

Meet the Lindens is a series of conversations / Q&A sessions with staff from Linden Lab, held as a part of the SL Birthday celebrations in-world. They provide opportunities for Second Life users to get to know something about the staff at the Lab: who they are, what they do, what drew them to Second Life and the company, what they find interesting / inspirational about the platform, and so on.

Tuesday, June 20th saw Landon Linden sit down with Saffia Widdershins, and this article hopefully presents some “selected highlights” of the chat, complete with audio extracts from my recording of the event. The official video of the event is embedded at the end of this article.

About Landon Linden

Landon Linden joined Linden Lab in August 2008, and is currently VP of Operations and Platform Engineering, based in the Lab’s Virginia offices. He has led the transition of live operations and the production platform to support the company’s new products. With a BSc in chemistry. he worked as a research chemist before moving into the IT sector. Since then, he has worked in telecommunications, launching numerous products.

For SL users, he’s possibly most recognised as the man responsible for re-opening the technology blogs the Lab publishes after major issues / outages occur. These had dried up after FJ Linden departed the Lab in 2011, and Landon revived them in 2014. April Linden has since taken over core responsibility for these posts since then.

Landon loves building large-scale systems, and says his passion for virtual worlds is fuelled by his interests in sociology and economics. As he notes, the nature of the work his teams undertake  – running the services, architecting them, improving them, migrating them where appropriate, etc., – is such that most of it goes sight unseen by users, unless there is a problem.

The Discussion

the initial part of the discussion looks at Landon’s background, his interest in sociology and economics – he notes that by working with the Linden Dollar and the Lab’s transactional services he’s learned a lot about economics – and touches on the Lab’s own studies with users.

In this latter point, Landon makes it clear that the Lab does not conduct direct social experiments on users, but obvious does monitor the use of services and capabilities such as the user on-boarding process, games like PaleoQuest, etc., to see how they are being used, where points of weakness lie which might be improved, what kind of metrics are being generated, and so on.

In terms of general SL trends, he makes the point of noting that – and contrary to claims otherwise – the Lab has seen a “considerable strengthening” of the Second Life economy over the last six months, probably sponsored in part by the arrival of Bento, which the Lab is obviously pleased to see.

This moves into a broader chat about the evolution of things like mesh and breedables, and how that helped grow Second Life, the way in which the Lab cannot always anticipate how new features will be used – but do try adapt to how users take them on and start using them.

Using Amazon  Services

One of Landon’s responsibilities has been to oversee and drive the evolution and enhancement of these supporting services and the infrastructure which supports them and Second Life. Most recently, this has included moving various services in to Amazon cloud.

The Lab has been a long-time user of Amazon services, and this current work not only involves moving services to Amazon, but also moving them to a container model, making them easier to test and deploy, whilst leveraging the flexibility offered by cloud-based services. These include reducing the complexity of having to manage a dedicated data centre environment to run the services, the complexities of having to manage capacity, plan ahead for growth and the purchase, delivery, installation and testing of new hardware, etc., in order to meet specific demands (as the cloud provider can “simply” turn on additional servers and facilities as they are required, and add them to the current billing.

Right now, the intention is not to reduce costs per se  in making the move – Landon rather describes the Lab and trying to break even – but is rather geared to leveraging AWS (and ECS?) and thus doing more, infrastructure-wise with the money the Lab has coming in.

Lab Working Environment

While he is based in Virginia, Landon spends a good deal of time at the Lab’s head office in San Francisco, and notes that while the Lab operates a number of office – Virginia, Boston, Seattle, San Francisco – a lot of people actually work from home, and the Lab has a relaxed approach to office-based work requirements – if it is possible to work from home and be more productive in doing so, there is no problem with doing this.

From his personal perspective, Landon views the Lab as the best place he has ever worked, describing his colleagues as “an amazing group of intelligent, passionate people”. Like others at the Lab have said, it is also a place where he tends to learn something every day, whether about technology, how SL is being used by the residents or about people.

This topic touches on the Lab’s history, going back to the late 1990s and attempts to build a VR / haptics system (aka “The Rig“).

General Q&A

What is being done to improve platform stability and performance?

Landon: We’re always working on these problems. One of the things that is frustrating for residents  – and it’s frustrating for me too – is that lag and crashing out seems to be like a perennial problem. And it is, but the reason it’s a problem is that it’s never, ever just one thing. It’s a near-infinite number of issues and problems, and we’re always working on trying to smooth those things out and reduce them, but it’s always ongoing work. And we’re always trying to balance being able to do new features versus performance improvements and stabilisation work, and I think we strike a pretty good balance there …

… This is going to come dangerously close to sounding like I’m blaming the residents for some of this stuff – and I’m not. But I think … it’s a very creative and expressive place, Second Life, and we really like people to be able to express themselves  in whatever way possible  – and within the confines of the law, at least! But that also means that the complexity of whatever it is that you’re doing, whether it’s in your region or in your parcel or on your avatar, can impact the people around you.  And so we’re trying to strike this balance of how can you express yourself without negatively impacting the people around you. And I think [Jelly Dolls] were a pretty good solution. And it also had the added benefit of feedback to the users, “Hey! Your avatar looks great, but maybe you should tone it down a bit.

Why can we have an unlimited inventory but only 60 groups?

Landon: Inventory is relatively cheap, you’re talking about a very small amount of storage when you have something sitting in inventory, and probably more importantly in the context of this question is the inventory doesn’t necessarily have to interact with other pieces of inventory. So you can pretty much just add anything in your inventory without bound, and UI problems notwithstanding, it doesn’t really have any negative impact on your experience and it certainly doesn’t impact anyone else.

When you’re talking about groups, you have this exponential impact on performance with the number of people who you’re adding into the group [particularly all the Group data which needs to follow you around SL so you can receive group notices, remain part of a group chat, etc.] … I think that’s the kind-of short and long of it. [Groups] have an impact on you and the people around you.

What is the number one cause of lag, and will improved server hardware improve SL?

Landon:  We’re always beefing up the hardware we’re using, and I can tell you the hardware is not a big factor at all in terms of lag. And this is going to be a really unsatisfying answer, but I can tell you that in my experience the single greatest contributor to lag is the network between you and wherever the server is.  So if you are physically far away from the server, you’re going to have a much more laggy experience. Most of our equipment – I dare say all of our equipment is in North America, and the west coast of North America at that. So if you’re in South America, you’re going to have more lag than some that’s sitting in Seattle, Washington. Likewise people who are in Europe and Africa are going to have a more laggy experience than people in North America.

… This is where I’m really going to get into trouble, because I don’t want to come out here and make a bunch of promises, because the things that I’m talking about are going to take probably years to do. But one of the things I absolutely have in the back of my mind is that once we get Second Life fully functioning on cloud services there is the possibility – and I will stress “possibility” – but there is the possibility we can co-locate regions more easily in other parts of the world, in south America or in Europe or in East Asia or Australia. And that would make the experiences for the people who are in those regions a lot better. The flip side to that is, if I’m moving the simulation closer to you and further away from somebody else, you’re making the lag worse for someone else.

… We did some analysis several years ago, regarding this. And what we saw was not a lot of geographic affinity for regions. One of the amazing things about Second Life is that people from all over the world come together and talk and get to know one another and chat and experience Second life together, and there’s not a lot of geographic affinity. There are a few notable exceptions to that, and I think language is one of those things; I think one of the exceptions is people who speak Portuguese, and then tend to almost exclusively come from Brazil. So we can say that if you have a region that caters to, or is attracted to Portuguese speakers, we would probably want to co-locate that region in Brazil.

This is just really stuff that we’re thinking about, there’s no hard plan to do any of this; I think we’ve got a lot of work to do before we can even considering doing something like that, but I’ve absolutely got that in the back of my mind.

Would LL ever consider adding any of the reliable language translation tools back into the viewer?

Landon: For what it’s worth, I’ve actually looked into some of that. I mean … there’s just some amazing tools that are becoming available now using AI machine learning, and I’m really interested in doing some things along those lines. That said, no promises, no commitments; I don’t control the product direction, so I’m looking at it just out of more-or-less professional curiosity and not something I’m actually planning on implementing.

But I think, to try to answer your question as best I can, I think it’s getting easier and easier to put translation and text-to-speech and speech-to-text services into your products, and I would hope that we get back to doing some of that – but no promises and no commitments, and I don’t control it anyway … I don’t make that call.