Meet the Lindens is a series of conversations / Q&A sessions with staff from Linden Lab, held as a part of the SL Birthday celebrations in-world. They provide opportunities for Second Life users to get to know something about the staff at the Lab: who they are, what they do, what drew them to Second Life and the company, what they find interesting / inspirational about the platform, and so on.
Tuesday, June 20th saw Landon Linden sit down with Saffia Widdershins, and this article hopefully presents some “selected highlights” of the chat, complete with audio extracts from my recording of the event. The official video of the event is embedded at the end of this article.
About Oz and Grumpity Linden
Oz Linden is the Technical Director for Second Life. He joined the company in 2010 specifically to take on the role of managing the open-source aspects of the Second Life viewer and managing the relationship with third-party viewers – in his previous role, he had been responsible for leading the company his was working for in taking their product from closed-source to open-source and then managing the technical side of the product as a open-source project for a number of years.
Over the first two years of his time at the Lab, he was primarily focused on the open-source viewer work and in refining the overall viewer maintenance process, before his role started expanding to encompass more and more of the engineering side of Second Life. When Work on Sansar started in earnest, he pro-actively campaigned within the Lab for the role he has now, with responsibility for managing all of the engineering side of the platform.
He came to Linden Lab out of a desire to do something “fun” after working in the telecommunication arena, notably with voice over IP systems (VOIP), which he defines as being “really interesting technology with some really fascinating challenge”, but in terms of it being fun, it really didn’t do what I wanted it to do.”
He classifies the attraction to working with Second Life as perhaps falling into three core areas: through the open-source nature of the viewer, he is directly involved with how SL users are using the viewer and what they do with it – which can often times take the Lab entirely by surprise; through the fact that the Second Life offers the challenge of trying to implement new technologies alongside of (rather than simply replacing) older technologies; and working with the operations team and others to ensure SL constantly evolves without (as far as is possible) breaking anything – a process he refers to and rebuilding the railway from a moving train.
Grumpity Linden is the Director of Product for Second Life, enjoying what she and Oz jokingly refer to as a “symbiotic relationship”. She actually started at Linden Lab in 2009 as a contractor working for The Product Engine, a company providing end-to-end consulting and software development services, and which support the SL viewer development. She became a “full-time Linden” almost three years ago.
As Director of Product she manages the product team, which oversees a wide range of SL-related activities alongside of Oz’s team. This can involve coordinating the various teams involved in bringing features and updates to Second Life (e.g. coordinating the engineering teams and the QA teams, liaising with legal, financial and compliance to ensure features and capabilities meet any specific requirements in those areas, etc.). This work can also involve looking at specifics within various elements of the overall SL product, such as UI design and layout, etc.
Grumpity has a background in psychology and computer science, but has worked in the oil and gas industry. On moving to the San Francisco area, she crossed over into working within the tech industry, eventually settling at Linden Lab as a contractor, working on the Viewer 2.0 project. She enjoyed working at the Lab so much, she resisted all attempts by her employers to move her elsewhere, finally joining the Lab full-time in 2015.
Like Oz, Grumpity is passionately committed to seeing Second Life continue.
How much control and input do you have over the direction of second life?
Grumpity: I will let Oz speak more to that, but Bento was conceived and reared and launched all through the efforts of Oz’s team and of Engineering. Certainly, Product took a part in defining that, but this is a great an example of one of the long-time Lindens [Vir Linden] suggesting this as a possibility and then this feature getting worked on.
There was a tonne of time spent defining that work with residents, which I’m also very proud of, I think we absolutely took the right path there, but as to the development of that project – Oz, do you want to speak to it?
Oz: Just to comment to that one point about Bento. The general direction of the project we started out with changed very significantly, once we got residence involved. The essential concept of extending the avatar skeleton and adding capabilities, that was the concept we began with, [but] the specific additions we made to the skeleton changed very dramatically after we got resident designers involved.
We were planning on doing a quite simplified hand, for example, and the designers came back to us and said, “look, we really need every joint in every finger”, and ultimately they convinced us that was the right thing to do, and in retrospect, it’s obviously worked out really well.
The broad question of who or how we set that direction; it’s one of the things that’s really great about working at the Lab … We have an incredibly collaborative process. Pretty much everyone involved, up to and including the residents – emphatically including the residents, I should say – is empowered to put forward ideas. And so our job isn’t so much thinking up what’s going to happen to Second Life, as it is from just picking from among a myriad of possibilities. We could have a staff of 500, and we wouldn’t have enough to do all of the really cool things that we might in theory be able to do.
So it’s picking and choosing, and we try to shift who we’re making happy at any given time, so we’re spreading it around a little bit … My job is to think about what the technology impact of anything is going to be, how difficult it’s going to be to do, and how long it’s going to take to do it; although even more so that most engineering groups, I think we’re really challenged in figuring out how long it’s going to take for things to happen.
And the Product team, headed by Grumpity, thinks about what the implications are for the way that affects the business, affects the activities people are already doing in Second Life – to the extent that we know! And we work together to pick from the things that are possible and can be done in a time frame that’s good. We try to make sure that we’re doing something new fairly regularly; so we can’t pile on everybody to a project that’s going to take two years, because then for two years, nothing would happen. Well, the company did that once, and we all know how that went…
But yeah, between us, we have a lot to say about it. There are aspects of the way that Second Life evolves that are not really our space. For example, we’re not heavily involved in Governance issues; we’re not heavily involved in thinking how much things may cost. That’s mostly other people. But how it works and what it can do – that’s what we spend our time on!
Grumpity: And to mention, we’re not necessarily heavily involved in compliance issues … but we do spend a lot of time trying to figure out how to minimise the impact of compliance while actually adhering to the needs.
As inventory in the viewer is just pointers to assets on the Lab’s servers, could Linden Lab provide a means for redelivering lost inventory items?
Oz: We have recently put out some changes that are intended to reduce some of the ways we think people were unintentionally deleting things, and we’ve fixed some bugs that may have been responsible for things going astray that shouldn’t [updates reviewed here & now in the release viewer] … She’s right, your inventory is a set of pointers to assets; we have the assets, we don’t have a record of what those pointers were. The pointers are ephemeral; they change dynamically, and we don’t today have a journal of what all the changes were that went through.
That’s an area of concern; unfortunately, solving that problem very likely falls into that project that would take at least two years category that I talked about before, that’s difficult to tackle on the whole. So we’re trying to find aspects of it that we can attack and improve … So we’re trying to find ways to do incremental steps that make inventory more and more robust. If I were to go to Grumpity and say, “this is what it would take to completely solve the inventory problem,” she would end-up saying we can’t commit that large a fraction of our resources for that long to that problem. So we have to find ways to break it down into small pieces, and that’s what we’re doing.
Unfortunately, that means we can’t say all the inventory problems will be solved by the middle of next week or even next year, necessarily.
Grumpity: We spent a lot of time investigating this recent uptick in reports of inventory going to Trash accidentally and getting deleted, and we’ve put in a bunch of viewer-side changes to prevent that, and Firestorm has merged those in. so please make sure your viewer is updated. The new Firestorm release [reviewed here] has all of them, and even some that we haven’t released but are in the latest Maintenance RC viewer [version 22.214.171.1247250 at the time of writing this transcript].
I would also like to use this platform to say that we absolutely need viewer logs from the session where the deletion or the disappearance of inventory happened, to continue to diagnose this problem. So if you’re in a position to provide those logs from the session where the inventory loss happened, please, please do. There are multiple JIRA already open – file a new one, reply on the forum, we’ll see all of them, and I will be thrilled to take up the cause and find out what has been going on.
Oz: That’s a point worth emphasising. We don’t keep all the logs for a long time; we couldn’t, they’re just too big. Id you report that three weeks ago, you lost 100,000 things, there is no hope whatsoever that we’re going to learn anything from that report. Those those logs are long gone; we cannot tell what you did or what happened to what you lost.
If you report it the day that you lose something, and we see that report – and we’re watching those reports, we have people who watch those reports all the time – and you attach a viewer log, and there’s a page on the wiki about how to find the logs. And you tell us “I was in this region, at this time, and the following stuff disappeared”, or even: “I was in this region at this time, and I knew that I had it then, but two hours later I noticed that it was all gone.” That gives a window where there’s some hope of us finding information about it. And we can use that information to figure out what happened.
We often will not be able to recover your lost items; occasionally it happens, but unfortunately it’s not the normal. But it would be an enormous help to us to get reports that have that kind of information on them promptly, so we can dig out and try to learn what happened and what went wrong, and then those cases at least, we can fix.
Grumpity: So, for the record. In all of the reported cases where we were able to get logs from the server-side for this inventory loss and actually find the log records for when the deletions happened; from our end it looks like it was a regular case of the user deleting inventory. So in order to figure out what’s going on, we absolutely need viewer logs so that we know what the viewer was doing and why those messages were sent to delete inventory, if you did not intentionally do it.
… Again, I’m going to use this soapbox to say we triage incoming bugs pretty much every [working] day, sometimes we skip a day when there are other things that get in the way. We triage incoming feature requests on a regular basis as well, not quite as frequently, and we pay attention to what’s going on. It is our hand on the pulse, and it is also your best bet for getting bugs addressed. If you write about a bug on the forum, maybe somebody else will file it, but may not, and maybe it will never get to us. If you are sure it’s a but – write a JIRA, and then we’ll see it.
Can you talk about the Windlight Enhancement Project?
Oz: That’s EEP! Enhanced Environment Project! E.E.P! That’s one I’ve been wanting to do for a long, long time, and I’m really excited that we’re making progress on it again.
So we’re going to be doing a bunch of things to provide some new characteristics of the environment that you can modify, and some new ways that you can manage those environment settings. Our belief and our hope is that we will be able to make them into assets, so you’ll be able to give someone else a sky setting object, and they’ll be able to apply it to their region, just the way you do now with, say, an avatar shape, and that’s a pretty cool thing that we never got around to doing on the earlier iterations of the project.
We have not finalised which of the grab bag of possible extensions we’re going end up doing, and we may end up with more than one iteration of the project, and do it in chunks.
The other thing we’re going to do is add changing the settings to the things an experience can do, so that if you’re in an experience, it can change your individual settings so that as you move through an experience, it can change whether or not there’s fog, what the character of the light is, and so on.
Would that affect day length?
Oz: That’s on the list. We’re going to be able to modify the length of the day; the current feeling is that it should be settable from between four real hours to 168, which is seven real-time days.
Will this go the other way? Setting a region’s time of day to match a real world location?
Is there going to be more work done on group chat?
Oz: We don’t really have anything on the short-term roadmap right now for that. There have been a few little bug fixes that have been looked at recently; we’ve recently adjusted how notices are sent out in an attempt to make them more reliable, mostly when you first log-in.
But that was a big project we did a year or so ago, and we’re mostly pretty happy with it. It may well be that some groups and some people on some simulators have very significant delays, but the statistics that we watch closely for how the group chat servers are performing are actually, looked at in the large, really quite good, so it doesn’t feel like an urgent problem right at the moment.
We keep watching it, and if people can identify specific situations, specific groups and specific times where there’s a very measurable problem, that’ll give us a starting point to understanding whether or not there’s something that we can do.
Would it be possible to have viewer floaters detachable from the main viewer window?
Oz: It’s all software, so in theory, it could be. But that’s a really be undertaking … The Second Life viewer and its whole UI system began long before that was a popular things to do, and it’s not really something that’s built into the infrastructure. It would be a pretty big undertaking to tackle that.
There was a third-party viewer team that attempted it, and had partial success at one point, and we followed that effort very, very closely. But they were only able to make it work for certain windows for certain times, and they were never able to support it in any way on a Mac, which is something we want to be able to continue to do.
So at this point, that falls into the really hard category.
Grumpity: We’ll take an open-source contribution!
Oz: Yeah, if someone figures out a way to do it, we’ll take that in a heartbeat.
Grumpity: One answer I do have to that is perhaps the solution to this problem lies not moving the windows outside of the main screen, but finding better ways of organising HUDS. Either creating some location where all the HUDS will attach, or perhaps as the marketplace evolves, designers will come up with better ways to control the mesh bodies and the mesh heads that will allow us not to walk around without so many HUDs blocking out the world.
Are you going to continue to support Linux?
Oz: Well, we made an announcement … it was quite a while ago, that the Linux viewer was being transitioned to a community support model. By which we mean, we’ll accept contributions to keep it working and we will integrated them if they’re good. And that’s how the Linux world works, and that’s how we intend to participate in it.
After we get our 64-bit viewer out, we’re going to make some changes to how we attempt to build the Linux viewer, that will help it to fit into Linux development models better than the way we’ve been doing it. We’ll have a general announcement about that, but we’ve talked about it at the third-party viewer meetings and on the open-source list. And we look forward to contributions that help us to make it better. I have to say, as a long-time open-source enthusiast, I’ve been disenchanted by the fact that what we get is a lot people saying, “why don’t you do Linux? Why don’t you do Linux?”, and we get almost no contributions to help.
So, I’m hoping that when we’ve changed our methodology to make those contributions much simpler to make, to make building the viewer on Linux easier, that will make it easier for people to contribute and participate in and improve the Linux support. I’m hoping that will happen.
Will SL be optimised for the new generation of multi-core CPUs (e.g. AMD’s Ryzen)?
Oz: We pretty much rely on that level of support from the drivers provided by the operating systems. There’s no reason why it shouldn’t support those things.
What we don’t plan to do is to adopt, at this point, other graphics frameworks like metal [or Vulkan] or other alternatives to OpenGL, just because that would be too big a forklift change to do. It would mean re-writing the entire rendering system … I mean, I’m not going to say never, because I try not to ever say never; but is not on the roadmap. Giant job.
Will 2FA be used to enhance account / viewer Security?
Oz: Yeah, we’ve talked about that. I don’t have a time-table for that … but like I said, we pick from what there is to do.
2FA for anybody who is not into the lingo is two factor authentication, so it means when you log-in you have to use a token or a text message or something to provide and extra piece of information beyond your password. Most banks require that now, a lot of things provide that as an option. And we’ve certainly talked about it; I mean certainly, our corporate accounts use 2FA …
So it’s in the category of things that come up so much, that it would be silly to think we won’t eventually pick that as one of the things to do, so yeah.
Has anything been done to allow creators to export from SL to Sansar?
Grumpity: We have not done any work to specifically allow creators to export things from Second Life to Sansar, and it’s not on the roadmap. Whether there will be an initiative that comes down to us that it is something we should attempt to support, I can’t say.
Can you tell us something about the Animated Objects (“Animesh”) project?
Oz: People do a lot of things in Second Life today that are pretty great. They make animals that move around, robots that unfold into something else, the examples just go on and on. Given the capabilities that we’ve got in Second Life today, it’s astonishing how much people do. And when you dig in to how they do it, how they make things look as though they’re bending and morphing, they’ve done an enormous amount of work that’s idiosyncratic to Second Life; it’s very, very Second Life specific. They haven’t been able to use tools from outside Second Life to do the work.
And frankly, those mechanisms are often much, much more expensive, and so it’s really hard to get an object to do the kind of things that an avatar does. You and I are sitting here, and thanks to the animation on this couch, I’m fidgeting as I talk and my head is moving around to watch things. We all take that for granted with avatars, but getting an object that’s not an avatar to do that stuff is a huge amount of work, and it’s pretty expensive in terms of its impact on the simulator and on the viewer, compared to the skeletal animation that the viewer does, where the computer says, move this bone, and then computes hoe the skin is affected by that movement. And so you see it as the whole body moves in a relatively natural looking way.
Getting that to be done using the primitive we have today, for thing that are not avatars, is really hard. It’s fantastically impressive how well people do it using what we have today. And the goal of the Animesh project – if we can call it that – is specifically to give people … the same skeleton avatars have now, and allow people to rig mesh skins onto whatever subsets of that skeleton they need, and then use the relatively inexpensive and easy to code – comparatively – mechanisms for animating it that are used for animating avatars … It’ll be significantly less than some of the mechanisms people are having to use now, [but] how large it will be is something we’ll measure when we get a little bit further along on the project.
Is anything being done to fix the profile feed issues (inability to upload snapshots, etc)?
Grumpity: There are known issues with snapshots not uploading and they are in our immediate queue of things that we’re working on … We hope that eventually we’ll be able to present a more robust solution than exists today. Right now it’s very flaky.
Why can we have an unlimited inventory but only 60 groups?
Oz: An item in your inventory is just a pointer to something in the asset store, and other than when your viewer is first starting and you’re loading it all up, it essentially has no impact on what you are doing. It’s passive. It’s like having something in a file cabinet: it doesn’t affect what you do in the rest of the office.
The fact that you’re in a group causes your viewer to keep track of a lot of information about what’s going on in the group, and causes the simulators – not just the one you’re on, but the one every member of the group is on, to do work to keep that information up-to-date.
One of the things we found when we did the project I mentioned before to improve the performance of group chat [is] we had to study very carefully where is time is being spent in the group chat system; it’s a separate system that works through the simulators, but behind them. And we’re constantly finding surprising things when we do that kind of study.
It turned out that the largest part of the load on the group chat system had nothing to do with actually sending chat messages. It was any time anyone in a group logged-in or logged-out, a message got sent to every simulator that had anybody in a group on it, and to every member of the group, regardless of where they were. And it turned out that something like 90% of all the messages going through the group chat system, were those log-in and log-out notifications. And the chat messages were not getting through because they were in line behind that other traffic.
So we made some changes that change how that impacted things, and one of the changes is that you don’t necessarily get all those notices any more, and we’d like to be able to expand that again in the future. And there are also some other operations you can do on groups that can cause really big and expensive queries to be made against the database that can interfere with other queries that are necessary to make things go.
So restricting the number of groups you can be in is, at this point, kind of a defensive measure for the infrastructure, because it has been our observation that people tend to join a lot of groups, and if we raised the number of groups, then people tend to join even more groups. And that’s not free; groups are not passive things that are just sitting in the filing cabinet. They’re very active things that are causing messages to be sent around the system. And we can’t eliminate that, because that’s some of the function of groups. That’s part of why we say more groups is a Premium benefit.
And we review all limits on a periodic basis. One of the regular activities that Grumpity and I many other people in Second Life do is to say, “OK, let’s look around. Can we change the limits?” So recently, we raised the number of prims you can have on a region, because we looked at it and we said, “you know what? We can actually afford to do that, it’s not going to have a bad effect on people,” and so we’ll do it.
Every limit is Second Life is something that we think about whether or not that we can change it, on a regular basis. One of the things that I’m particularly enthusiastic [about] and focused on right now is figuring out what the constraints are on the number of avatars you can have in a region and have everything work well, with the goal of eventually making that bigger.
Grumpity: Which, by the way, we already did change in effect, by adding the Premium access feature … [And] when we do look at limits, if we discover it’s a limit we can’t raise for all of Second Life, the second thing we say is, can we raise it for Premium? We’re very much interested in continuing to add value for Premium subscriptions, and so we regularly review what we can do for Premium. And we’ll hopefully have some exciting news coming up in that area soon.
Given you’ve said increasing prims doesn’t hurt, does it follow this big textures don’t matter?
Oz: big textures have a pretty significant impact on viewer behaviour. The server doesn’t care about textures, by and large. That’s a slight over-simplification, but mostly the simulator know from [different] textures; it just passes identifiers from them around. However, they can have a big impact on the viewer, in particular people running 32-bit viewers will run out of memory, because the amount of memory that gets taken up by lots of texture. And of course, the bigger they are, the more that is. When you make a texture go from 512×512 to 1024×1024, its four times as big and takes up that much more memory in your machine.
One of the things we thing will probably be possible, when we’ve got the people who can be onto the 64-bit viewers, is that we will be able to support larger textures; and that’s definitely something we’re already working on. We hope so, we’re not really sure what the impact will be, and we’re going to try to do some careful science to measure that.
Does “Jelly Dolls” (avatar complexity) mean people should jut get rid of their high-impact inventory, because no-one can see them?
Grumpity: This is really a personal decision on how you choose to deal with the avatar. I have to say the Jelly Dolls project has, overall, gone fantastically, and we’re very with the success that it’s had. It had the effect we were hoping for … First of all to make everyone in SL cognizant of the impact their render cost has on others, and by doing so, pressure merchants to actually cater to that need. and indeed, it worked, and we have little ticker tape parades in our personal chat messages in our group every time we would see someone say, “I revised my product because the rendering cost was so high, and now it’s the same thing, only at a fraction of the cost.” And this is exactly what we were hoping for. But we always try to leave the decision up to you. Do you want to keep your beautiful outfit that has a really high render cost? Can you tweak it to make it less expensive? Ultimately, it’s your decision.
And there are possibilities to always render avatars; anybody can always choose to render some specific avatars, so you and your friends will all see each other. but – by “pressure” of course I meant “encourage”, yes, thank you Xiola! And there are ways you can choose to see or not see heavy render cost avatars, and the decision is yours. We’re not going to go into your inventory and discard the stuff we consider poorly made.
Oz: One other comment on that. I think that it’s difficult for most people to appreciate what an enormously wide range of users we have, almost on any metric that you measure them. We have an unusually large range of ages, people of all ages use Second Life. People with many different level of experience of computers use Second Life. But in particular for this one, people with a huge range of graphics rendering capability.
There are of course some people – I’m not one of them – who have super high-end gaming rigs that can render glorious images at near infinite detail, with fantastically subtle lighting effects on gigantic screens, and it doesn’t hurt their performance even a little bit.
And then we people who have 10-year-old computers that are barely able to do the simplest kind of rendering, and there are lots of them. And so the Jelly Doll feature was designed to let everybody tune their own experience to the capabilities of their system, while – as Grumpity said – informing everybody what’s going on, so that they can tell.
If you and your friends all have high-end gaming rigs, and you’re the people you want to hang around with, you can just crank that baby up and have your fantastically expensive rendered avatars, and you’re not hurting anybody, and there’s no reason why we should put any limit on you. But if you’re going to go to a public club with lots and lots of people in it, there are going to be a lots of people around you who, if they don’t have a defence mechanism against you, you’re going to crush their viewer. You’re going to lag them out to the point where they’ll get a frame rate of have a frame a second. And we wanted to give them a way to say, “Those super laggy people? I need to be able to see, so I’ll do without seeing them in very much detail. It’s supporting that wide range of users that we’re trying to do.
By the way, if you’re one of those people out there who’s still got a machine that can only run a 32-bit operating system, or if you’re still running a 32-bit version of Windows, we are in fact exerting lots and lots of effort to continue to support you; but you will have a way better experience if you upgrade. It’s been quite dramatic when we test the 64-bit viewers.
Why were colours used for the Jelly Dolls, rather than avatar imposters? Have you done any studies to see if having lots of coloured blobs has impacted the new user experience?
Oz: We just made them as inexpensive as we could to render, and nice solid colours are really cheap …
Grumpity: … We did spend some time thinking about the new user experience and how that would affect it. But part of the way that’s ameliorated is by the amazingly low render cost starter avatars that continue to be put out; which to me are a shining example of what you can do for like 10,000 render cost.
Ruth [the original SL avatar form, now in the form of the Character Test Avatar] does live in the Library inventory, but that’s also not a new user experience I want … And I saw another comment go by. We do indeed spend time thinking about the changes we’re making in Second Life impact the new user experience and all of the new people coming in and the learning curve that awaits them. All of these things are absolutely a part of our headaches.
Is there a way to soften ban lines so when sailing / flying, you can roll off them? Could a navigation aid to ban lines be provided?
Oz: We actually had a feature request recently for whether or not you could change the distance at which you could see ban lines, and that’s something we may be able to do something about – seeing them on the mini-map is another interesting idea. But we haven’t got a short-term thing for that, but those are the kind of changes a third-party viewer could certainly experiment with, and we’d love to get contributions for that kind of change.
Grumpity: … And I would like to point out …. we did recently make a change that allows estate manager and owners to enforce rules about ban lines on the parcels in their estates. so if an estate owner doesn’t want anybody on their land to put in ban lines, so that it can be sailed around, for example, they now have this ability and are not forced to fly around and remove ban lines every day.
Hopefully, this solution is going to partially alleviate the problem, although no, we’ve never come up with a way to roll off the ban lines “softly”. I like the imagery, though!