On Saturday March15th 2014, the Firestorm team hosted a meeting and Q and A session to discuss the recent 4.6.1 release, provide updates on a number of issues, and answer audience questions.
While the meeting was recorded, the Firestorm team are aware that many of their users have hearing difficulties, and / or prefer to read text, so this transcript has been supplied on their behalf.
When reading, please remember:
- This is not a word-for-word transcript of the entire meeting. While all quotes given are as they are spoken in the video, to assist in readability and maintain the flow of conversation, not all asides, jokes, interruptions, etc., have been included in the text presented here
- In the interests of readability, topics in the transcript are not necessarily presented chronologically compared to the video. For example: questions asked during the various updates, etc., are presented in the Q and A section of the transcript, rather than at the point at which they were asked (unless directly relevant to the topic being discussed). Similarly, topics of discussion which came up during the Q and A session, but which were not tied to specific questions, have been placed under their own subject heading outside of the Q and A section
- If there are any sizeable gaps in comments from a speaker which resulted from asides, repetition, questions to others etc,, these are indicated by the use of “…”
- Timestamps are provided as guidance should anyone wish to hear the comments in full from any speaker on the video
- Questions /comments were made in chat while speakers were talking. This inevitably meant that replies to questions would lag well behind when they were originally asked. To provide context between questions and answers, questions in the transcript are given (in italics) at the point at which each is addressed by a member of the Firestorm team, either in voice or via chat.
Please note: This transcript is provided for informational purposes only. I am not an official member of the Firestorm team, and technical or support issues relating to Firestorm cannot easily be addressed through these pages. Such requests for assistance should be made through the in-world Firestorm Support groups or at the Firestorm support region.
The TL;DR Summary
The following is a brief summary of topics discussed. Timestamps in braces refer to times in the video where the relevant commentary can be heard. All sections are expanded upon in the main transcript – click on the timestamp to go to them.
- [0:0015] viewers are often subject to flase flagging by anti-virus programs as carrying a potential virus / Trojan. With the Firestorm 4.6.1, Norton anti-virus in particular had issues with viewer, prompting a positive response from Norton’s support
- [0:14:32] Mac issues update: work is being done on some Mac issues within the Lab, but there is no major project to address problems some users are having. Firestorm are somewhat stymied in dealing with issues due to both a lack of developers / developers with free time and because some of the issues are beyond their ability to resolve
- [0:31:00] Windows XP officially reaches its end-of-life on Aprial 8th, 2014. What does this mean for users on XP using Firestorm?
- [0:38:25] Even running a 32-bit viewer on a 64-bit OS yields stability improvements, although if you have a 64-bit version available, it’s obviously preferable to use that on a 64-bit OS
- [0:57:40] Firestorm are often critiqued on the frequency of releases. The team are moving to imporve things to a 3-monthly cycle, and there are reasons why a more frequent cycle may not be feasible
- [1:21:05] It remains that Firestorm will not offer nightly or weekly builds, because there are significant support issues
- [1:27:32] The team already try to release based on feature sets, however, a time-based cycle offers potentially better management of releases in keeping with the needs of the developers, QA and support
- [1:35:21] The target will therefore be a 3-monthly cycle of major releases, with possible interim releases with bug fixes or for special features, such as might be the case with the group ban functionality
- [1:58:53] With a target of a 3-monthly release cycle, it is probable that the next 2-3 releases are going to be primarily focused on incorporating features and capabilities coming out of the Lab, simply because there are so many of them: group bans, SSA updates, AIS v3, interest list, voice updates, etc.
- [2:01:55] The new download server has performed admirably with not craches or other issues.
- [1:55:40] Firestorm classes – with a new release just out, don’t forget there are Firestorm classes which cover all the new features, including things like the updated Contact Sets
- Questions and Answers: including information on clean installs / re-installs; using settings back-ups; troubleshhoting issues; the status of voice improvements; why group limits are unlikely to increase in the near future; helping Firestorm support, etc.
With thanks, as always, to North for the video.
Norton Anti-virus and False Positives
0:00:15 Jessica Lyon (JL): Anti-virus false positives is where we’re going to start today; specifically Norton this time around. Every release, we end up seeing from our users when they’re downloading that it’s Avast or it’s Norton or it’s MacAfee, or it’s this or it’s that [anti-]virus that is saying that it’s our website that’s malicious or the binaries that you’re downloading are malicious and sometimes, even as the user is downloading them, the anti-virus sends it straight to their virus vault so they can’t even access it. and it’s not just our binaries, sometimes it’s the binaries which are shipped with our binaries, which is slvoice.exe slplugin.exe …
0:01:13 JL: So, I came in here with the intent to rant, because this release is getting flagged by Norton anti-virus something fierce this time around. but a very unexpected thing happened last night. But before I talk about that, I want to talk about why I think – and believe me, it’s not just Firestorm that gets hit with these false positives; Linden Lab does, Singularity – all of the viewers do. and I’ll tell you what my theory is on that.
0:01:47 JL: If you look at how the heuristics of how a Trojan works, generally a Trojan hitch-hikes on the back of another application and connects to the Internet separately from that application. so if you look at the way the Second Life viewer works, you’ve got Firestorm.exe as the binary, but it also ships with two other executables within it that connect to the Internet as well, separate from the viewer, which are SLvoice and SLplugin. And I think – it’s just a theory – I think that why our viewers often get a false flag is because the heuristics of that kind-of look similar to how a Trojan works, and I think that’s why the false flags happen. And I call the false flags because trust me, Firestorm does not have a virus. Our website is not in any way malicious.
0:02:56 JL: Now I was going to do a big rant towards anti-viruses. We talked a little bit yesterday, after the third-party viewer meeting with Linden Lab, and Latif was there, a developer for Singularity, creator of Radegast client. And I just mentioned that I wanted to just punch Norton Anti-virus in the face, and he mentioned that Singularity has the same problems, and he made a suggestion. He said that something he tried was to flame and shame – it was Avast for them – flame and shame … the anti-virus people through Twitter. So last night, we sent a Tweet – it wasn’t rude [below]
0:04:10 JL: Within just a few minutes, we had a reply on Twitter from Norton support, giving us some suggestions and apologising for the trouble.
0:04:22 JL: And I was pretty surprised as to how quickly that came. And they suggested that the fill-out these forms … Now the problem with getting whitelisted is that we have eight binaries now. We have three Windows, 2 Mac and three Linux. And the way those forms work, is you have to attach the binary in question to the form. Which means you’ve got to fill the form out, I think in this case, you have to fill-out the form for each binary, and I’d have to do this every time we do a release for Norton alone. And then I’ve got to do it for Avast and God knows what other anti-virus programmes are out there. I have to go around to all of them and do this form. And on top of that, it takes time before they actually approve it, because they get lots of these, and they don’t have like a billion people to go through and verify that these things aren’t whitelisted. and it takes something like 2-3 months.
0:05:25 JL: By that time, we’ve got another release out, and we’ve got to go through the whole process again because they’re only white listing that binary. So once you have an update, you’ve got a new binary, and you’ve got to start the process all over again. And so the solution to it is just not really feasible. You’d have to have somebody who is just dedicated to doing that all the time; filling-out these forms and applying for white listing.
0:05:53 JL: So even if they do white list it right away, their customers, the people who use these anti-viruses, have to download the new definitions; so that white list has to wait until Norton or whoever has an update available for their users and those users have to go in and download the update, apply it, probably reboot their machine. So it’s a pretty ugly process and really frustrating.
0:06:21 JL: So I was prepared today to have a lot of nasty things to say about Norton this time, but it’s been Avast in the past, and other programmes. Anyway, so we politely flamed and shamed Norton on Twitter and we got a response pretty quick, and the suggestion was to fill-out these forms. And I had to step out for a few minutes, and when I came back, Ed had posted into one of the support chats that an avatar by the name of Norton Support had entered the region, this region, I kid you not.
0:07:04 JL: The Norton representative created a Second Life account, logged-in to Second Life for the first time, on Firestorm, never been in Second Life before, found this region, teleported here hoping he could find one of us. I kid you not. Oh my gosh!
0:07:25 JL: So Ed was like, “You need to get here. Now!” … and I thought maybe this is just a hoax; other people watch Twitter, they probably saw it. anyway, sure enough, it was Norton, in Second Life. Twitter works. And I have to give credit where credit is due, Norton is now on top of my respect-o-metre in regards to anti-virus programmes, even though I don’t use Norton. That kind of customer service is remarkable. Especially considering we don’t use Norton. We are not a customer of Norton. They didn’t need to do that, but the guy came in, he pointed me to the forms, I filled them out, because he was helping me, I only had to submit one binary, which was the one that’s been hit the most, which is our Windows 64-bit.
0:08:24 JL: He analysed himself, on the spot, he analysed all of our binaries to try to give us some tips on why they’re being flagged falsely, at least with Norton. Obviously, he can’t help with the other programs. And so then I filled-out the forms, he gave me his e-mail address, I e-mailed him the tracking numbers for those submissions and as of [Saturday March 15th] every future binary ever that we put out is not whitelisted. We will never, ever have another problem with Norton going forward.
0:09:00 Ed Merriman (EM): So you can say what you like about Norton anti-virus folks, but damn their support is good.
0;09:05 JL: That’s incredible. I was surprised they’d reply on Twitter. I had no idea they would actually come into Second Life. I mean this guy was a newbie. He actually took it upon himself to find our website get the viewer, log-in, all that stuff. Incredible. Really impressed.
0:09:30 JL: So that fixes the issue with Norton going forward. It does fix the issue with Avast, Bitdefender, AVG, and Kaspersky, and Avast … So at the very least you have my explanation as to why all viewers in Second Life frequently get false-flagged by anti-virus programs. I think it’s because of what the heuristics of these anti-virus software look for sort-of matches the way our viewers work with having other binaries that piggyback and connect to the Internet behind the scenes, like voice and SLplugin …
0:10:39 JL: I can’t speak unequivocally that no viewer in SL is going to contain a virus. But Firestorm won’t contain a virus, or Singularity … if we were to transmit a virus to our users … it wouldn’t be one person out of three hundred, everybody would be flagged, and it would be all over the Internet that Firestorm is transmitting a virus … If your anti-virus is flagging it and you look in the groups and you don’t see other people en masse saying it’s been flagged, it doesn’t have a virus.
0:11:30 JL: And we have a page, we have a wiki page on white listing. so basically, if your anti-virus is flagging us or other viewers, you’re going to have to go into your anti-virus program settings and find where it is you can add exceptions and add the viewer as an exception. And you may have to do that with each release as well, because the binary is named differently from one version to another. And I might also suggest that you complain to your anti-virus company … And I should add, if you’re on Norton, you are going to have to do an update to get the most recent definitions, whenever they release them, in order for Firestorm not to be flagged. The e-mail says they’ve added it to their definitions, but the definitions haven’t actually been released … they may put a whole bunch together and I don’t know how long that will take, and the e-mail had no suggestion of what kind of timeframe that would be.
0:13:29 JL: Anyway, Kudos, Norton Anti-virus. Good job Tim – that’s the name of the rep.
[VirSCAN.org is a good place to upload to check against the latest a/v signatures.]
Mac Issues and Firestorm Version Blocking
0:14:32 JL: What’s the future of Mac going forward since Cinder is no Longer with us? We do have Tonya, she’s a Mac developer, but Tonya’s availability is limited right now. But even if we still had Cinder, the issues that are plaguing Mac are more-or-less beyond our realm. most of them are related to Cocoa merge, were was after 4.4.2.
0:15:15 JL: We’ve brought this up many times now, almost every TPV meeting we have with Linden Lab we bring it up.
0:15:25 JL: So we’re going to be blocking version 4.4.0 in the next couple of weeks because of our stated rule that we’re only going to have three supported version on the grid at any given time. So 4.4.0 will get blocked and that will leave 4.4.2 as the next victim when we do our next release. And so Mac people who are plagued with these problems related to Cocoa, most of them are still using version 4.4.2 … it also doesn’t see Fitted Mesh and some other things… Materials. I don’t think it can see Materials, either.
0:16:08 JL: So some of the problems with Mac are, stability is a biggie, it’s crashing a lot. There’s a camera glitch with your ALT-camming, and various other problems. So we’re kind-of depending on Linden Lab to fix these, because they’re over our head and we don’t actually have a Mac developer who has the time to put into trying to fix these things. And the bad news is that all the viewers that have upgraded to Cocoa, and I think that’s all of them; I think Singularity has to, I could be wrong, I don’t know for sure. They all have the same problems.
0:17:01 JL: So Linden Lab has Mac developers, but their Mac developers are busy with other projects. I don’t think Linden Lab sees the Cocoa issues as being that bad. Oz [Linden] personally uses a Mac, he’s using the latest Mavericks operating system, and I think for most of the Mac issues, if you upgrade to Mavericks, are gone, except for the ALT-camming bug. I also understand that there are either people who cannot update their operating system or don’t want to for one reason or another.
0:17:44 JL: So I did some number crunching last night, and I took a look at our Google code page, which has all of our downloads prior to this one, and I looked at the 4.4.2 download numbers, how many downloads of Windows, Mac, Linux. It works out to about 9% of you use Macs. And that’s not a lot. And when it comes to prioritising with a big company like Linden Lab, they’re going to prioritise on the users they have the most of, which is Windows. So Mac is not going to be a huge priority. I wish I had better news for you guys.
0:18:35 JL: So I don’t know what else I can say to you guys, other than to file your bugs on Linden Lab’s JIRA. That JIRA is open now [but only for new bugs, unless the reporter of older bugs has opted to make their reports visible].
0:18:59 JL: This is one of those things that I think, that if it’s going to be fixed, Linden Lab need to see that there’s enough people having problems that justifies pulling a Mac developer off of whatever and focusing them on the Mac issues.
0:19:23 JL: How is [Firestorm] Mac development going with Cinder leaving? It’s not. Like I say, we’ve got Tonya, but Tonya’s availability is limited, she’s busy RL … As far as development goes, we don’t develop features specifically for Mac, features specifically for Windows, it’s more of a compatibility thing. We add-in a feature – we don’t even think about the operating system – we just add-in a feature and then we make sure that feature works on Windows, Mac and Linux. And so it’s not a matter of Mac development, it’s a matter of Mac compatibility with Firestorm. And as far as that’s concerned, we’re in sync with Linden Lab. All of our features work on Mac, aside from the bugs that you guys are suffering.
0:20:22 JL: I’ve noticed with this release, looking at statistics, I think it was less than about 5% of Mac people are downloading 4.6.1 – that’s even just downloading it to try it, less than 5% is actually trying 4.6.1. Probably most of you may be going back to 4.4.2 again because of the bugs. I just don’t know what to say to you guys. It’s a small percentage on Mac, which makes Mac not a priority with a big development company. So fingers crossed that things will get better. I’m sure they will.
0:21:10 JL: I did state during the [TPV Developer] meeting with Oz that we’re not going to block 4.4.2 until these major Cocoa issues are addressed. The unfortunate thing about that is doesn’t really solve much for you Mac folks, because you’re still not going to see Fitted Mesh, which I think will become popular over time, you’re not going to see Materials and whatever else is in the pipeline in the future. It’s not going to be an ideal thing, but I can tell you we’re not going to block 4.4.2 until the major issues with Mac are dealt with.
0:21:56 Lette Ponnier (LP): And also just to underline, it is not 4.4.2 that is being blocked at all right now, it is 4.4.0 [which is next to be blocked], which doesn’t actually have a lot of users anymore because 4.4.2 is the same but better. So 4.4.0.
0:22:18 JL: And I’m not promising when we’re going to that. I’ll say that if you’re using 4.4.0, I’ll say “why?” Because 4.6.1 is actually pretty good, 4.5.1 is pretty good … so not many people on 4.4.0, if you are on 4.4.0, get off, update to something newer. If you’re Mac, you may want to say with 4.4.2, we aren’t going to block 4.4.2 … until the major Cocoa issues are fixed.
What about Mac 64-bit?
0:23:49 JL: That was in progress, Cinder was working on that, she came across a few roadblocks. The challenge with making a 64-bit viewer is that you have to recompile all the libraries in 64-bit. Boost is a library that any developer working in Second Life hates with capital letters: HATES. It can be a real pain-in-the-butt. Cinder ran into some roadblocks with Boost and she wasn’t able to get past those roadblocks; we’re still hoping in the future we can get past them.
0:24:40 JL: Tonya is our only current Mac developer, and I’m sounding like a broken record, but she doesn’t have the availability she would like to have to continue on that Mac 64-bit path that we were hoping we would get to eventually. I will tell you that I’ll never say we’re never going to do Mac 64-bit anymore; it’s still on our planning board, it’s still something we absolutely want to do. But it’s not going to happen until we can get more availability from Mac developers or something like that.
0:25:19 JL: Linden Lab is very interested in 64-bit, by the way. They’ve been watching statistics and Singularity has 64-bit [Windows and Linux], we have it now; they’ve been watching both viewers very closely. They are very interested in it, Oz is pushing to get 64-bit support through Linden Lab. Vivox is apparently working on 64-bit voice, which Oz said Linden Lab would share with us even if Linden Lab doesn’t have 64-bit yet. They’ll share with us, so we’ll be able to have 64-bit voice for the Windows version, anyway … probably not Linux; Vivox isn’t putting much effort into Linux.
0:26:36 JL: So Linden Lab is interested in 64-bit and there’s a fair probably that Mac 64-bit won’t become available until Linden Lab does it unless things change with the open-source third-party viewers. Maybe somebody on Singularity will be able to do it, and maybe they’ll share those libraries with us, or maybe some other viewer; maybe they’ll have a Mac developer who can do that. Maybe Katharine Berry will be able to do it; she’s in university, I understand and busy with exams. but if there’s a person who can do that, Katharine Berry would be it, and Kat works on Exodus, for those of you who don’t know her. She’s a genius.
So don’t give up on it, it’s still going come, but it’s not going to come from us any time in the immediate future.
The 64 bit version seems really good except my frame rate is lower compared to the 32 bit version
0:27:34 JL: Some people report that, some people report higher frame rates, I think it depends on various things. Ansariel actually spotted something the other day in 64-bit through fast timers which might explain why some people might have a lower frame rate in windows 64-bit. I don’t know if she was able to look any further into that.
Ansariel Hiller (AH) via chat: Nicky had to disable some optimizations because of crashes and endless loops in 64-bit.
Gibson Firehawk, via chat: Nicky said on a JIRA ticket that it probably had to do with compiler flags she had to change for 64bit in certain parts of the viewer.
Question about voice user for Mac, you can’t do the same things as with Windows, replace the Vivox files with older versions? I always need to relog when I change my headset, and I’m not happy when I need to do that.
0:28:46 JL: Linden Lab has a release candidate with a new version of Vivox. We’ll probably make it available in the next week or two; we’ll make a wiki page for folks who want to try the new one. But Linden Lab hasn’t released it yet, so we’re not going to release it because it’s still being tested. Maybe the new issue will address what you’re talking about.
0:30:03 JL: If there’s one suggestion I could make to you Mac folks, it’s if you can update your operating system, update your operating system. Don’t be on 10.6 or 10.7. The older the operating system , the more issues you’re going to have.
0:31:00 JL: So I don’t know what the actual numbers are of users on Windows XP. I know that there’s enough of them on Windows XP 64-bit to complain that our Windows 64-bit doesn’t work for them. I’ll get back to that in just a moment.
0:31:33 JL: However many people there are on Windows XP, there’s not many of you, it’s going to be a pretty small number. But even small number, a small percentage in relationship to the amount of users we have still translates to quite a few people. And XP people are kind-of loud. You make yourselves heard when things don’t go your way.
0:32:05 JL: For those of you who don’t know, Windows XP support [from Microsoft] officially ends in April … And I do get that some people have incredible stability on Windows XP. I have not stats that I can look at myself, but Linden Lab assure us that by far, people on XP have the highest crash rate of any operating system.
0:32:41 EM: And I have something to say about that. I’ve had my alt on in excess of 200 hours more than once. so that’s a week at a time. So say what you want about the stability, it’s not true for everybody.
0:33:16 JL: So Microsoft is no longer supporting Windows XP. Are we going block Windows XP? Some of you may have noticed in the release notes the possibility of blocking Windows XP. I thought that might get your attention. If Linden Lab blocks XP, we’re going to block XP. But Linden Lab have said they’ve got no real interest in blocking Windows XP. They’re just going to let you guys suffer. The installer will still probably work for you, but it doesn’t mean the viewer is going to run well, it doesn’t mean that you’re not going to crash.
0:33:54 JL: By the way, if you’re running Firestorm on Windows XP and you’re crashing, you are hurting our crash rate. I don’t appreciate that, just sayin’!
0:34:09 JL: So we’re not going to block Windows XP unless Linden Lab does. We’re going to follow their suit in terms of what operating system they support and what they don’t, and so that way we can blame them, instead of being blamed.
0:35:09 JL: Firestorm 64-bit does not run on XP 64-bit. I am to blame, not Tank[master] Finesmith. Tank was pushing very hard that Windows XP 64-bit should be able to run Firestorm 64-bit. I wasn’t. And I’ll tell you why. You’re on a very old operating system. Please don’t expect new shiny to be delivered to you on an old, old, old operating system. I mean, how old is Windows XP? … Thirteen years? Upgrade your computer, guys. I know formatting and all that is a pain, but you can’t tell me that Windows XP is working that good for you.
0:36:37 JL: The Firestorm 32-bit will work on XP 64-bit. Always has, always will, unless Linden Lab decide to prevent that through the installer. Which is a possibility. It could happen. If you want to know for sure without a shadow of a doubt that you’ll always be able to use Firestorm, upgrade your operating system and you’ll never have to worry about it.
0:37:19 Tankmaster Finesmith (TF): I was just going to say Linden Lab have stated that of all the operating systems that they support, XP is by far the least stable of all of them, even [compare with] Mac and Linux. So updating really would help most people.
0:37:43 JL: I’ll tell you want else to. Don’t you dare come into our support groups or our blog post comments saying “Firestorm is really crashy and unstable for me and it sucks, and it’s useless and nobody should use it” if you’re running Windows XP. If you’re running Windows XP and saying all that stuff, you’re hurting us – and it’s your fault … please don’t do that.
0:46:06 JL: Here’s a link to the Third-party Viewer Meeting recording [at 24:42 and 28:00 in the recording – written notes here], and if you’re interested in Linden Lab’s point of view on windows XP, it’s there for those of you who want to look.
The Viewer and 64-bit Operating Systems
0:38:25 TF: Another thing that they say is that just using a 64-bit operating system even with a 32-bit viewer does offer significantly lower crash rates compared to the same version of Windows in 32-bit.
0:38:46 JL: Yeah, that’s interesting. Oz mentioned that as well. Just in our preliminary crash results from before we released 4.6.1, when it was in a preview group. Enough people in the preview group were on it that it showed-up in last week’s stats and the 64-bit has a significantly lower crash rate, really low. Although with the small number of people that was being counted, we expect that to go up. But it was significant. And Oz was suggesting that it may not be because the 64-bit version itself is stable, it’s that 64-bit operating system are stable. And now that we have a 64-bit version of Firestorm, people are downloading it, so we’re starting to see stats that are coming in that are separate from the typical Windows stats. Always before it was just “Windows”, how stable is Windows? But now we’ve got two versions, we’re setting two sets of numbers. We’re seeing Windows, and we’re seeing 64-bit.
0:40:04 JL: So it isn’t actually that the 64-bit Firestorm is more stable. It’s more a matter that 64-bit operating systems are more tolerant to things that would normally crash the viewer. So for example, all the viewers have memory leaks and whatnot, and we know the 32-bit has an out of memory crash, that at some point, you’re doing something and suddenly for some unknown reason the viewer requests more memory than the operating system can offer, and it goes Boom! It’s gone.
0:40:44 JL: With a 64-bit operating system and 64-bit Firestorm, that bug still exists, but you don’t actually run out of memory. so it’s not that we fixed the bug in 64-bit, it’s just that it’s more tolerant and won’t necessarily reach that crash point. And so it seems more stable, and for all intents and purposes it is more stable, it’s not that it’s any different. It’s just that 64-bit is more tolerant. And 64-bit operating systems are more tolerant to the memory crashes that the viewers have.
0:41:27 JL: So you can use 32-bit viewers on a 64-bit operating system, but I would suggest that if you have a 64-bit operating system, use the 64-bit viewer. Unless you need Havok for pathfinding or mesh with physics.
Firestorm Release Cycles
0:57:40 JL: This is a sensitive topic within our team. There are two sides to this, and I’m going to try to be fair to both sides.
0:58:17 JL: We do get people that complain that our release times are too long. And for those of you who don’t know, we’re averaging about four months between releases. That’s what a release cycle is, it’s the time between each update. We’re averaging every four months. And a lot of people are complaining, what’s the matter with us, why would we want to do release cycles so far apart…
0:58:53 JL: I can tell you unequivocally, no matter what side we’re on in the team – there’s a side that would like to do a release every week, and there’s a side that would like to do releases less frequently than every week. But I can tell you that all of the sides agree that every four months is too long. Our release cycle of every four months is not by choice. We’re not aiming to do a release every four months. It would be nice to do them – my side of the fence is every two months; if we can do a release every two months, that is the happy medium between four months and no months.
0:59:38 JL: And I’ll tell you why. And I’ll tell you that Ansariel, and there are people like Ansariel, wants to do a release every week or two … So let me make it clear: none of us are happy that our releases are taking four months. None of us; not even our support team. nobody is happy that releases are taking four months. It’s is too long, we all agree … so if you’re one of those people who have been complaining, please don’t come to us complaining why we’re choosing to do a release every four months. We’re not. It’s just working out that way. and it’s interesting that it happens to work out almost exactly four months every time. I don’t know how that happens.
1:00:28 JL: So there’s only so much we can do. Yeah, we’ve got some 80 people on the team. probably ten or twelve of them are developers. Of those developers, how many do you think are actually developing? If we had all twelve of our developers developing, we could be doing releases really quickly.
1:00:59 JL: We’ve got Ansariel, who does a ridiculous amount of work, just look at our repository. We’ve got Tank, who handles merges and compatibility issues and things like that. We’ve got Nicky, who handles server stuff, crash fixes, major, major issues that are too complicated for most of us, we’ve got Zi Ree who does some work, we’ve got Tonya who does some work; Tonya’s not been available, Zi Ree’s not been that available. And Tech, and Cinder was doing tonnes. We don’t have cinder now.
1:01:49 JL: How much do you think we can get done with so few developers, in a period of time? So I’ll tell you what happens. When we put out a release, generally the devs take a little bit of a break. They’ve worked hard and they deserve a break … so that’s two or three weeks where nothing is happening; we’re not getting anything done.
1:02:23 JL: During those 2-3 weeks our support team is hammered, crazy busy, going out of their minds, rage quitting because they’re burnt out, they can’t handle it any more – keep in mind support have to deal with all the people who are pissed off and feel entitled and all these different things … I was called some pretty nasty names in an offline because we didn’t provide the opportunity to install into a different directory for 64-bit. Which, by the way, yes we did. You’ve just got to look under the little button that says “Options”, because installing it into a different directory would be an option.
1:03:13 JL: Anyway. Support deals with people like this all the time. And it’s really bad when we do a release. Really, really bad. It’s when support gets hammered and we lose people on support because they just can’t take it anymore. Can you imagine if we did a release every two weeks? Or month?
1:03:37 EM: Every two weeks, and you wouldn’t have a support team.
1:03:40 JL: Exactly. And that’s my point. So, I have to balance out, we have three main departments on the team. We’ve got developers, we’ve got QA and we’ve got support. and all three of these departments have a different idea of what would work best for them, and my job is to choose one that works best for everybody but nobody’s actually fully happy with.
1:05:11 JL: If we did a monthly release, developers are going to get three weeks to work, QA is going to get one week to test. Is that going to be enough? … No … And that’s assuming QA find nothing wrong with the three weeks of work that the devs did, but I can assure you they will. so what happens is that in that one week of testing, they find these issues, and then we’ve got to get the developers to go in and fix the bugs. Let’s say that takes a week. Then another build has to be sent to QA, and they have to test those fixes. And quite often they find problems with those fixes, so then the next week the devs have to do some more work, and then QA has to do … you guys starting to see the problematics between development and QA? It gets complicated, it gets very difficult to get a release out without bugs. and I’ll get back to the fact that I’m really picky and want to do perfect releases, because that’s going to come up, and it’s certainly a valid point against me.
1:06:58 JL: So there’s the fact that there’s development time needed, and then there’s QA time, and as far as development time goes, with the few devs that we have who are active, we’re not going to get much done. How much do you guys like the idea of having updates, say, once a month with very few changes? How many of you would update if you didn’t really have anything cool to update to? We’re talking about maybe a feature and some fixes. Kind-of like an Agile process, little changes. Because shorter release cycles means you get less bang for your buck to download and install.
1:08:14 JL: This is a complicated topic because there’s a lot of different sides and a lot of different valid opinions that go against each other, and I’m certainly not going to say I’m right with my opinion.
1:08:44 JL: Let me give you Ansariel’s argument, which is a valid argument for more frequent releases: with less work done, there is less to test, and less impact on the users, and therefore less support. And that’s a valid argument … I don’t know if it’s true or not; to be honest, we’ve not tried it. I personally disagree to an extent, but I can’t say it from experience … Let me ask support, if we did monthly releases, how would they feel about that?
1:09:35 EM: They’d be burnt-out within two releases. When it comes to release time, we enter what I call the two weeks of hell. We have two weeks of total insanity, and it really is nuts … Let me ask you this, folks. How would you like to answer the same question 16 times in a hour, and they deal with someone who is really upset because there’s a bug affecting them terribly … so support goes through two weeks of hell after a release.
1:10:39 EM: People complain when we send them to the wiki, which has got all the answers; we get complaints every week, “Why do your support staff just send me a link to the wiki?” I’m sorry, folks, with the chat lag in the groups, we throw the links out, you can read it there; if we were to post it into the group, it would be a wall of text and it might not go through.
1:11:13 JL: So let me ask you this. Ed and Lette are our two main support folks who manage the rest of the support team, which is massive. How do you feel about a release every four months, currently what we’re ending-up averaging… I think you guys think every four months is too long as well. Am I right?
1:11:55 EM: Yes and no. It’s a lot of work leading up. For those who do the documentation, there’s a lot of work to be done, and we can’t do it until we’ve got a final build because we don’t want to go back changing everything over and over again.
1:12:16 JL: But again .. Ansariel’s argument, and the argument on that side of the fence, that if we do more frequent, then there’s less documentation changes that you have to do.
1:12:27 EM: That would cut down on the documentation. The other side of me would like more frequent releases, simply because it would be less work that way and less changes and fewer questions from the users. The problem being, every release we get, “Oh, what’s the piece of crap you’ve put out? The last version worked so much better!” Then the next person comes in and says, “Oh my god! this is absolutely wonderful!” How many of you remember the blocked release [4.4.1, June 2013] that was out for a day? Remember that? There were two changes [with the 4.4.2 release]. A bug fix and a debug setting switched off.
1:13:29 JL: And I’ll tell you what, those two things have no impact on anything.
1:13:35 EM: People said [of] 4.4.1, “oh it’s great!”. They get 4.4.2, and they say, “What’d you do?! It’s a piece of crap!” It was the same viewer, but going from one release to the other, “this one’s a piece of crap!” or “That one was a piece of crap, this one’s great!”
1:14:10 LP: The clean install topic is one of the reasons … we could probably be more frequent than every four months, but it’s a reason why a super-fast release cycle would create chaos. Because folks would come in, they’d have a problem that could have been caused by an unclean install and might be fixed by a clean re-install, but if they don’t know what that entails, and honestly, many people who think they’re doing clean installs actually aren’t. So they have this issue, and they think they did a clean install, they decide to roll-back to the previous version and after they do that, they still find that they still have the problem and they’re very upset.
1:15:04 LP: If we had a faster release cycle, people would probably be bouncing back and forth between different versions a whole lot more, trying to figure out which one works best for them because they’re experiencing a problem and not realising that their issue was largely caused by the way that they’re doing the installs. So we’ve got settings back-up now, which ought to make clean installs a lot easier and faster, but we still see these issues come up. So that would be one of my main concerns.
1:15:39 LP: Another one is trying to keep track of the versions. Firestorm is already a very complex and versatile viewer. You can make it work in so many different ways. you’ve got different log-in modes, you’ve got skins that put things in different places, and some don’t have the same things … We try our hardest to keep track of all those different things, and then when we’ve got different version and we’re trying to remember which bugs exist in which version so that we can tell people, “oh, you should be on this one, it’s a bug caused by the one that you’re on,” and so forth; it’s just a lot to keep track of. And we’re not going to be able to provide the best support we can possible give if we have more that we are actually responsible for remembering in that area.
1:16:47 JL: So, on the side of the fence that feels we can do much more frequent releases because there’s less work going into it, there’s fewer changes that need to be tested with less impact to the end user. With Chrome, Firefox, QuickTime updates, Flash updates, that is true. I mean, do you even notice when Chrome makes an update? It’s bam! It’s done, You just restart it, you don’t even think about it … when it comes to applications like that, that’s a different story. Second Life viewer updates are far more complicated than they should be … they should not be as complicated and varied as they are. Changing that, though, is quite non-trivial and I don’t know how updating your Second Life viewer could be make easier … like updating your Chrome or Firefox or whatever it is. But it isn’t. It’s complicated, it’s convoluted, it’s a royal pain in the butt for the user to do.
1:18:17 JL: Because it’s so difficult and such an awkward process, people get the placebo effect that the previous version was better than the current version or vice-versa; that whole example that Ed explained about 4.4.1 is a perfect example of placebo effect … you only thought it was different, it really wasn’t different. So every month just isn’t reasonable and it isn’t really fair to support. Every four months is not reasonable to developers, because developers want to get their stuff out sooner. Son my job in everything we do, is to try to find some comfortable middle ground.
1:19:08 JL: I would like to do a release every two months … That’s what I would like us to be able to put out; I think that’s fair. Fair for support, fair for devs, fair for QA, and fair to the users, because updating is a pain in the butt for you to. So I think every two months.
1:19:32 JL: Believe it or not, we’ve been aiming for that for a year now, and we still manage to hit an update every four months. Why is that? Well, I don’t have pay cheques to give QA, I don’t have pay cheques to give beta testers and I don’t have pay cheques to give to our developers and I don’t have pay cheques to give to our support people. we are all volunteers, and as I say, after a release, developers want to take a break and, generally speaking, they deserve to take a break, they’ve worked hard; especially with the few that we have that are active. But that break that they’re taking means that nothing is getting done.
1:20:16 JL: Now if this was a paid business, and everyone was being paid, you don’t get to take a break. You get a vacation every year. But if you’re making 60-80 grand a year, you’re expected to be working from eight in the morning until five at night and get your job done and have results. Frequently. And lots of them. And if we ran in a model like that, they we could do like Linden Lab does, have a new release out every [two] week[s], and we could force our users to update to that release.
Firestorm Nightly Build Offerings
1:21:05 JL: So … Public nightlies … The problem with that. Singularity does that. I think they have weeklies, don’t they? Weekly alphas? Singularity doesn’t have a support team. Linden Lab doesn’t do that, and if they did, Linden Lab has a support system in which the only way you can really get a hold of them is through support tickets, and support tickets can be quietly ignored. What other viewers do nightlies or weeklies? I can assure you that whoever they are, they don’t have a dedicated support team. We are the only third-party viewer in Second Life that provides live support, in group, unfiltered, ad-lib, we don’t know who’s coming next, we don’t know who’s going to ask what question. And that puts us in a different situation to any other kind of viewer project in Second Life. Because while we can say, “here’s a nightly, it’s completely unsupported, please don’t come to our support team asking for help with it, because we can’t support it, because it’s an alpha. we’re not going to answer your questions.”
1:22:35 JL: We can say that until we’re blue in the face. we can have bars across the screen when they’re running that viewer, which say, “Do not come and ask us questions or for help, because this is a nightly, it’s unsupported, it’s probably broken” – and it would be probably broken, I can assure you, if it’s nightly – do you think that’s going to stop people coming for help? I can assure you it won’t.
1:23:20 JL: So in our situation, we can’t really do public nightlies. We actually do nightlies internally. and so today Ansariel will commit some code that she’s been working on – speaking hypothetically -but this does happen all the time. A developer will commit half the work – because it’s not finished – but she wants to get it into the repository so that other people don’t write over it, for example. but it’s not finished. And if you go and download that nightly, you’ll have some floater panel that has some really cool-sounding feature, but it doesn’t actually work. and if that’s a public nightly, people are going to download it and say, “oh, what’s this really cool feature….oh, it doesn’t work… I’ve got to ask somebody why it doesn’t work. Where can I go? Who can I ask? Oh, I know, Firestorm has this public support group anybody can go in and ask!” And they’re going to come in and ask, and I can guarantee you that support are going to say, “What Are you talking about? … Where is that in the viewer?” And they’ll say, “It’s under this menu”, and support will say, “Well, I don’t have that in my viewer. Where are you seeing that?” And they’ll end up spending ten or fifteen minutes trying to work out what the hell you’re talking about before they realise you’re on an alpha that is not supported.
1:25:00 JL: And you have all the good intentions. You don’t mean to come in and cause headaches for support. you just have this question about this feature. You’ve got nobody to ask if you’re on a nightly. Don’t ask. But people will ask, and that’s why we don’t do nightlies. We just can’t support them, and saying we can’t support them does not stop people from coming in en masse and asking questions.
1:25:42 JL: Nightlies in the beta group. Beta is not an unmoderated group. Out QA runs it strictly, and rightly so. And they do a fantastic job, because more-or-less, we want beta testers to be testing specific things. If we just give them builds, and don’t get them to test anything specific.
1:26:20 JL: Let’s say for example, we’ve got half-finished code in the viewer and we give the beta testers a nightly. The beta testers are going to go in and say, “what’s this?” And they’re going to ask in group … and probably, QA won’t have a clue, they won’t know the answer, and all the beta testers are going to come in … all asking the same question to which we have no answer. You can imagine how frustrating that can become. It’s kind-of the same situation. Our model just doesn’t work that way. I’m not saying it’s the right model or the best model … but it’s just not going to work for public nightlies or even weeklies.
Releases Based on Features?
1:27:32 JL: Instead of doing releases every four months, what about doing releases based on the number of changes that get done in the viewer, once we hit a threshold of new stuff. that should be how we determine whether we do a release or not. Actually, to be honest, that is kind-of what we actually do. It just turns out to be every four months! It should be quicker.
1:28:06 JL: And how do we measure new features? For example, let’s use the SLShare really cool feature. There are some people in this audience who feel that is absolutely worth a release. And there are people in this audience who absolutely believe that it is completely and utterly useless to them. SLShare, by the way is the ability to link your SL avatar to your Facebook account…
1:26:45 JL: So how do we weigh that? How do we weigh what is and isn’t worth a release? It’s not even much of a question. We pretty much know what most people want in a release and we can more-or-less weigh-out what’s worth releasing. What we had a month ago was worth releasing. But we wanted to wait for HTTP, because that is really good, and had we not waited for it, and had a release without HTTP and then HTTP would have been out, and everybody wants HTTP. And then something, and something else, and something else.
1:29:23 JL: Why does it take us so long to keep up with Linden Lab? I’ll tell you why, in part. Because Linden Lab works on these things in secret. when they start to get a little bit closer, we might found out they’re working on it, and we might find out a little bit about it. But we don’t see the code, we don’t know what’s in the code, we don’t know how much code there is going to be. Linden Lab only makes the code available to us right before they put it out to release candidate, more-or-less. Sometime sooner, sometimes later. but we don’t see that code until Linden Lab is ready to release it into the wild.
1:30:07 JL: That doesn’t give us a whole lot of time to get it into our viewer and test it, and work out some of the bugs. no offence to Linden Lab, but Linden Lab … has a different threshold on what they think is ready for release versus what we think is ready for release. and that’s nothing against Linden Lab; Linden Lab has fewer users than we do, Linden Lab doesn’t have the kind of live support that we have. so they can release with big bugs and it won’t impact them as much as it will if we release with big bugs. We release with a little bug that affects one percent of the user base and that’s more than 10,000 people. We have to think about that.
1:31:01 JL: And so I’m really strict about what passes QA. And I can assure you that Takoda and Kate are even more strict on what passes QA. I have to supersede them sometimes and say, “OK that’s an annoying but, but we’re going to have to release with it.” And so again I have to find the comfortable middle ground that nobody is happy with, but they’re a little bit more happy than if I went to one extreme or the other.
1:31:38 JL: So QA is really important, we can’t just push out stuff untested … so that’s where that is.
1:32:42 EM: There’s been a suggestion made that we could do a full release every two to three months, and throw out a bug fix in between … no new features, just bug fixes on the in-between … that’s more work for the devs, though.
1:33:19 JL: It look like most people are OK with every three months.
1:33:24 EM: If it was bug fixes plus the regular stuff going on, they’d have to have two separate repro going on, one for the bug fixes … it’s a great suggestion, but do we have the manpower to do it?
1:33:43 JL: Well, the interest. Again, our devs aren’t being paid; I can’t tell them, “do this, or you ain’t getting paid…” I have to entice the developers with catnip or small mammals if they’re werewolves, depending on what creature they happen to be.
1:34:21 JL: the point I’m trying to get to is that every four months is definitely too long … and we are aiming for every two months. And if we aim for two months, we’ll probably manage to get a release out every three months. So that gives our devs two months, well maybe a month and a half and it give QA about a month … I think that’s fair. and support deals with a month and a half of hell, another month of recovery, and then preparing for another release. I think three months is suitable.
1:35:21 JL: So I’m going to tell you on the record, I try not to make statements that I regret. We are going to try to do a release every three months. That is going to be our target. We may have little minor releases in between, even.
1:35:38 JL: So, for example, we’re working a little bit with Linden Lab on group bans, the ability to moderate who can enter an open enrolment group so you can get rid of spammers and stuff, permanently without closing your group. So this is a really big feature, and it’s one of those features that everyone is going to kind-of need to adopt at the same time or thereabouts. Linden Lab is trying to get that done as soon as possible and released, and when they do, our plan is, providing it suits our quality standards and works the way we feel it should work, which is just to say that it’s useful, and not circumventable, we’ll issue an update which will probably be 4.6.2, that will have 4.6.1 code plus group ban code, and maybe a couple of little blocker fixes that came up in the 4.6.1 release.
Likely Features for Upcoming Releases
1:58:53 JL: If we’re going to succeed in a release every three months, say in the next year, we’re going to have to focus most of our efforts on merging-in the labyrinth of new stuff that Linden Lab is pushing out. And it’s all going to start coming out at the same time, most likely. And a lot of it is unstable still, even according to Linden Lab. So probably our focus for the next 6-8 months is just going to be trying to keep up with Linden Lab’s things.
1:59:30 JL: We’re talking the second part of the sunshine work, the avatar appearance stuff. We’re talking the interest list stuff, which is huge. We’re talking group bans, which is epically awesome as a feature. So don’t count on a lot of our own new features, unfortunately. I’m sure we’ll have some on the planning board, I’m just not sure we’re going to be able to get any included.
The Firestorm Download Server
2:01:55 JL: I am, you may have noticed, pretty happy today. And it’s not just because I think 4.6.1 has been relatively a good release aside from some very loud minority, it’s been a really solid release, I’m really proud of the team. My biggest concerned about this release was the download server, and it performed flawlessly. Absolutely beautifully. It was under a pretty heavy load for a while, 600Mbps, something like that. It was taxed for a while, but considering it’s not actually a download server … it’s not built specifically to send files. I didn’t actually get an e-mail from Hetzner, who we’re getting it from, saying, WTF are you doing to our server? Which I kind-of expected, as well.
[At the time of the meeting there had been over 102, 500 downloads from the server, with over 50% of them the windows 64-bit version, with the windows 32-bit version running a “close second”, with Mac at approximately 9% and Linux some 2-3%.]
1:55:40 Ed issues his usual encouragement for people to attend Firestorm classes, which are held at the Phoenix Firestorm Support region classroom and at Junkyard University. Class times are scheduled throughout the week at different times to meet the needs of different time zones, and recordings of classes can also be obtained on the Firestorm YouTube channel. These include classes on new and updated features, such as the revised Contact Sets in Firestorm 4.6.1. There is also the help area on the Support region where assistance can be sought.
Questions and Answers
Note: some of these questions below were asked early in the meeting, when the Firestorm team were running through announcements and updates. They have been moved here for ease-of-reference.
What is the problem with “Flash Player” not working on media viewers?
TF via chat: The web kit LL uses is ancient, it’s no longer supported by YouTube.
What is going on with the 64bit Havok libs?
0:26:10 JL: Havok libraries are something again we would need from Linden Lab. And they’re not going to do 64-bit Havok libraries until they’re doing 64-bit support. So it’ll come, but we can’t do anything with Havok; we’ve got the stub but were can’t do anything with it or recompile it. So that’s relying on Linden Lab.
What about the little message on the lower right where it says its deleting textures?
[This issue is common referred to as “texture thrashing” and is related to physical graphics memory. The Firestorm team have a troubleshooting page on their wiki, and I have an explanation on the usual causes here.]
0:43:10 JL: Basically it’s the viewer telling you you’re about to crash. Usually crashes happen without you expecting them. Usually the viewer doesn’t know it’s going to crash. When you get that message it means it knows it’s going to crash and it’s letting you know it’s about to crash, and then it crashes … Just like we can’t prevent other crashes, fixing this crash is really difficult, and by the way, Linden Lab has it to.
0:43:45 LP: But I believe the message in the corner is specific to us. Sometimes it makes people think the message is causing the crash, that something because of the message makes the crash more noticeable. But the purpose of the message is to warn you. So if you want to re-log before you actually crash, at least you’ll be able to save your settings and anything you’ve changed. Sometimes the message will come up and you don’t crash. That’s kind-of lucky when that happens. And if I’m not mistaken enabling texture compression help folks who have that problem chronically, although for some people that can degrade texture quality.
0:44:55 JL: So it boils down to if we didn’t have that message in there, people would just think they’re crashing like they ever crash. We added the message, and people think the message is causing the crash. Can you imagine if we added a message prior to every crash?
0:45:15 LP: Actually, that would be informative.
0:45:19 JL: It would, but we’d end up with this kind of reaction again, times a lot.
0:45:26 LP: But the message also gives clues as to what might be done in order to try to curtail the issue. It allows us to have that wiki page with the troubleshooting so that people know what they may be able to try. Troubleshooting of this type is not generally guaranteed, but you can at least try it.
Is the length of time you’re logged in a contributing factor in you possibly crashing?
0:46:54 JL: The viewer is not very good at managing memory., and that’s because the viewer has bugs in regard to memory management, and the longer you’re logged-in the more likelihood you’re going to crash. But then the same thing is true about when you’re driving a car. The more time you spend driving a car, the more likely you’re going to be in an accident. It’s just that the viewer’s not very good at managing memory; it’s very difficult to fix memory bugs, it’s very difficult to trace them because they’re often not reproducible. You can’t reliably reproduce it, you can’t do just one thing and you know that it’s going to reproduce the crash and you can watch to see what happens and then fix it. Memory issues are quite a bit more complicated than that, and they stem from the fact that the SL viewer is a very complicated layered mess of code.
So basically, the more memory you have the less chance of a crash?
0:48:41 JL: Yes, unless you’re on a 32-bit operating system, in which case having 20GB of RAM is not going to help you because it only uses 3-point-something.
How long is to long staying logged in?
0:48:57 EM: That really depends on your system and your set-up.
0:49:05 JL: And what you’re doing…. [Your] graphics card makes a huge difference in SL.
0:49:34 JL: Actually, let me address something else. We have a blog comment … along the lines of, I’m paraphrasing, but we hear this frequently: my computer is perfectly fine, it’s good computer, I can run every game on high settings, I can watch videos on YouTube, Firestorm doesn’t work very well, so you need to fix it. We get that a lot. Let me tell you something. SL cannot be compared to any kind of video game, at all. Ever. You cannot compare it to any kind of video game. I don’t care how well your computer plays Crisis or Pacman or any other kinds of games that are out there. I don’t care how well it plays them. Second Life is impossibly different to a video game.
0:50:41 JL: Video games – when you launch it, it’s loading all the graphics into memory, because it knows what you’re going to see when you go around that corner in the game, it already knows what’s going to be there, so it loads it into memory ahead of time.
0:50:54 JL: In Second Life, it knows nothing. The viewer has no idea what it’s going to see when it logs-in. So when it logs-in, it downloads everything over the Internet. Every step you take that renders-in to your draw distance, it downloads that, and then has to display it graphically in the viewer. And it’s dynamic, it’s constantly changing. You cannot compare Second Life to anything else that I can think of anywhere. I don’t care how well your computer plays some video game, it doesn’t mean it’s going to play second Life well.
But once it is loaded into memory, then?
0:51:43 EM: Once it’s loaded into memory, it’s in your memory, but when you go back around the corner it still has to look, because it might be gone by then. And on top of that, what a lot of people don’t seem to realise is not only can your hardware make a big difference as well as the settings in your computer and the settings in the viewer itself. But other software on your computer can interfere and cause all sorts of strange bugs, even crashes. So even having one simple little piece of software that you use once a week on your machine can cause problems.
0:52:33 EM: Skydrive was a good example. There’s several examples … just a simple little bug, but when you were building something and when you deleted something – I can’t remember if it was through the floater or using the keystroke command – it would turn off the AZERTY keyboard, and it would require a reboot of the computer to get the AZERTY keyboard back. So just because you’ve got the latest and great computer doesn’t mean the viewer is the problem.
0:53:22 JL: Another thing I often see: Firestorm was working fine yesterday, and it’s just getting worse and worse every day. What are you people doing wrong? I’ve got news for you. We don’t have the power to upgrade or change the viewer that’s installed on your computer remotely. We don’t do that. If Firestorm has changed from yesterday to today and you haven’t installed a new version of Firestorm, something has changed on your computer that has changed the performance of the viewer. Don’t come to us, we can’t help you; it’s something you did on your computer. And it’s not necessarily mean you did something intentionally. It’s doesn’t mean you downloaded something. Maybe your computer did a defrag last night and it’s changed things … there’s a zillion different things that could be happening in your computer that can change the way anything performs. We can’t make changes to the viewer while it’s installed on your computer. we can’t do that.
0:54:32 LP: The other two factors are network and server. So if something has changed since yesterday, it could be either of those two things also. And for some reason, network tends to be a sensitive one, people don’t want to hear it might be something wrong with their network … “oh my gosh, you’re offending me by suggesting I should reboot my modem and router!” But networks are dynamic. they are affected by a lot more than you might expect, especially if you’re on a wireless connection, and it absolutely can change from one day to the next, which is something the viewer can’t do.
0:55:10 LP: And as for server, if it’s a server issue, it may be affecting other people, it could be specific to the location in SL you’re in, so you can always try different regions. That’s always one of the things we suggestion doing for certain kinds of problems. In fact there are about a dozen things it can be, but the only thing it can’t be, if you’re experiencing a big change from one day to the next, is the viewer. That’s the one thing that it isn’t, out of everything that it might be.
I have that problem. used to be fine in ultra, now everything is slow. I’m guessing its network. my ISP not liking SL
0:56:02 JL: It could also be that your hard drive is slowing down. It could be that you’ve got something else running somewhere that’s using up resources on your computer, that takes away resources that you have for SL. There’s any number of a billion things that could be causing it. But what isn’t causing it, I can assure you, is that we didn’t change Firestorm remotely.
What about push updates like LL Viewer?
0:57:18 JL: Can you imagine if we forced you to update to a viewer version that doesn’t work for you? I don’t think that would go over well.
Why can’t the installer do clean installs?
1:38:00 JL: We don’t know where you installed the viewer. We know where the default location is, which is in your program files, Firestorm. A lot of people install the viewer somewhere else. So if we told the installer to go in and uninstall the default location, it’s not going to find it. and there’s more than that to it. The viewer puts a lot of files in the AppData folder on Windows. we don’t know what’s there.
1:38:56 JL: Let me give an example. We’ve had users set their default cache location to C: drive or their desktop. and I feel sorry for them. Because if they send it to their desktop, for example, and they launch the viewer, they’re going to have a zillion files on their desktop. And then they say, “Oh my gosh, that’s a mess. I’m just going to go in and hit clear cache in the viewer.” And what that does, is that deletes everything. And it doesn’t care if the file is already in use, the viewer very aggressive when it comes to clearing cache. It deletes everything. It’s unequivocally gone. And people have set their cache location to their C: drive – bye, C: drive!
1:39:48 JL: So I don’t know what files you might have put in your AppData folder, aside from what the viewer puts there. You might have your porn there. You might have your password file for your bank there. You might have all your childhood pictures or all the pictures and videos of your children growing up … and if the uninstaller goes in there and unequivocally deletes everything that’s there … you’re going to come to us, and you’re going to be very, very, pissed off. And I don’t know, from a legal standpoint, if we’re actually liable for having done that. Probably not, but I don’t want to find out. So that’s why we don’t do that. It’s a risk. And we’re all about protecting our users and trying not to harm your computer. and that’s just something that when you multiply the number of users we have, there’s a high likelihood it’s going to happen to at least somebody. Or lots of people. and that’s why we don’t do that.
1:41:28 EM: The other part of this is, we’re all about giving you options. we’re here to improve your experience, we’re not here to run your life. And one good thing about having to do a clean install manually is that hopefully, people are going to learn a little bit about their computers. And the more you know about your computer, the better it’s going to run, the same as the more you know about the viewer, the better it is going to run.
Can I make a cache folder on my desktop?
1:41:57 JL: You can make a cache folder anywhere you want. definitely put it in its own folder. Don’t just put in to a root somewhere,, because when you clear cache, I can guarantee it’s going to delete everything. It’s really aggressive.
1:42:27 JL: I absolutely hate that we have to recommend that you have to do a clean install. It’s unprofessional, among other things, and we’ve taken a lot of efforts to try to figure out why. Not everybody need to it, by the way, I’ve very rarely do a clean install and I never really have a problem. That doesn’t mean that you shouldn’t do it; some people do have the problem … I think most people don’t need to.
1:43:14 JL: But do this. Don’t do a clean install and log in, and the viewer’s slow launching or is acting weird or there is anything unusual. Don’t come to support until you’ve gone in and done a clean re-install properly. And then if the problem is still there, then come to support.
do you have to do a clean install for each account you have?
1:43:52 JL: No, just the one time.
1:44:00 LP: Part of the clean install is going to be clearing-out settings, which some are global and some are per account. but if you’re doing a proper clean install, you’re doing it manually, so you’re going to have that folder open, and you will need separate back-ups for each avatar and you will need to restore the per-account settings at the very least individually.
1:44:35 LP: So your cache clear and your global settings clear will be one-time, and then the per-account settings are one-by-one.
About clean installs. What does that do for backing-up settings? For example, I do a clean install and restore my settings. Is that a “clean” install?
1:44:47 EM (via chat): Yes.
Is it also a good idea to delete the Firestorm folder under AppData\Local?
1:45:26 EM: The only thing you really should keep is your chat logs. Other than that, get rid of the Firestorm folders. Both of them in AppData.
I didn’t see a restore for each account do anything
1:45:54 EM: A back-up on each account will do something, you also need to restore on each account for the per account settings. Your global settings are the same on all accounts, because they’re global. It’s the per-account one you have to back-up for each account and they restore for each account.
Can you use the same back-up settings for each account, or do you have to use separate ones?
1:46:36 EM: You use the same location. When you do your back-up, you log-in on one account, you do your back-up. you log-in on your next account, you back-up again. The global settings are the same. All you’re doing the second time is backing-up the per-account.
If you don’t do a clean install and later have issues, and just delete the AppData (settings) folder. Would that be equal to a clean install, or are settings stored in other parts of the cache location?
When installing 4.6.1, can I use my back-up to restore files from 4.4.2?
1:47:09 EM: Generally speaking, you can use them. Generally. Every so often we run into somebody who … has problems. My recommendation is this: use your back-ups on a new major release, and if you have problems, then go back to a clean slate and start all your settings.
1:47:44 JL: Basically it boils down to – and this is my opinion and support may not agree with me – don’t come to support unless you’ve tried fixing your weird issue by doing a proper clean install. follow the steps precisely on the wiki page, and if you still have a problem, then come to support.
1:48:38 EM: There’s a lot of recommendations that we make as support. Clean install, your bandwidth, things like that. Now these don’t apply to everybody. for example, the bandwidth thing. If you’re on a high-speed line, wired-in, 1500 is the highest we recommend. That doesn’t mean that you can’t set it higher. I personally know people who have it at 3,000 and don’t have any problems. But if you come to one of our support team and you have your bandwidth at 3,000 and you’re having problems, you know what we’re going to tell you. Drop your bandwidth to 1500 maximum, please. There’s no one size fits all.
1:49:45 Tonya Souther (TS): There’s not one size fits all … those settings are there because somebody at some time or another found that it needed to be tweaked for some case or another. That doesn’t mean you have to go through and tweak every setting. If you do that, you’re going to have problems. But the settings are there because at some point they were found to be necessary. If we tell you that you need to tweak a setting, that’s because doing that in our experience has fixed other problems like yours, so it’s worth a try.
1:50:37 EM: I’m going to take it a step further. Every machine is a little bit different. And I’ll give you a perfect example. We have somebody on the team with two Macbook Pros. One needs VBO and HTTP on, the other need VBO and HTTP off. They bought them at the same time, they’re both the same model, they’ve both got the same operating system on them. but one needs HTTP on, the other needs it off. There is no one size fits all.
1:51:14 EM: The recommendations the support team give you are based on our experience., and I’m going to use bandwidth as an example. People with their bandwidth set really high run into all sorts of interesting problems, like TP failures sometimes, all sorts of strange things. Packet loss, for example. Sometimes they don’t show packet loss but are experience all sorts of issues that are related to packet loss from what we understand.
1:51:50 JL: There a lot of things in the viewer that make zero sense. No matter how good a developer you are, they make no sense. I’ll give you an example. With 4.6.1, there’s a bug with fitted Mesh [FIRE-13071] … that causes the mesh to look stretched across the sim.
1:52:22 JL: So Nicky was trying to fix … an AMD crash … the reason we didn’t get to release on Sunday Night [March 9th] because we had enough people showing up with this issue of “The graphics driver has stopped responding but has recovered”. so it wasn’t the viewer crashing, it was the driver crashing – only AMD, which is ATi cards – and difficult to trace, because the viewer wasn’t crashing. So we were not getting any information from the viewer for it.
1:53:19 JL: So Nicky fixed that and was not happy about it, as it makes no sense why it’s fixed. Code a code standpoint, what she did doesn’t make sense why it fixed it … but there’s a lot of stuff that’s in the viewer that makes no sense. There’s a lot of functionality that shouldn’t work, and it does, or should work, but doesn’t.
1:53:50 EM: Changing your group tag.
1:53:52 JL: Yeah, changing your group tag is another example. There’s just a lot of stuff where logic necessarily doesn’t apply in Second Life.
1:54:04 TS: Well, OK. but you got to remember that Second Life is a system with an incredibly large number of moving parts, and lots of people have had their fingers in it and lots of people have had their fingers in the graphics drivers and lots of people have had their fingers in the operating systems – plural, there are several involved, even if you’re only running one [on your own computer], and on, and on, and on. [So] there’s an immense opportunity for people to screw things up, and invariably, that happens. There’s nothing to say it has to make sense looking at it externally. That kind of thing doesn’t make sense until you actually get to the approximate cause of what the actual bug is. and then, and only then, will it make sense. Except that the systems we deal with are complex enough and we have access to only small parts of them, there are a lot of things that will never make sense, because we’ll never see the proper cause of the problem.
When is voice updating? I have an issue with voice each time FS updates.
TF (via chat): the voice on 4.5.1 and 4.5.1 is newer than that on 4.4.2 or older Firestorm for Windows and Mac. Our next release will update voice again for those two operating systems. [Also see the Firestorm voice wiki page.]
When will the voice stuttering be fixed?
2:00:25 JL: Linden Lab has a new version of Vivox that they are currently testing. Tank wanted us to release it with this version, and he tried very hard and we had an argument and I won … because it’s just not tested … but again, part of my job is finding a happy medium and I will be willing to provide a download to those new voice files on a wiki page, which we haven’t done yet, but we will once Tank sends me those files … and we’re going to offer it on a wiki page and you can download it, it’ll have instructions … and your viewer will use the new voice files.
2:01:42 JL: We can’t actually do anything with voice. It’s closed source.
2:01:48 EM: As for Linux – no idea.
I have a question about the built-in AO. It used to freeze an avatar on sim crossing, it seems that that is fixed, walking over a sim border seems to work now. But flying over a sim border still freezes it. Any news on it?
2:05:30 TS: Sim crossings are still very much in the category of “here there be dragons” … there’s only so much we can do for the viewer, and it’s not a lot. We’re very heavily dependent on what the server does there.
Is it best to clear cache or leave it alone?
2:06:06 EM: Leave it alone unless it’s a problem with textures or inventory. [See also the cache wiki page.] …. and clearing your cache is part of a clean install.
Given this is hell week for support, is there anything we can do to help?
2:09:23 JL: If you see people asking questions in the support group and you’ve already seen that question before, because it’s repetitive, and you know the answer, feel free to answer. As long as you know it’s the right answer, like you’ve seen support answer it the same way, repeat the answer. If you see those questions come up in other groups that have nothing to do with Firestorm, because I know Firestorm at release time becomes a topic of conversation in most groups ... Because that’s the best way to help.
2:09:54 JL: A lot of people ask, “Can I donate? Can I give you guys money?” No. If you want to help us out, help other people out. Because that’s our goal, to help you help others, because that reduces the load on support. As long as you’re offering the correct answers. If you’re not sure, don’t answer. Because then you’re actually doing the opposite.
2:10!9 EM: What jess said is great. Point people to the wiki, help them out if you can. However, here’s one thing that I would request of all our users. If you see drama starting in the group “Oh your viewer sucks, yadda, yadda, yadda,” … Don’t all jump on the person. A single response would be fine, and then leave it alone. Just ignore them. Because all you’re doing is adding to the chat lag, adding to the drama, and making it harder to help people. Don’t feed trolls.
2:11:11 JL: And not everybody intends to be a troll. We’ve all been in that spot where we’re pissed off … about something. It doesn’t mean you’re a troll; you’re going to come into the group and voice your opinion and you’re very susceptible then to being argumentative with anything somebody responds to you with; so if you see somebody who is quite obviously pissed-off, sometimes the best thing to do is to say nothing at all.
Any thoughts on increasing groups to more than 42?
2:09:47 TS (via chat): That limit is server-side. In fact, the server tells the viewer how many groups you can join.
2:12:10 EM: Were you around when groups increased from 25 to 42? And did you notice the increase in chat lag in groups? And you want more?
2:12:16 JL: Let me just touch on that, about the complexities of just adding one additional group. So we have 42 as a limit right now, for those of you who don’t know why Linden Lab chose a limit of 42 … haven’t seen or are too young to remember The Hitch-hiker’s Guide to the Galaxy … and that’s because 42 is the answer to [life, the universe, and] everything.
2:13:59 JL: I’ll explain the complexities of going from 40 to 42. Every parcel that you walk across, every region that you cross, there is usually a group associated to the parcel or to that region. And then what the server has to do, is it has to look through all the groups that you belong to and determine if you belong to one of the groups associated with that parcel … If you are a member of a group belonging to that parcel, it has to then scan through all the roles in the group to find out if you’re the member of a role that has some kind of capability on that parcel.
2:13:40 JL: So you can imagine that when you’re flying through a region or your driving, and you’re passing parcels, the result is exponential … the additional load one extra group puts on the servers. so the fact that that we went from 25 to 42 was huge. And the reason that Linden Lab was able to do that was because they had just … M Linden [Mark Kingdon, former CEO prior to Rod Humble, and often unfairly maligned] invested a significant amount of money in new servers and backbone, and those servers were able to handle most of that load – but it still was some load. And it’s quite remarkable that it wasn’t as negatively impacting as it was.
2:14:32 JL: It’s not just a matter of adding a group, it’s actually adding one group adds tonnes and tonnes and tonnes more load you add to the server when you take into account the number of people that are crossing parcels at any given time.
What about text on your profile, adding more than ten groups? I’ve seen petitions going around about having more groups and picks put in
2:14:58 JL: I think that’s a storage thing. That certainly wouldn’t be an exponential thing … See, you guys have got to understand something. If we could give you more picks, we’d do it. Because it’s no skin off our backs because we work for free and we do stuff because we feel like doing it. For Linden Lab to give you more picks means investing $20,000 in wages for the developers to do it, and it would cost even more than that, and server costs and everything else. You’ve got to remember that in order for us to have Second Life, Linden Lab has to be profitable. And when it comes to deciding what features to work on and what features not to work on, they’ve got to look at what’s going to give them the best bang for their buck … Not that there’s any logical to tie Facebook in with Second Life…
2:17:03 EM: Actually there is … it’s exposure for them on Facebook. Free exposure in the long term.
2:18:43 TF: I do want to mention something I’ve seen in our blog comments regarding Facebook. There’s been some comments stating that Facebook doesn’t allow avatars on their site. That used to be true; they used to have a policy that you have to be a real person behind the account. But there was enough user backlash against that, that they’ve since recanted that. So you can use a Facebook account for just your avatar, that doesn’t have any of your real-life information on it. [Please see my comment below on this issue.]
The next Firestorm meeting will be in April, date TBA, at 16:00 SLT.