As always, please refer to the server deployment thread in the forums for the latest news and updates.
On Tuesday April 1st, the Main channel received the server maintenance package deployed to the Magnum RC in week 13.
On Wednesday April 2nd, all three RCs received the same maintenance update, which incorporates the bug fixes deployed to the Main channel and which also sees AIS v3 support returned on the Magnum RC alongside of BlueSteel and LeTigre.
The Sunshine / AIS v3 RC returned to the viewer release channel on Wednesday April 2nd with the release of version 3.7.5.288573 (download and release notes).
On Thursday April 3rd, the Lab also confirmed that Facebook had lifted to block on uploading snapshot to Facebook accounts following updates made by the Lab to both the viewer (available in all RC versions of the viewer) and the back-end intermediary sitting between SL and Facebook.
Group Chat
“I think we’re pretty close to a deploy and test on the main grid,” Simon Linden said during the Server Beta meeting on Thursday April 3rd, in reference to his work trying to improve group chat. He then added wryly, “but I thought that last week too.” Not that any non-deployment of the code to Agni for controlled testing should be taken as a bad sign. Rather the reverse, as Simon went on to say, “The test last week was really good … it helped point out some existing bugs in the system which I think I fixed this week.”
A further test was carried out on Aditi during the meeting yielding further logs to be checked, the outcome of which we’ll doubtless get to hear about in week 15.
Aditi Log-in Issue / Inventory Update Issue
When changing passwords to sync inventory between the main (Agni) and beta (Aditi) grids, the general recommendation is to change your password and, while you can immediately use it to log-into the main grid, wait around 24 hours in order for your password and inventory on Aditi to synchronise (or you can continue to use your old password to log-in to Aditi, and use the “unsynchronized” version of your inventory).
This is because a script is run once a day during a period of relatively low platform use (said to be between midnight and 02:00 SLT) which carries-out the synchronizing of passwords and inventory. However, a bug was recently filed (BUG-5563) indicating that passwords and inventory weren’t updating on Aditi even after 24-hours or more.
Commenting on the issue on March 31st, Maestro Linden said, “There seems to be something wrong with the nightly script that is supposed to keep the beta grid synchronized with the main grid.” When the issue was raised at the Server Beta meeting, Simon Linden indicated that Maestro may well have fixed the problem.
As always, please refer to the server deployment thread in the forums for the latest news and updates.
Main (SLS) Channel
On Tuesday April 1st, the Main channel received the server maintenance package deployed to the Magnum RC in week 13. This package comprises:
Crash mode fix.
Fixed a rare case in which certain users were unable to log in (BUG-5130)
Fix for a case in which multiple scripts in the same prim calling llTakeControls() with heterogeneous ‘accept’ and ‘pass_on’ parameters would not receive a control() event correctly in some cases (BUG-5281)
Updated LSL syntax file to use a new schema (fixes STORM-2000) related to a viewer in testing and apart of STORM-1831.
As the roll-out progressed, there were reports on the forum thread that some regions are having a double restart. After investigating the issue, Caleb Linden reported: “I got the word from engineering that the double rolls were manually triggered by accident but they shouldn’t compromise the sims’ performances once up. This only affected a small group of regions.”
Release Candidate Channels
On Wednesday April 2nd, all three RCs should receive the same maintenance update, which incorporates the bug fixes deployed to the Main channel and which also sees AIS v3 support returned on the Magnum RC, after it was disabled / removed in the week 13 deployment. There seems to be yet more confusion over that actual status of AIS v3 server-side. and whether this means it will actually be enabled on all three RCs, or just re-deployed.
SL Viewer Updates
The SL Voice RC returned to the release channel on Monday March 21st, with the arrival of version 3.7.5.288516. This viewer pulls in Vivox 4.6.x libraries instead of 4.5 for improved voice stability and to address Mavericks issues (download and release notes).
the Sunshine / AIS v3 Rc should return to the channel later in the week if all goes according to plan.
SL Share 2 Project Viewer
As I’ve reported here, on Tuesday april 1st, Linden Lab released the SL Share 2 project viewer, version 3.7.5.288424. This includes:
An option to connect to Twitter and upload tweets / snapshots from in-world to your Twitter account
An option to connect to Flickr and upload snapshots
An updated Facebook floater with revised snapshot upload tab incorporating the features found in the Flickr and Twitter snapshot uploads, and a new Friends tab.
An interesting element in this viewer is the use of filter effects which can be applied to snapshots, and which allow users to add their own filter options.
Group Chat and Group Bans
There’s little to report here, other than:
At the Simulator User Group meeting on Tuesday April 1st, Simon Linden said of the most recent group chat test, ” We had a good group chat test last week that gave me a few more things to fix. They weren’t new issues, but showed some more ways the current system can fail. So it was a good test.” It’s not clear whether this means the testing will now move to Agni and a selected group with a large number of users, or whether there will be further Aditi testing first.
With group bans, and as reported last time on the subject, Baker Linden is busy repairing the code that want broken / lost during a merge process and trying to make sure everything is back together and working properly,
A TPV developer meeting took place on Friday March 28th. The core items discussed in the meeting are reported below, with timestamps in the relevant paragraphs indicating the point at they are discussed in the video embedded here. My thanks as always to North for the latter.
Release Candidate Viewers Status
Stat Test RC and Google Breakpad RC
[00:10] The Stat Test release candidate viewer (version 3.7.5.288371 at the time of writing) is not intended to ever move to a formal release status. Its function is to help with identifying how big an impact a bug within the Google Breakpad RC was having on viewers by comparing crash rates.
The fix contained in the Stat Test viewer is now also in the Google Breakpad RC, which was updated on Thursday March 27th to version 3.7.5.288464 (download and release notes). It’s thought that this RC is getting “pretty close to final form”, something that will likely be confirmed once the new update has been in the release channel sufficient long enough for decent statistics to be gathered in week 14 (week commencing Monday March 31st).
Interest List RC and Merchant Outbox RC
[01:00] As noted in part 2 of this report, the Interest List RC returned to the release channel wither version 3.7.5.288404 (download and release notes). This RC had been removed from the release channel on March 21st due to the number of RC viewers in the channel at that time.
The Mechant Outbox RC, which contains fixes for accurately detecting Merchant status and improves recovery for Merchant Outbox errors, was updated on Friday March 28th to version 3.7.5.288408 (download and release notes). This update means that as of Friday March 28th, all RC viewer in the release channel were up-to-date with the current release version of the viewer.
Voice RC and Sunshine / AIS v3 RC
[01:35] These two release candidates were withdrawn from the release channel on March 21st to reduce the number of RC viewers vying for promotion to release status. These have yet to be updated with the current release viewer code, however, it is anticipated that the Voice RC will be returning to the release channel early in week 14, and that the Sunshine AIS will hopefully reappear later in the week.
The binaries for the Vivox updates are currently available to TPVs for testing an incorporation, so the removal of the Voice RC from the release channel should not have unduly impacted any work being undertaken by TPVs in updating or offering Vivox 4.6.x to Windows and Mac users. Unfortunately, Vivox have yet to supply updates for Linux.
Project Viewers
Project Zipper
[01:52] The Project Zipper “fast installer” viewer was updated to version 3.7.5.288507 on March 28th, also bringing it up to parity with the current release viewer, although it remains at project status (download and release notes).
Oculus Rift
The Oculus Rift project viewer is currently on a closed beta, and so the code is not publicly viewable. The beta is reported to be going “really well”, although other sources indicate that there is a lot of dissatisfaction in how the UI works, up to an including demands that the entire UI needs to be re-thought.
Given any change in the viewer UI tends to attract a high degree of negativity right out of the gate (and also given the fact I’ve not actually seen the “Riftlook” UI), it’s hard to say how many of the concerns are objective as opposed to subjective in nature (e.g. the issue is genuinely one of ease-of-use as opposed to it being one that is considered “unusable” because it’s either not how it’s worked in the past, or requires a re-training of muscle memory) . However, if the issues are objective and indicate significant issues of usability, it will be interesting to see how the Lab responds.
Upcoming Viewers
Group Bans
[02:24] The Group ban viewer is progressing, as noted in part 2 of this report, a new version is available through the Server Beta wiki page, and can be tested on the Morris region on Aditi. There is yet more work to be done on this, so it’s unlikely to move to an official project viewer or RC status in the short-term.
Snowstorm
A new Snowstorm RC viewer is in development, and Oz Linden is hoping to include a few more things in it. I believe this viewer may include the LSL syntax updates of STORM-1831, although the hope is apparently to get some more items into it alongside what is already there. As there are already so many viewers either in, or queuing-up for RC status, there is currently no rush to move this viewer forward.
Release Viewer
[04:02] Commenting on the current release viewer version: 3.7.4.288138, formerly the FmodEx Hotfix RC, Monty Linden indicated that there are almost a year of fixes the stream threading between the current viewer and previous versions. These include many fixes for issues where the viewer can drop a stream entirely. However, whether these include a fix for the issues where a stream containing externals ads (for cars, insurance, etc. – see FIRE-12798) can be completely dropped by the viewer and require a re-log in order to reconnection, is unclear. However, Monty has offered to take a poke at things.
SLS Share / Facebook Photos Upload
[07:26] As reported on March 20th, the ability for users to upload photos from the viewer to their Facebook accounts was disabled by Facebook, apparently because of the inclusion of SLurl with the photo uploads.
The Lab has made changes in the viewer and in the intermediate server layer which should address Facebook’s concerns, however, Facebook has yet to unblock the upload capability at their end, and at the time of writing there has been no word on when this might be done.
The viewer-side updates have yet to reach the release viewer, but are currently in all of the release candidates in the viewer release channel.
The SL Share photo upload panel as it is in the current release viewer (3.7.4.288138) on the left,complete with the option to include a SLurl; and as it appears in the RC viewers on the right, with the SLurl option removed to meet Facebook’s requirements
Mac Issues
[08:30] A request has been made for the Lab to provide a list of Mac / Cocoa issues to help TPVs keep track of things. As reported last time this issue was raised, the Lab are hampered in trying to deal with Cocoa-related issues, as their Mac developers are working on a number of projects, and so specific issues are dealt with individually, rather than as a part of an overall project.
HTTP Issues
Monty Linden:
[19:53] The Firestorm team asked if the Lab had any data on how well the HTTP code changes were going and if they were seeing anything unexpected with server-side loads. The prompt from this question came by the fact that as downloads of the Firestorm 4.6.1 release have increased, so the number of user complaints which appear as though they might be related to HTTP have increased. Monty Linden replied:
Not that I can tell. Nothing is getting to me if there is any such story. I’ve heard the anecdotes, and whenever I’ve looked into them, the stories behind them are generally pretty random. It’s people associating recent changes with behaviour they’re seeing and assuming some correlation where there isn’t any.
Now, that said, there are some patterns that continue to persist that have existed for the past few years that I’ve been dealing with transport issues. The simulators always have had problems in the lib services with connection quotas being exceeded, and that’s still going on today. A busy region on a busy sim hots is still a problem, but I can’t say it’s worse than before. It should be better, but the data I’m getting – which is almost none – doesn’t point to any problem. I’m sorry that’s not definitive, but it’s all I’ve got at this point.
It has been reported that HTTP downloads on sims which have a lot of textures and objects, but few avatars are significantly better than HTTP downloads where there is a large number (20+) avatars. In the latter circumstance, it has also been reported that switching back to UDP appears to improve downloads. However, Monty warned against this, as UDP puts significantly more load on the simulator, which can have a noticeable impact for every user connected to that simulator.
In discussing how to confirm whether or not users really are seeing HTTP issues, Monty continued:
The first thing would be to look at the log file. If you’re getting permanent failures in texture HTTP, it’s going to show up in the log file with a fail on a final retry with an error code and a bunch of related information.So you can pretty much confirm pretty quickly whether or not they’re getting serious permanent failures for texture operations … and these will be distinct from 404s. 404s are nots founds, and you’ll see those every now and then simply because the texture reference is not valid. But other reasons for a permanent failure would be interesting.
When I updated the texture console, I put mesh errors in, but I didn’t do anything about HTTP errors, but there would be another useful area to put things in the future for monitoring error counts. but for now the log file will have the truth. Let me know what you find, because that will point us in one direction or the other.
On our services, I absolutely definitely see busy sim hosts having their input channels saturated. And when that happens, you are going to have retries, and I’ve seen some pretty hard cases of that. Some of the big social things that get put on Second Life [such as RFL] often get put on concentrated sim hosts, and these really get victimised by this problem. But again, permanent failures, they’ll be in the logs and you can look at that. Beyond that, I’ll se what i can find out.
A problem with obtaining logs is that people will often encounter problems like this a few times and the revert back to an earlier release of the viewer. So, if you are using a viewer (SL or Firestorm) with the HTTP updates and encounters persistent rendering issues or failures when moving around SL, please consider taking a couple of minutes to fill-out a JIRA, attach your log file and submit it. Monty is very sensitive to HTTP issues, but in order to respond to them, he needs data, not anecdotes.
Other Items
AIS v3
[02:56] Oz Linden reported that ther server-side AIS v3 code will be enabled across all three RC channels (LeTigre BlueSteel and Magnum) in week 14. As BlueSteel and LeTigre had AIS v3 enabled with server update 14.03.07.287758 and Magnum with update 14.02.21.286985, I can only assume that it was disabled again with the Magnum update 14.03.12.288004 in week 13.
Male Avatar Chest Rendering Broken / Mesh Clothes Breaking on Male Avatars
This issue related to BUG-5537, where there is a discrepancy on how the AV mesh shape is rendered/viewed by oneself and by others in the male AV chest portion, so that while an item of rigged mesh clothing looks fine to the user when viewing his avatar, other people see parts of the avatar’s chest still poking through the outfit. There has also been discussion of the issue on the forums.
The Lab is currently investigating the issue, but have no definitive answer on the problem as yet.
there was only one update in week 13. The Magnum Rc received a new server maintenance project, comprising:
Crash mode fix.
Fixed a rare case in which certain users were unable to log in (BUG-5130)
Fix for a case in which multiple scripts in the same prim calling llTakeControls() with heterogenous ‘accept’ and ‘pass_on’ parameters would not receive a control() event correctly in some cases (BUG-5281)
Updated LSL syntax file to use a new schema (fixes STORM-2000) related to a viewer in testing and apart of STORM-1831.
SL Viewer
The StatTest viewer was updated on Thursday March 27th to version 3.7.5.288371. This viewer makes one change related to crash statistics gathering has no functional changes compared to the current release viewer (download and release notes).
Also on Thursday March 27th, the Interest List RC returned to the release channel, having been temporarily removed on March 21st due to the number of RCs occupying the channel. The return sees the viewer updated to version 3.7.5.288404 (download and release notes).
Group Bans List
Things have been a little quiet in this of late. speaking at the Server Beta meeting on Thursday March 27th, Baker Linden explained why. “I pushed a new version of the group ban viewer, but unfortunately, it’s still a tad unstable. I don’t know how I managed to mess up the merge this bad … I think I lost some code. Well, I’m sure I lost some code. So I’m trying to go back and fix all that, but it’s coming along.”
The latest versions of the test viewer can be obtained via the links on the Server Beta wiki page. This can be tested on Aditi, and the server-side code in active on Morris, where the Server Beta meetings are held (and the code for which has been updated to prevent people with group ban rights from banning themselves from the group…).
Group Chat
There was a further test of the group chat back-end changes during the Server Beta meeting on Thursday March 27th. This involved around the same number of people as the first test, with some remaining in one location and others moving between regions while chatting.
In both instances, viewer freezes were noted when swapping between the group chat sessions and between local chat and one or other of the group chat sessions. These would last between 2-10 seconds depending on the volume of chat going through the group channel session being swapped to, with the average freeze being around 5 seconds.
Commenting on progress with the work, Simon Linden said:
I’m cautiously optimistic I can increase the performance … which should help reliability. From what I can tell, it just can’t handle the load at times. At a small-scale group chat isn’t that difficult a problem but with possibly 42 groups per online AV, the numbers involved can start getting big … I didn’t write the group chat servers, but am just the lucky guy wading through it now. Like I said, I’m cautiously optimistic I can improve it. A re-write to a new engine would be a pretty large project … and has already been tried once.
Further testing should be progressing to Agni and a large group in week 14.
As always, please refer to the server deployment thread in the forums for the latest news and updates.
There was again no server deployment to the Main (SLS) channel on Tuesday March 25th. Of the three RC channels, there will be no change to either BlueSteel or LeTigre on Wednesday March 26th. However, the Magnum RC will be updated with a new server maintenance project, comprising:
Crash mode fix.
Fixed a rare case in which certain users were unable to log in (BUG-5130)
Fix for a case in which multiple scripts in the same prim calling llTakeControls() with heterogenous ‘accept’ and ‘pass_on’ parameters would not receive a control() event correctly in some cases (BUG-5281)
Updated LSL syntax file to use a new schema (fixes STORM-2000)
The LSL syntax file update for STORM-2000 is part of a large project being undertaken by Ima Mechanic with assistance from Oz Linden (see STORM-1831). This work is currently awaiting the release of viewer-side changes, which have been undergoing testing by LL’s QA.
A lightly-attended Simulator UG meeting, Tuesday March 25th
SL Viewer Updates
On Monday March 24th, the FmodEx Hotfix viewer (3.7.4.288138, release notes here) was updated to the de facto release viewer. The core updates in this viewer comprise:
Crash fix (MAINT-3703)
Update FmodEx library to 4.44.31
Additional work for MAINT-2718 (Linux viewer was using logging version of library)
Group Chat Work
During the Simulator User Group meeting on Tuesday March 25th, I asked Simon Linden about the outcome of the group chat tests run on Thursday March 20th, now that there’s been time to check the logs files. He said, “It didn’t have any surprises, but that was a good result. That test was mostly checking out the new statistics it gathered, which worked fine too.” He also confirmed that testing with a large group (most likely the Firestorm support group) is the next step in proceedings:
It’s been delayed a bit due to some people being out of the office. I’m hoping to do something next week … I think that’s the earliest possible. In the mean time I’m working on another layer in the back-end system that is inefficient, so I’m hoping to finish that up today or tomorrow and roll it in together.
Other Items
Revision to the TPVD Directory
The Third-party Viewer Directory received a change to the section listing viewers that report crash statistics. Whereas previously, viewers in this section had been ranked from best to worst crash rate (% of sessions that end in a crash), as from March 25th, they are now ordered from best to worst disconnect rate (% of sessions that end without the simulator seeing a logout).
Regions Size Trivia
During a discussion on vehicle design and region crossings, Simon Linde made mention of the factor that he once spent time looking at region sizes in SL. This isn’t the first time he’s mentioned this work. The last time he raised it, he went to far as to indicate that were regions ever to be resized, increasing them to 1 km a side rather than 1024 metres a side would be preferable, although this would impact the power of 2 approach taken with building SL. At the Simulator UG meeting, he reiterated this latter point, saying, “unfortunately the 256×256 region space is carved into the SL design all over … from the database to the simuators to the viewer and the messages. Changing that would be huge.”
There are a couple of things server-side making their way towards a release:
Fix for login issues of agents with number display names – this is a fix an issue where users with display names made-up of numbers are unable to log-in. “It was a subtle change in a low-level library that started treating those as numbers and not strings,” Simon Linden said of the issue’s origins.
Fix for llTakeControls() issues with multiple scripts with mixed parameters in same prim
These fixes are liable to testing on Aditi prior to moving to an RC.
SL Viewer
On Wednesday March 18th, the Hotfix viewer was updated to version 3.7.4.288138. This now lists three core issues as being addressed:
Crash fix (MAINT-3703)
Update FmodEx library to 4.44.31
Additional work for MAINT-2718 (Linux viewer was using logging version of library)
A typical Server Beta meeting
Group Chat
As indicated in part 1 of this week’s report, there is work underway to try to improve group chat. A small-scale test with a dozen people was carried out with some of the initial improvements during the Sever Beta meeting on Thursday March 20th. “This is really a test that shows it doesn’t fall over with minor load,” Simon Linden said of it.
During the test, those present were asked to send lots of short messages at speed to two test groups while moving around and between various regions (walking, flying, teleporting). Few issues appeared to be noted, at least none that Simon indicted he was interested in (e.g. messages showing delay between being sent and appearing), but as noted, it was a small group. In terms of what has been done with the code and plans for the future, Simon went on:
Basically I dug into the code and found some inefficient parts and cleaned them up. I also added more metrics, so I’m hoping it can show an improvement. I’m pretty sure it will be better, but it’s really hard to guess how much better. I’m hoping to test it on the main grid soon. It would get put on a server and then it would affect a fraction of the groups. I can pick one group to target and make sure it covers that group.
As chat is handled back dedicated back-end servers, no simulator or viewer change will be required to take advantage of this work as it progresses. If the logs from this initial test show nothing unpleasant or unexpected, and providing LL’s QA are happy, testing on the main grid could start in a week’s time.
Inara Pey: Living in a Modemworld 