As noted in part 1 of this report, week 16 should see a server maintenance update deployed which includes a fix for BUG-5533, “llTeleportAgent() and llTeleportAgentGlobalCoords() can break any script in any attached object that contains a change event.”
Commenting on this at the Server Beta User Group meeting on Thursday April 10th, Maestro Linden said:
Kelly found some race conditions which likely lead to the breakage, and the fix appears to work. Right now it’s on Aditi, in the DRTSIM-251 channel. Ahern is on that channel, though I found out that almost all of Ahern’s parcels have scripts disabled. However, “Tehama” is also on that channel, and does allow scripts on some of its parcels. It’s public access, so that’s a good place to check the fix out if you’re interested.
Server Beta Meeting, Thursday April 10th
SL Viewer
On Wednesday April 9th, the Lab release the VoiceMO release candidate viewer, version 3.7.6.288881 (release notes), into the viewer release channel. This RC combines the updates found in the Voice RC viewer (3.7.5.288516) and the Merchant Outbox RC viewer (3.7.5.288408), both of which at present also remain in the release channel for the time being.
Also on Wednesday April 9th, the Lab issued a new Maintenance viewer RC, version 3.7.6.288799, which includes some 54 MAINT fixes from the Lab – see the release notes for details. However, this does include a fix for the Male Avatar Chest Rendering Broken / Mesh Clothes Breaking on Male Avatars (BUG-5537) issue I covered in week 13. The fix for this is given as “MAINT-3896 Male Avatar chest rendering broken – chest shows bigger for onlookers, than to oneself – effects attachments and all worn mesh types”. This update also includes a change to remove the “Start Second LIfe now?” dialogue which is displayed after installing the viewer, and which resulted in the viewer being started with system Admin privileges on Windows systems.
Commenting briefly on things viewer, Simon Linden indicated that he’s also fixed the bug where terrain textures aren’t updated if you change the heights or textures, although I gather this has yet to reach a viewer.
Group Chat
A further Group Chat test was run on Aditi. “I’m looking into the overhead for group chat on maintaining that list of people in the chat,” Simon explained as the test kicked-off. “I think it becomes heavy load in a group with a lot of people on-line.” The test proceeded along similar lines to those carried out previously.
Group Bans
Baker Linden reports that most of the viewer-side bugs related to group bans have been fixed. however, Caleb Linden found an ugly bug in the back-end code whereby only the first name on a list of people being invited into a group would be checked to see if that person had been previously banned (and thus prevented from joining); anyone else further down the list would had been previously banned would be allowed to re-join on receiving the invitation.
A fix has yet to be written, but Baker doesn’t see it as a hard task to do, “My fix will check each one and if you’re banned, it will gracefully skip the banned agent’s invite,” he said when describing how things will work. Other Items
Aditi Log-in Issue / Inventory Update Issue
The script which should synchronise people’s passwords and inventories between Agni (the main grid) and Aditi (the beta grid) is not functioning correctly (see BUG-5563). As I’ve previously reported, this means that following a password change, people are finding they must continue to use their old password to access Aditi and are not seeing their inventory update. Commenting on the issue, Maestro Linden said, “We’re not sure why it’s not working – the script which is supposed to synchronize the accounts says that it updates them on Aditi.” So at the moment any fix is dependent on the Linden carrying out further checks to ascertain what is going on.
Feature Request: Hide All Objects Outside Parcel
A request for a parcel control feature which, when set, would cause the viewer to ignore and not render all objects outside of a parcel has been put forward (see BUG-5671) and is drawing strong debate.
On the one hand, people feel this could greatly enhance performance when entering regions that are massively loaded with objects and textures (because the viewer would simply ignore everything outside of the parcel where the option is enabled).
On the other hand, some people feel that such an option could negatively impact the feeling of SL as a place (imagine teleporting around regions and all you see are the objects on the parcel you’re in, the rest is just terrain).
One suggestion put forward is that rather than making this a parcel option (and thus forced on anyone entering a parcel where it is enabled), such a capability is, if possible, made a viewer-side option, so that people get a more flexible choice as to what is rendered / what they see. I have to say that this potentially offers the most attractive option were such a capability to be taken-up by LL.
It’s pretty simple again this week: there are no server deployments scheduled for either the Main (SLS) channel (which remains on maintenance project 14.03.12.288004) or the RCs (which collectively remain on 14.03.28.288552). As always, please refer to the server deployment thread in the forums for the latest news and updates / issues.
SL Viewer
The de facto release viewer updated on Monday April 7th, with the promotion of the Google Breakpad RC, version 3.7.5.288464 (release notes).
As a result of this promotion, the StatTest viewer (formerly version 3.7.5.288371), which was never intended for promotion as a release viewer but issued as a means of assisting with bug-fixing the Google Breakpad code, has been removed from the viewer release channel.
Group Ban Lists
“I’ll be working with Maestro this week to try to get the group ban services and back-end stuff grid-wide on Aditi,” Baker Linden informed the Simulator User Group meeting on Tuesday April 8th. He went on: “I have a handful of smallish bugs to finish up, and then I’ll look at my options for development viewers and such, so group ban will be grid-wise soon [on Aditi] (I hope, depending on the amount of releases in the pipe). And then after we see there’s no big issues, we’ll get that pushed out to Agni.”
Other Items
Aditi Log-in Issue / Inventory Update Issue
As reported in part 2 of the my week 14 updates, there has been an issue in getting passwords and inventory to correctly sync on the Aditi beta grid following a password change. Normally, a script is run on daily basis during periods of relatively low SL use (around midnight-02:00 SLT) which should synchronise a user’s password and inventory between Agni and Aditi. However, several people had noted that their Aditi passwords / inventories were not updating despite several days passing after making password change (see BUG-5563).
It had been thought this issue had been dealt with in week 14; however, a forum thread notes it is still causing problems for some people.
Transaction History Oopsie
There was a brief issue with Transaction History pages on Tuesday April 8th which caused some consternation when it happened, although it was quickly rectified.
The problem first came to light when merchants noticed that their Transaction History page was no longer showing totals or options to download the history in anything other than .CSV format (the previous options had been .XML or .XLS). Further issues were noticed as time went on, as noted in a Commerce forum thread and also in a JIRA (BUG-5664).
The root cause of the problem appears to have been the URL for the familiar Transaction History page being swapped for a new page. The concerns this raised were sufficient for the URL to be reverted back to the original a little over an hour after the problem was first noticed, allowing people to once more access the familiar Transaction history page. Whether the change of URL is indicative of an upcoming change that is in preparation, or simply a mistake on the part of someone at the Lab, is unclear.
As always, please refer to the server deployment thread in the forums for the latest news and updates.
On Tuesday April 1st, the Main channel received the server maintenance package deployed to the Magnum RC in week 13.
On Wednesday April 2nd, all three RCs received the same maintenance update, which incorporates the bug fixes deployed to the Main channel and which also sees AIS v3 support returned on the Magnum RC alongside of BlueSteel and LeTigre.
The Sunshine / AIS v3 RC returned to the viewer release channel on Wednesday April 2nd with the release of version 3.7.5.288573 (download and release notes).
On Thursday April 3rd, the Lab also confirmed that Facebook had lifted to block on uploading snapshot to Facebook accounts following updates made by the Lab to both the viewer (available in all RC versions of the viewer) and the back-end intermediary sitting between SL and Facebook.
Group Chat
“I think we’re pretty close to a deploy and test on the main grid,” Simon Linden said during the Server Beta meeting on Thursday April 3rd, in reference to his work trying to improve group chat. He then added wryly, “but I thought that last week too.” Not that any non-deployment of the code to Agni for controlled testing should be taken as a bad sign. Rather the reverse, as Simon went on to say, “The test last week was really good … it helped point out some existing bugs in the system which I think I fixed this week.”
A further test was carried out on Aditi during the meeting yielding further logs to be checked, the outcome of which we’ll doubtless get to hear about in week 15.
Aditi Log-in Issue / Inventory Update Issue
When changing passwords to sync inventory between the main (Agni) and beta (Aditi) grids, the general recommendation is to change your password and, while you can immediately use it to log-into the main grid, wait around 24 hours in order for your password and inventory on Aditi to synchronise (or you can continue to use your old password to log-in to Aditi, and use the “unsynchronized” version of your inventory).
This is because a script is run once a day during a period of relatively low platform use (said to be between midnight and 02:00 SLT) which carries-out the synchronizing of passwords and inventory. However, a bug was recently filed (BUG-5563) indicating that passwords and inventory weren’t updating on Aditi even after 24-hours or more.
Commenting on the issue on March 31st, Maestro Linden said, “There seems to be something wrong with the nightly script that is supposed to keep the beta grid synchronized with the main grid.” When the issue was raised at the Server Beta meeting, Simon Linden indicated that Maestro may well have fixed the problem.
As always, please refer to the server deployment thread in the forums for the latest news and updates.
Main (SLS) Channel
On Tuesday April 1st, the Main channel received the server maintenance package deployed to the Magnum RC in week 13. This package comprises:
Crash mode fix.
Fixed a rare case in which certain users were unable to log in (BUG-5130)
Fix for a case in which multiple scripts in the same prim calling llTakeControls() with heterogeneous ‘accept’ and ‘pass_on’ parameters would not receive a control() event correctly in some cases (BUG-5281)
Updated LSL syntax file to use a new schema (fixes STORM-2000) related to a viewer in testing and apart of STORM-1831.
As the roll-out progressed, there were reports on the forum thread that some regions are having a double restart. After investigating the issue, Caleb Linden reported: “I got the word from engineering that the double rolls were manually triggered by accident but they shouldn’t compromise the sims’ performances once up. This only affected a small group of regions.”
Release Candidate Channels
On Wednesday April 2nd, all three RCs should receive the same maintenance update, which incorporates the bug fixes deployed to the Main channel and which also sees AIS v3 support returned on the Magnum RC, after it was disabled / removed in the week 13 deployment. There seems to be yet more confusion over that actual status of AIS v3 server-side. and whether this means it will actually be enabled on all three RCs, or just re-deployed.
SL Viewer Updates
The SL Voice RC returned to the release channel on Monday March 21st, with the arrival of version 3.7.5.288516. This viewer pulls in Vivox 4.6.x libraries instead of 4.5 for improved voice stability and to address Mavericks issues (download and release notes).
the Sunshine / AIS v3 Rc should return to the channel later in the week if all goes according to plan.
SL Share 2 Project Viewer
As I’ve reported here, on Tuesday april 1st, Linden Lab released the SL Share 2 project viewer, version 3.7.5.288424. This includes:
An option to connect to Twitter and upload tweets / snapshots from in-world to your Twitter account
An option to connect to Flickr and upload snapshots
An updated Facebook floater with revised snapshot upload tab incorporating the features found in the Flickr and Twitter snapshot uploads, and a new Friends tab.
An interesting element in this viewer is the use of filter effects which can be applied to snapshots, and which allow users to add their own filter options.
Group Chat and Group Bans
There’s little to report here, other than:
At the Simulator User Group meeting on Tuesday April 1st, Simon Linden said of the most recent group chat test, ” We had a good group chat test last week that gave me a few more things to fix. They weren’t new issues, but showed some more ways the current system can fail. So it was a good test.” It’s not clear whether this means the testing will now move to Agni and a selected group with a large number of users, or whether there will be further Aditi testing first.
With group bans, and as reported last time on the subject, Baker Linden is busy repairing the code that want broken / lost during a merge process and trying to make sure everything is back together and working properly,
A TPV developer meeting took place on Friday March 28th. The core items discussed in the meeting are reported below, with timestamps in the relevant paragraphs indicating the point at they are discussed in the video embedded here. My thanks as always to North for the latter.
Release Candidate Viewers Status
Stat Test RC and Google Breakpad RC
[00:10] The Stat Test release candidate viewer (version 3.7.5.288371 at the time of writing) is not intended to ever move to a formal release status. Its function is to help with identifying how big an impact a bug within the Google Breakpad RC was having on viewers by comparing crash rates.
The fix contained in the Stat Test viewer is now also in the Google Breakpad RC, which was updated on Thursday March 27th to version 3.7.5.288464 (download and release notes). It’s thought that this RC is getting “pretty close to final form”, something that will likely be confirmed once the new update has been in the release channel sufficient long enough for decent statistics to be gathered in week 14 (week commencing Monday March 31st).
Interest List RC and Merchant Outbox RC
[01:00] As noted in part 2 of this report, the Interest List RC returned to the release channel wither version 3.7.5.288404 (download and release notes). This RC had been removed from the release channel on March 21st due to the number of RC viewers in the channel at that time.
The Mechant Outbox RC, which contains fixes for accurately detecting Merchant status and improves recovery for Merchant Outbox errors, was updated on Friday March 28th to version 3.7.5.288408 (download and release notes). This update means that as of Friday March 28th, all RC viewer in the release channel were up-to-date with the current release version of the viewer.
Voice RC and Sunshine / AIS v3 RC
[01:35] These two release candidates were withdrawn from the release channel on March 21st to reduce the number of RC viewers vying for promotion to release status. These have yet to be updated with the current release viewer code, however, it is anticipated that the Voice RC will be returning to the release channel early in week 14, and that the Sunshine AIS will hopefully reappear later in the week.
The binaries for the Vivox updates are currently available to TPVs for testing an incorporation, so the removal of the Voice RC from the release channel should not have unduly impacted any work being undertaken by TPVs in updating or offering Vivox 4.6.x to Windows and Mac users. Unfortunately, Vivox have yet to supply updates for Linux.
Project Viewers
Project Zipper
[01:52] The Project Zipper “fast installer” viewer was updated to version 3.7.5.288507 on March 28th, also bringing it up to parity with the current release viewer, although it remains at project status (download and release notes).
Oculus Rift
The Oculus Rift project viewer is currently on a closed beta, and so the code is not publicly viewable. The beta is reported to be going “really well”, although other sources indicate that there is a lot of dissatisfaction in how the UI works, up to an including demands that the entire UI needs to be re-thought.
Given any change in the viewer UI tends to attract a high degree of negativity right out of the gate (and also given the fact I’ve not actually seen the “Riftlook” UI), it’s hard to say how many of the concerns are objective as opposed to subjective in nature (e.g. the issue is genuinely one of ease-of-use as opposed to it being one that is considered “unusable” because it’s either not how it’s worked in the past, or requires a re-training of muscle memory) . However, if the issues are objective and indicate significant issues of usability, it will be interesting to see how the Lab responds.
Upcoming Viewers
Group Bans
[02:24] The Group ban viewer is progressing, as noted in part 2 of this report, a new version is available through the Server Beta wiki page, and can be tested on the Morris region on Aditi. There is yet more work to be done on this, so it’s unlikely to move to an official project viewer or RC status in the short-term.
Snowstorm
A new Snowstorm RC viewer is in development, and Oz Linden is hoping to include a few more things in it. I believe this viewer may include the LSL syntax updates of STORM-1831, although the hope is apparently to get some more items into it alongside what is already there. As there are already so many viewers either in, or queuing-up for RC status, there is currently no rush to move this viewer forward.
Release Viewer
[04:02] Commenting on the current release viewer version: 3.7.4.288138, formerly the FmodEx Hotfix RC, Monty Linden indicated that there are almost a year of fixes the stream threading between the current viewer and previous versions. These include many fixes for issues where the viewer can drop a stream entirely. However, whether these include a fix for the issues where a stream containing externals ads (for cars, insurance, etc. – see FIRE-12798) can be completely dropped by the viewer and require a re-log in order to reconnection, is unclear. However, Monty has offered to take a poke at things.
SLS Share / Facebook Photos Upload
[07:26] As reported on March 20th, the ability for users to upload photos from the viewer to their Facebook accounts was disabled by Facebook, apparently because of the inclusion of SLurl with the photo uploads.
The Lab has made changes in the viewer and in the intermediate server layer which should address Facebook’s concerns, however, Facebook has yet to unblock the upload capability at their end, and at the time of writing there has been no word on when this might be done.
The viewer-side updates have yet to reach the release viewer, but are currently in all of the release candidates in the viewer release channel.
The SL Share photo upload panel as it is in the current release viewer (3.7.4.288138) on the left,complete with the option to include a SLurl; and as it appears in the RC viewers on the right, with the SLurl option removed to meet Facebook’s requirements
Mac Issues
[08:30] A request has been made for the Lab to provide a list of Mac / Cocoa issues to help TPVs keep track of things. As reported last time this issue was raised, the Lab are hampered in trying to deal with Cocoa-related issues, as their Mac developers are working on a number of projects, and so specific issues are dealt with individually, rather than as a part of an overall project.
HTTP Issues
Monty Linden:
[19:53] The Firestorm team asked if the Lab had any data on how well the HTTP code changes were going and if they were seeing anything unexpected with server-side loads. The prompt from this question came by the fact that as downloads of the Firestorm 4.6.1 release have increased, so the number of user complaints which appear as though they might be related to HTTP have increased. Monty Linden replied:
Not that I can tell. Nothing is getting to me if there is any such story. I’ve heard the anecdotes, and whenever I’ve looked into them, the stories behind them are generally pretty random. It’s people associating recent changes with behaviour they’re seeing and assuming some correlation where there isn’t any.
Now, that said, there are some patterns that continue to persist that have existed for the past few years that I’ve been dealing with transport issues. The simulators always have had problems in the lib services with connection quotas being exceeded, and that’s still going on today. A busy region on a busy sim hots is still a problem, but I can’t say it’s worse than before. It should be better, but the data I’m getting – which is almost none – doesn’t point to any problem. I’m sorry that’s not definitive, but it’s all I’ve got at this point.
It has been reported that HTTP downloads on sims which have a lot of textures and objects, but few avatars are significantly better than HTTP downloads where there is a large number (20+) avatars. In the latter circumstance, it has also been reported that switching back to UDP appears to improve downloads. However, Monty warned against this, as UDP puts significantly more load on the simulator, which can have a noticeable impact for every user connected to that simulator.
In discussing how to confirm whether or not users really are seeing HTTP issues, Monty continued:
The first thing would be to look at the log file. If you’re getting permanent failures in texture HTTP, it’s going to show up in the log file with a fail on a final retry with an error code and a bunch of related information.So you can pretty much confirm pretty quickly whether or not they’re getting serious permanent failures for texture operations … and these will be distinct from 404s. 404s are nots founds, and you’ll see those every now and then simply because the texture reference is not valid. But other reasons for a permanent failure would be interesting.
When I updated the texture console, I put mesh errors in, but I didn’t do anything about HTTP errors, but there would be another useful area to put things in the future for monitoring error counts. but for now the log file will have the truth. Let me know what you find, because that will point us in one direction or the other.
On our services, I absolutely definitely see busy sim hosts having their input channels saturated. And when that happens, you are going to have retries, and I’ve seen some pretty hard cases of that. Some of the big social things that get put on Second Life [such as RFL] often get put on concentrated sim hosts, and these really get victimised by this problem. But again, permanent failures, they’ll be in the logs and you can look at that. Beyond that, I’ll se what i can find out.
A problem with obtaining logs is that people will often encounter problems like this a few times and the revert back to an earlier release of the viewer. So, if you are using a viewer (SL or Firestorm) with the HTTP updates and encounters persistent rendering issues or failures when moving around SL, please consider taking a couple of minutes to fill-out a JIRA, attach your log file and submit it. Monty is very sensitive to HTTP issues, but in order to respond to them, he needs data, not anecdotes.
Other Items
AIS v3
[02:56] Oz Linden reported that ther server-side AIS v3 code will be enabled across all three RC channels (LeTigre BlueSteel and Magnum) in week 14. As BlueSteel and LeTigre had AIS v3 enabled with server update 14.03.07.287758 and Magnum with update 14.02.21.286985, I can only assume that it was disabled again with the Magnum update 14.03.12.288004 in week 13.
Male Avatar Chest Rendering Broken / Mesh Clothes Breaking on Male Avatars
This issue related to BUG-5537, where there is a discrepancy on how the AV mesh shape is rendered/viewed by oneself and by others in the male AV chest portion, so that while an item of rigged mesh clothing looks fine to the user when viewing his avatar, other people see parts of the avatar’s chest still poking through the outfit. There has also been discussion of the issue on the forums.
The Lab is currently investigating the issue, but have no definitive answer on the problem as yet.
there was only one update in week 13. The Magnum Rc received a new server maintenance project, comprising:
Crash mode fix.
Fixed a rare case in which certain users were unable to log in (BUG-5130)
Fix for a case in which multiple scripts in the same prim calling llTakeControls() with heterogenous ‘accept’ and ‘pass_on’ parameters would not receive a control() event correctly in some cases (BUG-5281)
Updated LSL syntax file to use a new schema (fixes STORM-2000) related to a viewer in testing and apart of STORM-1831.
SL Viewer
The StatTest viewer was updated on Thursday March 27th to version 3.7.5.288371. This viewer makes one change related to crash statistics gathering has no functional changes compared to the current release viewer (download and release notes).
Also on Thursday March 27th, the Interest List RC returned to the release channel, having been temporarily removed on March 21st due to the number of RCs occupying the channel. The return sees the viewer updated to version 3.7.5.288404 (download and release notes).
Group Bans List
Things have been a little quiet in this of late. speaking at the Server Beta meeting on Thursday March 27th, Baker Linden explained why. “I pushed a new version of the group ban viewer, but unfortunately, it’s still a tad unstable. I don’t know how I managed to mess up the merge this bad … I think I lost some code. Well, I’m sure I lost some code. So I’m trying to go back and fix all that, but it’s coming along.”
The latest versions of the test viewer can be obtained via the links on the Server Beta wiki page. This can be tested on Aditi, and the server-side code in active on Morris, where the Server Beta meetings are held (and the code for which has been updated to prevent people with group ban rights from banning themselves from the group…).
Group Chat
There was a further test of the group chat back-end changes during the Server Beta meeting on Thursday March 27th. This involved around the same number of people as the first test, with some remaining in one location and others moving between regions while chatting.
In both instances, viewer freezes were noted when swapping between the group chat sessions and between local chat and one or other of the group chat sessions. These would last between 2-10 seconds depending on the volume of chat going through the group channel session being swapped to, with the average freeze being around 5 seconds.
Commenting on progress with the work, Simon Linden said:
I’m cautiously optimistic I can increase the performance … which should help reliability. From what I can tell, it just can’t handle the load at times. At a small-scale group chat isn’t that difficult a problem but with possibly 42 groups per online AV, the numbers involved can start getting big … I didn’t write the group chat servers, but am just the lucky guy wading through it now. Like I said, I’m cautiously optimistic I can improve it. A re-write to a new engine would be a pretty large project … and has already been tried once.
Further testing should be progressing to Agni and a large group in week 14.
Inara Pey: Living in a Modemworld 