On Tuesday, May 19th the Main (SLS) channel received the server maintenance package previously deployed to the three RC channel, comprising Internal server logging changes, back-end system bug fixes and a change to Reply-To e-mail addressing on snapshots. There were no RC deployments on Wednesday, May 20th.
The Attachments Viewer RC (Project Big Bird) was updated to version 184.108.40.2061943 on Thursday May 21st. As noted in part 1 of this week’s report, the initial RC release of this viewer had an elevated crash rate compared to the current release viewer, including a crash-on-exit bug, so this release will hopefully address those issues.
A fix for issues around BUG-9130, where some people were unable to see any posts in some or all of there group chats, including their own posts, while everyone else in the same group could see their posts, has started to be deployed across the chat servers, and should be completed on Friday, May 22nd.
“The chat servers got stuck with bad info about where the sender was, so the messages never reached them,” Simon Linden said at the Server Beta User Group meeting on Thursday, May 21st, reiterating an explanation given at a recent Simulator UG meeting. “And unfortunately it wouldn’t fix with relogging or even a chat server restart.”
“Loading…” Issue with Names in Group Chats
This is a viewer-side problem which causes avatar names to appear as “Loading” under certain circumstances in group chat (see BUG-3829 and STORM-2114). A contribution by Ansariel Hiller is currently with the Lab and is expected to be released as a part of the next Snowstorm contributions viewer, which is expected to appear soon.
Region Restart Glitch
There has been something of a rise in reports of regions experiencing issues following recent following restarts – most noticeably caps failures. This is something the Lab is looking into, and Simon commented, “we have a suspicion that after rolls, as that server host starts up regions, it’s doing enough of them at about the same time that things get overloaded. It’s still a theory but makes some sense why we’d get cap failures like that.”
The following notes are primarily taken from the TPV Developer meeting held on Friday, March 27th, a video of which is included towards the end of the article (my thanks as always to North for recording it and providing it for embedding), and from the Server Beta meeting held on Thursday, March 26th. Any time stamps contained within the following text refer to the TPV developer meeting video.
Server Deployments Week 13 – Recap
As always, please refer to the deployment thread in the forums for the latest updates / news.
On Tuesday, March 24th, the Main (SLS) channel received the server maintenance package deployed to the three RCs in week 12, comprising updates which allow the Lab to make various configuration changes without having to necessarily run a rolling restart when they have done so. It contains not actual functional changes to the simulator software
On Wednesday, March 25th, the three RC channels received the same new server maintenance package, which is focused on inventory loss issues, and provides the Lab with better error detection and logging, improving their ability to look at some of the failure places and the removal of unused code. This updates does notremove the server-side messaging used in support of RTLP.
SL Viewer Update
Avatar Layers Project Viewer
Vir Linden’s work on a new global limit for system layer clothing was released as a project viewer, version 220.127.116.119805. With this viewer, a user can wear any combination of clothing layers (wearables), up to a maximum of 60, rather than being limited to (in general, and as with the official viewer) to a maximum of 5 items per layer type. Note that these changes do not apply to body part wearables (skin, shape, hair, eyes), for which the limit is still one of each, and do not affect attachments, for which the limit is still 38 total.
[07:18] There is already an update in the pipe for this viewer, which should be appearing next week.
Camera Positioning / Handling
[05:12] While there are no specific details as yet, the lab is hoping to put some work into improving camera positioning and handling in the not too distant future, in the hope of removing various glitches and issues.
Build Tools Viewer
[05:54] There have been a few fixes added to this viewer (currently version 18.104.22.1689443), so a further update to the release candidate version is with the Lab’s QA team and should be appearing in week #14 (week commencing Monday, March 30th).
Maintenance Release Viewer
[06:29] Currently at version 22.214.171.1249845, the latest Maintenance release viewer has a range of issues, many of which have hopefully been addressed with a series of fixes, so an update to that viewer is also with the Lab’s QA team. However, given the scope of the updates, it is proving a little harder to pass the QA process.
Experience Tools Viewer
[06:50] The Experience Keys / Tools viewer (currently version 126.96.36.1999338) is being merged-up with the latest release version of the viewer code (version 188.8.131.529635). The updated version should also be appearing (again as an RC) in week #14.
[17:27 – 19:50] There is an interesting discussion on the viewer code which, for anyone interested in how the viewer has developed over the years, and how much of it dates back some 14 years.
[00:00] There was a pile-on test of the new Viewer-Managed Marketplace capability on Aditi in week #12, and Brooke Linden was at the TPV Developer meeting to provide feedback. The pile-on test did not reveal any significant issues in terms of performance.
However, there is still a viewer / simulator / marketplace communications issue which has to be resolved, which may take another couple of weeks to fix. After that, there are two grid deployments which need to take place: one for the VMM code itself, and one for updates to the Advanced Inventory System (AIS), so it is unlikely VMM will be fully deployed within the next month to two month, and the project viewer (currently version 184.108.40.2068865) is unlikely to progress through a release candidate to release status until after the server components have been deployed.
[07:32] Simon Linden has been continuing to work on the group chat code, and all of his current updates should have been deployed to the back-end group chat servers. A broad consensus is that the issues which recently occurred as a result of some changes have been reversed, and that the group chat service as a whole is now running a lot better, both in terms of the early performance improvements Simon made, and with regards to the overall stability of the service and the servers.
[08:24] There is a further round of updating in the planning, but these require a platform upgrade to be carried out for the group chat service first. Therefore, unless unless the latest set of updates deployed by the Lab start to show issues, the engineering team will be switching focus for the immediate future, and will return to working on group chat once the necessary upgrade work has been completed.
Experience Keys / Tools
[09:20] One of the items the engineering team want to focus on in particular is Experiences, and getting the remaining back-end issues sorted out so that Experiences can be properly deployed.
[09:59] There will be a further round of voice updates which are expected to appear in a project viewer “shortly”. They include (but are not limited to) things like general code clean-up to prevent unnecessary list loading, removal of media messaging in person-to-person calls (which has never worked), fixes for issues related to microphone volume and improvements to the microphone test so that you can now hear yourself when testing your microphone, and improvements for hot swapping microphones / headsets.
[13:58] There is some confusion over whether or not a fix to voice designed to prevent someone’s voice channel being “left behind” when teleporting between regions has actually worked. It had been thought that the fix for this had been deployed in later 2014. However, bug reports are still being made still reporting issues (see BUG-8543 and STORM-2109), prompting the Lab to re-examine the status of the fix.
[19:54] Voice package updates from Vivox are also expected to be forthcoming in the future as well.
Restore To Last Position (RTLP)
[21:08] There have been around 400 responses to the Firestorm call for feedback on how people use the Restore To Last Position functionality found in some TPVs. As I’ve previously reported, the Lab had been considering deprecating the server-side message RTLP uses as an overall part of on-going work to reduce the amount of inventory loss issues (real or perceived) which can occur.
Firestorm’s call is helping the Lab to better understand how, as faulty as it might be, RTLP does fulfil a range of useful / valid use cases. Commenting on the fact the he has been reading through the feedback, Oz Linden said:
[21:49] Well, I understand that there are user scenarios that need to be addressed and need to be better supported. Whether the existing feature is the way to do that or not, I still consider to be an open question. I do want to take those use cases and work back through that process [of determining how best to serve them].
So the Lab still isn’t going to do anything “quickly” either way on RTLP, and people needn’t worry about RTLP vanishing / breaking “suddenly”.
In the meantime, they are working on other changes intended to address various rezzing failure situations. This work is more server-side focused, although it may be a while before updates appear on the grid as the exact nature of the updates is still being determined.
[23:42] Oz also again thanked everyone who responded to the Lab’s call for feedback on inventory losses in general, defining the feedback as “really, really useful”.
The following notes are primarily taken from the TPV Developer meeting held on Friday, March 13th, a video of which is included at the end of the article (my thanks as always to North for recording it and providing it for embedding), and any time stamps contained within the following text refer to it.
Server Deployments Week 11 – Recap
As always, please refer to the sever deployment thread for the latest updates and information.
There was no Main (SLS) channel deployment on Tuesday, March 10th
On Wednesday, March 11th, all three RC channels received the same new server maintenance package comprising “internal improvements for premium users”.
The Avatar Hover Height viewer reached the release channel on March 10th, with the release of an RC version (220.127.116.119635). Avatar Hover Height allows you to adjust the vertical position of your avatar within some preset limits. See the wiki page and my overview.
This brings the total number of RC viewers in the viewer release channel to four, however:
[0:41] It is unlikely the Maintenance RC viewer (currently version 18.104.22.1689610, released on March 6th) will be promoted without further update, as it has been found to contain a significant number of additional bugs which require fixing
[0:51] As the Avatar Hover Height RC viewer has only just been released, it is unlikely that the Lab will have enough stats on it to judge whether or not it can be promoted to the de facto release viewer in the immediate future; it is therefore likely to remain at RC status for at least another week, although initial reports suggest it is stable and doesn’t hide any unpleasantness
[01:07] The back-end support for Experience Keys / Tools still isn’t ready for the service to go live, although the Lab is making further progress with whatever needed to be done; it is therefore remains unlikely the that Experience Keys viewer (currently version 22.214.171.1249338, released on March 9th) will be promoted to the de facto release viewer until such time as the remaining back-end work has been completed.
Tools Update Viewer and XP Users
[01:20] This potentially means that the Tools Update RC viewer (currently version 126.96.36.1999443, released on March 4th) may be promoted to the de facto release viewer in week #12.
When this happens, it will obviously mean that all future builds of the official viewer will be made using the new tool chain and autobuild process. This in turn means that any Windows version of the viewer built using the new tools set (which includes MS visual Studio 2013) will not run on any version (32-, or 64-bit) version of Windows XP. To this end, the installer is being set so that it requires a minimum of Windows Vista with Service Pack 2 installed, in order for it to successfully install the viewer.
Note that this is not a deliberate attempt to block XP users from Second Life; it is purely the result of the Lab moving towards the use of up-to-date tools for building the viewer (and which will yield positive benefits elsewhere, such as with greater tool commonality between the Lab and TPVs), and some of these tools do not support windows XP due to its age and it no longer being actively supported by Microsoft.
[16:54] Some TPVs may investigate / opt to build the viewer somewhat manually using the new tool chain in such a way that it can be used on XP, but this is reportedly requires a “very large amount of work” to achieve, requires a lot of command line input, an avoidance of VS 2013, and is “really hacky”.
[03:28] The Viewer-Managed Marketplace project viewer (currently version 188.8.131.528865, released on February 13th) is liable to be updated in week #12 as a result of further fixes and updates that came out of the last round of testing
[04:20] The Mesh Importer project viewer (currently version 184.108.40.2068441, released on February 3rd), is currently undergoing further update with new fixes and will be updated as a project viewer in the near future.
Avatar Layers Global Limit
[04:41] In response to BUG-6258, “Popularity of Mesh Attachments Facilitates Need For More Alpha Layers”, the Lab is working to implement a new “global” limit to the number of system clothing layers an avatar can wear.
Under the current system, there are 12 types of clothing layers or wearables (alpha, tattoo, undershirt, shirt, jacket, underpants, pants, gloves, socks, skirts, shoes, and physics), with (generally) up to 5 of each type of wearable able to be worn at the same time, giving a maximum of 60 wearables that can worn simultaneously.
Under the new code being developed by Vir Linden, a new “global” limit of 60 wearable layers is being set per avatar, and users will be able to wear any number / combination of layers up to that limit (so you could opt to wear 60 jacket layers if you wanted, or 10 each of alpha, shirt, pants, gloves, jacket and socks, for example).
This update requires changes to both the viewer and to the server-side Appearance (SSA) service. The viewer-side changes are updates to the viewer’s logic, so it is purely checking the number of worn layers against the global limit of 60, rather than limits set for individual layers. The SSA changes will similarly support the new “global” use of clothing layers, but will also continue to support the 5-per-layer limit for viewers that do not adopt the newer code, or require a longer lead time in order to adopt the new viewer code, once it is available, thus providing a measure of “back compatibility”. The viewer code is expected to appear in a project viewer once it, and the back-end changes have cleared the Lab’s QA team.
[09:29] As noted in my recent updates, changes made to the group change service in the last two weeks unexpectedly resulted in up to 20% of messages failing to be delivered correctly. Simon Linden spent a fair amount of time during week #10 stabilising things and delivering further updates to try to correct the problem. As a result, in what has been called an “educational” two weeks for the Lab, the situation has been largely reversed, although some problems still remain.
The Server Beta User Group meeting on Thursday, March 12th, saw a further set of updates from Simon undergo testing on the Beta grid, and during the TPV Developer meeting on Friday, March 13th, Oz indicated that the Lab will probably undertake a further round of “serious” upgrading of all the technology associated with group chat before they declare the project in any way “finished”. This will likely involve putting the back-end service support group chat on more up-to-date hardware and OS environments.
As always, please refer to the sever deployment thread for the latest updates and information.
There was no Main (SLS) channel deployment on Tuesday, March 10th
On Wednesday, March 11th, all three RC channels should receive the same new server maintenance package comprising “internal improvements for premium users”.
When asked during the Simulator User Group meeting on Tuesday, March 10th whether the “internal improvements for premium users” was related to the change to the in-world receipt of off-line IMs, as mentioned at the last SBUG meeting, Simon Linden could only say, “I’m not supposed to announce anything, so I can’t go into details … but one thing we’re looking at this year is ways to make premium accounts better. This may or may not do something like that eventually.”
SL Viewer Updates
A new Maintenance RC viewer, version 220.127.116.119610,was released on March 6th. This includes multiple fixes and improvements as listed on the release notes and download page.
The Experience Keys viewer updated to version 18.104.22.1689338 on Monday, March 9th, maintaining parity with the current release viewer.
Although the Experience Tools viewer has been updated (see above), there is still no news on when Experiences might be fully deployed. In order to help build interest in Experiences a suggestion has been put forward to enable Experiences to be rated in terms of the number of people actively joining them (see BUG-6911), which could be optionally shown (at the Experience creator’s discretion in things like search listings, allowing people to judge Experiences by their popularity.
The Lab has considered allowing users to rate Experience themselves in a future update – but as point out in the JIRA comments, such a system could be open to gaming, much like the old avatar popularity ratings. BUG-6911 has been imported by the Lab, but it is currently unclear if the idea will be carried forward.
As also noted in my last updates, recent changes to the group chat service have seen up to a 20% failure rate in delivered messages. Simon Linden spent a fair amount of time during week #10 stabilising things once more, and notes that the situation taught the Lab more about how things might fail. He currently has a set of updates which may further improve things, and these are liable to be tested at the next Server Beta User Group meeting.
Names Vanishing from Ban Lists
There have been reports of avatars added to a region / estate ban list or have been previously muted suddenly dropped from the list without an action on the part of the list owner. This might be connected to the old issue of bans made using radar on some older versions of v1-style viewers (notably Phoenix) failing to “stick”, or it may be something else, such as a failure to correctly update a ban / mute list.
Commenting on the subject at the Simulator User Group meeting, Simon said, “we’ve heard reports of that and have looked into it … if you ever can narrow down an instance of that happening, please note it in a JIRA … Our logs will record info about those changing but we have to know where and when to look
“I’m making wild guesses, but I think it would be either the viewer or the simulator making an update to the ban list, and somehow having bad data. Perhaps an incomplete list gets into the picture, and using that as a basis for the update it drops people. Our logs will show events like “MrNoisy was added” and “MrGoodBehaviour was removed” but finding the event is the missing part of the puzzle.
“If you have multiple regions in the estate, there’s another issue of having the changes sent out to all the regions. We’ve seen failures there and I know it’s been worked on a few times (and suspected in some of these reports).”
So, if you do encounter a situation involving an apparent ban list failure, and can log the exact circumstances / details, please consider raising a bug report.