SL Project updates week 13/2: TPV Developer meeting – HTTP, VMM and more

Matoluta Sanctuary (Flickr) – blog post

The following notes are primarily taken from the TPV Developer meeting held on Friday, March 27th,  a video of which is included towards the end of the article (my thanks as always to North for recording it and providing it for embedding), and from the Server Beta meeting held on Thursday, March 26th. Any time stamps contained within the following text refer to the TPV developer meeting video.

Server Deployments Week 13 – Recap

As always, please refer to the deployment thread in the forums for the latest updates / news.

• On Tuesday, March 24th, the Main (SLS) channel received the server maintenance package deployed to the three RCs in week 12, comprising updates which allow the Lab to make various configuration changes without having to necessarily run a rolling restart when they have done so. It contains not actual functional changes to the simulator software
• On Wednesday, March 25th, the three RC channels received the same new server maintenance package, which is focused on inventory loss issues, and provides the Lab with better error detection and logging, improving their ability to look at some of the failure places and the removal of unused code. This updates does not remove the server-side messaging used in support of RTLP.

SL Viewer Update

Avatar Layers Project Viewer

Vir Linden’s work on a new global limit for system layer clothing was released as a project viewer, version 3.7.26.299805. With this viewer, a user can wear any combination of clothing layers (wearables), up to a maximum of 60, rather than being limited to (in general, and as with the official viewer) to a maximum of 5 items per layer type. Note that these changes do not apply to body part wearables (skin, shape, hair, eyes), for which the limit is still one of each, and do not affect attachments, for which the limit is still 38 total.

[07:18] There is already an update in the pipe for this viewer, which should be appearing next week.

Camera Positioning / Handling

[05:12] While there are no specific details as yet, the lab is hoping to put some work into improving camera positioning and handling in the not too distant future, in the hope of removing various glitches and issues.

Build Tools Viewer

[05:54] There have been a few fixes added to this viewer (currently version 3.7.26.299443), so a further update to the release candidate version is with the Lab’s QA team and should be appearing in week #14 (week commencing Monday, March 30th).

Maintenance Release Viewer

[06:29] Currently at version 3.7.26.299845, the latest Maintenance release viewer has a range of issues, many of which have hopefully been addressed with a series of fixes, so an update to that viewer is also with the Lab’s QA team. However, given the scope of the updates, it is proving a little harder to pass the QA process.

Experience Tools Viewer

[06:50] The Experience Keys / Tools viewer (currently version 3.8.0.299338) is being merged-up with the latest release version of the viewer code (version 3.7.26.299635). The updated version should also be appearing (again as an RC) in week #14.

Viewer Code

[17:27 – 19:50] There is an interesting discussion on the viewer code which, for anyone interested in how the viewer has developed over the years, and how much of it dates back some 14 years.

Viewer-Managed Marketplace

[00:00] There was a pile-on test of the new Viewer-Managed Marketplace capability on Aditi in week #12, and Brooke Linden was at the TPV Developer meeting to provide feedback. The pile-on test did not reveal any significant issues in terms of performance.

However, there is still a viewer / simulator / marketplace  communications issue which has to be resolved, which may take another couple of weeks to fix. After that, there are two grid deployments which need to take place: one for the VMM code itself, and one for updates to the Advanced Inventory System (AIS), so it is unlikely VMM will be fully deployed within the next month to two month, and the project viewer (currently version 3.7.25.298865) is unlikely to progress through a release candidate to release status until after the server components have been deployed.

Group Chat

[07:32] Simon Linden has been continuing to work on the group chat code, and all of his current updates should have been deployed to the back-end group chat servers. A broad consensus is that the issues which recently occurred as a result of some changes have been reversed, and that the group chat service as a whole is now running a lot better, both in terms of the early performance improvements Simon made, and with regards to the overall stability of the service and the servers.

[08:24] There is a further round of updating in the planning, but these require a platform upgrade to be carried out for the group chat service first. Therefore, unless unless the latest set of updates deployed by the Lab start to show issues, the engineering team will be switching focus for the immediate future, and will return to working on group chat once the necessary upgrade work has been completed.

Experience Keys  / Tools

[09:20] One of the items the engineering team want to focus on in particular is Experiences, and getting the remaining back-end issues sorted out so that Experiences can be properly deployed.

Voice Updates

[09:59] There will be a further round of voice updates which are expected to appear in a project viewer “shortly”. They include (but are not limited to) things like general code clean-up to prevent unnecessary list loading, removal of media messaging in person-to-person calls (which has never worked), fixes for issues related to microphone volume and improvements to the microphone test so that you can now hear yourself when testing your microphone, and improvements for hot swapping microphones / headsets.

[13:58] There is some confusion over whether or not a fix to voice designed to prevent someone’s voice channel being “left behind” when teleporting between regions has actually worked. It had been thought that the fix for this had been deployed in later 2014. However, bug reports are still being made still reporting issues (see BUG-8543 and STORM-2109), prompting the Lab to re-examine the status of the fix.

[19:54] Voice package updates from Vivox are also expected to be forthcoming in the future as well.

Restore To Last Position (RTLP)

[21:08] There have been around 400 responses to the Firestorm call for feedback on how people use the Restore To Last Position functionality found in some TPVs. As I’ve previously reported, the Lab had been considering deprecating the server-side message RTLP uses as an overall part of on-going work to reduce the amount of inventory loss issues (real or perceived) which can occur.

Firestorm’s call is helping the Lab to better understand how, as faulty as it might be, RTLP does fulfil a range of useful / valid use cases. Commenting on the fact the he has been reading through the feedback, Oz Linden said:

[21:49] Well, I understand that there are user scenarios that need to be addressed and need to be better supported. Whether the existing feature is the way to do that or not, I still consider to be an open question. I do want to take those use cases and work back through that process [of determining how best to serve them].

So the Lab still isn’t going to do anything “quickly” either way on RTLP, and people needn’t worry about RTLP vanishing / breaking “suddenly”.

In the meantime, they are working on other changes intended to address various rezzing failure situations. This work is more server-side focused, although it may be a while before updates appear on the grid as the exact nature of the updates is still being determined.

[23:42] Oz also again thanked everyone who responded to the Lab’s call for feedback on inventory losses in general, defining the feedback as “really, really useful”.

HTTP Pipelining Improvements and Issues

[33:42] In the last set of viewer updates for HTTP pipelining, Monty Linden implemented a change to reduce the delay time (from 150 seconds to 60 seconds) which occurs between re-tries on texture / mesh fetching should the process stall. However, he indicated the real fix for this would be for the Lab to add a curl update to the viewer. This will likely occur once the Lab has ensured all of Monty’s HTTP code changes have been fully implemented within the viewer.

[39:00] Some anti-virus packages have also been identified as causing issues for HTTP pipelining (e.g. Norman AVE suite and Forticlient – see BUG-8711 and BUG-8631 as examples). A problem here is that issues with AV software and firewalls can occur purely as a result of how the user configures the software, so it is hard for the Lab to deal directly with such problems.

Attachment Reliability

[43:05] As I’ve reported in week #8 and week #9, there are on-going issues related to attachments. Vir Linden been addressing these. The work should be appearing in a project viewer at some point, but this is unlikely to be in the next week or two.

[49:15] The Lab do not claim that the fixes will resolve all issues of attachments randomly detaching following a TP or of failing to attach, as there are many external factors which may also be causing race conditions to occur which lead to such issues (such as the connection through the Internet between the viewer and LL’s servers and how well it handles messaging), but that are hopeful the update will address many of the problems they can identify and deal with.

 Other Items

Rendering / Rezzing Failures and the CDN

There have been increased reports of rendering / rezzing issues over the past month, or so, many of which have been attributed to the CDN, although some might equally appear to be the result of  possible local caching issues as well. However, reporting on the forums and in the JIRA (BUG-8767),  Samual Wetherby appears to have identified an issue involving Florida-base ISP Mediacom and the “local” CDN node, which can be traced to a specific time slot. While this is not a problem LL can fix directly, they are planning further improvements to the CDN configurations in the coming weeks which may help.

Support Liaison

[25:10] The matter of possible closer support liaison between the Lab’s support staff and TPVs was again raised. Izzy Linden, having fed the idea up to the support managers at LL following the original suggestion, There is concern within the Lab that establishing such liaison outside of what might be regarded as “normal” support channels could lead to issues of support staff being sidetracked into dealing with matters outside of the usual process / workflow, etc., although there is an acknowledgement that there are cases where such closer liaison could help in dealing with certain types of issue.

Premium Memberships

[29:40] Paige Theseus raised a suggestion that Premium membership packages should be available as gift which can be bought for others. This segued into a broader discussion on how Premium accounts are blocked from logging-in to SL should billing fail (land and / or Premium renewal), rather than falling back the Basic. Izzy Linden puts forward a reasoned case as to how the Lab handles matters, and why they handle “delinquent” Premium accounts  in the manner they do, and what might be a means to improve things.

Oz linden Interview

Episode #60 of the Drax Files Radio Hour includes an interview with Oz Linden, recorded prior to his appearance at the VWBPE conference, where he responded to questions from the audience about the Lab viewer and viewer release process, and sought to understand more of the issues the education sector have with the viewer (such as how automatic  / mandatory updates can be problematic, etc). In the interview, he revealed some astonishing information on the amount of avatar, mesh and texture data Second Life handles:

We recently switched to delivering mesh and texture data, and a little bit further ago avatar texture data, through content delivery networks. That has proven to be very, very successful; the amount of data our content delivery network delivered to users in January [2015] was pretty close to 1.9 petabytes.

However you look at it, that’s an stunning amount of information. There’s some interesting commentary and observations made during the interview, which commences at the 34:40 mark into the podcast; so if you’ve not already listened, why not hop over to TDFRH blog and have a listen.

Firestorm Releases

Things are in a state of flux, but it currently looks as if the next release of Firestorm (currently delayed from March) will now be classified a “beta” release, and will include all current code found in the LL viewer. It will then be followed-up by a “full” release, which may include additional updates from the Lab, such as attachment fixes and VMM. depending on the overall status of such work. There are also likely to be two limited “preview” releases of the viewer, aimed at testing for possible issues related to AIS v3 and HTTP pipelining, in order for the Firestorm team to determine the percentage of users who may be affected by such issues.