SL project updates week 11 (1): server deployments, viewer, and interest list updates

Server Deployments

Second Life Server (Main channel)

On Tuesday March 12th, the Main channel received Baker Linden’s large object rezzing project which had been deployed to BlueSteel and LeTigre in week 10. This project is designed to improve simulator handling of “large” (as a file size / complexity, rather than physical object size) so that the simulator does not stall / choke when handling one or more such objects. makes sim performance smoother while objects are being rezzed. Further details on the project are in my week 10 update, and the server release notes are available in the SL wiki.

Release Candidate Channels

On Wednesday March 13th, the Release Candidate channels should be updated as follows:

  • BlueSteel and LeTigre: both of these channels should be receiving the same server maintenance package, intended to fix a common crash mode – release notes
  • Magnum should receive an update to the server maintenance package it received in week 10, with further improvements / fixes. These include the removal of the fix for VWR-786, which rather than correctly fixing the known issue (IMs to friends do not respect their privacy settings) resulted in all IMs to non friends returning the “User is not online” message, regardless as to whether the recipient was online or not. Release notes for the package are on the SL wiki

SL Viewer

There has been no major viewer movement since the last update in week 10.

CHUI, the Communication Hub User Interface

CHUI looks set to be merged-in to the Snowglobe code, with Oz Linden stating he was hoping to start on this on Monday 11th March. There are concerns as to how LL’s ongoing work with the viewer might impact TPVs going forward. As it is, CHUI is liable to remain in the SL beta viewer for a while (and there is expected to be one more CHUI release into beta, as perviously noted).

Materials Project

Work is continuing on clearing the current issues within the viewer code, with a further push of the non-public viewer expected this week, which may resolve some of the problems.

The Potential LL Roadmap for Viewer Releases

While things are always in a state of flux, the potential order of viewer releases from LL’s perspective is currently veering towards:

  • Materials is now unlikely to “be seen” until after the code has merged with the CHUI code  – this follow-on from the SSB code currently undergoing a merge with CHUI and the move to merge CHUI with the Snowglobe code mentioned above
  • This does not mean that a materials project viewer will not appear prior to CHUI reaching the SL release viewer; rather it means that when a materials project viewer appears, it is likely to have CHUI incorporated into it
  • That said, materials will likely only arrive in the release version of the SL viewer after CHUI has been formally released and (most likely) SSB has been deployed
  • Other updates – FMODex is currently awaiting CHUI as well, but has no clear release date; the same is true of the Mac Cocoa project. Currently, it appears as if these are unlikely to reach mainstream release until after CHUI has been formally released.

Most TPVs are currently focusing on the Server-side Baking (SSB) integration, as this has a significantly greater impact on viewers in terms of the impact on  users than CHUI (although the latter is by far the most complex update as it involves a lot of code refactoring as well as CHUI updates). As such, it is likely to be a while before CHUI starts appearing in the majority of third-party viewers (although Kokua has already merged with it, and now has the CHUI code in the beta branch of its code).

Merchant Outbox Project Viewer

As reported over the weekend, Linden Lab has re-issued the Merchant Outbox project viewer, updated to the 3.4.4 viewer code, but which does not incorporate CHUI. This release is purely to assist merchants who are encountering issues in migrating to Direct Delivery now that the initial retirement of Magic Boxes has been announced.

Those who have / are encountering problems in migrating to Direct Delivery can obtain the viewer from the Alternate Viewers wiki page.

This project viewer will be withdrawn at some point in the future, and will not impact other viewer releases.

Server-side Baking

A further reminder that there will be a further SSB pile-on / load test on Thursday March 14th, following-on, as with the last test, from the Server Beta meeting on Aditi. For wishing to participate:

  • The test is liable to be in much the same format as the first test
  • Those participating should be running the latest version of the official SSB project viewer (3.4.5.271419)
  • Participants should have a number of outfits of system clothing, preferably with multiple layers, which they can swap between during the course of the test. Library outfits are acceptable, but LL are keen for people to use their own outfits to add greater weight to the tests
  • Clearing the viewer cache prior to the test is suggested, but not an absolute requirement.
The first SSB pile-on / load test (image courtesy of Latif Khalifa
The first SSB pile-on / load test (image courtesy of Latif Khalifa

The project also seems to be going through a further informal name change: originally referred to as “Server-side Baked Texture Generation & Storage”, the project has generally been shortened to “Server-side Baking”, but is now tending to be referred to as “Server-side Appearance project”. I’ll be continuing to refer to it as “Server-side Baking” or “SSB” for ease of reference.

Interest List Work

Issues

There are increasing reports of odd behaviour which appears to be possibly tied-in to the recent interest list updates. These include:

  • An issue whereby a user can teleport into a somewhat busy region and see everything normally – other avatars moving around, objects, etc., and animations play, but the use fails to see their avatar physically move around, even though others do. This doesn’t happen on every teleport to busy regions, and LL are having problems reproducing it (see below)
  • Further region crossing issues, which have been seen to particularly affect aircraft (but which are not limited to such, other fast-moving vehicles can be affected):

  • Reports that scripted “clear visible cache” tools (such as the “rockets” commonly found across the grid which fire one up to high altitude then back to the ground to trigger a render refresh) and reducing the viewer’s Draw Distance to zero no longer work as a means of correcting apparently deformed avatars  / objects (e.g. you see a malformed avatar nearby, so drop Draw Distance to zero and bring it back up quickly and see the avatar render properly)

In terms of these issues:

Avatar movement issues following teleport: this first has been noted by the Lab and they have requested assistance in reproducing. Those able to do so are requested to file a bug report, supplying: the region on which they encountered the issue, the time at which the issue occurred, and the viewer / simulator details (which can be copied directly from Help -> About [viewer name]

Region crossings in fast-moving vehicles: LL were having problems reproducing this issue, but have recently had success in doing so as a result of (non-public) BUG-1814 being filed. Following Maestro Linden’s comment in the week 10 deployment thread linking the issue to interest list changes, I took the opportunity to ask Andrew Linden for his thoughts on the matter during the Simulator User Group meeting on Tuesday 12th March. He replied:

Inara, I think it is possible that problem crossing regions boundaries on vehicles is related to interest list changes. I had an idea on something that might fix the problem …. I suspect it has something to do with long ping-times… some kind of race condition … I guess that is what I’ll be working on tomorrow.

If it is a race condition / long ping times issue, this might explain why many of those experiencing the problem appear to be based outside of the USA, with several of the reports  / complaints in particular coming out of Europe and the UK. Given Andrew has additional work to complete on upcoming interest list work (see below), they may not be further updates on this problem in time for the Server Beta meeting on Thursday 14th March.

“Clear visible cache” issues: the issue with “clear visible cache” tools no longer working may be down to the fact that they force the simulator to send KillObject messages, but the interest list code is no longer responding to the messages as it did previously. Andrew explained the matter thus:

The KillObject message has two meanings; it is ambiguous. It means either (1) the object was deleted or (2) the server thinks the object is no longer in your interest list, and will stop sending updates for it. in either case the viewer is supposed to delete the object from its render pipeline. So the new interest list code still sends KillObject messages, however it is a bit lazier about getting around to sending them, especially when the viewer’s camera is still moving around [as it would be in tracking the motion of something like a “clear visible cache rocket]. That is, the “clear cache” rockets would probably work if they go way up, sit up there at rest for about 5 seconds, and then come back down.

Andrew further suggests the same might be true when reducing Draw Distance to zero – that holding it at zero for 5 seconds or so before pushing it back up to a preferred default – might better force updates to be handled as expected. He also noted that while the viewer allows Draw Distance to be set to zero, it is worth remembering that the server forces a minimum draw distance of 10 metres for its own logic – so zapping DD down and up for objects within this distance may not resolve any issues anyway.

Upcoming RC Deployment

Andrew Linden’s additional work on the interest list code has cleared QA and looks set for a Release Candidate channel deployment in week 12. These changes focus on:

  • Improved scene rendering
  • Better reporting on moving objects outside of the camera’s field-of-view
  • Changes to object cacheing

The improved scene rendering should particularly help those using lower viewer bandwidth settings, although the overall improvements should be much broader than this. “If you login to a region with a clear cache the objects near you will show up much faster,” Andrew explained at the Simulator User Group meeting.

thth
The latest work on interest list should help with nearby object rendering, particularly for those using lower viewer bandwidths

To test the capability Andrew is attempting to use the same region on both the beta (Aditi) grid and the main (Agni) grids so that direct comparisons can be made (with the Agni region running the current code and the Aditi region the new code). As such, the region needs to have broadly the same number of objects within it. However, the region initially selected is rated Adult, which raised concerns over accessibility at the Simulator User Group meeting, so Andrew is going to attempt to move things to a suitable pairing between Aditi and Agni where the region rating will not be an issue, and will be reporting back on this at the Server Beta meeting on Thursday 14th March. I’ll provide an update on the rest region name at that time.

Better reporting on moving objects outside of the camera’s field-of-view comprises two updates.

With the current release of the code, the server stops sending updates for moving objects which are outside of the immediate field of view for the camera / viewer; “full” updates only resume when the object either enters the field of view or the camera is rotated such that the object enters the field of view. This can lead to a variety of odd results from the user’s point-of-view: object appearing to “jump” from one point to another on entering the field of view as the camera is rotated, or pathfinding characters failing to show-up on the mini-map until they enter the field of view, etc.

Under the new code, updates for moving objects will once again get sent to the viewer, but at a significantly lower rate than was the case before the new interest list code was deployed.

The second improvement to this comes in the use of “caxhe probes”, which Andew explains thus:

The server sends what we’re now calling “cache probes” (ObjectUpdateCached messages) for the cacheable objects behind you and if they are in cache then the viewer will put them into the render pipeline, even if they are out of view, at least… the viewer puts them on the mini-map.

The changes to object cacheing are intended to improve the handling and re-use of cached objects.

I’ve expanded the definition of what is “cacheable” for the object cache data. Currently it is the case that “cacheable” means it is static and doesn’t have any script. The reason for this is that we’ve basically got two big lists of objects on the server: “passive” objects that don’t move and don’t have scripts and “active” objects that are moving or have scripts.

The two lists were being used for two different purposes: (1) for script execution and (2) for interest list sorting cacheable vs non-cacheable.

I changed the interest list to look on both lists and send cache probes and updates for any object that hasn’t changed in a couple of minutes which greatly expands the set of cacheable objects, so there should be more cache hits going forward.

Expect to hear more on this going forward.