Category: SL Tech

Server Deployments

Week 47 marks Thanksgiving in the USA so as reported last time, there have been no server-side deployments for the Release Candidate or main channels, and no rolling restarts. This is liable to continue into week 48 (week commencing Monday 26th November), as there is unlikely to be any deployment to the main channel. There will, however, be deployments to the RC channels, details TBA.

HTTP Updates – Texture Fetching

After indicating that there would be no viewer releases during week 47 in the run-up to Thanksgiving, the Lab rolled out the first of the 3.4.3 beta releases  – 3.4.3267135 – on November 20th. The major change to this is the inclusion of the first phase of Monty Linden’s new HTTP-based texture fetch capability, designed to significantly improve texture rezzing within the viewer. As the release notes state:

A new scheme for performing HTTP operations is introduced with this release. It is intended to reduce crashes and stalls while performing HTTP operations and generally enable performance and reliability improvements in the future. In this release, it is being used by the viewer’s texture retrieval code. Our expectation is that it will provide consistent and predictable downloading of textures. As well as the usual problem reporting, we’re also interested in confirmation of improvements where this release improves your experience.

The code for these improvements has already started appearing in some TPVs, and will doubtless be available across all flavours of the viewer in the near future.

Observable improvements in rezzing times have been reported by those who have used the project viewer releases of this code, so it should yield benefits for those using the beta. Monty Linden, who is handling this project is apparently now working on further improvements to the server-side of the equation, which should see additional improvements in the future.

Also pushed out during the week was a new version of the development viewer – 3.4.3.267201.

It is currently not clear when the renewed roll-out od beta and development viewers will result in updates appearing with the production version of the viewer, I believe that this may be additionally delayed while other requirements are put in place related to the Steam link-up (the code for the Steam link-up already having been merged into the beta viewer).

Volumetric Pathfinding

Also during the Tuesday Simulator meeting on November 20th, the question of volumetric pathfinding came up, and how pathfinding might be extended into the air, to allow birds, etc., and under Linden water. There are a range of issues with doing this – perhaps the biggest being the actual demand. There is also the matter of keeping birds and the like from crashing into buildings and skyborne objects, or in keeping fish in the water.

During the meeting, Baker Linden passed a question on the subject to Falcon Linden and indicated that Falcon felt, “It’d be about 3 months of work to get volumetric pathfinding — and that still wouldn’t handle dynamic avoidance (which is the hard part). Theoretically, it’s not that hard — it’s having to rework some Havok systems to work with intermediate data.”

This doesn’t mean that the work is about to be undertaken in any way whatsoever – just that were LL to consider it, getting the basics going for volumetric pathfinding going would take around three months. However, even then, unless the issue of dynamic hazard avoidance, it is unlikely this is something we’ll be seeing in SL for a while yet.

Server-side Object Rezzing Performance

Baker Linden indicated that he has started looking at server-side object rezzing. This work isn’t connected to Andrew Linden’s Interest list work, which is related to which assets the simulator should be loading ready for rezzing, but is rather focused on reducing the server-side lag which is induced when an object physically rezzes in a region. As Baker explained during the meeting, “If you get a really complex object, with many large meshes, or large LLSD files, it takes a while to rez into the world. I’m trying to reduce that.”

There are no timescales associated with the work, although it is expected that it will include avatar attachments as well as in-world objects have less of a performance / fps hit on the region when rezzing complex items, particularly in Baker can get the parsing of large object files to work asynchronously, which currently does not occur. Whether this will translate to visible viewer-side improvements is debatable.

SL Issues

Homestead Performance / Memory Issues

There have been growing reports of region performance issues occurring across the grid. These primarily appear to be impacting Homestead regions – although it can be encountered on full regions as well.

Essentially the problem manifests itself (for most users) when they find they are unable to rez objects in-world and / or as attachments, while raw prims created in-world may rez, but are reported as turning phantom on creation. The issue appears to be large and abnormal memory usage by the region’s physics system, although the precise causes as to why it is occurring are currently unknown.

Regions are allocated a fixed amount of memory that can use. In the case of a full simulator, this is about 1GB, while Homesteads are allocated around 250MB. Generally, physics memory usage for a region – even a busy one – is around 40-80MB. However, on affected Homestead regions, the physics memory use is reaching or exceeding 200MB.

When the physics memory for a simulator gets abnormally high (close to or on 90% of the allowed maximum) internal region safeguards kick-in and prevent object rezzing in an attempt to limit further calls of the region’s memory and keep it alive. This is the behaviour people are witnessing in their regions. The safeguards themselves are designed to help prevent regions from becoming unstable during griefing attacks. However, the problems people are experiencing appear to be entirely unrelated to any form of griefing, and are thus causing a certain amount of head-scratching at Linden Lab.

Reed Linden, in responding to a support request from Motor Loon, provides clear guidelines on what to do if you have a Homestead and experience these issues. It is thought that the most likely culprit for the problem is an unidentified memory leak, but this has yet to be confirmed. Reeds comments regarding particle systems are fascinating. Particles tend to be more viewer-intensive than server, and as many commented at the Simulator User Group meeting on Tuesday 20th November, it would take something bizarre to be going on for particles to be impacting region performance; however, at least one region affected by the issue appears to have a large number of particle emitters in operation.

A further interesting twist came at the meeting itself, when a pathfinding snake and a number of pathfinding characters were rezzed, and the region suffered a severe performance hit (sharp FPS drop experienced by all attendees, sharp increase in both physics memory use and time taken to ping the region) which appeared to be linked with the snake (which was set to follow its creator around) re-calculating its path to both follow its creator and avoid other avatars / objects. However, when the snake was re-rezzed a short time later, no similar issue was noticed, with the region using around 116MB physics memory, with no other outward performance issues.

I the meantime, and as the Linden dev team continue to investigate the issue, if you experience this kind of problem, please ensure you raise a support ticket, supplying as much information as possible, including region name / simulator version (from HELP > ABOUT (either Second Life or the name of your viewer), the time the problem occurred, how it manifested and, if possible, information from the Statistics bar: memory used, FPS and physics performance details, etc.