Due to the offline e-mail issue involving scripted objects, as reported in my last news update, there has been no Main Channel deployment this week. Two RC deployments are currently planned for Wednesday 12th December, however. These are:
BlueSteel and LeTigre: should receive the same maint-server project that rolled to Magnum in week 49, with bug fixes arising from that deployment. The release notes are available for review
Magnum should receive a superset of the changes scheduled for BlueSteel and LeTigre, which includes extra bug fixes, including stability improvements and a memory leak fix.
The only new feature new to Magnum is an increase in the allowed animation asset size – the 60KB size limit on animation assets has been raised to 120KB. This change is to allow for longer and more complex animations to be made in the future, once an viewer-side update to allow 60-second animation loops has been implemented. Magnum’s release notes can be read here.
So be sure to read them 🙂 (with thanks to Whirly Fizzle for the link)
Update on Key Region Issues
Physics Memory / Region Performance
As reported last time, the physics memory issues affecting some regions, which I reported in week 47, had been tracked down by Simon Linden to a Havok issue related to navmesh rebakes. His fix for this problem cleared QA and forms a part of the RC deployments for the 12th December, together with a fix for a low-level threading problem within the simulator code which has also been causing region crashes.
Offline IMs from In-world Objects Failing to Forward to E-mail
This issue, linked to llInstantMessage(llGetOwner(), caused the RC deployments in week 49 to be rolled back on Thursday 6th December. A fix has been developed and tested and is included in all three RC deployments planned for Wednesday 12th December.
Code Freeze / No Change Windows
Again, to re-iterate from my last report, there will be no server-side code changes over the holiday period as follows:
Week 52 – commencing Monday December 24th
Week 1, 2013 – commencing Monday December 31st
Simon Linden still hoped that one of the code being deployed to the RC channels this week can be rolled to the Main Channel in week 51. There will likely be a further update on this following the Thursday Server Beta UG.
JIRA / Bug Tracker Update
Linden Lab are still mulling the September closure of the old public JIRA system. Since the initial shut-down, things have opened up a little. Additional JIRAs have been left open as read only beyond the initial triage, while others have been opened and have had their comments enabled in order to allow feedback – such as the CHUI JIRA, which is being very constructively used for comments and feedback and shows how, in an ideal world, the system might work.
Currently, it appears that “nothing definitive” has been decided on the change, although it has been under internal discussion.
Feedback from those in the two JIRA support groups (developers who have significantly contributed code and those who have in the past supplied significant support in handling JIRAs) has been interesting. It appears that the number of feared duplicates on issues has been a lot smaller than had been feared. The overall quality of input given using the new form also appears to have been significantly improved since it was introduced.
Things have been relatively quiet viewer-wise with only two updates to the official viewer branches. The development viewer rolled to 3.4.4.267614 December 4th, while on December 5th, the beta viewer rolled to 3.4.3.267755. The latter included a good crop of updates, including a number of graphics and GPU support related changes, and the long-await snapshot tiling fix.
The MAINT-628 fix in action: an image taken with the latest beta viewer running in deferred mode and at a resolution of 3500×2154 pixels, well above the 1400×900 of my monitor – and no tiling! There are limits to how well the fix works at ultra-high resolutions, as noted in the JIRA comments included in my report on the release.
Server Deployments
Following-on from last week’s RC deployment issues, there was no main channel deployment on Tuesday December 4th, although a number of regions were restarted during the course of the day.
Wednesday December 5th saw the same maintenance release rolled to all three RC channels. This comprised the release originally aimed at Magum in week 48 and which included all bugs fixes for the problems which required the roll-back on Thursday 29th November. Initial statistics for this update during the brief time it was available last week showed a clear improvement in stability, and this seems to have continued with this week’s release, although there has been one major issue come to light and is under investigation.
This relates to IM messages sent by scripted objects failing to trigger e-mails to the object’s owner if they are off-line. The problem appears to be related to the use of llInstantMessage(llGetOwner(), and appears to affect regions on all three RC channels, but not every case where llInstantMessage(llGetOwner() is used appears to be affected.
Currently, it is thought that a fix will be available for deployment during week 50, and should reach the RC channels om Wednesday December 11th.
Details of the week’s RC release can be found in the release notes and in the forum discussion thread (including some discussion on the current scripted object / e-mail issue).
After a smooth deployment to the main channel on Tuesday 27th November, things got a little unsettled on Wednesday 28th November with the deployments to the RC channels. As noted in part 1 of this report, these were supposed to comprise a maint-server release to BlueSteel and LeTigre, with the same package and a few extras going to Magnum.
The problems started during the actual deployment on Wednesday, wherein after successfully updating Magnum and BlueSteel, the deployment team started noticing issues unrelated to the deployment which caused Coyot Linden to call off the LeTigre roll-out until things were sorted. However, as Maestro Linden takes up the story:
Then there were reports in the forums about offline IM emails from objects being broken; if an object sent you an IM and you were offline, the offline email would contain all the usual details *except* for the message. This bug affected both BlueSteel and Magnum since they both shared the responsible change. Then, after digging into offline emails a bit more, we noticed that the ‘To’ field of offline emails would show the object owner’s name instead of the recipient’s name … which was a little confusing … Anyway, these bugs were kind of bad, but we weren’t sure that they were worth the trauma and downtime of an emergency rollback …But then this morning, we became aware of a 3rd bug, from support. It turned out that deeding parcels to groups was failing.
It was this third bug which was deemed sufficiently serious enough to warrant a roll-back of the RC deployments, which took place on Thursday 29th November, with the result that all four channels are now running on the same release – 12.11.09.266804. Kelly Linden has fixes for all three issues, but they are currently in testing, and the plan is to try again next week with the RC deployments.
Region Performance / Memory Issues
The physics memory issues which I reported in week 47, and then provided an update on in Part 1 of this report have received further attention from Linden Lab. The problem has been with some regions experiencing severe physics memory bloat within a short time of being restarted, with the result being that they rapidly reach a threshold of memory use (~230MB for homesteads, ~920MB for full regions) which prevents rezzing of any objects, in-world or attached.
In investigating the issue, Simon Linden located a source of memory leak related to the Havok system which may address the issue, and is hopeful he has found a fix. Commenting on the matter at the Server Beta meeting on 29th November, “I’m in the midst of hacking a special test mode for a region … I’m going to make it continuously re-bake the navmesh and terrain data. I’ll let that run overnight and see what happens … I’m not going to claim victory quite yet … this is like that point where you hit the zombie hard, but you have to see if it comes back again.”
Aditi Grid Log-in Issues
It had been hoped that the Aditi situation, wherein problems with inventory-related data is preventing people from being able to log-in to the beta grid – would be discussed at the Server Beta meeting, but these were somewhat side-lined by discussions on other projects, most notably Interest Lists (work on which prevented Andrew Linden from digging into the matter following the Simulator User Group meeting on Tuesday 27th November).
While Maestro was able to confirm that a member of Linden Labs is, “Working on a script to help the Aditi inventory situation”, little more information was provided at the meeting due to other ongoing conversations. Hopefully, this matter will be picked up next week.
PATHBUG-183
In working on Interest Lists, Andrew Linden had hoped he’d sorted a fix for PATHBUG-183, which relates to offscreen physical objects flying across your in-world view. The code for the fix has been available on Ahern, on the Aditi grid, but there are reports that while the problem may have been somewhat reduced in scale, it is still very apparent. Andrew hopes to look into this during week 49.
Region “Flicker” and Content “Warping”
There has been some issues related to viewing neighbouring regions and crossing region boundaries. These issues take a number of forms, and are related to both server and viewer issues:
The entire neighbouring region “flickers” if you are near its edge
After crossing between regions, the content of the region you have just left seems to appear in the region you’ve just entered, usually somewhat warped / deformed
“Cachable” objects (e.g. unscripted and non-physical) objects vanish from your view of a region on leaving it
A region appears to “reset” (re-renders) itself shortly after leaving it
Sometimes on crossing between regions, objects from the region you’ve just left seem to appear warped and deformed in the region you’ve just entered……only to disappear after a few seconds / as you approach them
The first two of these issues are considered to be viewer-related bugs which are thought to have been resolved in code currently in the viewer development. The final item in the list is related to a server issue which Andrew Linden believes to have been fixed as a part of his work on Interest Lists. The “cacheable” objects problem was also considered to be a viewer-side bug which has also been addressed in viewer development, but in considering the problem at the Server Beta meeting, Andrew Linden thought there might actually be more than one bug responsible and indicated he would be looking into this some more.
After indications from LL that there may not be a Main channel deployment on Tuesday 27th November, restart commenced as the deployment made to the RC channels last week went ahead as per the usual schedule.
Wednesday 28th November should see the three main RC channels updated as follows:
BlueSteel and LeTigre: should receive a maint-server project. There are a few new flags for the LSL function llGetObjectDetails(), but the most important changes are some fixes for physics and mesh-based crash modes – see the server release notes
Magnum should receive the say package, with additional stability improvement changes – see the Magnum server release notes.
As usual there is a forum thread for the week’s server deployments.
Viewer News
Release Viewer
The 3.4.2 viewer code finally reached the release (production) version of the LL viewer with the release of 3.4.2.267137 on Monday 26th November, which I briefly reviewed here.
Beta Viewer
The beta viewer, now cleared of the crash issue bottleneck, moved rapidly through the 3.4.2 code base prior to Thanksgiving in the US, as previously reported in the news updates, and then reached 3.4.3 with the surprise release of 3.4.3.267135 during Thanksgiving week, after it had been indicated there would be no viewer releases during the week due to decreased support staff availability during the long weekend period. As reported last week, this release includes the first phase of Monty Linden’s HTTP texture fetch project, which should see people experiencing significantly faster texture rezzing when in-world.
CHUI Viewer
The CHUI – the Communications Hub User Interface – project viewer is due to go through another couple of iterations before moving towards a development / beta viewer code merge. There has already been one update since the project viewer, which is aimed at improving the capabilities and reliability of in-world text and Voice conversations, first appeared.
CHUI: potentially a couple more iterations to come
While he has not followed the project first-hand, Oz Linden believes CHUI to be nearing a “feature complete” status. The advice is that if you haven’t tried it out and wish to give feedback, now is the time to do so.
Nalates Urriah provides an update on some of the ongoing work around the mesh deformer. In the meantime, speaking at the Open Development User Group meeting on Monday the 26th November, Oz linden responded to a question from White Rabbit as to what garments are still required for testing by saying, “That’s a great question. I’m setting up a meeting with the people responsible for avatars to try to get a proper acceptance test defined for both that and STORM-1800.” STORM-1800 relates to the vertex weights of the default avatar character mesh.
While Oz didn’t specify a date for the meeting, those with a direct interest in either supplying mesh clothing for testing or in the JIRA should be hearing from him in the near future on the meeting details.
Week 47 marks Thanksgiving in the USA so as reported last time, there have been no server-side deployments for the Release Candidate or main channels, and no rolling restarts. This is liable to continue into week 48 (week commencing Monday 26th November), as there is unlikely to be any deployment to the main channel. There will, however, be deployments to the RC channels, details TBA.
HTTP Updates – Texture Fetching
After indicating that there would be no viewer releases during week 47 in the run-up to Thanksgiving, the Lab rolled out the first of the 3.4.3 beta releases – 3.4.3267135 – on November 20th. The major change to this is the inclusion of the first phase of Monty Linden’s new HTTP-based texture fetch capability, designed to significantly improve texture rezzing within the viewer. As the release notes state:
A new scheme for performing HTTP operations is introduced with this release. It is intended to reduce crashes and stalls while performing HTTP operations and generally enable performance and reliability improvements in the future. In this release, it is being used by the viewer’s texture retrieval code. Our expectation is that it will provide consistent and predictable downloading of textures. As well as the usual problem reporting, we’re also interested in confirmation of improvements where this release improves your experience.
The HTTP texture fetching code is now available in the latest SL beta viewer (3.4.3.367135)
The code for these improvements has already started appearing in some TPVs, and will doubtless be available across all flavours of the viewer in the near future.
Observable improvements in rezzing times have been reported by those who have used the project viewer releases of this code, so it should yield benefits for those using the beta. Monty Linden, who is handling this project is apparently now working on further improvements to the server-side of the equation, which should see additional improvements in the future.
Also pushed out during the week was a new version of the development viewer – 3.4.3.267201.
It is currently not clear when the renewed roll-out od beta and development viewers will result in updates appearing with the production version of the viewer, I believe that this may be additionally delayed while other requirements are put in place related to the Steam link-up (the code for the Steam link-up already having been merged into the beta viewer).
Volumetric Pathfinding
Also during the Tuesday Simulator meeting on November 20th, the question of volumetric pathfinding came up, and how pathfinding might be extended into the air, to allow birds, etc., and under Linden water. There are a range of issues with doing this – perhaps the biggest being the actual demand. There is also the matter of keeping birds and the like from crashing into buildings and skyborne objects, or in keeping fish in the water.
During the meeting, Baker Linden passed a question on the subject to Falcon Linden and indicated that Falcon felt, “It’d be about 3 months of work to get volumetric pathfinding — and that still wouldn’t handle dynamic avoidance (which is the hard part). Theoretically, it’s not that hard — it’s having to rework some Havok systems to work with intermediate data.”
This doesn’t mean that the work is about to be undertaken in any way whatsoever – just that were LL to consider it, getting the basics going for volumetric pathfinding going would take around three months. However, even then, unless the issue of dynamic hazard avoidance, it is unlikely this is something we’ll be seeing in SL for a while yet.
Server-side Object Rezzing Performance
Baker Linden indicated that he has started looking at server-side object rezzing. This work isn’t connected to Andrew Linden’s Interest list work, which is related to which assets the simulator should be loading ready for rezzing, but is rather focused on reducing the server-side lag which is induced when an object physically rezzes in a region. As Baker explained during the meeting, “If you get a really complex object, with many large meshes, or large LLSD files, it takes a while to rez into the world. I’m trying to reduce that.”
There are no timescales associated with the work, although it is expected that it will include avatar attachments as well as in-world objects have less of a performance / fps hit on the region when rezzing complex items, particularly in Baker can get the parsing of large object files to work asynchronously, which currently does not occur. Whether this will translate to visible viewer-side improvements is debatable.
SL Issues
Homestead Performance / Memory Issues
There have been growing reports of region performance issues occurring across the grid. These primarily appear to be impacting Homestead regions – although it can be encountered on full regions as well.
Essentially the problem manifests itself (for most users) when they find they are unable to rez objects in-world and / or as attachments, while raw prims created in-world may rez, but are reported as turning phantom on creation. The issue appears to be large and abnormal memory usage by the region’s physics system, although the precise causes as to why it is occurring are currently unknown.
Physics memory use can be monitored via the Statistics floater (CTRL-SHIFT-1)
Regions are allocated a fixed amount of memory that can use. In the case of a full simulator, this is about 1GB, while Homesteads are allocated around 250MB. Generally, physics memory usage for a region – even a busy one – is around 40-80MB. However, on affected Homestead regions, the physics memory use is reaching or exceeding 200MB.
When the physics memory for a simulator gets abnormally high (close to or on 90% of the allowed maximum) internal region safeguards kick-in and prevent object rezzing in an attempt to limit further calls of the region’s memory and keep it alive. This is the behaviour people are witnessing in their regions. The safeguards themselves are designed to help prevent regions from becoming unstable during griefing attacks. However, the problems people are experiencing appear to be entirely unrelated to any form of griefing, and are thus causing a certain amount of head-scratching at Linden Lab.
Reed Linden, in responding to a support request from Motor Loon, provides clear guidelines on what to do if you have a Homestead and experience these issues. It is thought that the most likely culprit for the problem is an unidentified memory leak, but this has yet to be confirmed. Reeds comments regarding particle systems are fascinating. Particles tend to be more viewer-intensive than server, and as many commented at the Simulator User Group meeting on Tuesday 20th November, it would take something bizarre to be going on for particles to be impacting region performance; however, at least one region affected by the issue appears to have a large number of particle emitters in operation.
A further interesting twist came at the meeting itself, when a pathfinding snake and a number of pathfinding characters were rezzed, and the region suffered a severe performance hit (sharp FPS drop experienced by all attendees, sharp increase in both physics memory use and time taken to ping the region) which appeared to be linked with the snake (which was set to follow its creator around) re-calculating its path to both follow its creator and avoid other avatars / objects. However, when the snake was re-rezzed a short time later, no similar issue was noticed, with the region using around 116MB physics memory, with no other outward performance issues.
I the meantime, and as the Linden dev team continue to investigate the issue, if you experience this kind of problem, please ensure you raise a support ticket, supplying as much information as possible, including region name / simulator version (from HELP > ABOUT (either Second Life or the name of your viewer), the time the problem occurred, how it manifested and, if possible, information from the Statistics bar: memory used, FPS and physics performance details, etc.
Wednesday November 14th saw the same code deployed to all three RC regions in preparation for the US Thanksgiving week code freeze (see below). This primarily brought all three RCs to the same code level, release-wise and fixed a bug discovered in week 45.
There will be no rolling restarts in week 47 (week commencing Monday 19th November) due to the code freeze.
SL Viewer Update
The beta viewer rolled to the last of the 3.4.2 releases on Thursday November 15th, with the release of 3.4.2266995. Providing the crash statistics remain good (they were very positive for the first 24 hours), it will be going to the production (release) viewer QA in week 47 and should result in the release of a new version of the viewer shortly after the Thanksgiving weekend.
This beta contains a lot of updates and improvements, as well as a wide range of fixes for issues encountered with earlier releases up to and including the previous beta release, 3.4.2.266708, issued on November 8th. For a comprehensive list of changes, please refer to the release notes.
At the same time, the development viewer rolled to 3.4.3.267061, marking a further update of the development viewer to the 3.4.3 code, which should be appearing in the next release of the beta viewer. This include the new viewer-side code for the HTTP texture fetch project developed by Monty Linden as a part of the Shining Project improvements.
The code for faster texture fetching / rezzing has moved from a project viewer into the viewer-development stream and is present in the 3.4.3267061 Ddevelopment viewer release and should appear in the beta viewer after the US Thanksgiving weekend
As the holiday period is approaching, viewer releases will be slowing down, but the aim remains to try to clear the backlog of waiting merges and updates by the New Year with a view to resuming their more usual cycle of releasing a development / beta update around every three weeks. Once things are back on track, LL will be looking more closely once again at the question of disabling tcmalloc completely within the viewer.
Code Change Freezes
The official dates for code change freezes during the upcoming holiday periods currently stand at:
Week 47 – Monday 19th November through to Sunday 25th November
Week 51 – Monday 17th December through Sunday 23rd December
Week 52 – Monday 24th December through Sunday 30th December.
No status has yet been given for the New Year week (Monday 31st December through Sunday 6th January 2013.
Avatar Baking
Bake fail: work on new service progressing
On Friday 16th November, Nyx Linden provided a brief update on the Avatar Baking project, which again forms a part of the Shining project improvements to Second Life. In short:
The viewer code is starting to look stable. However, merging the new code into the existing viewer code is liable to be somewhat more “painful” (Nyx’s term) than had been hoped
Work is progressing on the server-side elements (the composite baking server), with the code reaching a point where avatar texture can be generated server-side
Currently, the plan is to continue working on both sides of the equation, with the aim of ensuring the new code is successfully merged into the viewer development branch, and then offering it in a “very alpha” form to TPVs so that they can start merging it into their code and assist with testing. As this happens, one or two regions on the beta (Aditi) grid will be set-up to allow testing on the new service.
There is still no time frame for the appearance of the viewer code in the development branch or any test regions on Aditi, but in Nyx’s view, both are liable to be on the horizon “pretty soon”. Overall, the plan still remains to have at least a two month period from when the code is made available for testing purposes through to the official implementation of the new service.
Interest List Demonstration and Weirdness
The focus of this project is to optimise the data being sent to the viewer, information already cached on the viewer and the manner in which that data is used in order to ensure it is used more efficiently so that things rez both faster and in a more orderly manner than is currently the case.
This work is being undertaken by Andrew Linden in a number of stages, the first being to clean-up the code related to information sent to the viewer from the simulator relating to objects in the camera’s viewing range such that only objects actually in the camera frame are updated (if updates are required) and that objects outside of the current camera frame are not updated, thus reducing the amount of data both the server and the viewer need to process, which should lead to performance improvements.
It is possible to visualise the update process using a viewer debug setting (Develop > Show Info > Show Updates to objects or Ctrl+Alt+Shift+U) to show object updates being received by the viewer. This shows updates in three colours: red, which indicates the viewer is receiving a “full” update on the object (generally because it is being “seen” / is within draw distance) for the first time; blue, which indicates the viewer already has data on the object and is receiving “terse” updates relating to changes in the object’s appearance / position relative to the viewer’s camera position; and green, which indicates the object has been deleted / remove from the camera view, and updates are about to cease.
On the 15th November, Andrew used this debug setting, together with a set of scripted “bouncing” cubes to demonstrate his improvements to this update process. Observers were asked to focus their camera on the cubes, which were initially static and had no coloured data stream associated with them. The cubes were then set bouncing, which generated a stream of blue “terse” updates, indicating the motion of the cubes in the viewer was being updated. However, when observers angled their camera to view the space above the cubes (so the cubes were not directly in their world view), the update stream ceased – indicating the viewer was no longer receiving update data for the cubes.
This may seem a small change, but it does dramatically decrease the amount of information the viewer has to process relating to in-world objects, and should result in performance improvements within the viewer. In the future, further work will be undertaken to enhance the interest list code even further – such as prioritising the order in which objects in the world view are actually rezzed, so that items closest to the camera view are rezzed first, etc.
HUD Issues
Following the demonstration, however, some people started noticing an odd issue: they could right-click on the centre of their screen and reveal a prim attachment belonging to someone else’s HUD. Chieron Tenk was the first to raise the issue, although Ana Stubbs also quickly reported the same problem.
Chieron Tenk posted an image of the problem: a prim appears in the centre of his viewer which, on inspection, appears to be linked to a HUD worn by Rex Cronin
After initial confusion, investigation revealed it was possible for anyone to find they had random prims from other people’s HUDs appearing on their screen, simply by attaching a HUD or a prim to their own screen. Concerns were further raised when it appeared that events might be able to be triggered if the prim was touchable.
I find I have a prim belonging to Ana Stubb’s Mystitool appearing on my screen
The issue appears to be tied to changes made to the interest list code on the test region, and is obviously going to be the subject of further investigation on the Lab’s part prior to the interest list project being carried forward.
