After indications from LL that there may not be a Main channel deployment on Tuesday 27th November, restart commenced as the deployment made to the RC channels last week went ahead as per the usual schedule.
Wednesday 28th November should see the three main RC channels updated as follows:
BlueSteel and LeTigre: should receive a maint-server project. There are a few new flags for the LSL function llGetObjectDetails(), but the most important changes are some fixes for physics and mesh-based crash modes – see the server release notes
Magnum should receive the say package, with additional stability improvement changes – see the Magnum server release notes.
As usual there is a forum thread for the week’s server deployments.
Viewer News
Release Viewer
The 3.4.2 viewer code finally reached the release (production) version of the LL viewer with the release of 3.4.2.267137 on Monday 26th November, which I briefly reviewed here.
Beta Viewer
The beta viewer, now cleared of the crash issue bottleneck, moved rapidly through the 3.4.2 code base prior to Thanksgiving in the US, as previously reported in the news updates, and then reached 3.4.3 with the surprise release of 3.4.3.267135 during Thanksgiving week, after it had been indicated there would be no viewer releases during the week due to decreased support staff availability during the long weekend period. As reported last week, this release includes the first phase of Monty Linden’s HTTP texture fetch project, which should see people experiencing significantly faster texture rezzing when in-world.
CHUI Viewer
The CHUI – the Communications Hub User Interface – project viewer is due to go through another couple of iterations before moving towards a development / beta viewer code merge. There has already been one update since the project viewer, which is aimed at improving the capabilities and reliability of in-world text and Voice conversations, first appeared.
CHUI: potentially a couple more iterations to come
While he has not followed the project first-hand, Oz Linden believes CHUI to be nearing a “feature complete” status. The advice is that if you haven’t tried it out and wish to give feedback, now is the time to do so.
Nalates Urriah provides an update on some of the ongoing work around the mesh deformer. In the meantime, speaking at the Open Development User Group meeting on Monday the 26th November, Oz linden responded to a question from White Rabbit as to what garments are still required for testing by saying, “That’s a great question. I’m setting up a meeting with the people responsible for avatars to try to get a proper acceptance test defined for both that and STORM-1800.” STORM-1800 relates to the vertex weights of the default avatar character mesh.
While Oz didn’t specify a date for the meeting, those with a direct interest in either supplying mesh clothing for testing or in the JIRA should be hearing from him in the near future on the meeting details.
Week 47 marks Thanksgiving in the USA so as reported last time, there have been no server-side deployments for the Release Candidate or main channels, and no rolling restarts. This is liable to continue into week 48 (week commencing Monday 26th November), as there is unlikely to be any deployment to the main channel. There will, however, be deployments to the RC channels, details TBA.
HTTP Updates – Texture Fetching
After indicating that there would be no viewer releases during week 47 in the run-up to Thanksgiving, the Lab rolled out the first of the 3.4.3 beta releases – 3.4.3267135 – on November 20th. The major change to this is the inclusion of the first phase of Monty Linden’s new HTTP-based texture fetch capability, designed to significantly improve texture rezzing within the viewer. As the release notes state:
A new scheme for performing HTTP operations is introduced with this release. It is intended to reduce crashes and stalls while performing HTTP operations and generally enable performance and reliability improvements in the future. In this release, it is being used by the viewer’s texture retrieval code. Our expectation is that it will provide consistent and predictable downloading of textures. As well as the usual problem reporting, we’re also interested in confirmation of improvements where this release improves your experience.
The HTTP texture fetching code is now available in the latest SL beta viewer (3.4.3.367135)
The code for these improvements has already started appearing in some TPVs, and will doubtless be available across all flavours of the viewer in the near future.
Observable improvements in rezzing times have been reported by those who have used the project viewer releases of this code, so it should yield benefits for those using the beta. Monty Linden, who is handling this project is apparently now working on further improvements to the server-side of the equation, which should see additional improvements in the future.
Also pushed out during the week was a new version of the development viewer – 3.4.3.267201.
It is currently not clear when the renewed roll-out od beta and development viewers will result in updates appearing with the production version of the viewer, I believe that this may be additionally delayed while other requirements are put in place related to the Steam link-up (the code for the Steam link-up already having been merged into the beta viewer).
Volumetric Pathfinding
Also during the Tuesday Simulator meeting on November 20th, the question of volumetric pathfinding came up, and how pathfinding might be extended into the air, to allow birds, etc., and under Linden water. There are a range of issues with doing this – perhaps the biggest being the actual demand. There is also the matter of keeping birds and the like from crashing into buildings and skyborne objects, or in keeping fish in the water.
During the meeting, Baker Linden passed a question on the subject to Falcon Linden and indicated that Falcon felt, “It’d be about 3 months of work to get volumetric pathfinding — and that still wouldn’t handle dynamic avoidance (which is the hard part). Theoretically, it’s not that hard — it’s having to rework some Havok systems to work with intermediate data.”
This doesn’t mean that the work is about to be undertaken in any way whatsoever – just that were LL to consider it, getting the basics going for volumetric pathfinding going would take around three months. However, even then, unless the issue of dynamic hazard avoidance, it is unlikely this is something we’ll be seeing in SL for a while yet.
Server-side Object Rezzing Performance
Baker Linden indicated that he has started looking at server-side object rezzing. This work isn’t connected to Andrew Linden’s Interest list work, which is related to which assets the simulator should be loading ready for rezzing, but is rather focused on reducing the server-side lag which is induced when an object physically rezzes in a region. As Baker explained during the meeting, “If you get a really complex object, with many large meshes, or large LLSD files, it takes a while to rez into the world. I’m trying to reduce that.”
There are no timescales associated with the work, although it is expected that it will include avatar attachments as well as in-world objects have less of a performance / fps hit on the region when rezzing complex items, particularly in Baker can get the parsing of large object files to work asynchronously, which currently does not occur. Whether this will translate to visible viewer-side improvements is debatable.
SL Issues
Homestead Performance / Memory Issues
There have been growing reports of region performance issues occurring across the grid. These primarily appear to be impacting Homestead regions – although it can be encountered on full regions as well.
Essentially the problem manifests itself (for most users) when they find they are unable to rez objects in-world and / or as attachments, while raw prims created in-world may rez, but are reported as turning phantom on creation. The issue appears to be large and abnormal memory usage by the region’s physics system, although the precise causes as to why it is occurring are currently unknown.
Physics memory use can be monitored via the Statistics floater (CTRL-SHIFT-1)
Regions are allocated a fixed amount of memory that can use. In the case of a full simulator, this is about 1GB, while Homesteads are allocated around 250MB. Generally, physics memory usage for a region – even a busy one – is around 40-80MB. However, on affected Homestead regions, the physics memory use is reaching or exceeding 200MB.
When the physics memory for a simulator gets abnormally high (close to or on 90% of the allowed maximum) internal region safeguards kick-in and prevent object rezzing in an attempt to limit further calls of the region’s memory and keep it alive. This is the behaviour people are witnessing in their regions. The safeguards themselves are designed to help prevent regions from becoming unstable during griefing attacks. However, the problems people are experiencing appear to be entirely unrelated to any form of griefing, and are thus causing a certain amount of head-scratching at Linden Lab.
Reed Linden, in responding to a support request from Motor Loon, provides clear guidelines on what to do if you have a Homestead and experience these issues. It is thought that the most likely culprit for the problem is an unidentified memory leak, but this has yet to be confirmed. Reeds comments regarding particle systems are fascinating. Particles tend to be more viewer-intensive than server, and as many commented at the Simulator User Group meeting on Tuesday 20th November, it would take something bizarre to be going on for particles to be impacting region performance; however, at least one region affected by the issue appears to have a large number of particle emitters in operation.
A further interesting twist came at the meeting itself, when a pathfinding snake and a number of pathfinding characters were rezzed, and the region suffered a severe performance hit (sharp FPS drop experienced by all attendees, sharp increase in both physics memory use and time taken to ping the region) which appeared to be linked with the snake (which was set to follow its creator around) re-calculating its path to both follow its creator and avoid other avatars / objects. However, when the snake was re-rezzed a short time later, no similar issue was noticed, with the region using around 116MB physics memory, with no other outward performance issues.
I the meantime, and as the Linden dev team continue to investigate the issue, if you experience this kind of problem, please ensure you raise a support ticket, supplying as much information as possible, including region name / simulator version (from HELP > ABOUT (either Second Life or the name of your viewer), the time the problem occurred, how it manifested and, if possible, information from the Statistics bar: memory used, FPS and physics performance details, etc.
During August / September a griefing object in the form of a “freebie gift” started circulating in-world. Called ”..::ExDepart::.. Gift Package 2012”, it is essentially a spoofing/griefing item which, when rezzed will create more items citing you as the owner, and attempt to pass them out.
Since it first appeared, the item has appeared in a number of variants, of which “-[L4L]- Gestures & Walkers (Freebies) <3” is one.
If you receive either item – do not rez it. Instead, file an abuse report, citing Governor Linden as the abuser, list any pertinent information on the object – and remember the person you received it from most likely did not create it, then delete it.
Similarly, if you receive any other item with a similar format of name, or which gives rise to suspicions on your part – particularly if the receipt of the item is unexpected – contact the person who sent it to you first and verify with them that they have legitimately sent you something before attempting to rez the item.
If you receive such an item and rez it, you will need to locate it and delete it – and you may find it has spawned several copies of itself. Typically, one rezzed, the item will move itself to around 4000m above ground, and will continue to spam you with items. To locate and delete the object(s):
Check the incoming dialogue pop-ups associated with the incoming items. These should provide the co-ordinates where the item is located. Alternatively, decline the object’s offer, and the co-ordinates should be recorded in local chat.
Position yourself on the ground using the X and Y coordinates for the object, then:
Either fly up to the Z coordinate for the object, or
Rez and cube, sit on it and use EDIT to elevate the cube to the Z coordinate, or
If you have a viewer which supports command line instructions, use gtp x y z – replacing X, Y and Z with the object’s coordinates
Once in the location, enable Beacons for Scripted Objects (CTRL-ALT-SHIFT-N). You should see a small box with crosshairs on the offending object
Go to Edit and drag a selection box around the object to select it.
Press Delete to remove the object.
Note there may be more than one copy of the object nearby, so you may have to repeat the steps above to remove all of them.
The use of the object is known to the Lab, with Simon Linden commenting at the Simulator User Group meeting on Tuesday 20th November that, “It’s pretty much an ugly social-engineering griefer ploy.”
Miro Collas, of the Phoenix / Firestorm team has put together detailed advice on the object and removing it on the Firestorm wiki, from which the instructions in this post have been drawn.
There have been ongoing issues with regions not rendering within the World map. The precise reasons why this is the case is currently unclear; Andrew Linden has been trying to look into the matter since it was reported at the last of the Friday Simulator User Group meetings on the 9th November, but work on Interest Lists has kept him busy.
The problems appear to be twofold: tiles for some regions either entirely fail to generate in the World map, or their appearance is linked to the level of zoom being used.
The first issue is demonstrated with the region Sunny Point, which simply does not appear on the World map at all.
Sunny Point – failing to appear on the World map
With the second issue, Qie Niangao reports that strips of regions on the World map can effectively vanish at certain zoom levels (see JIRA SVC-8115), with some regions of the Zindra adult content apparently never having been drawn at some zoom levels of the Map.
Regions located close to Harshap demonstrate a part of the problem, in that they will appear in the map when zoomed fully in, but step out once on the zoom, and a strip of regions will vanish.
Regions near Harshap: visible when zoomed in…
While the issue has only recently come up for discussion at the Simulator UG meeting, the problem appears to have been persistent for a good while, and some have reported the map is missing strips of region tiles from as many as eight different locations.
Harshap: step out a level with zoom, and a strip of regions vanish, and are not re-rendered.
The World map is generated via a process which images regions from an altitude of around 350m (hence why builds above this altitude do not appear on the map). The information is then scaled for rendering at a number of levels to represent the different zoom levels within the World map floater. Currently, it appears is if the problem lies with the generation of these different zoom levels, at least as far as the “missing strips” issues is concerned. The data from the process is also used for generating map images at maps.secondlife.com, with the result that issues can also occur when viewing map segments there.
As it stands, the possible causes for the problem are still under investigation by LL personnel. However, anyone encountering problems with their region(s) failing to render properly on the World map should consider raising a bug report and /or attending the Tuesday Simulator UG meeting (the Friday meeting is now discontinued).
Regions absent from the World map, as imaged by MartinRJ Fayray for the Simulator User Group meeting on Tuesday 13th November
Deployments this week are being delayed some 24 hours, which means the main channel deployment will not take place until Wednesday 31st October, and the RX channels on Thursday November 1st. This has led to concerns that the main channel deployment / restarts may impact any Halloween events taking place across the grid on Wednesday 31st. Simon Linden promised to raise the concerns at an internal LL meeting taking place later on the 30th October, however, at the time of writing, the Grid Status page reports that the deployment will go ahead, starting at 05:00 SLT on the 31st.
In the meantime, the deployments for the week remain largely as previously indicated:
The main channel should receive the code currently on BlueSteel and Magnum. As mentioned above, the code has no externally visible changes but has some system level adjustments – release notes
LeTigre will receive a further update for the code currently running on it, which will include a number of bug fixes and pathfinding updates – release notes
On Friday 26th October, Simon Linden indicated to me that Magnum should be receiving the code planned for week 43 (the llSensor() problem has been fixed) which will include Baker Linden’s Group Services code currently on Snack, however, as of the Simulator User Group meeting on the 30th October, the final release notes for Magnum had yet to be published, so the update may still be in a state of flux
BlueSteel should get the same code as is on LeTigre.
Interest List
Andrew Linden has fixed a bug wherein some child prims in linkset fail to rez and he has carried out further work on performance issue he reported last time. This turned out to be an issue with the code which caused the simulator to send a full update of everything within view to the viewer each time an avatar within visual range moved. Understandably, on crowded regions, this led to performance issues.
The code is in the process of being revised to ensure it only calls for “terse” updates to be sent to the viewer, which will help ensure more relevant information is sent to viewers when updating, which should reduce the performance hits.
Group Services
Baker Linden, speaking at the Simulator User Group on Tuesday 29th October, said that the server-side code for this project, which should improve the load times and editing abilities for very large group lists, seems to be working “moderately well” since the deployment to the Snack RC channel last week. However, some bugs have been found, Commenting on these, Baker explained that, “The group name, description, and other things don’t load right now on the Snack RCs.” However, the bugs have been investigated and fixes found, which should be merged into the code ahead of the planned deployments on Thursday November 1st. The fixes have also been tested on Aditi, where they’ve been found to induce a slight lag on group loading.
For the time being, testing this now code continues to require a dedicated SL project viewer (available for Windows, Mac and Linux), until such time as issues with the SL beta viewer code can be fixed and merges with viewer project code resumed / made available to TPV for integration.
Avatar Baking
Work is still progressing, Nyx Linden confirmed, talking at the content Creation User Group on Monday 28th October. How far down the road the work is, is unclear. The server-side of things will apparently be using the code being developed for the viewer, and it is this which is the focus of attention for the present, as has been the case for some time.In talking to TPV developers on the subject the last time the matter came up, Oz Linden confirmed LL’s plan is still to give TPVs “at least” 2 months (eight weeks) notice prior to anything being rolled out for testing, in order to give TPVs sufficient time to incorporate the code into project viewers of their own and assist with the overall testing.
SL Issues
Mesh Alpha Issue
Theresa Tennyson demonstrates the skinned mesh / alpha issue
Also during the Content Creation meeting, a problem with alpha textures as applied to worn skinned meshes was brought up. Theresa Tennyson demonstrated the issue during the meeting, which somewhat resembles the old invisiprim / alpha issue.
Siana Gearz suggested two possible causes for the issue, “[The] first is that rigged mesh transparent surfaces appear to be drawn before prim transparent surfaces. [The] second issue is that the shader apparently writes depth for whole fragment, not just for relatively in transparent pixels.”
Nyx Linden requested a JIRA item be raised for the issue, again highlighting the problem with the recent JIRA changes, in that outside of those with access privileges to the new system, no-one could actually confirm if a JIRA had already been raised.
ETA 31st October: Seems a public JIRA on the matter is available – see MartinRJ’s comments which follow this article. Many thanks, Martin!
With thanks to Theresa Tennyson for the Simulator UG meeting transcript
There is a lot going-on server-wise at the moment, so best to break it down by heading.
Server Rebalancing
The Lab is currently engaged in a rebalancing exercise in an attempt to put neighbouring regions on the same server and generally do a logical organizing of the grid to help improve various aspects of performance. Speaking at the Server Beta User Group on Thursday 18th October, Oskar Linden explained that there has been a lot of moving around (in terms of regions) within and between the Lab’s co-location facilities, and so the re-balancing is warranted and needed.
This work takes time – a rebalancing operation earlier this year took around 6 weeks to complete. It requires that regions are organized into groups and then generally moved twice: the first time to a temporary sever, the second to the target machine, each move requiring a reboot, so people may notice additional and unexpected restarts to regions they are in as the work progresses. Two moves are required because the server topology is so tight, it often isn’t possible to move regions directly from one server to a target server, so an intermediary is required.
While this work will take time to complete, the result should be improvements in stability and performance with the likes of teleports, etc, and even improved region crossings.
More on the Server Deployments for Week 42
Main channel: Oskar provided some more information on the main channel update of Tuesday 16th October, saying, “The main channel got a tweak this week, but it was a really small change, and no sim code got changed. We recognised that we had some inefficient SQL queries where large groups were concerned, so we optimised them, and the effect was quite noticeable. The databases are more responsive [and] this helps at all levels.” He went on to clarify that these changes were not Baker Linden’s Group Services code changes, after some in the meeting appeared to think this might have been the case
BlueSteel received the updates which were tested in the network pile-on test in week 42. At the time, I commented that teleporting seemed a lot faster, but that might have been a placebo effect of being on Aditi. It was. Commenting on the test, Oskar said, “There were no simulator changes in that test code. We were just testing backend tweaks.”
Magnum received no update per se, as previously reported, but was merged with trunk and then redeployed
LeTigre received the biggest update, which included new LSL functions ad updates, and most importantly of all, a new version of Havok (see below). Of the LSL functions, Maestro had a warning about the new OBJECT_PATHFINDING_TYPE parameter in the pathfinding command llGetObjectDetails, “We misspelled a constant, OPT_UNKNOWN, so we plan to fix that.” The fix will probably be next week.
Havok
As mentioned above, Havok on LeTigre was updated to version 2012.1. The update enables Havok’s built-in terrain optimisation and should lead to improved performance as a result of the physics shape of the terrain being simplified. Prior to the deployment, there were concerns that it would lead to issues with mesh vehicles trying to cross between regions running different version fo Havok, as has previously been the case.
As reported in part one of this update, these concerns led to Andrew Linden contacting the deployment team in LL to check whether it would be possible to ensure none of the Blake Sea regions remained on LeTigre while two versions of Havok running on the grid to help alleviate at least some of the pain people would feel when using mesh vehicles there. This apparently happened, whether it was before or after the deployment is unclear, as some people did report issues following the roll-out. There was also a little confusion as to what had been swapped where.
At the Server Beta meeting, Oskar gave the impression that all Blake Sea regions were on LeTigre. However, at the Simulator User Group meeting on Friday 19th October, Andrew Linden indicated that records showed none of the Blake Sea regions are running on LeTigre, although they are spread across the other channels. Given that there were (according to Andrew) around six Blake Sea regions running on LeTigre to start with, it would appear to make sense that they have been rotated off to another channel, rather than attempting to rotate all of Blake Sea on to LeTigre.
Please use the page numbers below left to continue reading this article
Inara Pey: Living in a Modemworld 