As reported last week, Wednesday October 3rd saw a massive problem hit the LeTigre Release Candidate channel, which impacted over 1200 regions. This most visibly manifested itself as a large number of items (including partial builds) being returned to people’s inventories as a result of regions being seen as “full” by the software as a result of an error in the prim accounting code. This saw disruption across the grid throughout Wednesday and into Thursday, partially because those regions impacted by the error not only required a corrective deployment of RC code (from BlueSteel), but also had to be manually restored to a state prior to the LeTigre deployment occurring.
Since the problem occurred, Linden Lab has not only been looking into the bug within the prim accounting software, but also at their internal processes in terms of why the error wasn’t picked-up prior to the LeTigre deployment going ahead, and also in terms of what steps can be taken to curtail such a massive disruption in the future, should ever a similar problem occur, and how regions can be restored in a less manually intensive manner. Even so, sorting out a solution which fits every possible scenario by which a deployment may go wrong isn’t easy.
Speaking at the Sim / Server User Group meeting, Simon Linden commented on the matter thus, “The tough thing with SL testing is running it on all those combinations – it’s just never possible to have complete test coverage. The RC channels are actually designed to be representative of the whole grid, so we try to keep a mix of the different types of regions like full, mainland, Linden Homes, etc. On one hand, the system worked … we found a problem before it got to the whole grid, which might have been how this would have happened before we started the Release Channels. But it really was so bad we definitely want to catch something like that even earlier. So … we’ve had a bunch of meetings discussing what and why it happened, and have some better tests added to the regular test pass so this specific problem won’t happen again.”
Options to prevent a similar issue occurring again in the future which have apparently been discussed at these meetings include:
Improving the testing carried on Aditi. Part of the problem here is that as a representative testing environment, Aditi is very much smaller and much less diverse than the main grid, and as such it is harder to test for all possible failure conditions which may occur when deploying code to the main grid
Adding alarms to the deployment process so that when things do go wrong, such as a large number of object returns occurring, the process will automatically stop itself before the damage becomes widespread
Altering the deployment process so that code is initially rolled out to a subset of each Release Channel, prior to it being paused for a few hours to see if there are any reports from users of unexpected or undesirable results , and only resuming the deployments if it appears nothing untoward has happened
While it is the first time this particular problem has occurred in terms of selective region object returns, it has prompted the Lab into looking at ways and means to initiated an automated restore process in order to make the rollback of affected regions more time-efficient and less intensive.
It remains to be seen which of these – and any other ideas – which have been discussed at the Lab are implemented. However, it should be remembered that even with the best will in the world, and given the dynamic nature of Second Life with all the user-created content and scripting, it is impossible for Linden Lab to take into account every single possible error which may occur with a server deployment, and provide a means of avoiding it. Even so, as a result of the LeTigre event, LL are looking to further improve how server code is both tested and deployed in the future, and provide the means to better flag any negative impact occurring during a deployment in order to allow remedial action to be determined and actioned in a more timely manner.
As many are aware, there was a major error in this week’s LeTigre Release Channel deploy. Apparently, the root cause of the problem lay in the server-side prim account code, which Simon Linden describes as having “blown up” on the LeTigre RC channel. This resulted in a large number of items (including partial builds) being returned to people’s inventories as a result of regions being seen as “full”. The problem required a two-stage recovery:
LeTigre regions were rolled back to a state prior to the faulty deployment, and were then updated with the BlueSteel code also deployed on Wednesday October 3rd. This helped to determine the extent of the damage (a total of some 1200 regions)
The regions damaged by the land impact miscalculation were then restore to a state prior to the roll-out of the original faulty LeTigre code. These had to be restored manually, which took a considerable time
There is further post-mortem work going to to try and discover why this error did not reveal itself when the code deployed to LeTigre was being tested on Aditi, and whether there is anything specific to the regions impacted by the error which may have triggered it. Thought is now also being given to managing large scale region restorations, despite this being the first time there has been such a massive issue of this kind occurring across the grid.
Current RC plans for next week call for the same maintenance release to be made to all three RC channels, which Simon Linden describes as, “Mostly internal changes but [which] does include a minor update for the physics engine library … It’s almost all updating libraries … we’ve been using a fairly old set of compilers and such to make some of the development builds of the servers, and this brings us to more recent code.” Further details on the deploy should be available next week in the Second Life Server section of the Technology forum.
SL Viewer
As indicated in part one of this report earlier this week, problems have continued with the Beta viewer code and high crash rates. Work has been ongoing to try and locate the probable cause(s), some of which included the temporary return of tcmalloc. While not actually a cause of the crash issues, having tcmalloc disabled was affecting efforts to reproduce the problems. a beta release was made on the 3rd/4th October (3.4.1.265434), which is proving to be a lot more stable than previous versions, and which happens to have tcmalloc enabled.
The current plan is for a further beta release to be made, most likely on Monday 8th October, which should see tcmalloc turned off once more (if not removed). Should this also prove to be stable, the fixes it contains will be merged back into the development viewer code, and this will clear the way for clearing the backlog of code merges for both the beta and development viewers. It may also see a further 3.4.1 release version of the viewer being made.
Among the projects awaiting merging into the development and beta viewer code are:
The Steam support changes, which have been available within a development viewer stream, and which are described as “mostly cosmetic”. There is apparently a version of the viewer on Steam, but it is not available for general viewing / download, and is presumably there for testing purposes
Monty Linden’s HTTP library (texture fetch) code
Baker Linden’s Group Services project code
Apple OSX 10.8 Mountain Lion support work, including gatekeeper compatibility
Bug fixes and further regionalisation work.
Previous plans for these releases called for them to be made under the 3.4.2 code base. While this wasn’t discussed at the TPV/Dev meeting, one assumes this is still the case. However, speaking at the TPV Dev meeting on Friday October 5th, Oz Linden indicated that the order, etc., in which waiting merges will be cleared hasn’t been fully defined, and will be the subject of internal conversations next week at the Lab.
Avatar Baking Project
Bake fail: a familiar problem for many
There is still no major news on this project, although work is continuing both on the viewer and on the server code.
The plan remains to provide TPV developers with access to the viewer code at least 8 weeks ahead of any initial deployment of the server-side code to an Agni release channel. This is to allow TPVs time to merge the code into their viewers and participate in ongoing testing of the new service.There is a possibility that that viewer code will be available sufficiently well ahead of things in order for TPVs to be able to use it alongside the testing on Aditi (beta grid), depending on the status of the beta grid tests and how development of the viewer code progresses.
Please use the page numbers below left to continue reading this report
The server deploys planned for Wednesday September 26th have been cancelled. The news was given in a brief update to the the Sever Deploys blog post, which simply read:
UPDATE: There were blocking bugs found in both the RC’s planned for release this week. There will be no releases Wednesday morning. There will be no rolling restarts.
We found blocking issues during our pre-RC smoke tests. These issues will block the Wednesday morning RC releases. Regions will not be restarted.
Classified as maintenance releases, the deploys were to have included back-end configuration work designed to help SL run better on new and future hardware, and Baker Linden’s new Group Services code.
The postponement is the second time RC deploys have been cancelled in the last two weeks, with those planned for the week commencing 17th September being cancelled as a result of failing to pass QA testing.
As a result of last week’s RC cancellation, there was no main channel deploy on Tuesday 25th September. While the RC channel deploys might be rescheduled for later this week, depending on the severity of the reason for them being cancelled in the first place, if they do not take place then it is probable that there will be no main channel roll-out again next week.
Main Channel: Sever release 12.07.24.262437 – Tuesday 31st July
This should see a further roll-out of the LSL functions related to the Advanced Creator Tools. This release will see the addition of three new LSL functions:
These new LSL functions work with the current runtime permissions system, and are precursor to future work with experience permissions. More information about the runtime permission is here:PERMISSION_TELEPORT.
This is a roll-out of the code deployed to LeTigre and BlueSteel last week. As with both of those channels last week, the code will be enabled on the main channel regions following the deploy (although LL retain the capability to disable it).
Magnum RC: Further Pathfinding Roll-out – Wednesday August 1st
Roll-out due to commence: 07:00 SLT
A further roll-out of the server-side pathfinding code, with fixes. Currently the wiki notes for this channel appear to be stalled on the 12.07.24.262484 release.
Note that the viewer-side pathfinding tools are now available in the latest Development Viewer.
BlueSteel RC – Wednesday August 1st
Re-start due to commence: 08:30 SLT
There are no changes to this channel. It will have the same code as the main channel.
LeTigre RC: Infrastructure Project update – Wednesday August 1st
Roll-out due to commence: 09:30 SLT
Oskar comments: “This channel will have an infrastructure project that has no intentional changes to existing behaviour. There are perhaps unintentional changes to existing behaviour. If you find some please let us know!”
Update July 18th: The Magnum RC roll-out has been delayed until Thursday July 19th. Oskar may supply a reason on the deployment thread in the forums – keep an eye on that for updates (with thanks to Wolf Baginski).
Main Channel Release
Tuesday 17th July sees the a roll-out of LSL functions related to the Advanced Creator Tools. This release will see the addition of three new LSL functions (comments taken from the release notes):
llAttachToAvatarTemp(integer attach_point): Follows the same convention as llAttachToAvatar, with the exception that the object will not create inventory for the user, and will disappear on detach, or disconnect. It should be noted that when an object is attached temporarily, a user cannot ‘take’ or ‘drop’ the object that is attached to them. The user is ‘automatically’ made the owner of the object. Temporary attached items cannot use the llTeleportAgent or llTeleportAgentGlobalCoords LSL functions
llTeleportAgent(key agent_uuid, string lm_name, vector landing_point, vector look_at_point): Teleport Agent allows the script to teleport an agent to either a local coordinate in the current region or to a remote location specified by a landmark. If the destination is local, the lm_name argument is a blank string. The landing point and look at point are respected for this call. If the destination is remote, the object must have a landmark in its inventory with the teleport agent script. lm_name refers to the name of the landmark in inventory. This function cannot be used in a script in an object attached using llAttachToAvatarTemp
llTeleportAgentGlobalCoords(key avatar, vector global_coordinates, vector region_coordinates, vector: Teleports an agent to region_coordinates within a region at the specified global_coordinates. The agent lands facing the position defined by look_at local coordinates. A region’s global coordinates can be retrieved using llRequestSimulatorData(region_name, DATA_SIM_POS). This function cannot be used in a script in an object attached using llAttachToAvatarTemp.
The new LSL functions work with the current runtime permissions system and are precursor to future work with experience permissions. More information about the runtime permission is here:PERMISSION_TELEPORT.
The keen-eyed will note that these are the functions that were rolled-out to the Magnum RC channel in May, and which were subsequently abused for griefing purposes. However, Linden Lab have added a new capability to the functions – what is described as an “on / off” switch which is available only to Linden Lab personnel, and which allows the functions to be enabled / disabled (the functions were also rolled-out to the Le Tigre RC on July 11th with the “on / off” switch capability). As the release notes make clear, the functions are disabled by default in the roll-out, and will presumably remain that way until such time as the updated permissions system has been rolled-out.
The release also includes three bug fixes (again, as specified in the release notes):
SCR-342: llTeleportAgent() does not fail gracefully when specifying an invalid landmark name
SVC-7966: Magnum RC, llTeleportAgent gives a wrong message
SVC-7987: llTeleportAgent always points in the positive Y direction on teleport.
Pathfinding release: Magnum and Le Tigre
On Wednesday 18th July, the Magnum RC will get a further roll of the pathfinding code and Le Tigre will apparently get the same code as well. At the time of writing, the actual release note pages on the SL wiki for Magnum and Le Tigre still reflected the releases for July 11th and the forum post announcing the release did not show any specific changes from the forum post relating to the July 11th release. Any alternations which may have been made following the difficulties some initially encountered on the Magnum RC following that roll-out are therefore hard to identify. This ma change prior to the actual roll-out.
Pathfinding is drawing closer to a release across the main grid, and preparation work for the roll-out – which will constitute one of the biggest changes to SL – is underway on several fronts. This article is intended to be a high-level update on various elements of the project, gleaned from a variety of sources.
Magnum RC Roll-out
On Wednesday July 11th, the server-side pathfinding code was rolled-out to the Magnum Release Channel. There had been some predictions that this could lead to significant problems and issues as a result of the issues given within the release notes
Following the release, issues were experienced, notably with mesh vehicles, as reported on the forum thread discussing the releases for the week, and which have been rapidly responded to by Linden Lab personnel. there are still concerns around the roll-out and potential impact, and Linden Lab are continuing to monitor.
In discussing the RC toll-out at the TPV/Developer meeting held on Friday July 13th, Lorca Linden, the Associate Producer responsible for the project, commented: “OK, so pathfinding did go in RC on Magnum on Wednesday [July 11th]. As a whole, things are looking really, really good. We’re seeing very few crashes, the performance is working great we are seeing issues with some vehicles – definitely not all. That’s the only major hitch that we’re looking into, but as a whole the RC has been going quite smoothly.”
New Viewer Tools
As mentioned above, Lorca Linden (together with Stinson Linden and Prep Linden) from the pathfinding project attended the Friday TPV/Developer meeting on the 13th July, where they specifically discussed the viewer-side pathfinding tools. The tools are covered in detail in a new wiki page from Linden Lab, and may already be familiar to those who have been working on the pathfinding beta. They are currently in the Pathfinding Project Viewer, and will need to be incorporated in to TPVs as well. The wiki page provides comprehensive notes on the tools, complete with screen shots; the following in intended to provide a high-level summary and some background notes for those unfamiliar with the core elements of pathfinding, and to provide an overview of what this means for viewers going forward.
Navmesh and the Rebake Tool
For those not familiar with the term, navmesh is short for navigation mesh. This is a representation of a region’s geometry generated and used by the physics simulator to determine paths for pathfinding characters. The navmesh can be somewhat fluid in nature, depending upon what is going on in a region and what is being changed; a new path for a character, for example, will change a region’s navmesh. When this happens, the navmesh for the region needs to be updated, which can take some six hours if left to update automatically.
To overcome this when pathfinding is rolled out, one of the new tools that will be appearing in the viewer will be a Rebake Region button. This will automatically appear at the bottom of the viewer window of all users within a region when the navmesh requires updating – regardless as to who may actually have altered the navmesh.
Rebake region button for navmesh updates (with thanks to Linden Lab)
Once baking has commenced, the button will fade on all viewers on which is displayed, indicating that an update is in progress (and preventing someone else from initiating a rebake). Once the rebake is complete and the navmesh is updated, the button will vanish from viewers.
Object Attribute Tools
By default, a navmesh treats all resident-made objects within the region in which it is active as obstacles that pathfinding characters must manoeuvre around. Obviously, this may not always be the case; there will be objects (e.g. stairs, ramps, sidewalks, floors, etc.), pathfinding characters need to traverse, climb, etc. This is achieved by altering the pathfinding atrributes associated with an object, and some of the new viewer tools are to allow this to be done and to also allow users to examine in-world objects to determine their status vis-a-vis pathfinding and how pathfinding characters will react to them.
These tools take the form of menu options and additional panels located in the Build and Object Profile floaters.
Dedicated Floaters
Also included in the tools are three dedicated pathfinding floater panels:
Linsket floater: designed to give advanced users and builders the ability to customize an area to achieve interesting effects with pathfinding-enabled characters
Character floater: designed primarily to help users to locate characters moving throughout a region and to identify the CPU cost of characters affecting the performance of a region
View / Test floater: intended for advanced users who are building pathfinding-enabled objects and characters.
Pathfinding characters floater (with thanks to Linden Lab)
Tool Status and TPV Integration
As mentioned above, these tools are all currently available in a Project Viewer. However, it is anticipated that they will be appearing in a Linden Lab beta viewer in “one to two weeks” (Lorca Linden). The tools themselves are regarded as feature complete by LL, and Lorca encouraged TPV developers at the meeting to consider integrating them into their viewers sooner rather than later.
Integrating the new tools in TPVs is liable to be in two parts:
An initial release containing the tools required for setting object attributes, etc.
A follow-up release incorporating the use of the Havok libraries Linden Lab is establishing and which will be made available under the new sub-licence arrangement.
The reasons for this are two-fold:
The attribute tools, etc., are vital for optimising pathfinding within regions and ensuring everything works correctly (e.g. to ensure pathfinding characters can climb the stairs they’re supposed to by climbing or walk along the prim sidewalk they are supposed to walk along, etc.
The Havok libraries are not yet available, although Oz hopes to have them in a position where he can talk in more detail to TPVs about them “pretty soon”, and while it is nice to be able to visualise the navmesh, etc., it is not quite such vital part of the pathfinding process.
Universal Tools
Alongside the viewer tools, pathfinding will see a set of universal tools rolled-out in console format. These will be available to region owners and estate managers and will allow them to change an entire class of object in a region to have certain pathfinding attributes once pathfinding goes live. Linden Lab are approaching this in terms of having all non-scripted objects set them to be static obstacles that pathfinding characters must manoeuvre around, while anything that is scripted is set to “dynamic”, as it is thought to be moving.
This obviously doesn’t fit all cases – vendor boards, for example are scripted, but they are hardly what can be termed “moving” objects. Indeed, it might be argued that the majority of scripted objects within a region are non-moving, and therefore should have their pathfinding attribute set to “static”. However, LL feel they have no way of easily differentiating between a non-moving scripted object and a moving scripted object, and thus feel that setting all scripted objects to “dynamic” is the better option and allowing the attribute to then be modified through the viewer where necessary, as setting them to “static” could result in a worse overall behaviour case within a region.
Other Tools and Items
Alongside the above, Linden Lab have previously indicated that they will be making the following available as pathfinding rolls-out:
A set of script templates used for the creatures found in the Wilderness areas
A script for a “master rezzer system”.
The latter is a means by which region performance and the number of pathfinding characters rezzed in the region can be monitored, and which will reduce the number of pathfinding characters within a region in response to the region’s performance / number of avatars within the region.
Potential Timescales for Roll-out
During the TPV/Developer’s meeting, Lorca outlined some potential dates for pathfinding. note that these are currently potential, and shouldn’t be taken as tablets of stone:
The pathfinding tools should be available in one to two weeks in a beta viewer
The server-side release is dependent upon how well (or otherwise) the current release to the Magnum RC progresses, and may potentially come within the next two-to-four weeks, but certainly no sooner than two weeks.
Again these are not confirmed dates, and may well change in the next couple of weeks – particularly if major issues are found with the Magnum RC roll-out.
