Tag: Server Updates

SL projects update week 41 / 2

Posted on by Inara Pey

This item is a follow-on from part one, published earlier this week.

More Server News

At the Thursday Beta Grid User Group meeting (Thursday October 11th), and prior to the network optimisation tests, Oskar gave further news in the  serve deploys for week 41. On Tuesday 9th October, the main channel received code previously on BlueSteel, which in keeping with Simon Linden’s comments at the Monday Sim / Server UG meeting, Oskar referred to as being, “A pretty small release, just some server crash mode fixes; stability ++.”

On Wednesday October 10th, BlueSteel and LeTigre received a fix to some database queries that were really slow when accessing really large groups (note these were not Baker Linden’s Group Services code, that is being looked at as a deployment in week 42).

Monday 16th October may see some restarts on the grid in order to shuffle some regions onto new hardware, with the servers having more and faster CPU cores, which will increase the number of simulators running on the new servers, but they’ll be running on faster CPU cores.

Interest Lists and Object Caching

The short-version update for this comes from Andrew Linden, speaking at the Server Group meeting on Friday 12th October, “I thought I’d have something working this week… it isn’t quite working right. You can see it not working on Ahern on Aditi…” (!)

Interest list changes: easing the pain of random rezzing

He went on more seriously to explain that while the new code is working correctly for the most part, and that rezzing orders should be improved / faster, there are some problems with objects which should be in view of an avatar not showing up and a major issue around teleporting into high ground.

When the latter happens, you effectively arrive “underground” (presumably at the default “ground level” for the simulator  – 21 metres in the case of unterraformed land). The simulator then calculates where you should be and moves your avatar appropriately. With the new code, this has the effect of breaking the server’s notion of the camera – where it is and what it can see – which is used to figure out what objects to send to the viewer. This means that the camera itself cannot be moved or updated.

There have been some performance tests on an older version of the code, which have been mixed, as Andrew also explained, “here were two performance tests run on an earlier version. One test (mostly empty region with about 30 avatars running around) showed a slight decrease in performance… about 5% worse. Another test (crowd of avatars NOT looking at a pile of dynamic objects behind them) showed about 40% improvement (less time spent running the interest list). So I went back to the code to try to fix the first test, and I think I’ve got something that will be as good or better all around.”

The code will also see changes as to how the camera behaves and in the resultant level of detail. Andrew is currently working on limiting the distance the camera cam be moved away from the avatar. Note this is not limiting Draw Distance, but limiting the distance the camera can be freely moved independently of the avatar. He’s considering 128 metres to be the likely range. There are two reasons for this.One is to prevent the camera wandering into regions which are more than one neighbouring region away, the other is because as the camera moves laterally, detail levels degrade, because object detail is tied to the avatar’s position (hence why, when you zoom a great distance, buildings and objects may only appear to partially rez, etc.). Under the new system, object detail will be tied to the camera, so that little degradation is experienced. However, in order for this to work, the camera must be kept within a reasonable distance of the avatar; if it is moved too far, the detail will start to degrade once more (presumably because of the volume of data the viewer is trying to handle).

Mesh Deformer

On Thursday 11th October at the Open/Dev meeting, Darien Caldwell outlined her ideas for using base shape info exported from Second Life when uploading rigged meshes.

If this works, it will essentially mean that rather than being restricted to using a default base female or male shape when uploading rigged meshes, creators will be able to download a human shape as an XML file (permissions allowing), and then specify this shape when uploading rigged meshes.  The basic code for handling the upload with specific avatar shape information has already been added to the deformer by Qarl Fizz, so Darien is focusing on the best way to use it, her work going into a fork of the existing Mesh Defromer project viewer.

Avatar shapes can currently be exported from a viewer via DEVELOP -> AVATAR -> APPEARANCE TO XML (again subject to the permissions system). This saves the avatar shape data as an XML file, which contains the settings from the appearance sliders, and which is automatically saved to your computer (generally to  C:\Users[USERNAME]\AppData\Roaming\SecondLife\user_settings for Windows).

To associate an avatar shape .XML file with a mesh, Darien is proposing a further revision to the mesh uploader floater, and has provided an early mock-up as to how it might look.

New option to associate an avatar shape XML file with a mesh on upload (image courtesy of Darien Caldwell)

More work is required the flesh-out this idea, including, as Oz noted at the Open/Dev meeting, making the shape export option more obvious for people to use, which will more than likely see it moved out of the Develop menu, wherein it is currently nested. The .XML file itself is not suitable for use directly in most 3D modelling programmes, so how the exported data might be used with these when creating mesh items remains to be seen. nevertheless, if successful, Darien’s approach may offer a more fine-tuned solution to developing mesh clothing to a range of shapes.

Other items

Viewer and FMOD

The use of FMOD has been the subject of much discussion within the TPV/Dev meetings of late. FMOD is used within the sound system for the Viewer, and until now, Linden Lab has provided a script which pulls library files from an FMOD repository for use in viewer builds. However, following what appears to have been a clean-up of their archives, FMOD have removed the some of the legacy files required for this, as reported in JIRA OPEN-150.

Some viewer developers have already started using FMODex within their builds (e.g Singularity 1.7.0+), which also addresses issues with sound quality as well. Other TPVs are looking at possibly integrating this work into their builds.

It currently appears as though Linden Lab themselves are looking to integrate FMODex, as they see this very much as something which needs to be addressed. Speaking at the TPV/Dev meeting on Friday 5th October, Oz Linden stated: “I got around to forwarding the JIRA on that to our engineering manager for Second Life, and he agreed with me that it is something we should definitely do something about. I’m not sure what the time-table on that will be, it’s going to go into the hopper for the next ‘Things we should do something about, what priority are they compared to all the other things we should do something about’ meeting, which happens weekly.” While openAL has also been suggested as an alternative, it does seem more likely that FMODex will be adopted, something which was hinted at by Oz when talking at the Open/Dev meeting on Thursday 11th October.

Teleport Timeouts

Baker Linden has been looking into the issue of teleport timeouts, and has managed to pin down one cause as a reproducible bug. He’s not sure as to whether it can be fixed, and is currently investigating further as to why it is happening.

 

Network Optimisation: LL seek assistance

Posted on by Inara Pey

Linden Lab are in the process of changing the manner in which network traffic is handled within Second Life, and require assistance in testing the changes made to date in order to ensure various services are functioning correctly prior to rolling-out the changes to the grid as a whole.

To this end, Oskar Linden posted the following request in the Server forum late on Wednesday 10th October:

Linden Lab has made some changes to the way regions handle network traffic. We need your help to test. This will help us insure that regions are communicating appropriately over the network. Mainly we are concerned with agents entering via direct login, teleport, and region crossing. As well as other functions such as IMs, Voice, and Group chat.

Tomorrow, October 11th, at 4PM Pacific time (right after the server beta user group) we will conduct these tests with as many people as we have. Testing will take place on ADITI and require out of world communication we will be coordinating via IRC. This will require an IRC client connected to EFNET in the channel #sltest. If you don’t know how to do that you have until tomorrow afternoon to figure it out. 🙂

The details on the tests are here:

 – https://wiki.secondlife.com/wiki/Networking_Optimization_Pile_On_Tests

I hope you can come and help us test these new changes.

__Oskar

The tests will comprise three parts:

  • Direct log-in test to a pre-determined location on the beta grid.
  • A group chat test via the Second Life Beta in-world group.
  • A teleport test to a pre-determined location.

Note that Voice is also indicated as one of the services to be tested, but no details on what this will entail have as yet been included in the test notes – please check both the note and Oskar’s forum thread for possible updates on this ahead of testing.

Those wishing to take part in the tests will need to:

  • Be members of the Second Life Beta in-world group
  • Be able to log-in to the Aditi beta grid

The tests will be coordinated on IRC using the EFnet channel #sltest, and those involved in the tests will need to be able to access this channel either via the EFnet website or through an IRC client.

Accessing the #sltest channel on the EFnet website – note you do not require a registered account; you can access the channel using any suitable nickname

The tests are due to commence at 16:00 SLT.

Note that these changes are not related to the region lag issue sudden and massive lag spikes, as reported in the Server forum threads, but rather appear to be part of ongoing network-related work.

SL projects update week 41 / 1

Posted on by Inara Pey

Server Deploys

The main channel today received the same maintenance release made last week to BlueSteel (and which was used to help fix the LeTigre problems). speaking at the Server / Simulator User Group, Simon Linden described the deployment as, “A very minor change … from one of the RCs. It’s really not any functional change but was needed to go grid-wide before another one goes into RC.”

Following week 40’s issues with LeTigre, the three primary Release Candidate channels will be getting updated as follows:

  • Magnum, which has code to help sims run better on new hardware,  will be getting the same change that went to the main channel
  • BlueSteel and LeTigre will get the same update, which is described as “pretty minor but important”, being a back-end query optimisation designed to assist with database loads.

Currently, the prim accounting issue is still being worked on, which affecting scheduling what will be available in deployments. Passing comment on how deployment packages are put together, Simon explained that, as a rule, bug fixes tend to be put together as far as possible, prior to going to QA and then to an RC channel. He went on to say that what goes into other releases can be variable, “There’s a judgement call on how much gets bundled together, and a bunch of things go into the decision, like how overloaded the QA guys are, how many other things are trying to get to RC, risks of one part blocking it (like now), stuff like that”. In the meantime, LL are actively looking at ways to both prevent a recurrence of this problem and to improve the RC channel deployments as a whole.

SL Viewer

The promised new beta release (3.4.1.265642) reached the release point on Monday October 8th. This release sees tcmalloc disabled once more,  but otherwise appears to be the same code as the previous beta. It is intended to be a further stability testing / confirmation release, and as such will remain available for the next couple of days as Linden Lab gather data on its performance.Tcmalloc has been disabled, rather than removed, as it has apparently been useful in helping to trace issues within the viewer code, and LL wished to retain the ability to re-enable it in case they needed to re-enable it to help identify problems within the viewer in future.

Assuming this release proves stable, and assuming that plans outlined by Oz Linden have not undergone significant change, it should clear the way for the unblocking of various code merges that have been awaiting the stability / memory leak issue to be resolved. As previously reported, the precise order in which code merges will be made / released is unclear, but Linden Lab have a significant amount of updates waiting in the wings, including Steam support changes, Monty Linden’s HTTP library updates, Baker Linden’s Group Services project code, Apple OSX Mountain Lion support (including gatekeeper compatibility), and more.

Steam updates – one of the viewer merges waiting to be released

Under the original plans for the beta viewer, project viewer code was to start merging into the viewer with the 3.4.2 release code. As there was no OpenDev meeting on Monday 8th October, it is assumed this is still the case, however, the precise order of the merges is due for discussion this week within the Lab, and a clearer indication of the order may be available by the Thursday OpenDev meeting, and will be given in part 2 of this report, if that is the case.

Group Services Project

Due to the problems experienced with the leTigre deployment in week 40, Baker Linden’s Group Services code did not receive a proper deployment to a Release Channel. It is not due to be released this week, but should be in an RC deployment aimed at week 42, although as it has been bundled with the LeTigre deployment which had problems, this may be delayed further while the prim accounting error is looked into.

It is currently unclear as to whether the delay with the RC roll-out will influence when the Group Service viewer code (currently available in a dedicated project viewer) will be merged into the development  / beta branches of the SL viewer code; again more should be known on this following Thursday’s OpenDev meeting.

Materials Processing

Continues to progress, with little to report at this time. The feature set for the initial release still has yet to be published, and the wiki page for materials processing is due for further update. Concerns were raised over one statement relating to the use of colour, to whit:

“Color a solid color for the surface; not used if a Texture is also specified.”

The concern was that whereas it is currently possible to specify both a texture and a colour for a given object or object face, the wiki implies that under material processing, it will become either / or. However, this appears to be an error in the wiki, and both options will remain available.

Linkability Bug

As reported in my last update, and while not strictly a project, the bug which is currently allowing prims to be linked over distances greater than 54m has been investigated, and a fix is expected to be rolled-out to the RC channels for this in week 42.

 

The “LeTigre event” and seeking to safeguard deployments

Posted on by Inara Pey

As reported last week, Wednesday October 3rd saw a massive problem hit the LeTigre Release Candidate channel, which impacted over 1200 regions. This most visibly manifested itself as a large number of items (including partial builds) being returned to people’s inventories as a result of regions being seen as “full” by the software as a result of an error in the prim accounting code. This saw disruption across the grid throughout Wednesday and into Thursday, partially because those regions impacted by the error not only required a corrective deployment of RC code (from BlueSteel), but also had to be manually restored to a state prior to the LeTigre deployment occurring.

Since the problem occurred, Linden Lab has not only been looking into the bug within the prim accounting software, but also at their internal processes in terms of why the error wasn’t picked-up prior to the LeTigre deployment going ahead, and also in terms of what steps can be taken to curtail such a massive disruption in the future, should ever a similar problem occur, and how regions can be restored in a less manually intensive manner. Even so, sorting out a solution which fits every possible scenario by which a deployment may go wrong isn’t easy.

Speaking at the Sim  / Server User Group meeting, Simon Linden commented on the matter thus, “The tough thing with SL testing is running it on all those combinations – it’s just never possible to have complete test coverage. The RC channels are actually designed to be representative of the whole grid, so we try to keep a mix of the different types of regions like full, mainland, Linden Homes, etc. On one hand, the system worked … we found a problem before it got to the whole grid, which might have been how this would have happened before we started the Release Channels. But it really was so bad we definitely want to catch something like that even earlier. So … we’ve had a bunch of meetings discussing what and why it happened, and have some better tests added to the regular test pass so this specific problem won’t happen again.”

Options to prevent a similar issue occurring again in the future which have apparently been discussed at these meetings include:

  • Improving the testing carried on Aditi. Part of the problem here is that as a representative testing environment, Aditi is very much smaller and much less diverse than the main grid, and as such it is harder to test for all possible failure conditions which may occur when deploying code to the main grid
  • Adding alarms to the deployment process so that when things do go wrong, such as a large number of object returns occurring, the process will automatically stop itself before the damage becomes widespread
  • Altering the deployment process so that code is initially rolled out to a subset of each Release Channel, prior to it being paused for a few hours to see if there are any reports from users of unexpected or undesirable results , and only resuming the deployments if it appears nothing untoward has happened
  • While it is the first time this particular problem has occurred in terms of selective region object returns, it has prompted the Lab into looking at ways and means to initiated an automated restore process in order to make the rollback of affected regions more time-efficient and less intensive.

It remains to be seen which of these – and any other ideas –  which have been discussed at the Lab are implemented. However, it should be remembered that even with the best will in the world, and given the dynamic nature of Second Life with all the user-created content and scripting, it is impossible for Linden Lab to take into account every single possible error which may occur with a server deployment, and provide a means of avoiding it. Even so, as a result of the LeTigre event, LL are looking to further improve how server code is both tested and deployed in the future, and provide the means to better flag any negative impact occurring during a deployment in order to allow remedial action to be determined and actioned in a more timely manner.

With thanks to Baz deSantis.

SL projects update week 40 / 2

Posted on by Inara Pey

Server Deploys

As many are aware, there was a major error in this week’s LeTigre Release Channel deploy. Apparently, the root cause of the problem lay in the server-side prim account code, which Simon Linden describes as having “blown up” on the LeTigre RC channel. This resulted in a large number of items (including partial builds) being returned to people’s inventories as a result of regions being seen as “full”. The problem required a two-stage recovery:

  • LeTigre regions were rolled back to a state prior to the faulty deployment, and were then updated with the BlueSteel code also deployed on Wednesday October 3rd. This helped to determine the extent of the damage (a total of some 1200 regions)
  • The regions damaged by the land impact miscalculation were then restore to a state prior to the roll-out of the original faulty LeTigre code. These had to be restored manually, which took a considerable time

There is further post-mortem work going to to try and discover why this error did not reveal itself when the code deployed to LeTigre was being tested on Aditi, and whether there is anything specific to the regions impacted by the error which may have triggered it. Thought is now also being given to managing large scale region restorations, despite this being the first time there has been such a massive issue of this kind occurring across the grid.

Current RC plans for next week call for the same maintenance release to be made to all three RC channels, which Simon Linden describes as, “Mostly internal changes but [which] does include a minor update for the physics engine library … It’s almost all updating libraries … we’ve been using a fairly old set of compilers and such to make some of the development builds of the servers, and this brings us to more recent code.” Further details on the deploy should be available next week in the Second Life Server section of the Technology forum.

SL Viewer

As indicated in part one of this report earlier this week, problems have continued with the Beta viewer code and high crash rates. Work has been ongoing to try and locate the probable cause(s), some of which included the temporary return of tcmalloc. While not actually a cause of the crash issues, having tcmalloc disabled was affecting efforts to reproduce the problems. a beta release was made on the 3rd/4th October (3.4.1.265434), which is proving to be a lot more stable than previous versions, and which happens to have tcmalloc enabled.

The current plan is for a further beta release to be made, most likely on Monday 8th October, which should see tcmalloc turned off once more (if not removed). Should this also prove to be stable, the fixes it contains will be merged back into the development viewer code, and this will clear the way for clearing the backlog of code merges for both the beta and development viewers. It may also see a further 3.4.1 release version of the viewer being made.

Among the projects awaiting merging into the development and beta viewer code are:

  • The Steam support changes, which have been available within a development viewer stream, and which are described as “mostly cosmetic”. There is apparently a version of the viewer on Steam, but it is not available for general viewing / download, and is presumably there for testing purposes
  • Monty Linden’s HTTP library (texture fetch) code
  • Baker Linden’s Group Services project code
  • Apple OSX 10.8 Mountain Lion support work, including gatekeeper compatibility
  • Bug fixes and further regionalisation work.

Previous plans for these releases called for them to be made under the 3.4.2 code base. While this wasn’t discussed at the TPV/Dev meeting, one assumes this is still the case. However, speaking at the TPV Dev meeting on Friday October 5th, Oz Linden indicated that the order, etc., in which waiting merges will be cleared hasn’t been fully defined, and will be the subject of internal conversations next week at the Lab.

Avatar Baking Project

Bake fail: a familiar problem for many

There is still no major news on this project, although work is continuing both on the viewer and on the server code.

The plan remains to provide TPV developers with access to the viewer code at least 8 weeks ahead of any initial deployment of the server-side code to an Agni release channel. This is to allow TPVs time to merge the code into their viewers and participate in ongoing testing of the new service.There is a possibility that that viewer code will be available sufficiently well ahead of things in order for TPVs to be able to use it alongside the testing on Aditi (beta grid), depending on the status of the beta grid tests and how development of the viewer code progresses.

Please use the page numbers below left to continue reading this report

Pages: 1 2

Second Life RC channel server deploys cancelled

Posted on by Inara Pey

The server deploys planned for Wednesday September 26th have been cancelled. The news was given in a brief update to the the Sever Deploys blog post, which simply read:

UPDATE: There were blocking bugs found in both the RC’s planned for release this week. There will be no releases Wednesday morning. There will be no rolling restarts.

Oskar Linden also added a comment:

We found blocking issues during our pre-RC smoke tests. These issues will block the Wednesday morning RC releases. Regions will not be restarted.

Classified as maintenance releases, the deploys were to have included back-end configuration work designed to help SL run better on new and future hardware, and Baker Linden’s new Group Services code.

The postponement is the second time RC deploys have been cancelled in the last two weeks, with those planned for the week commencing 17th September being cancelled as a result of failing to pass QA testing.

As a result of last week’s RC cancellation, there was no main channel deploy on Tuesday 25th September. While the RC channel deploys might be rescheduled for later this week, depending on the severity of the reason for them being cancelled in the first place, if they do not take place then it is probable that there will be no main channel roll-out again next week.