SL project news 9 (1): servers, SSB, materials and Aditi

Deployments for Week 9

It’s a “lightweight” week for deployments, with all channels receiving one of two maintenance packages.

Server Life Server (Main) Channel

Region restart notifications: the "old" stayle (top), and the "new" style (bottom)
Region restart notifications: the “old” style (top), and the “new” style (bottom)

On Tuesday February 26th, the Second Life Server (SLS / Main channel) received the server maintenance project which was on all three RC channels last week.  This project has miscellaneous bug fixes, along with an impoved restart notification (which will only be apparent when this new version restarts for the next update) – see the image on the right.

While this is much better than the previous notification, it does suffer from the fact it will still fade from the viewer’s window and become a chiclet whether or not the OK button is clicked – and thus can still be missed if attention is not on the screen when it appears. To help avoid this, it is possible to set a sound for the alert using the Debug Setting UISndAlert, which requires the UUID of the sound, which coule be one you’ve uploaded to inventory yourself (Firestorm additionally has UISndRegionRestart, which can be set through Debug Settings or Preferences > Sound & Media > UI Sounds 2).

This release  means the until the RC channel deployments take place (see below), almost the entire grid is running the same server code – the only exceptions being a few “special” regions (such as the Linden Realms regions) – release notes.

Release Candidate Channels

On Wednesday February 27th, all three Release Candidate channels should receive a new server maintenance package which includes a fix for a crash mode – release notes (BlueSteel).

As always, there is a forum thread for discussing the deployments.

Server-side Baking

Following-on from his initial feedback given at the TPV Developer meeting on Friday 22nd February, Nyx Linden gave a further update on the recent Sever-side Baking (SSB) pile-on / load test when chairing the Content Creation User Group meeting on Monday 25th February

Thanks all for everyone who helped out with the pile-on test of server appearance last Thursday. The results tended to be that when the system worked, it was nice and speedy, however there were a number of issues that prevented it from working for everyone all the time. We got a good bit of data on the bugs we knew were there as well as a couple new ones. we’ve been crunching on the results and are starting to attack the issues, so it was definitely successful in achieving what we wanted: more data :). There will be future rounds, so thanks in advance for anyone who comes back to help us test again.

Z-offset Issue and SUN-38

As noted in the comments following my quick SSB test using currently available TPVs with SSB enabled, there is an issue with the “Z-offset” capability common to many third-party viewers, which allows the vertical height of an avatar above the ground to be adjusted, such that sits and kneels don’t leave the avatar apparently floating in the air, and which allow those with very tall / giant avatars or very small / petite avatars and those wearing full body mesh to similarly adjust their vertical placement relative to the ground / floor.

The problem was reported to Nyx in week 8, and SUN-38 was subsequently raised by Henri Beauchamp on February 24th to officially log the matter. Commenting on the problem during Content Creation meeting, Nyx Linden stated. “It hasn’t escaped our notice. We’re considering a couple different approaches … we’re considering a few different options. Suggestions appreciated, but we haven’t officially settled on an approach we’re going to commit to publicly just yet.” He also suggested that any questions on the subject be asked again next week, presumably by which time LL will have had time to consider the problem in detail, and hopefully consult with TPV developers.

Interest List Updates

Andrew Linden is continuing to work on the interest list code, with more refinements to come, as well as looking at a number of bug fixes. The updates about to go to LL’s QA include changes which should make scenery load faster  / more correctly when first logging-in to SL or teleporting to a region.

Aero Pines Park
Future updates to the interest list code should see scenery in your world view rezzing a lot faster

Continue reading “SL project news 9 (1): servers, SSB, materials and Aditi”

SL projects week 8 (3): Viewer, materials and SSB load test

SL Viewer Updates

Release and Beta Viewers

The release and beta version of the viewer are effectively on a par with one another at this point in time, following the roll-out of SL viewer 3.4.5.270263 on February 14th. There is currently nothing “in” beta at the moment in terms of specific SL projects.

Development Viewer and CHUI

The development viewer and the development version of the CHUI (Communications Hub User Interface) project viewer are also pretty much on a par, and it is anticipated that the CHUI code will be merged-up to viewer development “any minute now”, to use LL’s parlance, although a date has not been indicated. The viewer development code branch is pretty much waiting for this to happen, and CHUI remains in pole position as far as LL’s code merge plans are concerned, so potentially there could be more news on this in week 9.

Project Cocoa

Work is progressing on Project Cocoa within LL. This is a rarely talked-about project to update LL’s Mac support to the Mac OSX Cocoa API specifically for OSX 10.8 support, and remove dependencies on old Mac APIs which are not well-supported any more. The overall goal of this project, as commented on by Widely Linden is to, “Get people building cleanly with 10.8,” although OSX 10.6 will continue to be supported, although it will no longer be possible to build a Mac viewer using 10.6 once this project has been deployed. Widely also commented that there is a project viewer and source code for this work, which interested parties “should snag.”.

Vivox Update

Work is underway to update the SLvoice plugin to use the latest release of Vivox. This should bring with it a number of benefits including: security updates, stability improvements (although perhaps not improved connection reliability), better echo cancellation and – anecdotally, at least – better voice quality. There is no ETA on when this project will be deployed.

FMODex

Linden Lab continue to work on utilising FMODex as a replacement for FMOD.

Materials Project

There has been significant progress in fixing the known outstanding issues on the project which are standing in the way of a public project viewer and viewer code appearing. Speaking at the TPV Developer meeting on Friday 22nd February, Oz Linden said, “Our list of things which must be fixed before we can hand it out to people is now down to one.” However, there is still no estimated date as to when a project viewer and source code will actually appear Real Soon NowTM, which appears to put them both closer than Pretty SoonTM and Real SoonTM on the LL scale of things :).

Materials processing: with one remaining issue to fix, a project viewer now really should not be that far away. In the meantime the server code is fully deployed to the main grid
Materials processing: with one remaining issue to fix, a project viewer now really should not be that far away. In the meantime the server code is fully deployed to the main grid

As has been reported in my server-side news for the week, the server code for materials is deployed to the whole of the main grid, and so the system will be usable as soon as project viewer surfaces.

Server-side Baking

What is likely to be the first in a number of Server-side Baking load / pile-on tests took place on Thursday February 21st. Results were, at best, mixed, for a variety of reasons.

The test was held in the Sunshine project test regions on Aditi, immediately following the Server Beta User Group meeting. Those participating were asked to use the latest iteration of the official project viewer, which had been set-up for LL to do a certain amount of data logging. Anyone encountering issues was asked to raise a JIRA under the SUN project, listing issues encountered, with the viewer session log attached.

the test was in two parts:

  • Part one: performed on a region still running on a region using the current baking system, this saw people change between three of four outfits so that some baseline data could be obtained at the LL end of things. As this was using the current baking system, the usual baking issues were apparent
  • Part two: performed on a region running the new baking service, this again saw people changing between a number of outfits, this time monitoring and reporting on their own experiences.

Results were, it is fair to say, mixed. They were also not helped by the fact that Aditi itself has significant issues with inventory, etc., which made the test considerably more complicated than perhaps needed to be the case (for example, people were getting “object failed to rez”-style messages and other errors as items could not be fetched from inventory, etc.).

SSB load test: mixed results (image courtesy of Latif K
SSB load test: mixed results (image courtesy of Latif Khalifa

As an overall load test on the service itself, this should have generated some interesting numbers for LL with at least 40 people participating in the test at its peak. Commenting on the test on Friday 22nd February, Nyx Linden said, “A big thank you to everyone who participated in the pile-on yesterday. We got a lot of data out of it, [and] it looks like the majority of the issues were inventory-related, and we’re going to be digging into those. Anecdotal evidence suggests that when the system worked, it worked pretty darn well; but there were some people who had more trouble than others … We are looking into the remaining issues; we’re going to be fixing them as quickly as possible.”

While Nyx indicated that the majority of problems were inventory-related, he also stated that he and his team were still digging into the data to see if the problems were purely related to the known issues with Aditi’s inventory handling, or whether some of the issues are apparent in the inventory system itself, either on the server-side of things or within the viewer itself.

Continue reading “SL projects week 8 (3): Viewer, materials and SSB load test”

SL project news – week 8 (1): server releases, viewer, SSB load test, interest lists

Server Deployments – week 8

The deployments scheduled for the week commencing Monday 18th February are detailed below. Please note that due to Monday 18th being a holiday in the United States (Presidents Day), the deployments will be taking place one day later than usual.

Main (Second Life Server / SLS) Channel

The Main channel should receive the following two projects:

  • The Interest List Improvement project, which has been on the Magnum RC channel for the past few weeks
  • Server-side support for materials processing.

Note that there is still no publicly available project viewer to take advantage of the materials project code, although there may be news on this later in the week.

This deployment should take place on Wednesday 20th February – release notes.

Bluesteel and LeTigre Release Candidate (RC) Channels

Baker Linden in a change from his usual rooster avatar
Baker Linden in a change from his usual rooster avatar

Bluesteel should receive Baker Linden’s object rezzing code changes, which were reported here in week 1. These updates have nothing to do with the interest list code changes. Baker describes the aim of the work as, “Hopefully significantly decreasing lag spikes when rezzing large, complex objects. Large does not necessarily imply size, but size of the files being read. When an object is rezzing, we have to parse the object / mesh files and create our in-world objects with that data.”

Until now, reading and parsing of any files related to objects which require rezzing has been on the main thread. When several such objects requiring rezzing at the same time, the simulator stalls. Baker has been moving the reading / parsing operation to a background thread in the expectation that rezzing multiple “large” (again, in terms of file size, not the size of the object itself) objects will not choke the simulator, causing performance issues.

These deployments should take place on Thursday 21st February – release notes (Bluesteel).

Magnum Release Candidate (RC) Channel

Magnum should receive the same maintenance server update deployed in week 7 to LeTigre, intended to fix miscellaneous crash modes.This deployment also includes the following:

  • An improvement to the rolling restart notifications so that they appear in an alert format (as with manual region restarts) rather than an easily missed notification. This change will only be apparent in restarts following the code deployment restart (as per JIRA SVC-7759)
  • A fix to an encroachment / return problem:  if you’re banned from the neighbour’s parcel, you couldn’t select  / return items that encroached on your parcel (see JIRA SVC-496)
  • Instant messages are now truncated to 1024 bytes to prevent certain types of delivery failure. Currently, the IM database supports larger messages than the delivery system can handle. This change will enforce a limit of 1024 bytes when processing messages coming into the database as well as those being sent out.

This deployment should take place on Thursday 21st February – release notes.

SL Viewer News

The release version of the SL viewer moved to the 3.4.5 code base on the 14th February, with the release of viewer 3.4.5.270263 (release notes). At the same time, the Server-side Baking project viewer received its second update with the release of version 3.4.5.270409, od which more below.

The CHUI viewer received a further update to the development version, reaching 3.4.6.270520 on February 18th. This project is currently the next in line to merged into the viewer-dev code base (development viewer) and then into the beta code base.

Server-side Baking Load Test

Serer-side baking - load test February
Serer-side baking – load test February 21st

A reminder that if all goes according to plan, there should be a special load test for Server-side Baking on Thursday February 21st, and volunteers are being sought to assist.

This will take place on the SSB test regions on the beta grid (Aditi), immediately following the Server Beta User Group meeting which take place at 15:00 SLT on Thursdays in  Morris, also on Aditi with the aim of placing the SSB code under a stress test representative of the loads it will face when deployed to the main grid, with people routinely changing outfits, updating their appearance, enter / leave regions running the SSB code (given that the grid will, for a time, be running both the current avatar baking service and SSB as the latter is initially deployed), and so on.

Test Requirements

While final details of the test have yet to be confirmed, key requirements for those wishing to participate in the test are as follows:

  • Participants must be able to log-in to Aditi and attend the Sunshine test regions from 16:00 SLT onwards (participants can attend the Server Beta UG meeting ahead of the test if they wish)
  • Participants must be running the latest version Server-side Baking project viewer (version 3.4.5.270409 or later) – this viewer has been specifically configured to report statistics required by LL for the test
  • Participants should have a number of outfits of system clothing, preferably with multiple layers, which they can swap between during the course of the test. Library outfits are acceptable, but LL are keen for people to use their own outfits to add greater weight to the tests
  • Clearing the viewer cache prior to the test is suggested, but not an absolute requirement.

“if you have specific failures we’ll ask for your viewer logs, otherwise just running through the test will help us gather data,” Nyx added when explaining what is required by way of feedback from those opting to take part.

Further details can be found in my SSB Load Test announcement.

Continue reading “SL project news – week 8 (1): server releases, viewer, SSB load test, interest lists”

SL project news: week 4 (1): server, permissions, terrain and teleporting

Server Deployments Week 4

There was no Main Channel deployment on Tuesday 22nd January due to the issues encountered in week 3, which saw the interest list code on BlueSteel and LeTigre rolled back due to stability issues, and the maint-server release on Magnum having problems of its own (including regions not showing up in search, etc.).

There will be RC deployments on Wednesday 23rd January, as follows:

  • BlueSteel and LeTigre should receive the region crossing improvement project.  This project makes sim performance smoother when objects and avatars cross between regions.  This is the same project which was on BlueSteel and LeTigre in week 2; the only change is a fix for a crasher – release notes (BlueSteel)
  • Magnum should receive the interest list improvement project.  This update should reduce the bandwidth usage of viewers due to object updates, and should improve simulator performance, especially in sims with many connected avatars.  This is the same project which was on BlueSteel and LeTigre briefly in week 3; the only change is a fix for two crash modes – release notes
3rd time lucky? Interest List updates due for deployment to Magnum on Wednesday 23rd January
3rd time lucky? Interest List updates due for deployment to Magnum on Wednesday 23rd January

According to Simon Linden, speaking at the Simulator User Group on Tuesday 22nd January, the issue(s) relating to Magnum regions / parcels failing to show up in search also has a fix, which is apparently included in code due to be deployed to those regions, although there is no comment on this in the release notes themselves at the time of writing.

SL Viewer

Work continues on the viewer development side of things with further updates an merges and new versions appearing almost every other day, but little is currently filtering through to the beta viewer code branch at present.

Similarly, the CHUI developement viewer is going through a rapid series of updates, reaching 3.4.4.269264 on Tuesday 22nd January, although updates from the last series of development releases have yet to reach the CHUI project viewer.

Work has been promised to update the Sunshine (avatar baking) viewer code, whether this has been done or not is unclear as there was no Content Creation meeting on Monday 21st January due to Martin Luther King’s birthday; however, the current release of the project viewer remains 3.4.4.268071 at the time of writing.

Merges are also still awaiting on the Mesh project viewer in order to bring that back into line with more recent viewer-dev code updates.

Continue reading “SL project news: week 4 (1): server, permissions, terrain and teleporting”

SL project news: week 2 (3): server, mesh and materials

Server Deployments – week 2

As noted in the update to part 2 of this week’s report, the planned deployment of two new releases to the RC channels didn’t go as anticipated. Originally, it had been intended that BlueSteel and LeTigre would received the new threaded region crossing code while Magnum would receive Andrew Linden’s interest list code improvements.

Interest List Deployment Cancellation

The interest list deployment was cancelled after the 11th hour discovery of some bugs with the code. Speaking at the Server Beta meeting on Thursday 10th January, Maestro Linden described the main issues as – ironically – being connected with region crossings, and with object updates.

In the first, anyone crossing between regions several times in a vehicle would experience all of their non-rigged attachments disappearing from their world view, with the viewer itself eventually physically detaching them. Not only did this cause confusion as to what was happening with attachments for those experiencing the issue, it also resulted in some avatars ending up naked following a relog.

Interest listcode: bugs led to deployment cancellation on Wednesday 9th January
Interest list code: bugs led to deployment cancellation on Wednesday 9th January

The second problem was slightly more complicated, if potentially more rare. In it, if User A had an object on the ground and User B looked at then turned their camera away such that the object was no longer on their screen, User A could then wear the object as an attachment and teleport away; however, when User B subsequently turned their camera back to where the object had been, they would still see it on the ground despite the fact it had been taken away. What happened if User A (now wearing the object) teleported back to User B wasn’t actually tested.

As a result of both of these issues and the cancellation of the interest list deployment, Magnum received the same region crossing package as intended for BlueSteel and LeTigre.

Region Crossing Code Issues

However, the bad news did not end there, as Maestro Linden explained, “After a few hours, we saw that the [region/sim] crash rate was way too high.” As a result, the threaded region crossing code was disabled via a configuration change to the servers without the need to rollback the release. Once this had happened, region crash rates returned to “normal” levels. In all the new region crossing code was active for around five hours before being disabled once more.

Analysis of the crash rate revealed it to be linked to avatars crossing to / from heavily scripted regions. While the new code was extensively tested on Aditi, the regions there were not excessively loaded with scripts during testing, and so the problem did not manifest. However, subsequent testing with the test regions running heavy script loads did result in them also crashing, confirming the problem.

At the time of writing, Kelly Linden believes he has a fix for the issue; if so, it should hopefully find its way to the RC channels in week 3, commencing Monday 14th January.

Continue reading “SL project news: week 2 (3): server, mesh and materials”

SL project news: week 2 (2): Server deployments and network issues

Update:  The Interest list release to Magnum has been postponed due to a last-minute bug being found. Magnum has instead received the same code  as deployed to Bluesteel and LeTigre, containing the threaded region crossing update.

Server Deployments for Week 2

Server deployment recommenced this week, with a main channel roll-out on Tuesday 8th January, which saw that channel get the same code as released to the RC channels just before the Christmas / New Year break and as reported here.

Wednesday January 9th will see two major deployments to the RC channels. LeTigre and BlueSteel should be receiving a package which includes threaded region crossing code while Magnum gets Andrew Linden’s work on Interest Lists.

The threaded region crossing code should help improve simulator performance when avatars are region crossing to and from a region. However, and as previously noted, while the code did give clear improvements when crossing between regions on foot, the results were less positive when using vehicles during a recent test. Whether this has been improved as a result of those tests remains to be seen.

Release notes are available for both for LeTigre and Bluesteel, and the forum discussion thread is available for feedback / questions.

Interest List Deployment

The Interest List work should reduce the bandwidth usage of viewers due to object updates, and should improve simulator performance, especially in sims with many connected avatars and which is busy in terms of object updates.

This is the first phase of work planned around interest lists and object caching as a part of the Shining project. Andrew’s work has primarily been focused on improving the manner in which object updates are handled, etc., in order to provide improved performance. However, even with this initial phase of the work, there should be some improvements in the actual order in which objects appear in world (those closer to your camera position appearing prior to items further away – although there is far more work to be done in this regards before the project is finished. 

As it stands, there are already some updates to the work in the offing, as Andrew indicated at the Simulator User Group meeting on the 8th January, saying, “I’ve got some interest list work that didn’t quite make it into the Magnum RC … this work is almost ready for testing, so I’ll be trying to get it into a maintenance branch or something for a follow-up release.”

Interest list updates should be deployed to the Magnum RC on Weds 9th Jan.
Interest list updates should be deployed to the Magnum RC on Weds 9th Jan.

The new interest list code will see the final removal of the “legacy cloud” layer (at around the 170-200m mark). While this has been disabled viewer-side since around the introduction of Viewer 2, the data relating to the legacy cloud layer has still been sent out by the servers, allowing viewers which still incorporate the necessary code to render the clouds in-world. However, the server-side code relating to legacy clouds has been removed from the interest list code, so viewers will no longer be able to render the clouds.

The Magnum release notes are available here.

Continue reading “SL project news: week 2 (2): Server deployments and network issues”