Server-side Appearance – Page 11 – Inara Pey: Living in a Modemworld

SL projects week 8 (3): Viewer, materials and SSB load test

Posted on February 23, 2013August 22, 2013 by Inara Pey

SL Viewer Updates

Release and Beta Viewers

The release and beta version of the viewer are effectively on a par with one another at this point in time, following the roll-out of SL viewer 3.4.5.270263 on February 14th. There is currently nothing “in” beta at the moment in terms of specific SL projects.

Development Viewer and CHUI

The development viewer and the development version of the CHUI (Communications Hub User Interface) project viewer are also pretty much on a par, and it is anticipated that the CHUI code will be merged-up to viewer development “any minute now”, to use LL’s parlance, although a date has not been indicated. The viewer development code branch is pretty much waiting for this to happen, and CHUI remains in pole position as far as LL’s code merge plans are concerned, so potentially there could be more news on this in week 9.

Project Cocoa

Work is progressing on Project Cocoa within LL. This is a rarely talked-about project to update LL’s Mac support to the Mac OSX Cocoa API specifically for OSX 10.8 support, and remove dependencies on old Mac APIs which are not well-supported any more. The overall goal of this project, as commented on by Widely Linden is to, “Get people building cleanly with 10.8,” although OSX 10.6 will continue to be supported, although it will no longer be possible to build a Mac viewer using 10.6 once this project has been deployed. Widely also commented that there is a project viewer and source code for this work, which interested parties “should snag.”.

Vivox Update

Work is underway to update the SLvoice plugin to use the latest release of Vivox. This should bring with it a number of benefits including: security updates, stability improvements (although perhaps not improved connection reliability), better echo cancellation and – anecdotally, at least – better voice quality. There is no ETA on when this project will be deployed.

FMODex

Linden Lab continue to work on utilising FMODex as a replacement for FMOD.

Materials Project

There has been significant progress in fixing the known outstanding issues on the project which are standing in the way of a public project viewer and viewer code appearing. Speaking at the TPV Developer meeting on Friday 22nd February, Oz Linden said, “Our list of things which must be fixed before we can hand it out to people is now down to one.” However, there is still no estimated date as to when a project viewer and source code will actually appear Real Soon Now^TM, which appears to put them both closer than Pretty Soon^TM and Real Soon^TM on the LL scale of things :).

Materials processing: with one remaining issue to fix, a project viewer now really should not be that far away. In the meantime the server code is fully deployed to the main grid

As has been reported in my server-side news for the week, the server code for materials is deployed to the whole of the main grid, and so the system will be usable as soon as project viewer surfaces.

Server-side Baking

What is likely to be the first in a number of Server-side Baking load / pile-on tests took place on Thursday February 21st. Results were, at best, mixed, for a variety of reasons.

The test was held in the Sunshine project test regions on Aditi, immediately following the Server Beta User Group meeting. Those participating were asked to use the latest iteration of the official project viewer, which had been set-up for LL to do a certain amount of data logging. Anyone encountering issues was asked to raise a JIRA under the SUN project, listing issues encountered, with the viewer session log attached.

the test was in two parts:

Part one: performed on a region still running on a region using the current baking system, this saw people change between three of four outfits so that some baseline data could be obtained at the LL end of things. As this was using the current baking system, the usual baking issues were apparent
Part two: performed on a region running the new baking service, this again saw people changing between a number of outfits, this time monitoring and reporting on their own experiences.

Results were, it is fair to say, mixed. They were also not helped by the fact that Aditi itself has significant issues with inventory, etc., which made the test considerably more complicated than perhaps needed to be the case (for example, people were getting “object failed to rez”-style messages and other errors as items could not be fetched from inventory, etc.).

SSB load test: mixed results (image courtesy of Latif Khalifa

As an overall load test on the service itself, this should have generated some interesting numbers for LL with at least 40 people participating in the test at its peak. Commenting on the test on Friday 22nd February, Nyx Linden said, “A big thank you to everyone who participated in the pile-on yesterday. We got a lot of data out of it, [and] it looks like the majority of the issues were inventory-related, and we’re going to be digging into those. Anecdotal evidence suggests that when the system worked, it worked pretty darn well; but there were some people who had more trouble than others … We are looking into the remaining issues; we’re going to be fixing them as quickly as possible.”

While Nyx indicated that the majority of problems were inventory-related, he also stated that he and his team were still digging into the data to see if the problems were purely related to the known issues with Aditi’s inventory handling, or whether some of the issues are apparent in the inventory system itself, either on the server-side of things or within the viewer itself.

Continue reading “SL projects week 8 (3): Viewer, materials and SSB load test” →

Nyx confirms SSB load test

Posted on February 20, 2013August 22, 2013 by Inara Pey

Nyx Linden has sent out an e-mail confirming the Server-side baking load test scheduled for Thursday 21st February, and which I’ve posted about on a couple of occasions recently.

The e-mail reads in full:

Nyx Linden - SSB load test call — Nyx Linden – SSB load test call

Greetings all,

In preparation for rolling out server-side avatar appearance, we’ll be running a short test tomorrow afternoon. If you are available, or know someone who is, please come to the server user group (http://wiki.secondlife.com/wiki/Server_Beta_User_Group) with the latest sunshine viewer (http://wiki.secondlife.com/wiki/Project_Sunshine-Server_Side_Appearance version number 270409). We will run the test after the server meeting, for those able to stick around. You will need several outfits that your avatar can switch between and will do so both on the old system and the new system. Also please clear your cache before attending.

Please use our latest viewer as it has additional statistics gathering code that will allow us to calculate load patterns and measure the improvements expected for later releases.

Let me know if there are any questions!

Nyx Linden

If you are interested in participating, please check my report on the test announcement, which outlines the aim of the test, what is likely to be taking place, and what is required.

You should check your ability to log-into Aditi well in advance of the test, and confirm you have a number of outfits (preferably with multiple layers) in your Aditi inventory.

Additional Information

Server-side Baking test regions on Aditi
SL wiki page for SSB (includes download links for the viewer)
SLurl for the Server Beta meeting on Aditi (15:00 SLT Thursdays)

SL project news – week 8 (1): server releases, viewer, SSB load test, interest lists

Posted on February 18, 2013August 22, 2013 by Inara Pey

Server Deployments – week 8

The deployments scheduled for the week commencing Monday 18th February are detailed below. Please note that due to Monday 18th being a holiday in the United States (Presidents Day), the deployments will be taking place one day later than usual.

Main (Second Life Server / SLS) Channel

The Main channel should receive the following two projects:

The Interest List Improvement project, which has been on the Magnum RC channel for the past few weeks
Server-side support for materials processing.

Note that there is still no publicly available project viewer to take advantage of the materials project code, although there may be news on this later in the week.

This deployment should take place on Wednesday 20th February – release notes.

Bluesteel and LeTigre Release Candidate (RC) Channels

Baker Linden in a change from his usual rooster avatar

Bluesteel should receive Baker Linden’s object rezzing code changes, which were reported here in week 1. These updates have nothing to do with the interest list code changes. Baker describes the aim of the work as, “Hopefully significantly decreasing lag spikes when rezzing large, complex objects. Large does not necessarily imply size, but size of the files being read. When an object is rezzing, we have to parse the object / mesh files and create our in-world objects with that data.”

Until now, reading and parsing of any files related to objects which require rezzing has been on the main thread. When several such objects requiring rezzing at the same time, the simulator stalls. Baker has been moving the reading / parsing operation to a background thread in the expectation that rezzing multiple “large” (again, in terms of file size, not the size of the object itself) objects will not choke the simulator, causing performance issues.

These deployments should take place on Thursday 21st February – release notes (Bluesteel).

Magnum Release Candidate (RC) Channel

Magnum should receive the same maintenance server update deployed in week 7 to LeTigre, intended to fix miscellaneous crash modes.This deployment also includes the following:

An improvement to the rolling restart notifications so that they appear in an alert format (as with manual region restarts) rather than an easily missed notification. This change will only be apparent in restarts following the code deployment restart (as per JIRA SVC-7759)
A fix to an encroachment / return problem: if you’re banned from the neighbour’s parcel, you couldn’t select / return items that encroached on your parcel (see JIRA SVC-496)
Instant messages are now truncated to 1024 bytes to prevent certain types of delivery failure. Currently, the IM database supports larger messages than the delivery system can handle. This change will enforce a limit of 1024 bytes when processing messages coming into the database as well as those being sent out.

This deployment should take place on Thursday 21st February – release notes.

SL Viewer News

The release version of the SL viewer moved to the 3.4.5 code base on the 14th February, with the release of viewer 3.4.5.270263 (release notes). At the same time, the Server-side Baking project viewer received its second update with the release of version 3.4.5.270409, od which more below.

The CHUI viewer received a further update to the development version, reaching 3.4.6.270520 on February 18th. This project is currently the next in line to merged into the viewer-dev code base (development viewer) and then into the beta code base.

Server-side Baking Load Test

Serer-side baking - load test February — Serer-side baking – load test February 21st

A reminder that if all goes according to plan, there should be a special load test for Server-side Baking on Thursday February 21st, and volunteers are being sought to assist.

This will take place on the SSB test regions on the beta grid (Aditi), immediately following the Server Beta User Group meeting which take place at 15:00 SLT on Thursdays in Morris, also on Aditi with the aim of placing the SSB code under a stress test representative of the loads it will face when deployed to the main grid, with people routinely changing outfits, updating their appearance, enter / leave regions running the SSB code (given that the grid will, for a time, be running both the current avatar baking service and SSB as the latter is initially deployed), and so on.

Test Requirements

While final details of the test have yet to be confirmed, key requirements for those wishing to participate in the test are as follows:

Participants must be able to log-in to Aditi and attend the Sunshine test regions from 16:00 SLT onwards (participants can attend the Server Beta UG meeting ahead of the test if they wish)
Participants must be running the latest version Server-side Baking project viewer (version 3.4.5.270409 or later) – this viewer has been specifically configured to report statistics required by LL for the test
Participants should have a number of outfits of system clothing, preferably with multiple layers, which they can swap between during the course of the test. Library outfits are acceptable, but LL are keen for people to use their own outfits to add greater weight to the tests
Clearing the viewer cache prior to the test is suggested, but not an absolute requirement.

“if you have specific failures we’ll ask for your viewer logs, otherwise just running through the test will help us gather data,” Nyx added when explaining what is required by way of feedback from those opting to take part.

Further details can be found in my SSB Load Test announcement.

Continue reading “SL project news – week 8 (1): server releases, viewer, SSB load test, interest lists” →

SL project news: week 7 (2): Server-side Baking load test

Posted on February 15, 2013August 22, 2013 by Inara Pey

Server Deployments

the server deployments planned for week 7 all went ahead as scheduled.These comprised:

Main channel: (Second Life Server / SLS): received the maint-server package focused on crashes fixes which was deployed in week 6 to LeTigre – release notes
Bluesteel: retained the materials processing project code and received the same fixes and updates being deployed to the SLS channel (above) – release notes
Magnum: retained the interest list project code and received the same fixes and updates being deployed to the SLS channel (above) – release notes
LeTigre received a new maintenance server update to fix miscellaneous crash modes – release notes. This deployment also included the following:
- An improvement to the rolling restart notifications so that they appear in an alert format (as with manual region restarts) rather than an easily missed notification. This change will only be apparent in restarts following the code deployment restart (as per JIRA SVC-7759)
- A fix to an encroachment / return problem: if you’re banned from the neighbour’s parcel, you couldn’t select / return items that encroached on your parcel (see JIRA SVC-496)
- Instant messages are now truncated to 1024 bytes to prevent certain types of delivery failure. Currently, the IM database supports larger messages than the delivery system can handle. This change will enforce a limit of 1024 bytes when processing messages coming into the database as well as those being sent out.

Feedback on all the deployments has so far been muted, with only a couple of issues having been reported via the forum thread,

There is no news on deployments for week 8 (commencing Monday, 18th February), as the meeting to determine upcoming deployments does not take place at the Lab until Friday of the current week.

Server-side Baking (SSB) Load Test

On Thursday February 21st there will be a special load test for Server-side baking, and LL are looking for volunteers to help.

This will take place on the SSB test regions on the beta grid (Aditi), immediately following the Server Beta User Group meeting which take place at 15:00 SLT on Thursdays in Morris, also on Aditi.

The aim is to place the SSB code under a stress test which is representative of how SSB will be used once it is deployed to the main grid – with people routinely changing outfits, updating their appearance (as SSB handles appearance updates differently to the current service), enter / leave regions running the SSB code (given that the grid will, for a time, be running both the current avatar baking service and SSB as the latter is initially deployed), and so on.

“We have a few other internal stress tests, but wanted to do one with real-world conditions on real connections,” Nyx linden explained when announcing the test opportunity. “The test will run through switching from the old system to the new system, which is a transition where issues may pop up. if enough people are changing outfits simultaneously it should get us some valid data.”

Test Requirements

While final details of the test have yet to be confirmed, key requirements for those wishing to participate in the test are as follows:

Participants must be able to log-in to Aditi and attend the Sunshine test regions from 16:00 SLT onwards (participants can attend the Server Beta UG meeting ahead of the test if they wish)
Participants must be running the latest version Server-side Baking project viewer (version 3.4.5.270409 or later) – this viewer has been specifically configured to report statistics required by LL for the test
Participants should have a number of outfits of system clothing, preferably with multiple layers, which they can swap between during the course of the test. Library outfits are acceptable, but LL are keen for people to use their own outfits to add greater weight to the tests
Clearing the viewer cache prior to the test is suggested, but not an absolute requirement.

Aditi Log-in

As has been reported in this blog on a number of occasions, the Aditi grid is itself subject to a number of issues, both in terms of access and inventory support.

If you have not logged-in to the beta grid at all, or in the last several months and wish to participate in the SSB tests, it is recommended that you test your Aditi log-in (you use the same user name and password as you use to access the main (Agni) grid) sooner rather than later. If you find you are unable to log-in, then you should try changing your password. to refresh your Aditi access (this should also update your beta grid inventory).

HOWEVER, be warned that this process isn’t always successful, due to the issues mentioned above, and that it can take up to 48 hours before you can access Aditi, and even then, your inventory may not be successfully updated.

If you have recently updated your password and have reasonable inventory access on Aditi, the recommendation is that you don’t force any inventory update by running a further password change, as there is a risk you may either lose access to Aditi or that you may become subject to inventory change failures.

Those who wish to participate in the load test, and who encounter either issues with their Aditi inventory or accessing the beta grid can, as a last resort, contact Nyx Linden at least 24 hours ahead of the test. Nyx will then endeavour to see if LL can fix matters.

Other Items

Vanishing Regions

Last updated in SL project news: week 6 (2), wherein diagonally adjacent regions fail to render until such time as the observer moves to a region immediately bordering the “missing” region. This has been an ongoing problem for some time, as reported in SVC-8130, and commenting on it at the Server Beta meeting, Simon Linden indicated that the maint-server code deployed to LeTigre may help with some of the issues being encountered, but also admitted, “There’s been some improvement but it looks like there are still bugs to chase there.”

Missing regions: new Maint-server code may help...? — Missing regions: new Maint-server code may help…?

There is a forum post on the matter – if you are encountering this issue in a reproducible manner, and in lieu of SVC-8130 being open to comment, you might want to note your experiences on the thread.

SL project news week 7 (1): Server deploys, viewer updates

Posted on February 12, 2013August 22, 2013 by Inara Pey

Server Deployments for Week 7

On Tuesday February 12th, the Main channel (Second Life Server / SLS) received the maint-server package focused on crashes fixes which was deployed in week 6 to LeTigre – release notes.

On Wednesday February 13th the three RC channels are scheduled to receive the following packages:

Bluesteel: retains the materials processing project code – please see the SL Viewer section below for information regarding the current status of viewer development work for thi project – and should receive the same fixes and updates being deployed to the SLS channel (above) – release notes
Magnum: retains the interest list project code and should receive the same fixes and updates being deployed to the SLS channel (above) – release notes
LeTigre should receive a new maintenance server update to fix miscellaneous crash modes – release notes. This deployment also includes the following:
- An improvement to the rolling restart notifications so that they appear in an alert format (as with manual region restarts) rather than an easily missed notification. This change will only be apparent in restarts following the code deployment restart (as per JIRA SVC-7759)
- An avatar banned from a parcel can now select an object located in that parcel if they have ‘Return’ powers over that object (see JIRA SVC-496)
- Instant messages are now truncated to 1024 bytes to prevent certain types of delivery failure. Currently, the IM database supports larger messages than the delivery system can handle. This change will enforce a limit of 1024 bytes when processing messages coming into the database as well as those being sent out.

As always, a forum thread has been created for discussions / feedback on the deployments.

SL viewer Updates

Beta Viewer

While the current beta release of the 3.4.5 code remains stable, problems are being reported in merging various project code updates with the 3.4.5 code base, some TPV developers are reporting issues attempting to merge the 3.4.5 code into their viewer builds, notably with Linux 64-bit and Windows, although the same issue does not appear to be occurring with Mac OSx.

In the meantime, one more 3.4.5 beta release of the SL viewer is anticipated during week 7.

CHUI

As noted in part 3 of the week 6 report, the CHUI project viewer is now largely on a par with the development viewer, although it has yet to be actually merged with the main viewer-dev code trunk. This is due to happen very soon. In week 6, Oz Linden commented that it is anticipated that CHUI will be the first of the upcoming projects to merge with the beta viewer code, and would then be in for a (quot) “Good long run” in beta. This week, it has been hinted that CHUI could actually have a lengthy run in both the development viewer and the beta viewer as final priorities are sorted out.

Server-side Baking Project Viewer

Problems remain in merging the SSB code with viewers using RLVa, although good progress is being made in addressing them, although they have yet to be merged-up to a TPV which supports RLVa and have a version under development using the SSB code.

In the meantime, the latest updates to the viewer code from the Lab were pushed to the project repositories on Monday 11th February, and a new version of the Sunshine project viewer for Windows emerged on Tuesday February 12th – Sunshine 3.4.5.270231, with Mac and Linux builds to follow.

Materials Viewer

Problems remain with the materials processing viewer-side code. Precise details are not clear, but the latest private (LL and project members only) Windows builds are being reported as being, “Conspicuously more unstable than the Mac builds”, in that they tend to crash when connecting to a materials-enabled server. Investigations into the problems (notably by TPV developers engaged in the project) are ongoing to try to locate the underpinning cause.

Materials processing: viewer issues still being resolved, so no public project viewer as yet

There are a couple more features to go into the materials viewer as well prior to it reaching a public project viewer status, but these are apparently being held-off until the current server interaction problems are resolved.

Experience Permissions

July saw the launch of the first phase of the Advanced Creation Tools, also referred to as experience tools. Following problems with an initial deployment of the tools in June, which resulted them being exploited as a means of griefing, the “first phase” of the release saw the tools implemented with existing permissions system in place, with the intention of updating the permissions system to allow the tools to be more fully used “in the future”. Since then, it has been something of a waiting game to see if / when the promised new experience permissions would be deployed, and my own last report on the general status of the permissions system being in December 2012.

Questions are still being asked at various LL meetings, he most recent being tabled at the Content Creation User Group on Monday 11th February. In reply, Nyx Linden indicated he would attempt to get an update from Falcon Linden, who is leading this project.

Object Inventory Issues

Some people may have encountered problems when trying to transfer a number of inventory items into a box (prim rezzed in-world) or similar. There are two known issues with this:

The first is (non-publicly viewable) JIRA BUG-383: “When HTTP Inventory is disabled, mass selecting a bunch of inventory items and attempting to add to an objects contents will fail”. This will apparently happen if the HTTP inventory setting in a viewer is changed without cache also being cleared
The second is (non-publicly viewable) JIRA SEC-1041, which is apparently (according to Kelly Linden) related to “An internal TCP error between sim and dataserver.” The problem manifests with the in-world error-message “Inventory creation on in-world object failed” on trying to transfer a large number of items (e.g. 50+) from inventory into the contents of a prim in one go.

As the JIRA on both of these issues are non-viewable, their status isn’t clear, other than the problems described in them can be reproduced. SEC-1041 is the more serious of the two problems and might result in a crash, although it will not result in physical damage to a person’s inventory.

SL projects news week 6 (3): viewer and CHUI, SSB, materials

Posted on February 9, 2013August 22, 2013 by Inara Pey

SL Beta Viewer and CHUI

As noted in part 2 of this report, a new beta release of the SL viewer was made on February 6th. The crash rates for this version, some 48 hours on from the release, are reported by Oz Linden as being “remarkably low”, with some 1400 users / half-a-million minutes logged against it to date. If this continues through the weekend, the chances are the code will go to a release version of the viewer early in week 7 (week commencing Monday 11th February).

If things go according to plan, there is likely to be one more 3.4.5 beta release prior to a merge taking place with the Communications Hubs User Interface (CHUI) code.

Currently, the CHUI code is now fully merged with the viewer-dev code, although it is slightly out-of-step with the beta code. Once the merge to beta has been made, and with everything else going on with the viewer, it is anticipated that CHUI will have “a nice long run,” as a beta release.

Server-side Baking and Pile-on Test Pre-announcement

SSB: looking to solve avatar bake fail

Nyx Linden and his team continue to work on server-side baking (SBB), both on the server and the viewer side of things. An update to the project viewer and viewer code was made back on January 30th, and Nyx has indicated that another push is due early in week 7.

The major news on this project is that a pile-on test is in the planning stage. While dates and times have yet to be confirmed, this is being targeted for later in February on Aditi, and the Lab is looking to have people running a Server-side baking viewer (e.g. the project viewer or any TPV which has implemented the necessary viewer-side code) attend and help provide a thorough load test for the new code.

The test will require viewers to be enabled for certain types of data logging in order to ensure LL gather the relevant information. Details on what will be required will be passed on to those TPVs which indicate they can participate in the test. For those individuals wishing to help out, but whose preferred viewer has not enabled the SSB code by the time of the test, the LL project viewer will have the necessary logging options already enabled. Further details on the proposed test will be published here as they become available.

In terms of TPV integration, it has been confirmed that the SSB code changes for the viewer impact RLVa. Details of the overall impact is unclear, other than RLVa is affected in several areas and Kitty Barnett is working to fix issues. However, this does mean that integration of the SSB code has slowed for some TPVs.

Materials Processing and Project Viewer

As previously reported, the server-side code is on the BlueSteel RC channel, and there are no apparent issues with the code which might prevent its wider distribution. There are still some issues to resolve with materials processing, which means that a project viewer may be delayed another week or so.

Some of the problems appear to have been network related. When testing recent versions of the pre-release materials viewer on a materials-enabled region of the Main grid, it was noted that they (quote) “significantly added” to the network load for the test region, impacting performance. These problems required additional investigation in order to resolve, and it is hoped that they have now been “wrestled to the ground”. There were also some reports of a couple of things still to be fixed in the UI, although it appears that, again, these have now been dealt with.

Given this, and while a publicly available project viewer appearing in week 7 is “not out of the question”, the Lab appear to be looking at week 8 (commencing Monday 18th February) as the week in which a materials project viewer will surface. In the meantime, the recommendation remains that TPVs do not pull code from the current materials code repository for integration into their own viewers.

The look of things to come: Materials Processing should have a public viewer available in the next two weeks