SL project news 8 (2): servers and issues

Update February 24th: Metabolt 0.9.64.0 (Beta) was released on February 23rd to address the issue of nearby objects not being recorded following theserver-side interest list updates.

Update February 22nd:  Radegast 2.8 has been released, which both provides support for server-side baking and resolves the interest list releated issue of failing to correctly report nearby objects noted in the first poart of this report.

Week 8 Main (Second Life Server / SLS) Channel Deployment

The expected Main channel (SLS) deployment took place on Wednesday 20th February, as anticipated. The SLS channel deployment came a day late as a result of the Presidents Day holiday in the US – Main channel deployments will be back to their normal Tuesday slot from week 9 (week commencing Monday 25th February.

Issues

  • There is a report that both the Radegast and Metabolt text clients are either incorrectly seeing local objects, or failing to see them at all, possibly as a result of the interest list code deployment.
  • There is a bug in the  interest list code which means that child prims within a linkset do not always render from a distance – you need to cam in towards them for them to appear, and they can then vanish again on camming away. A fix for this is expected in the “next round” of interest list updates.

Week 8 Release Channel Deployments

There have been signficant changes to the RC channel deployments occurring on Thursday 21st February (again a day late due to Presidents Day).

Due to a last-minute bug being found in Baker Linden’s large object rezzing code updates, the package intended for both BlueSteel and LeTigre has been pulled. Instead, these channels will receive the code targeted for Magnum, together with the updates made to the Main channel.

Diagonal Region Rendering Issues

I’ve reported on this on a number of occasions recently, and JIRA SVC-8130 is still open on the matter. The incidence of occurrences seems to be on the increase, with more people having anecdotal evidence of problems. Following his comments in the second part of my week 7 report, Simon believes a fix is in the works, with Simon reporting at the Simulator User Group meeting on Tuesday 19th February that it is, “With QA now.”

Missing regions: bug fix on the way ...
Missing regions: bug fix on the way …

Region Crossing Issues

Numerous problems are being reported with region crossings, some of which appear to be on the increase of late.

Content “Warping”

I first reported on this in week 48 of 2012, as one of a number of region crossing issues being reported. When crossing between regions, the contents of the region you’ve just left suddenly “warp” into the region you’ve just entered, appearing for a time and perhaps somewhat deformed.

When the matter was first raised, Andrew Linden thought it might be more a server-side issue than a viewer-side problem – although all the issues raised at that time seemed to be split between possible issues within the server code and the viewer code.

warp-4
Sometimes on crossing between regions, objects from the region you’ve just left seem to appear “warp” into the region you’ve just entered, and may appeared deformed…

In looking at the problem as a server-side issue, Andrew had hoped that the matter would be resolved via the interest list code roll-out. However, it has persisted despite the interest list code having been deployed to an RC channel, and is now largely regarded as a viewer-side issue. The status of any fix for the problem is unclear.

...only to disappear after a few seconds / as you approach them
…only to disappear after a few seconds / as you approach them

Camera Position Lost on Region Crossings

There has ben a return / increase in issues of the camera position getting “messed up” when crossing between some regions in a vehicle. This can take a variety of forms, most notably with the camera suddenly shifting to show a side view of the vehicle while driving, rather than the usual forward-looking view SVC-7684 provides the details, complete with a video demonstrating the issue. It appears that fixes for this problem have thus far been on a per-region basis, as there is no clear indicator as to the underlying issue – although it is recognised enough such that Motor Loon now incorporates a scripted workaround for the problem in lieu of a fix, which forces the correct camera positioning to be re-applied on a script crossing. The Lab have been trying to investigate the issue, but are having a hard time reproducing it in a consistent manner. Those routinely experiencing the problem on given region crossings should consider filing a JIRA specifying the issue and region crossings where it occurs and reference SVC-7684.

Increase in Crossing Disconnects and Other Issues

The Simulator User Group meeting on Tuesday 19th February saw an increase in the number of other region crossing issues, with people reporting several issues, ranging from a disconnect on attempting a region crossing through to an increase in teleport failures when moving between issues and getting stuck in the corner of a region and spinning during a region crossing. Again, it is unclear what may be causing these issues, some of which have been reported as increasing somewhat sharply over the last 2-4 weeks.

Sudden Region Lag

It has been a while since I last updated on this issue, as reported in both a forum thread by Toysoldier Thor and others and in a JIRA BUG (BUG-355). In sort, the problem remains, with live event venues reporting issues, and the Lab uncertain as to the cause. When the issue was raised at the Simulator User Group meeting, Simon Linden replied, “Hmm, that sounds like it needs a fresh look. It seems like there’s a spike in networking traffic, performance goes bad and thus all sorts of things don’t work well.”

Interestingly, the four regions of the recent OBR in SL event did not suffer any such issues, despite a constant and high volume of traffic, although these had apparently received additional attention from the Lab to help ensure problems might be minimised.

Server-side Baking Load Test

A final reminder that there will be a Server-side Baking load test on Aditi on Thursday February 21st, following the Server Beta meeting. For details, please see my original announcement.

Nyx confirms SSB load test

Nyx Linden has sent out an e-mail confirming the Server-side baking load test scheduled for Thursday 21st February, and which I’ve posted about on a couple of occasions recently.

The e-mail reads in full:

Nyx Linden - SSB load test call
Nyx Linden – SSB load test call

Greetings all,

In preparation for rolling out server-side avatar appearance, we’ll be running a short test tomorrow afternoon. If you are available, or know someone who is, please come to the server user group (http://wiki.secondlife.com/wiki/Server_Beta_User_Group) with the latest sunshine viewer (http://wiki.secondlife.com/wiki/Project_Sunshine-Server_Side_Appearance  version number 270409). We will run the test after the server meeting, for those able to stick around. You will need several outfits that your avatar can switch between and will do so both on the old system and the new system. Also please clear your cache before attending.

Please use our latest viewer as it has additional statistics gathering code that will allow us to calculate load patterns and measure the improvements expected for later releases.

Let me know if there are any questions!

Nyx Linden

If you are interested in participating, please check my report on the test announcement, which outlines the aim of the test, what is likely to be taking place, and what is required.

You should check your ability to log-into Aditi well in advance of the test, and confirm you have a number of outfits (preferably with multiple layers) in your Aditi inventory.

Additional Information

Firestorm meeting 13th February, 2013 – video and transcript

Update February 25th: As per a comment from Jessica Lyon, Firestorm have now merged the Server-side Baking code and updates to RLVa into one of their private repositories.

firestorm-logoOn the 13th February, 2013, the Firestorm team hosted a question-and-answer session so they could outline the current status of the Firestorm viewer, the issues the team (and SL) are facing, and outline plans for both the immediate future and longer-term as well as address questions from the audience.

While the meeting was recorded, the Firestorm team are aware that many of their users have hearing difficulties, and / or prefer to read text. It is because of this that this transcript has been provided, otgether with the original recording itself. When reading it, please remember:

  • This is not a word-for-word transcript of the entire meeting. While all quotes given are as they are spoken in the video, to assist in readability and maintain the flow of conversation, not all asides, jokes, interruptions, etc., have been included in the text presented here
  • If there are any sizeable gaps in comments from a speaker which resulted from asides, questions to other speakers or requests for images to be displayed, these are indicated by the use of “…”.
  • Timestamps are provided as guidance should anyone wish to hear the comments in full from any speaker on the video
  • Questions were asked in chat during the meeting and while speakers were talking. This inevitably meant that replies to questions would lag well behind when they were orginially asked. Therefore, to provide context for both questions and answers, questions have been included in the transcript and timestamped at the point at which each is addressed by a member of the Firestorm team
  • The meeting proper commences at 16:36 into the video, and this is the point at which the transcript starts.

Video courtesy of Northspring.

Continue reading “Firestorm meeting 13th February, 2013 – video and transcript” →

SL project news – week 8 (1): server releases, viewer, SSB load test, interest lists

Server Deployments – week 8

The deployments scheduled for the week commencing Monday 18th February are detailed below. Please note that due to Monday 18th being a holiday in the United States (Presidents Day), the deployments will be taking place one day later than usual.

Main (Second Life Server / SLS) Channel

The Main channel should receive the following two projects:

  • The Interest List Improvement project, which has been on the Magnum RC channel for the past few weeks
  • Server-side support for materials processing.

Note that there is still no publicly available project viewer to take advantage of the materials project code, although there may be news on this later in the week.

This deployment should take place on Wednesday 20th February – release notes.

Bluesteel and LeTigre Release Candidate (RC) Channels

Baker Linden in a change from his usual rooster avatar
Baker Linden in a change from his usual rooster avatar

Bluesteel should receive Baker Linden’s object rezzing code changes, which were reported here in week 1. These updates have nothing to do with the interest list code changes. Baker describes the aim of the work as, “Hopefully significantly decreasing lag spikes when rezzing large, complex objects. Large does not necessarily imply size, but size of the files being read. When an object is rezzing, we have to parse the object / mesh files and create our in-world objects with that data.”

Until now, reading and parsing of any files related to objects which require rezzing has been on the main thread. When several such objects requiring rezzing at the same time, the simulator stalls. Baker has been moving the reading / parsing operation to a background thread in the expectation that rezzing multiple “large” (again, in terms of file size, not the size of the object itself) objects will not choke the simulator, causing performance issues.

These deployments should take place on Thursday 21st February – release notes (Bluesteel).

Magnum Release Candidate (RC) Channel

Magnum should receive the same maintenance server update deployed in week 7 to LeTigre, intended to fix miscellaneous crash modes.This deployment also includes the following:

  • An improvement to the rolling restart notifications so that they appear in an alert format (as with manual region restarts) rather than an easily missed notification. This change will only be apparent in restarts following the code deployment restart (as per JIRA SVC-7759)
  • A fix to an encroachment / return problem:  if you’re banned from the neighbour’s parcel, you couldn’t select  / return items that encroached on your parcel (see JIRA SVC-496)
  • Instant messages are now truncated to 1024 bytes to prevent certain types of delivery failure. Currently, the IM database supports larger messages than the delivery system can handle. This change will enforce a limit of 1024 bytes when processing messages coming into the database as well as those being sent out.

This deployment should take place on Thursday 21st February – release notes.

SL Viewer News

The release version of the SL viewer moved to the 3.4.5 code base on the 14th February, with the release of viewer 3.4.5.270263 (release notes). At the same time, the Server-side Baking project viewer received its second update with the release of version 3.4.5.270409, od which more below.

The CHUI viewer received a further update to the development version, reaching 3.4.6.270520 on February 18th. This project is currently the next in line to merged into the viewer-dev code base (development viewer) and then into the beta code base.

Server-side Baking Load Test

Serer-side baking - load test February
Serer-side baking – load test February 21st

A reminder that if all goes according to plan, there should be a special load test for Server-side Baking on Thursday February 21st, and volunteers are being sought to assist.

This will take place on the SSB test regions on the beta grid (Aditi), immediately following the Server Beta User Group meeting which take place at 15:00 SLT on Thursdays in  Morris, also on Aditi with the aim of placing the SSB code under a stress test representative of the loads it will face when deployed to the main grid, with people routinely changing outfits, updating their appearance, enter / leave regions running the SSB code (given that the grid will, for a time, be running both the current avatar baking service and SSB as the latter is initially deployed), and so on.

Test Requirements

While final details of the test have yet to be confirmed, key requirements for those wishing to participate in the test are as follows:

  • Participants must be able to log-in to Aditi and attend the Sunshine test regions from 16:00 SLT onwards (participants can attend the Server Beta UG meeting ahead of the test if they wish)
  • Participants must be running the latest version Server-side Baking project viewer (version 3.4.5.270409 or later) – this viewer has been specifically configured to report statistics required by LL for the test
  • Participants should have a number of outfits of system clothing, preferably with multiple layers, which they can swap between during the course of the test. Library outfits are acceptable, but LL are keen for people to use their own outfits to add greater weight to the tests
  • Clearing the viewer cache prior to the test is suggested, but not an absolute requirement.

“if you have specific failures we’ll ask for your viewer logs, otherwise just running through the test will help us gather data,” Nyx added when explaining what is required by way of feedback from those opting to take part.

Further details can be found in my SSB Load Test announcement.

Continue reading “SL project news – week 8 (1): server releases, viewer, SSB load test, interest lists” →

Viewer release summary 2013: week 7

This summary is published every Monday and is a list of SL viewer / client releases (official and TPV) made during the previous week. When reading it, please note:

  • It is based on my Viewer Round-up Page, a list of  all Second Life viewers and clients that are in popular use (and of which I am aware) and which are recognised as adhering to the TPV Policy
  • By its nature, this summary will always be in arrears
  • The Viewer Round-up Page is updated as soon as I’m aware of any releases / changes to viewers & clients, and should be referred to for more up-to-date information as the week progresses
  • The Viewer Round-up Page also includes comprehensive links to download pages, blog notes, release notes, etc., as well as links to any / all reviews of specific viewers / clients made within this blog.  

Updates for the week ending: 17 February, 2013

  • SL Viewer updates:
      • Current release version updated to 3.4.5.270263 on February 14th –  release notes
      • Development viewer updated to 3.4.6.270268 on February 14th
      • CHUI development viewer reached release 3.4.6.270520 on February 18th
      • Sunshine (server-side baking project) project viewer updated to 3.4.5.270409 on February 14th – wiki page
  • Cool VL updated on the 9th of February and again on the 16th to arrve at:
    • Stable version: 1.26.6.11
    • Legacy version (v2.6 renderer): 1.26.4.54
    • Experimental version: 1.26.7.11
    • Release notes
  • Lumiya released version 2.4.3 on February 16th – core updates: flexiprim support; teleport to selected avatar; improved notification settings; corrected attachment positions on avatars  –  release notes
  • Metabolt updated to 0.9.62.0 (Beta) – February 17th – core updates:creator UUID now displayed for objects in Object Manager; radar now displays all avatars in the region; upgraded to the latest version of libopenmv; optimisations and bug fixes – release notes

Discontinued Viewers

  • Phoenix officially reached end-of-line for SL on December 31st – read more here
  • Zen viewer was withdrawn from the SL TPV directory and all repositories shutdown on January 27th, 2013.

Related Links

SL project news: week 7 (2): Server-side Baking load test

Server Deployments

the server deployments planned for week 7 all went ahead as scheduled.These comprised:

  • Main channel: (Second Life Server / SLS): received the maint-server package focused on crashes fixes which was  deployed in week 6 to LeTigre – release notes
  • Bluesteel: retained the materials processing project code and received the same fixes and updates being deployed to the SLS channel (above) – release notes
  • Magnum: retained the interest list project code and received the same fixes and updates being deployed to the SLS channel (above) – release notes
  • LeTigre received a new maintenance server update to fix miscellaneous crash modes – release notes. This deployment also included the following:
    • An improvement to the rolling restart notifications so that they appear in an alert format (as with manual region restarts) rather than an easily missed notification. This change will only be apparent in restarts following the code deployment restart (as per JIRA SVC-7759)
    • A fix to an encroachment / return problem:  if you’re banned from the neighbour’s parcel, you couldn’t select  / return items that encroached on your parcel (see JIRA SVC-496)
    • Instant messages are now truncated to 1024 bytes to prevent certain types of delivery failure. Currently, the IM database supports larger messages than the delivery system can handle. This change will enforce a limit of 1024 bytes when processing messages coming into the database as well as those being sent out.

Feedback on all the deployments has so far been muted, with only a couple of issues having been reported via the forum thread,

There is no news on deployments for week 8 (commencing Monday, 18th February), as the meeting to determine upcoming deployments does not take place at the Lab until Friday of the current week.

Server-side Baking (SSB) Load Test

Serer-side baking - load test February
Serer-side baking – load test February 21st

On Thursday February 21st there will be a special load test for Server-side baking, and LL are looking for volunteers to help.

This will take place on the SSB test regions on the beta grid (Aditi), immediately following the Server Beta User Group meeting which take place at 15:00 SLT on Thursdays in  Morris, also on Aditi.

The aim is to place the SSB code under a stress test which is representative of how SSB will be used once it is deployed to the main grid – with people routinely changing outfits, updating their appearance (as SSB handles appearance updates differently to the current service), enter / leave regions running the SSB code (given that the grid will, for a time, be running both the current avatar baking service and SSB as the latter is initially deployed), and so on.

“We have a few other internal stress tests, but wanted to do one with real-world conditions on real connections,” Nyx linden explained when announcing the test opportunity. “The test will run through switching from the old system to the new system, which is a transition where issues may pop up. if enough people are changing outfits simultaneously it should get us some valid data.”

Test Requirements

While final details of the test have yet to be confirmed, key requirements for those wishing to participate in the test are as follows:

  • Participants must be able to log-in to Aditi and attend the Sunshine test regions from 16:00 SLT onwards (participants can attend the Server Beta UG meeting ahead of the test if they wish)
  • Participants must be running the latest version Server-side Baking project viewer (version 3.4.5.270409 or later) – this viewer has been specifically configured to report statistics required by LL for the test
  • Participants should have a number of outfits of system clothing, preferably with multiple layers, which they can swap between during the course of the test. Library outfits are acceptable, but LL are keen for people to use their own outfits to add greater weight to the tests
  • Clearing the viewer cache prior to the test is suggested, but not an absolute requirement.

“if you have specific failures we’ll ask for your viewer logs, otherwise just running through the test will help us gather data,” Nyx added when explaining what is required by way of feedback from those opting to take part.

Aditi Log-in

As has been reported in this blog on a number of occasions, the Aditi grid is itself subject to a number of issues, both in terms of access and inventory support.

If you have not logged-in to the beta grid at all, or in the last several months and wish to participate in the SSB tests, it is recommended that you test your Aditi log-in (you use the same user name and password as you use to access the main (Agni) grid) sooner rather than later. If you find you are unable to log-in, then you should try changing your password. to refresh your Aditi access (this should also update your beta grid inventory).

HOWEVER, be warned that this process isn’t always successful, due to the issues mentioned above, and that it can take up to 48 hours before you can access Aditi, and even then, your inventory may not be successfully updated.

If you have recently updated your password and have reasonable inventory access on Aditi, the recommendation is that you don’t force any inventory update by running a further password change, as there is a risk you may either lose access to Aditi or that you may become subject to inventory change failures.

Those who wish to participate in the load test, and who encounter either issues with their Aditi inventory or accessing the beta grid can, as a last resort, contact Nyx Linden at least 24 hours ahead of the test. Nyx will then endeavour to see if LL can fix matters.

Other Items

Vanishing Regions

Last updated in SL project news: week 6 (2), wherein diagonally adjacent regions fail to render until such time as the observer moves to a region immediately bordering the “missing” region. This has been an ongoing problem for some time, as reported in SVC-8130, and commenting on it at the Server Beta meeting, Simon Linden indicated that the maint-server code deployed to LeTigre may help with some of the issues being encountered, but also admitted, “There’s been some improvement but it looks like there are still bugs to chase there.”

Missing regions: new Maint-server code may help...?
Missing regions: new Maint-server code may help…?

There is a forum post on the matter – if you are encountering this issue in a reproducible manner, and in lieu of SVC-8130 being open to comment, you might want to note your experiences on the thread.

Related Links