2019 SL User Groups 18/1: Simulator User Group

Toshi Farms; Inara Pey, March 2019, on FlickrToshi Farmsblog post

Server Deployments

As always, please refer to the server deployment thread for the latest news.

  • There was no deployment of the SLS (Main) channel on Tuesday, April 20th, leaving all regions on that channel on server update 19#19.04.22.526534 comprising performance improvements (possible fixes for teleport / region crossing issues) and additional internal logging.
  • On Wednesday, May 1st, all three main RC channels should be updated to server maintenance package 19#19.04.25.526669, primarily intended to correct the simulator-side EEP regressions that resulted of the roll-back of Thursday, April 18th. This update also includes a number of other internal fixes.

SL Viewer

There have been no viewer updates at the time of writing this update, leaving the viewer pipelines as follows:

  • Current Release version 6.2.0.526190, formerly the Estate Access Management RC viewer, dated April 12, promoted April 17 NEW. – see my EAM overview for more information
  • Release channel cohorts:
  • Project viewers:
  • Linux Spur viewer, version 5.0.9.329906, dated November 17, 2017 and promoted to release status 29 November – offered pending a Linux version of the Alex Ivy viewer code.
  • Obsolete platform viewer, version 3.7.28.300847, May 8, 2015 – provided for users on Windows XP and OS X versions below 10.7. This viewer will remain available for as long as reasonable, but will not be updated with new features or bug fixes.

My.secondlife.com

My second Life has been down since the weekend of April 27/28th, 2019. This has affected all access to the site, the upload of snapshots and in-world web-based profiles.

Linden Lab is aware of the issue, and working to resolve the outage.

Teleport Disconnects

The teleport disconnect issue is now being seen as a race condition,with Simon Linden commenting:

The TP issue looks like a race condition … we “fixed” 3 other bugs and made that worse.

To which Oz Linden added:

And we’ve already put in place a bunch of improvements to our RC monitors to help prevent a similar recurrence (with more coming soon) … Doubtless Murphy will find new ways to avoid our monitors from time to time, but one tries.

The lab now feel they have investigated the issue and gathered sufficient data around it to be able to perform a post-mortem on the situation to (hopefully) reduce the likelihood of such an occurrence in the future.

Script Processing Issues

This was raised at the April 26th TPVD meeting as well. There have been numerous reports of script run time issues, with some reporting that problems only started occurring following the roll-back on April 18th, 2019. BUG-226851 outlines some of the problems.

As per the TPVD meeting, the Lab are not aware of anything that may have changed to impact script run time (particularly on Full regions); but the problem has been somewhat exacerbated by the issue being bounced between the JIRA and support (note how the above report has been closed, referencing the matter back to support).

Advertisements

2019 SL User Groups 17/1: SUG – teleport disconnects update

Puddlechurch; Inara Pey, March 2019, on FlickrUmiblog post

Server Deployments

From the server deployment thread for the week:

We are working on the TP & sim crossing disconnect issue, and making several changes over this week.  These may be a little bit more disruptive than our usual grid roll process and we apologise in advance for the inconvenience of those.  We’ll do our best to keep this disruption to a minimum. We have a simulator update which we will roll to BlueSteel and LeTigre on Tuesday.  Depending on the results we will make additional plans for gridwide rolls as soon as practical. We thank you for your patience and fully realize both the urgency and the frustration this has been causing.

Thus far, there has been a deployment to BlueSteel and LeTigre – server update 19.04.22.526534. Check the deployment thread for further updates.

SL Viewer

There have been no viewer updates at the start of the week, leaving the viewer pipelines as follows:

  • Current Release version 6.2.0.526190, formerly the Estate Access Management RC viewer, dated April 12, promoted April 17 NEW. – see my EAM overview for more information
  • Release channel cohorts:
  • Project viewers:
  • Linux Spur viewer, version 5.0.9.329906, dated November 17, 2017 and promoted to release status 29 November – offered pending a Linux version of the Alex Ivy viewer code.
  • Obsolete platform viewer, version 3.7.28.300847, May 8, 2015 – provided for users on Windows XP and OS X versions below 10.7. This viewer will remain available for as long as reasonable, but will not be updated with new features or bug fixes.

Teleport Disconnects

Let’s see … for server news, many of us are focused on the teleport issues … I hate to promise anything, but we all know a lot more about TP internal problems than we used to 🙂 . We do understand they’re annoying and a frustrating problem – we definitely want to fix it and make it better.

– Simon Linden, Simulator User Group, April 23rd

From the above statement, it should be clear that the Lab is devoting a lot of time to teleport disconnects. There have been mixed reports on the outcome of the deployment made on Thursday, April 18th, 2019, with some indicating matters have improved, others reporting no real change. In particular, people visiting Fantasy Faire and travelling around the new Linden Homes continent appear to be faring a lot better than had been the case. Which is not to say the issue has in anyway been resolved – hence the continuing work.

While it had been indicated that the recent operating system update may have played a role in the problems, Simon again referenced the timing issue with region crossings, whilst also mentioning the potential for the simulator side of EEP possibly also playing a role in things.

One of the puzzles we’re trying to sort out is if somehow the environmental work caused the TP problems – the timing of the release is suspicious but the functionality _should_ be different …. The disconnects are a problem where the viewer and 2nd region don’t start talking as they should [so something is out of sync] or there’s a failure to communicate. I know, for example, the 2nd region is waiting for the viewer to connect and get a message … that never happens …

He continued:

It doesn’t seem to be closely associated with AV complexity … that said, the more complex your AV is, the more work it needs to change regions. It’s always been better to have fewer scripts and data for teleports and region crossings.

The problem is still trying to pin down actual potential causes, with disconnects remaining inconsistent in terms of reproduction., again as Simon noted:

It’s frustratingly inconsistent. It’s a lot easier to fix something that breaks all the time … to fix it, and to know when you fixed it. For example, before this meeting we ran a test that had probably 400 or so successful teleports, no disconnects … that’s good, but not proof of a fix.

Mazidox Linden, from the Lab’s QA team further qualified the Lab’s problem:

We need in the neighbourhood of 10000 teleports to have any kind of real statistical confidence, just as a reference point 🙂 .

EEP / Windlight Issues

Following the Thursday deployment, many region holders / designers noted significant differences in how their region windlights were being rendered. The didn’t get much in the way of discussion during the meeting, however, Rider Linden offered an apology for the situation, noting that the focus on the TP issue(s) resulted in some unexpected regressions. However, it’s not currently clear what might be done to deal with this issue.

2019 SL User Groups 16/1: SUG – teleport disconnects update

Salt Water; Inara Pey, March 2019, on FlickrSalt Waterblog post

Server Deployments

As always, please refer to the server deployment thread for updates and news.

  • On Tuesday, April 9th the SLS (Main) channel was updated to server maintenance package 19#19.04.09.526122, previously deployed to the main RCs in week #15, and comprising logging improvements related to teleporting.
  • RC deployments were still TBA/TBC at the time of writing; I’ll update when more information is available.

SL Viewer

At the time of writing, there had been no viewer updates to mark the start of the week, leaving the SL viewer pipelines as follows:

  • Current Release version 6.1.1.525446, formerly the Love Me Render RC viewer, dated March 26, promoted April 2 NEW.
  • Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
    • Estate Access Management (EAM) RC viewer version 6.2.0.526190, April 12.
    • EEP RC viewer version 6.2.0.526104, April 11.
    • Bakes on Mesh RC viewer, version 6.1.1.525409, March 26.
    • Teranino Maintenance RC viewer version 6.1.1.525401, March 20.
  • Project viewers:
  • Linux Spur viewer, version 5.0.9.329906, dated November 17, 2017 and promoted to release status 29 November – offered pending a Linux version of the Alex Ivy viewer code.
  • Obsolete platform viewer, version 3.7.28.300847, May 8, 2015 – provided for users on Windows XP and OS X versions below 10.7. This viewer will remain available for as long as reasonable, but will not be updated with new features or bug fixes.

It is anticipated the EAM release candidate viewer could be  promoted to de facto release status this week.

Teleport Disconnects and Region Crossing Issues

Following the Simulator User Group meeting on Tuesday, April 9th, the Lab asked those attending to engage in a stress test of teleports to help gather data on the ongoing disconnect issues. Commenting on the results at this week’s meeting, Simon Linden also reiterated what the Lab believe is happening:

We have some good info on the underlying causes and are working on fixes, but don’t have a real improvement yet. it usually fails as the viewer and 2nd region don’t start communication properly, so I’ve been digging through that code and all the ways it can fail … that’s my life lately … I’m sure I’ll have something new for testing in a day or two so hopefully next week’s updates will be better, or at minimum give better information.

When asked if the issues with vehicle crossings are related to the same potential cause, he added:

 I haven’t looked specifically at vehicles so I can’t say, but it’s possible, yes. I tried a boat trip around a new Linden Homes continent and it was a challenge, to put it politely.

In the meantime, new code that might help relieve the issue was deployed to the Cake mini-RC channel on Tuesday, April 16th. Those wishing to test crossings on this channel can do so on the following regions: Morris, Arnthrud, Croak, Gouda, Impeller, Kasba, Oak Bay, Sapars, Yongma. Constructive feedback can be sent via IM or note card to Simon Linden.

When testing, please wait around 2-3 minutes between each teleport to allow you to fully disconnect from the region you have just departed, particularly if you intend to just bounce back and forth between the same two regions (data is retained for a short time by a region you’ve just left, in case you want to rapidly want to TP back, and this data needs to be dropped during testing, as it can affect results).

 

 

 

2019 SL User Groups 15/1: SUG; teleport disconnects

Sol Farm; Inara Pey, February 2019, on FlickrSol Farm blog post

Update, April 10th: as per Bugsly Linden’s forum comment, the RC regions are receiving a maintenance update – 19#19.04.09.526122, which contains additional logging to help capture data on the region crossing issues.

Server Deployments

As always, please refer to the server deployment thread for updates and news.

  • On Tuesday, April 9th the SLS (Main) channel was updated to server maintenance package 19#19.03.28.525781, comprising the simulator operating system update and EEP.
  • RC deployments – at the time of writing, there was a degree of conflict here:
    • The deployment thread states there is no planned deployment to the three main RC channels, and all three should remain on server maintenance package 19#19.03.28.525781.
    • At the Simulator User Group meeting, Simon Linden indicated there will likely be two deployments, stating:

We’re doing last-minute discussions but will likely have updates out tomorrow in the RC channels. One won’t have any visible changes, but has some logging that might help sort out the region change issue [teleport disconnect?].

The other is a maintenance release, the most interesting thing there is some additions to llGetObjectDetails() that give Animesh information.

    • I’ll update should the deployment thread be refreshed with new information.

UDP  Asset Messaging Deprecation

It was indicated in the week #14 deployment notes that 19#19.03.28.525781 would contain the removal of UDP Asset message handling from the simulator code, meaning that anyone using very old viewers still reliant on UDP messaging for assets will not be able to receive any updates for the following: animations, body parts, clothing, gestures, landmarks, meshes and sounds.

However, testing after the deployment of this package suggests that UDP messaging is still being handled through older viewers (notably the SL Linux Spur and Obsolete Platforms viewers) and on regions on any of the RC channels or the SLS (Main) channel. Whether this is due to a back-end issue (a stale cache?) or something else, is unclear. Commenting on the situation, Simon Linden noted:

I’m not sure what’s going on … it was _supposed_ to be disabled with the update that went to the full grid today.

SL Viewer

There have been no updates to the current SL viewer pipelines, leaving things as follows:

  • Current Release version 6.1.1.525446, formerly the Love Me Render RC viewer, dated March 26th, promoted April 2nd.
  • Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
    • Estate Access Management (EAM) RC viewer, version 6.2.0.525891, April 4th.
    • EEP RC viewer version 6.2.0.525808, April 1st.
    • Bakes on Mesh RC viewer, version 6.1.1.525409, March 26th.
    • Teranino Maintenance RC viewer version 6.1.1.525401, March 20th.
  • Project viewers:
  • Linux Spur viewer, version 5.0.9.329906, dated November 17th, 2017 and promoted to release status 29th November, 2017 – offered pending a Linux version of the Alex Ivy viewer code.
  • Obsolete platform viewer, version 3.7.28.300847, May 8th, 2015 – provided for users on Windows XP and OS X versions below 10.7. This viewer will remain available for as long as reasonable, but will not be updated with new features or bug fixes.

Teleport Disconnects

Teleport disconnects continue to be a problem, and the Lab is continuing to investigate, having issued a blog post on the matter on April 8th. Commenting on the situation at the Simulator User Group meeting, Simon Linden said:

Currently we know that the 2nd region and the viewer aren’t communicating as they should when it fails … it seems like you get transferred to the 2nd region OK, but the viewer just doesn’t get connected properly … The first region thinks all is well and it’s done, the 2nd gets the TP data but never starts talking right with the viewer.

Following the Simulator User Group meeting, a stress test was attempted, with groups of people teleporting between a group of three regions, with mixed results (some reported disconnects, others had no issues).

15-minute Disconnects

Older viewers have additionally been suffering with “regular” disconnects roughly every 15-minutes, which may be related to the deployment of the latest simulator operating system updates, and might be getting conflated with the teleport disconnects to some degree. However, for Singularity users, there are fixes in the works, and the suggestion is that, if possible, people try installing one of the more recent nightly builds.

2019 SL User Groups 14/1: SUG; teleport disconnects

On The Other Side; Inara Pey, February 2019, on FlickrOn The Other Sideblog post

Server Deployments

As always, please refer to the server deployment thread for updates and news.

  • On Tuesday, April 2nd, the SLS (Main) channel was updated to server maintenance package 19#19.03.15.525315, comprising the simulator operating system update and EEP.
  • On Wednesday, April 3rd, the three RC channels should be updated with at least one server maintenance package (details TBA at the time of writing). Updates should include:
    • The removal of UDP Asset message handling across all RC channels – see below.
    • Some changes that might help with some disconnect issues older viewers have been experiencing, and which appears to be connected to some TCP message handling between the simulator and the viewer.

Removal of UDP Asset Message Handling

The removal of UDP message handing sees the removal of server-side support for asset (inventory) messaging via UDP. All maintained viewers should be using HTTP for all asset handling – and this has been the case for a good few years for several asset types, with the last few moving to HTTP in 2017; as such these viewers will not be impacted by this deployment.

However, it does mean that anyone using very old viewers still reliant on UDP messaging for assets will not be able to receive any updates for the following: animations, body parts, clothing, gestures, landmarks, meshes and sounds whilst connected to regions on any RC channels; this includes the two legacy viewers provided by Linden Lab, the Linux Spur viewer and the Obsolete platform viewer.

In particular, and since  some specific body parts are required to render avatars, user of these older viewers will see their avatar either as a cloud or the “Ruth” avatar, and will be unable to change from it.

SL Viewer

  • The Love Me Render viewer, version 6.1.1.525446, was promoted to de facto release status on Tuesday, April 2nd.
  • The EEP RC viewer updated to version 6.2.0.525808 on Monday, April 1st.

The rest of the viewer pipelines remain as follows:

  • Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
    • Bakes on Mesh RC viewer, version 6.1.1.525409, March 26.
    • Estate Access Management (EAM) RC viewer, version 6.2.0.525404, March 25.
    • Teranino Maintenance RC viewer version 6.1.1.525401, March 20.
  • Project viewers:
  • Linux Spur viewer, version 5.0.9.329906, dated November 17, 2017 and promoted to release status 29 November – offered pending a Linux version of the Alex Ivy viewer code.
  • Obsolete platform viewer, version 3.7.28.300847, May 8, 2015 – provided for users on Windows XP and OS X versions below 10.7.

Teleport Disconnects

Teleport disconnects continue to be a problem. Some are pointing to the EEP deployment as the cause, and also citing the use of derendering the sky (Advanced menu > Rendering Types > unchecking Sky or CTRL-ALT-SHIFT-6). This seems to work for at least some of those experiencing the issue. However, Maxidox and Rider Linden have been looking into this, with Rider commenting:

The sky thing is much the same as putting a banana in your ear to keep the alligators away.

In other words, while derendering the sky my offer a workaround for the problem, it doesn’t actually resolve the issue.

Simon Linden added:

It’s definitely possible that any difference in network traffic might make the system behaviour change – but I don’t think changing the sky settings produces that much network usage.

The failures I’ve investigated seem to be falling into a middle-limbo where the first region thinks it’s handed you off successfully, but the 2nd doesn’t get all the data it expects and eventually times out.

One suggestion is to keep examinations of physical region crossing disconnects separate to TP disconnect, on account of the former tending to be more generally unpredictable due to the added data being transferred (vehicle as well as avatar(s)) and the manner in which the hand-offs are made. A further possible conflation of figures is the ongoing Singularity disconnect issue.

At the moment, the Lab believe that have one statistic that may show the issue, but as it is only recorded once a day, they are waiting to gather further data from it before determining if it is revealing a possible issue / cause.

2019 SL User Groups 13/1: Simulator User Group

Originalia; Inara Pey, February 2019, on FlickrOriginaliablog post

Server Deployments

As always, please refer to the server deployment thread for updates and news.

  • There was no deployment to the SLS (Main) channel on Tuesday, March 26th, leaving it on server maintenance package 19#19.03.07.525089. Previously deployed to BlueSteel and LeTigre, this update contains:
    • Internal fixes.
    • The Environment Enhancement Project code.
  • On Wednesday, March 27th, the BlueSteel and LeTigre RC channels should be updated with server maintenance package 19#19.03.22.525531, comprising:
    • Internal Fixes
    • Removal of UDP Asset message handling – see below
  • There is no planned deployment to the Magnum RC channel, which should remain on server maintenance package 19#19.03.15.525315, comprising:
  • The new operating system update.
  • The Environment Enhancement Project code and all bug fixes for it.

Removal of UDP Asset Message Handling

The BlueSteel and LeTigre RC deployment sees the removal of server-side support for asset (inventory) messaging via UDP. All maintained viewers should be using HTTP for all asset handling – and this has been the case for a good few years for several asset types, with the last few moving to HTTP in 2017; as such these viewers will not be impacted by this deployment.

However, it does mean that anyone using very old viewers still reliant on UDP messaging for assets will not be able to receive any updates on regions on either BlueSteel or LeTigre – this includes the two legacy viewers provided by Linden Lab, the Linux Spur viewer and the Obsolete platform viewer, neither of which contain all of the HTTP asset fetching code.

If you hear someone on a BlueSteel or LeTigre region complaining they are constantly a cloud on those region, suggest to them they check their viewer, and perhaps move to a more recent version.

SL Viewer

  • Bakes on Mesh reached RC status with the release of version 6.1.1.525409 on Tuesday, March 26th.
    • For those not familiar with this project, Bakes on Mesh is a new feature to allow system avatar baked textures to be shown on mesh attachments. Currently you need this viewer to use it.
  • The Love Me Render RC viewer updated to version 6.1.1.525446 on Tuesday, March 26th.
  • The Estate Access Management RC viewer updated to version 6.2.0.525404 on Monday, March 25th.

The rest of the viewer pipelines remain as follows:

  • Current Release version 6.1.0.524670, formerly the BugSplat RC viewer February 13, promoted February 28 No Change.
  • Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
  • Project viewers:
  • Linux Spur viewer, version 5.0.9.329906, dated November 17, 2017 and promoted to release status 29 November – offered pending a Linux version of the Alex Ivy viewer code.
  • Obsolete platform viewer, version 3.7.28.300847, May 8, 2015 – provided for users on Windows XP and OS X versions below 10.7.