Tag: Simulator User Group 2020

The following notes were taken from the November 12th Simulator User Group meeting.

Server Updates and Cloud Uplift

Please reference to the server deployment thread for the latest updates.

• On Tuesday, November 10th, the uplifted AWS RC channels were updated with simulator release 551942. This version includes some cloud configuration changes that these may improve some of the performance metrics, but otherwise should not be anything user visible.
• On Wednesday, November 11th, around 50% of the SLS channel will be transitioned to AWS services, also running simulator version 551942.

SL Viewer

The Start of the week has seen no change to the current crop of official viewers, leaving the as follows:

• Current release viewer version 6.4.10.549686, formerly the Mesh Uploader RC promoted on October 14 – No Change.
• Release channel cohorts:
  • Cachaça Maintenance RC viewer updated to version 6.4.11.551711,on November 6.
• Project viewers:
  • Project Jelly project viewer (Jellydoll updates), version 6.4.11.551213, November 2.
  • Custom Key Mappings project viewer, version 6.4.10.549685, November 2.
  • Legacy Profiles viewer, version 6.4.11.550519, October 26.
  • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
  • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
  • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

Teleport Issues

The teleport issues – particularly AWS-to-AWS regions – are still proving problematic for some.

There are reports from some AWS-hosted regions of issues with TPs manifesting with other issues – rezzing problems, errors trying to add items to object contents. All seem to be rectified by a region restart (hardly the best solution), before things start going awry once more.

The Lab have added more logging to the simulator so they can further analyse the problem(s).

The following notes were taken from the November 3rd Simulator User Group meeting.

Server Updates and Cloud Uplift

Please reference to the server deployment thread for the latest updates.

• There are no planned deployments to the simulators running on the Lab’s core SLS channel.
• RC deployments are planned as follows:
  • On Wednesday, 4th November all simulators on the LeTigre and BlueSteel RC channels should become AWS hosted.
  • On Thursday, 5th November all simulators on the Magnum RC channel should also become AWS hosted.
  • However, at the time of the meeting, it was not clear if all of the RC channels would be running the same version of simulator software.

The current work on migration is such that as per Private Regions Available in Limited Quantity (via Linden Lab), private regions are once more being made available.

SL Viewer

The Start of the week saw the following viewer updates on Monday, November 2nd:

• The Jellydoll project viewer updated to version 6.4.11.551213.
• Custom Key Mappings project viewer updated to version 6.4.10.549685.

The rest of the official viewers in the pipelines remain as follows:

• Current release viewer version 6.4.10.549686, formerly the Mesh Uploader RC promoted on October 14 – No Change.
• Release channel cohorts:
  • Cachaça Maintenance RC viewer, version 6.4.11.551139, issued October 27.
• Project viewers:
  • Legacy Profiles viewer, version 6.4.11.550519, October 26.
  • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
  • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
  • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

In Brief

• Group Chat: LL deployed updates to the group chat service in an attempt to relieve at least some of the issues that groups have been experiencing over the last several weeks. Testing has suggested the group chat sessions should be faster and more reliable than has been experienced within some groups (notably those with large memberships). However, the issue remains open pending further observation / feedback.
• TP failures continue, and are being noted by the Lab, although not at the rates at which users appear to be experiencing them.
  • However, the nature of the beast means that at present, correlation of all the logs involved in a teleport has to be done manually, and this is impacting the Lab’s ability to arrive at a potential root cause (or causes).
  • Once the majority of cloud migration work has been completed, and if the matters hasn’t been resolved, Simon Linden may set-up another round of TP testing by users as we’ve seen in past issues of teleport issues.

The following notes were taken from the October 20th Simulator User Group meeting.

Server Updates and Cloud Uplift

• There are no planned deployments to the regions running within the Lab’s co-lo facilities.
• Starting on Tuesday, October 27th, all regions on the “AWS channel” will be updated with a simulator release  – version 551155 – that includes extended logging capabilities and a range of fixes, as indicated in the release notes.

For details of the deployment of main grid regions running via AWS cloud services, please see the following:

• A Light in the Cloud: A Migration Update – Linden Lab, October 22nd.
  • Second Life Cloud Uplift: April Linden updates – this blog, October 22nd (expansion on the above).
• Uplift Update – Linden Lab, October 16th.
  • Oz Linden posts on Second Life cloud uplift status – this blog, October 16th (expansion on the above).

Issues

The grid has seen a range of issues coincident to the uplift work that may or may not be related to that work (as they are occurring across regions that are both now running on AWS and those still running at the Lab’s co-lo facility, although some appear to happen more frequently within AWS region.) The latest of these is object failing to rez when pulled from inventory. However, Mazidox Linden indicated that LL believe they have an handle on this issue, and the AWS region simulator version may have address it.

The most significant  issues (again which may or may not be related to the cloud uplift) being experienced comprise:

• Group chat failures.
• Teleport failures.
• Rezzing issues.
• Slow opening of legacy profiles.
• Slow opening on the World Map (hopefully fixed with the 551155 release).

A further issue that has been noted, but yet to be confirmed, is that idle scripts appear to be taking up to to between 16% and 26% more script timing on regions running on AWS when compared to regions still operating out of the Labs co-lo facility.

Uplift Status

• As per April Linden’s blog post, around 1,000 region on the main grid that are running on AWS.
  • Thus far, LL have only seen minor issues from their perspective, with those regions are running with “only minor hiccups”.
• The deployment this week will see the total number of regions on AWS increase to between 2,000-2,500.
• The updates required to allow the simulator to run on AWS systems represents – in Mazidox Linden’s words – “”the largest change to the simulator [software]  ever.”
• Maxidox also confirmed there is an upcoming change that will see simulators running on AWS given a synthetic hostname rather than the viewer just giving the IP address of the AWS compute node in URI form.  The synthetic hostname will map to how LL describes simhosts internally.

SL Viewer

The Start of the week has seen the following viewer updates:

• The Cachaça Maintenance RC viewer updated to version 6.4.11.551139 on Tuesday, October 27th, 2020.
• The Legacy Profiles Viewer updated to version 6.4.11.550519 on Monday, October 26th, 2020.

The rest of the official viewers in the pipelines remain as follows:

• Current release viewer version 6.4.10.549686, formerly the Mesh Uploader RC released October t and promoted on October 14 – No Change.
• Project viewers:
  • Project Jelly project viewer (Jellydoll updates), version 6.4.10.549690, October 1.
  • Custom Key Mappings project viewer, version 6.4.5.544079, June 30.
  • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
  • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
  • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

In Brief

• Some users (notably Firestorm users) are claiming that uplift is “breaking” region crossings, and that Firestorm should offer a higher default bandwidth allowance. However:
  •  The bandwidth setting applies to UDP only – the majority of data related to avatars, attachments and objects are handled via HTTP through the CDN, over which the UDP bandwidth has no influence.
  • While UDP messaging is involved in region crossings, changing the bandwidth default is viewed as potentially unwise whilst the uplift for is in progress, as it would largely be impossible to objectively tell if the change has improved matters.

The following notes were taken from the October 20th Simulator User Group meeting.

Cloud Uplift Update

There are no formal simulator deployments for week #43. This is to make way for selected regions (both RC and Main channel) to be migrated to AWS services (the cloud). This means those regions that are transitioned will be restarted.

For details of the initial announce on the uplift, please refer to Lab expanding number of regions on AWS.

Speaking at the simulator User Group meeting on October 20th, Maxidox Linden provided the following update:

This morning we started our medium-size (at least by the size of a world) movement of regions to cloud based simhosts. Specifically we now have over 300 simulators running in the cloud. We’re looking at how quickly we can move more, as well.

For this round we went with a mixture of extremely high impact regions and extremely low impact “filler” regions, to stress various parts of our systems like the region allocation process and how we pack regions into a host.

In future rounds it’s likely we’ll be focusing on specific Release Candidate channels.

Basically: My *goal* (I’m not saying it’s what we’re doing, but I’m saying it’s what I’m trying to do) is to make this the worst it gets, and it only gets better from here.

– Mazidox Linden

Bullet Notes on Uplift

• As of the October 20th SUG meeting, some 300 regions have been transitioned to the cloud, representing a mix of region in terms of use.
• Performance issues have been warned about because there are a number of back-end systems that have yet to be uplifted (and presumably need the simulators in place before they can be), which may have some impact.
• The hope is that some of these additional services will be transitioned when they can, rather than being “left until last”.
  • One of those earmarked for transition that has been causing some angst are the servers managing the KVP database for experiences.  There have already bee reports of degraded experience performance on AWS-hosted regions, so the hope is to get the KVP database moved, with “quiet a bit of effort” going into it.
• Those who have had regions uplifted and feel they are suffering adversely from the move can request a roll back to being hosted in the Lab’s co-lo. However:
  • As all regions will be running on AWS services by year’s end, LL would rather region holders bear with any issues they may encounter whilst things are in a state of flux, and if they are encountering specific issues, to work with the Lab to try and identity and rectify matters.
  • Those who feel they need a roll-back to the lab’s co-lo facility should file a support ticket, providing a clear indication of why they would like their region(s) rolled back, and the issues they are encountering.

During the transition (as now) either the uplifted or data centre regions will have some penalty … we expect that will mostly disappear by the time we’re done

– Oz Linden on possible simulator (and other) performance impacts
during region migration

• The Lab’s aim is to continue to make the transitioning of services from their co-lo to AWS services as invisible to users as possible (that is, you shouldn’t really be able to tell the difference between a service now running on AWS and when it was running via the Lab’s co-lo).

SL Viewer

There have been no updates to the current list of official viewers, leaving the pipelines as follows:

• Current release viewer version 6.4.10.549686, formerly the Mesh Uploader RC released October t and promoted on October 14 – No Change.
• Release channel cohorts:
  • Cachaça Maintenance RC viewer, version 6.4.10.549752, issued October 1.
• Project viewers:
  • Project Jelly project viewer (Jellydoll updates), version 6.4.10.549690, October 1.
  • Custom Key Mappings project viewer, version 6.4.5.544079, June 30.
  • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
  • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
  • Legacy Profiles viewer, version 6.3.2.530836, September 17, 2019. Covers the re-integration of Viewer Profiles.
  • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

In Brief

• Group chat issues are being worked on, but is pending the uplift work.
• The long-promised increase in sound loop support from 10 seconds to 30seconds, first promised a fair while ago now, is … pending the uplift work – although it is rumoured (and subject to confirmation) to be a Premium Plus benefit.