The following notes were taken from the Tuesday, December 1st Simulator User Group (SUG) meeting.
Server Updates
On Monday, November 30th, the simulators on the SLS Main channel were all updated to version 552571, matching the RC channels.
Originally, it had been planned to follow this with a tight series of rolls over the next few days, with simulators updated in batches at roughly 16-hour intervals. However, at the time of writing this is on hold, as Rider Linden explained:
That plan, like so many others, is now up in the air. There are a couple of last minute snags that we ran into and so we’re evaluating what the best way to do these deploys is going to be and we do not yet have an answer. We will keep you all informed on the blog where things settle out.
This means that – again at the time of writing – the rolls listed on the grid status pages are TBD, and subject to update from LL.
No Change Window
Normally, December sees a limited number of deployments for the simulators, with a “no change” window coming into effect over the holiday period (usually from just before the Christmas break to just after New Year’s day).
However this year, thanks to the migration work, this are apparently more fluid, and dates – if any – have yet to be announced.
SL Viewer
The Start of the week has seen no change to the current crop of official viewers, leaving them as follows:
Current release viewer version 6.4.11.551711, formerly Cachaça Maintenance RC viewer promoted on November 12 – No Change.
Release channel cohorts:
Custom Key Mappings project viewer, version 6.4.12.552100, November 12.
Project viewers:
Simple Cache project viewer, version 6.4.11.551403, issued on November 12.
Project Jelly project viewer (Jellydoll updates), version 6.4.11.551213, November 2.
Legacy Profiles viewer, version 6.4.11.550519, October 26.
Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.
We know what’s going on but unfortunately it’s going to take a bit of time before we can have a fix. Some of the other work on getting to the cloud needs to get done first. It’s complicated by how inter-dependent our systems are.
On Tuesday November 24th, RC channels were updated to simulator version 552571, comprising “some bug fixes and internal tweaks”. However, one fix should correct the inability to correctly set permissions (e.g. edit rights) for friends.
There is no planned SLS Main channel deployment for week #48.
Week #49
If all goes to plan at the Lab, week #49 (commencing Monday, November 30th) will see a daily series of rolling restarts across the grid. Due to start on Monday, this will be batches of regions being restarted, with the plan that restarts are staggered to avoid too much disruption, and if everything proceeds smoothly, each region should only be restarted once at some point in the week.
Commenting on the restarts, Maxidox Linden stated:
I think it will be something like at most 14-16 hours between rolls. Though as Rider says unless something changes we don’t plan to roll the same region in two successive rolls. And we’re going to do our best to avoid times with tons of people on-line if we can.
These deployments are liable to occur at 16-hour intervals.
Uplift Status
As per my blog post from week #48 (see: LL confirms Second Life regions now all on AWS), whilst all regions are now running on AWS services, the work in transitioning all of the Second Life back-end services is not complete, and LL are still “operating with one foot in either camp” – and this may be exacerbating the problems currently being experienced by some.
Another factor could be the different communications routes between viewers and servers following the move from operating out of the Lab’s co-lo facility in Arizona to the Amazon centre in Oregon. For some, this has definitely resulted in a noticeable increase in basic ping times to / from the servers, although for others, this has barely changed.
Commenting on the general state of play, Mazidox Linden observed:
We are not yet at what I would call “Final uplift performance” (that is to say, without any explicit attempts at tuning performance and behaviour of system communication). There is still plenty of stuff making that round trip over hundreds of miles to the data centre, slowing things down.
In this respect, some of the issues people are noticing at the moment may be down to the fact that LL haven’t as yet started fine tuning things, and are unlikely to do so until all services are running via AWS. As such, some of the problems people are noting might be down to this. In this respect, Oz Linden noted:
We’re much more focused right now on getting things other than the simulators uplifted and fixing anything that really breaks. Performance problems are a step down in priority until that’s done, but we won’t forget about them.
SL Viewer
The Start of the week has seen no change to the current crop of official viewers, leaving the as follows:
Current release viewer version 6.4.11.551711, formerly Cachaça Maintenance RC viewer promoted on November 12 – No change
Release channel cohorts:
Custom Key Mappings project viewer, version 6.4.12.552100, November 12.
Project viewers:
Simple Cache project viewer, version 6.4.11.551403, issued on November 12.
Project Jelly project viewer (Jellydoll updates), version 6.4.11.551213, November 2.
Legacy Profiles viewer, version 6.4.11.550519, October 26.
Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.
In Brief
Group Chat: There has been an update to the Group chat servers which the Lab hopes will help alleviate the issues of the last couple of months. The hope is that as a result of this, things should be somewhat better, although it is acknowledged things are “not perfectly solid” as yet.
Map Tiles: there is a known issue with in-world Map tiles failing to update. At the time of writing, there was no ETA on when a fix will be implemented.
Teleports: people are still reporting teleport failures, although data collected by the Lab using additional logging apparently shows the overall level of teleport failures as being back to “normal” after the recent spike.
Textures: Slow texture loads: people are reporting slower than usual texture loading. When isn’t currently understood (given textures are among the data coming to users via the CDN, so not directly a part of the AWS transition). The speculation offered by Oz Linden is that where texture messaging is concerned, it may not be going as fast as LL like.
Scripts: there have also bee reports of some regions initially showing improved script performance, only to apparently drop back to “pre-uplift” levels of processing. Commenting on this, Maxidox Linden stated:
If you’re seeing changes to scripts run the likely explanation is that there is contention for shared system resources beneath the simulator layer. That is something we had only mild control over before and have even less control over now.. That said, it’s on our radar. I can’t promise that even when someone gets time to look at it there will be anything we can *do*, but we are aware, and we’re not ignoring it. …
I mean, it is almost possible that we’re calculating that number wrong Lucia, because we have certainly changed the hardware the simulators are running on in ways that the people who made that statistic probably never imagined. I’m not going to swear that is or isn’t happening, but it will certainly be one of the many things we look at.
On Tuesday November 17th, the AWS RC channels were updated to simulator version 552183, which includes internal configuration changes, and the outcome of this deployment is being monitored.
On Wednesday, November 18th, the rest of the SLS Main channel may be migrated to running on AWS, with simulators on that channel also running sever update package 551942, which will mean all main grid (Agni) regions will be running via AWS. However, this is currently fluid – check the deployment thread for updates.
There may be a further deployment on Thursday, November 18th. Again, check the server deployment thread for updates.
Additional Notes
Due to known issues with regions running on AWS, the Lab will continue to run Debug1 and Debug2 from their co-location facility, for residents who need to to use for workarounds to these issues.
It is hoped that the configuration changes will help improve the recent TP failure and group chat issues many have been experiencing – however, this is dependent on the above-noted monitoring of the simulator update.
SL Viewer
The Start of the week has seen no change to the current crop of official viewers, leaving the as follows:
Current release viewer version 6.4.11.551711, formerly Cachaça Maintenance RC viewer promoted on November 12 – NEW.
Release channel cohorts:
Custom Key Mappings project viewer, version 6.4.12.552100, November 12.
Project viewers:
Simple Cache project viewer, version 6.4.11.551403, issued on November 12.
Project Jelly project viewer (Jellydoll updates), version 6.4.11.551213, November 2.
Legacy Profiles viewer, version 6.4.11.550519, October 26.
Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.
The following notes were taken from the November 12th Simulator User Group meeting.
Server Updates and Cloud Uplift
Please reference to the server deployment thread for the latest updates.
On Tuesday, November 10th, the uplifted AWS RC channels were updated with simulator release 551942. This version includes some cloud configuration changes that these may improve some of the performance metrics, but otherwise should not be anything user visible.
On Wednesday, November 11th, around 50% of the SLS channel will be transitioned to AWS services, also running simulator version 551942.
SL Viewer
The Start of the week has seen no change to the current crop of official viewers, leaving the as follows:
Current release viewer version 6.4.10.549686, formerly the Mesh Uploader RC promoted on October 14 – No Change.
Release channel cohorts:
Cachaça Maintenance RC viewer updated to version 6.4.11.551711,on November 6.
Project viewers:
Project Jelly project viewer (Jellydoll updates), version 6.4.11.551213, November 2.
Custom Key Mappings project viewer, version 6.4.10.549685, November 2.
Legacy Profiles viewer, version 6.4.11.550519, October 26.
Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.
Teleport Issues
The teleport issues – particularly AWS-to-AWS regions – are still proving problematic for some.
There are reports from some AWS-hosted regions of issues with TPs manifesting with other issues – rezzing problems, errors trying to add items to object contents. All seem to be rectified by a region restart (hardly the best solution), before things start going awry once more.
The Lab have added more logging to the simulator so they can further analyse the problem(s).
The Start of the week saw the following viewer updates on Monday, November 2nd:
The Jellydoll project viewer updated to version 6.4.11.551213.
Custom Key Mappings project viewer updated to version 6.4.10.549685.
The rest of the official viewers in the pipelines remain as follows:
Current release viewer version 6.4.10.549686, formerly the Mesh Uploader RC promoted on October 14 – No Change.
Release channel cohorts:
Cachaça Maintenance RC viewer, version 6.4.11.551139, issued October 27.
Project viewers:
Legacy Profiles viewer, version 6.4.11.550519, October 26.
Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.
In Brief
Group Chat: LL deployed updates to the group chat service in an attempt to relieve at least some of the issues that groups have been experiencing over the last several weeks. Testing has suggested the group chat sessions should be faster and more reliable than has been experienced within some groups (notably those with large memberships). However, the issue remains open pending further observation / feedback.
TP failures continue, and are being noted by the Lab, although not at the rates at which users appear to be experiencing them.
However, the nature of the beast means that at present, correlation of all the logs involved in a teleport has to be done manually, and this is impacting the Lab’s ability to arrive at a potential root cause (or causes).
Once the majority of cloud migration work has been completed, and if the matters hasn’t been resolved, Simon Linden may set-up another round of TP testing by users as we’ve seen in past issues of teleport issues.
The following notes were taken from the October 20th Simulator User Group meeting.
Server Updates and Cloud Uplift
There are no planned deployments to the regions running within the Lab’s co-lo facilities.
Starting on Tuesday, October 27th, all regions on the “AWS channel” will be updated with a simulator release – version 551155 – that includes extended logging capabilities and a range of fixes, as indicated in the release notes.
For details of the deployment of main grid regions running via AWS cloud services, please see the following:
The grid has seen a range of issues coincident to the uplift work that may or may not be related to that work (as they are occurring across regions that are both now running on AWS and those still running at the Lab’s co-lo facility, although some appear to happen more frequently within AWS region.) The latest of these is object failing to rez when pulled from inventory. However, Mazidox Linden indicated that LL believe they have an handle on this issue, and the AWS region simulator version may have address it.
The most significant issues (again which may or may not be related to the cloud uplift) being experienced comprise:
Group chat failures.
Teleport failures.
Rezzing issues.
Slow opening of legacy profiles.
Slow opening on the World Map (hopefully fixed with the 551155 release).
A further issue that has been noted, but yet to be confirmed, is that idle scripts appear to be taking up to to between 16% and 26% more script timing on regions running on AWS when compared to regions still operating out of the Labs co-lo facility.
Uplift Status
As per April Linden’s blog post, around 1,000 region on the main grid that are running on AWS.
Thus far, LL have only seen minor issues from their perspective, with those regions are running with “only minor hiccups”.
The deployment this week will see the total number of regions on AWS increase to between 2,000-2,500.
The updates required to allow the simulator to run on AWS systems represents – in Mazidox Linden’s words – “”the largest change to the simulator [software] ever.”
Maxidox also confirmed there is an upcoming change that will see simulators running on AWS given a synthetic hostname rather than the viewer just giving the IP address of the AWS compute node in URI form. The synthetic hostname will map to how LL describes simhosts internally.
SL Viewer
The Start of the week has seen the following viewer updates:
The Cachaça Maintenance RC viewer updated to version 6.4.11.551139 on Tuesday, October 27th, 2020.
The Legacy Profiles Viewer updated to version 6.4.11.550519 on Monday, October 26th, 2020.
The rest of the official viewers in the pipelines remain as follows:
Current release viewer version 6.4.10.549686, formerly the Mesh Uploader RC released October t and promoted on October 14 – No Change.
Project viewers:
Project Jelly project viewer (Jellydoll updates), version 6.4.10.549690, October 1.
Custom Key Mappings project viewer, version 6.4.5.544079, June 30.
Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.
In Brief
Some users (notably Firestorm users) are claiming that uplift is “breaking” region crossings, and that Firestorm should offer a higher default bandwidth allowance. However:
The bandwidth setting applies to UDP only – the majority of data related to avatars, attachments and objects are handled via HTTP through the CDN, over which the UDP bandwidth has no influence.
While UDP messaging is involved in region crossings, changing the bandwidth default is viewed as potentially unwise whilst the uplift for is in progress, as it would largely be impossible to objectively tell if the change has improved matters.
