2020 SUG meeting week #48: Uplift update

Time Remains, October 2020 – blog post

The following notes were taken from the Tuesday, November 24th Simulator User Group meeting.

Server Updates

Please also refer to the week’s server deployment thread.

  • On Tuesday November 24th,  RC channels were updated to simulator version 552571, comprising “some bug fixes and internal tweaks”. However, one fix should correct the inability to correctly set permissions (e.g. edit rights) for friends.
  • There is no planned SLS Main channel deployment for week #48.

Week #49

If all goes to plan at the Lab, week #49 (commencing Monday, November 30th) will see a daily series of rolling restarts across the grid. Due to start on Monday, this will be batches of regions being restarted, with the plan that restarts are staggered to avoid too much disruption, and if everything proceeds smoothly, each region should only be restarted once at some point in the week.

Commenting on the restarts, Maxidox Linden stated:

I think it will be something like at most 14-16 hours between rolls. Though as Rider says unless something changes we don’t plan to roll the same region in two successive rolls. And we’re going to do our best to avoid times with tons of people on-line if we can.

These deployments are liable to occur at 16-hour intervals.

Uplift Status

As per my blog post from week #48 (see: LL confirms Second Life regions now all on AWS), whilst all regions are now running on AWS services, the work in transitioning all of the Second Life back-end services is not complete, and LL are still “operating with one foot in either camp” – and this may be exacerbating the problems currently being experienced by some.

Another factor could be the different communications routes between viewers and servers following the move from operating out of the Lab’s co-lo facility in Arizona to the Amazon centre in Oregon. For some, this has definitely resulted in a noticeable increase in basic ping times to / from the servers, although for others, this has barely changed.

Commenting on the general state of play, Mazidox Linden observed:

We are not yet at what I would call “Final uplift performance” (that is to say, without any explicit attempts at tuning performance and behaviour of system communication). There is still plenty of stuff making that round trip over hundreds of miles to the data centre, slowing things down.

In this respect, some of the issues people are noticing at the moment may be down to the fact that LL haven’t as yet started fine tuning things, and are unlikely to do so until all services are running via AWS. As such, some of the problems people are noting might be down to this. In this respect, Oz Linden noted:

We’re much more focused right now on getting things other than the simulators uplifted and fixing anything that really breaks. Performance problems are a step down in priority until that’s done, but we won’t forget about them.

SL Viewer

The Start of the week has seen no change to the current crop of official viewers, leaving the as follows:

  • Current release viewer version 6.4.11.551711, formerly Cachaça Maintenance RC viewer promoted on November 12 – No change
  • Release channel cohorts:
    •  Custom Key Mappings project viewer, version 6.4.12.552100, November 12.
  • Project viewers:
    • Simple Cache project viewer, version 6.4.11.551403, issued on November 12.
    • Project Jelly project viewer (Jellydoll updates), version 6.4.11.551213, November 2.
    • Legacy Profiles viewer, version 6.4.11.550519, October 26.
    • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

In Brief

  • Group Chat: There has been an update to the Group chat servers which the Lab hopes will help alleviate the issues of the last couple of months. The hope is that as a result of this, things should be somewhat better, although it is acknowledged things are “not perfectly solid” as yet.
  • Map Tiles: there is a known issue with in-world Map tiles failing to update. At the time of writing, there was no ETA on when a fix will be implemented.
  • Teleports: people are still reporting teleport failures, although data collected by the Lab using additional logging apparently shows the overall level of teleport failures as being back to “normal” after the recent spike.
  • Textures: Slow texture loads: people are reporting slower than usual texture loading. When isn’t currently understood (given textures are among the data coming to users via the CDN, so not directly a part of the AWS transition). The speculation offered by Oz Linden is that where texture messaging is concerned, it may not be going as fast as LL like.
  • Scripts: there have also bee reports of some regions initially showing improved script performance, only to apparently drop back to “pre-uplift” levels of processing. Commenting on this, Maxidox Linden stated:
 If you’re seeing changes to scripts run the likely explanation is that there is contention for shared system resources beneath the simulator layer. That is something we had only mild control over before and have even less control over now..  That said, it’s on our radar. I can’t promise that even when someone gets time to look at it there will be anything we can *do*, but we are aware, and we’re not ignoring it. …
I mean, it is almost possible that we’re calculating that number wrong Lucia, because we have certainly changed the hardware the simulators are running on in ways that the people who made that statistic probably never imagined. I’m not going to swear that is or isn’t happening, but it will certainly be one of the many things we look at.

 

2020 CCUG meeting week #47 summary

Ruegen, October 2020 – blog post

The following notes were taken from my audio recording and chat log of the Content Creation User Group (CCUG) meeting held on Thursday, November 19th 2020 at 13:00 SLT. These meetings are chaired by Vir Linden, with dates available via the SL Public Calendar.

The venue for meetings is the Hippotropolis camp fire.

SL Viewer

At the time of writing, the current selection of official viewers (release, RC and projects) was as follows:

  • Current release viewer version 6.4.11.551711, formerly Cachaça Maintenance RC viewer promoted on November 12 – NEW.
  • Release channel cohorts:
    •  Custom Key Mappings project viewer, version 6.4.12.552100, November 12.
  • Project viewers:
    • Simple Cache project viewer, version 6.4.11.551403, issued on November 12.
    • Project Jelly project viewer (Jellydoll updates), version 6.4.11.551213, November 2.
    • Legacy Profiles viewer, version 6.4.11.550519, October 26.
    • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

General Viewer Notes

  • The Custom Key Mapping viewer is thought to be reaching a point where it can be promoted as the de facto release viewer.
  • A new Love Me Render (LMR) RC / project viewer is due to be issued. LMR-5 will include non-EEP bug fixes.
  • The Copy / Paste project viewer is on hold as a result of running into UI complications, and is currently awaiting further work by the viewer UI team, or possibly a decision to scrap the viewer entirely.

ARCTan / Jellydoll / Imposter Avatars

  • Vir has most recently looking at timers / frame rate variability.
  • As a reminder:
    • The current ARCTan work is focused on the viewer-side updates to avatar complexity calculations.
    • Work on providing in-world object rendering costs (LOD models, etc.) which might affect Land Impact will be handled as a later tranche of project work, after the avatar work.

Graphics

  • Work has commenced on separating out the rendering of viewer UI elements from scene rendering.
    • The reason for this is that the UI currently gets rendered on per frame basis, which is performance expensive on what is essentially a relatively static element of the viewer.
    • The aim is to revise the UI such that it is only re-drawn was needed, and it is hoped this will result in a performance gain in terms of viewer frame rates.
    • When this work is in a condition where it can be released, it will appear in its own project / RC viewer, and not as a part of the Love Me Render graphics-specific viewer pipeline.
  • No news on work for replacing OpenGL.

In Brief

  • There may be an issue with the viewer ignoring avatar custom joint positions if they do not have a weight associated with them. This appears true for .DAE files exports from Maya, and possibly with exports from Blender 1.7.9.a / 1.7.9.b (although Avastar may automatically compensate for unweighted custom joints by given them a default weighting). However, it is not clear if the issue lies with the modelling software or the mesh uploader, and its been requested that those experiencing these issues check the mesh upload data to see if anything is being pruned.

Date of Next Meeting

  • Thursday, December 3rd.

2020 SUG meeting week #47: uplift

Paradise on Sea, October 2020 – blog post

The following notes were taken from the Tuesday, November 17th Simulator User Group meeting.

Server Updates and Cloud Uplift

Please also refer to the week’s server deployment thread.

  • On Tuesday November 17th, the AWS RC channels were updated to simulator version 552183, which includes internal configuration changes, and the outcome of this deployment is being monitored.
  • On Wednesday, November 18th, the rest of the SLS Main channel may be migrated to running on AWS, with simulators on that channel also running sever update package 551942, which will mean all main grid (Agni) regions will be running via AWS. However, this is currently fluid – check the deployment thread for updates.
  • There may be a further deployment on Thursday, November 18th. Again, check the server deployment thread for updates.

Additional Notes

  • Due to  known issues with regions running on AWS, the Lab will continue to run Debug1 and Debug2 from their co-location facility,  for residents who need to to use for workarounds to these issues.
  • It is hoped that the configuration changes will help improve the recent TP failure and group chat  issues many have been experiencing – however, this is dependent on the above-noted monitoring of the simulator update.

SL Viewer

The Start of the week has seen no change to the current crop of official viewers, leaving the as follows:

  • Current release viewer version 6.4.11.551711, formerly Cachaça Maintenance RC viewer promoted on November 12 – NEW.
  • Release channel cohorts:
    •  Custom Key Mappings project viewer, version 6.4.12.552100, November 12.
  • Project viewers:
    • Simple Cache project viewer, version 6.4.11.551403, issued on November 12.
    • Project Jelly project viewer (Jellydoll updates), version 6.4.11.551213, November 2.
    • Legacy Profiles viewer, version 6.4.11.550519, October 26.
    • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

 

2020 TPVD meeting week #46: summary

Dya’s Southern Twilight, October 2020 – blog post

The following notes are taken from the TPV Developer meeting held on Friday, November 13th, 2020. These meetings are generally held every other week, unless otherwise noted in any given summary. The embedded video is provided to Pantera – my thanks to her for recording and providing it. Time stamps are included with the notes will open the video at the point(s) where a specific topic is discussed. Note these summaries are not intended to be a full reporting on all topics discussed, but focus on those items that are more directly user-facing.

Another exceptionally brief meeting, with some discussion in chat, so please refer to the video as well.

SL Viewer News

[0:09-3:27]

On Thursday, November 12th:

  • The Cachaça Maintenance RC viewer, version 6.4.11.551711,dated November 6th, was updated to de facto release status.
  • The Custom Key Mappings viewer was promoted to RC status with version 6.4.12.552100 on November 12th.
  • The Simple Cache project viewer, version 6.4.11.551403, was issued.
    • This version of the viewer uses a less complex file caching system to save local copies of frequently used assets such as meshes, sounds and animations. There should be no user facing differences in this viewer.

The remaining Projects viewers currently available remain as follows:

  • Project Jelly project viewer (Jellydoll updates), version 6.4.11.551213, November 2.
  • Legacy Profiles viewer, version 6.4.11.550519, October 26.
  • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
  • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
  • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

General Viewer Notes

  • Legacy Profiles viewer – still trying to get “unstuck”, but now caught with an issue apparently related to the migration work. The hope is that this can go to RC once a fix is in.
  • Jellydolls: Vir is still working on fixing some final issues before this promoted to RC.
  • A new Maintenance RC viewer should be appearing in the next week.
  • Further non-EEP graphics updates are pending a further Love Me Render RC viewer, but no ETA on arrival at this time.

In Brief

  • [3:31-6:13] LL have said they’ll look at the Moon texture flipping upside down after reaching midnight – this had previously been referenced as “expected behaviour”.
  • [6:37-18:10] Group chat issues (being able to send messages but not receive) may be account-specific, particularly following the Wednesday, November 11th deployments, although the issue is not believed to be deployment-related, but possibly network related.
    • Requests have been made to make group chat failures / lock-out a priority.
    • There is a suspicion at the Lab that the group chat issues and teleport failure issues may have a related underlying cause, particularly as they have similar characteristics and started at roughly the same time. However, LL do not have a definitive diagnosis for either or a possible link.
    • [20:09-20:48] As a part of the migration to AWS, LL added a new API that allows them to restart individual groups, that may act as a quick / short term fix as issues are being investigated, and might be something that can be extended to support to use, if test show it does help with problems on a intermediary basis.
    • Currently, the issues being experienced by the Firestorm team are impacting their ability too make a further release.
  • [18:23-18:41] The slow loading of legacy profiles via AWS regions is thought to be because the back-end support for legacy profiles has yet to be migrated to AWS.
  • [22:24-23:10] BUG-229623 “[AWS] Avatar permission changes do not stick on AWS regions”- no time frame on when this will be fixed. It is believed to be mostly a viewer-side issue, although it’s acknowledged the “simulator doesn’t do quite the right thing in the cloud”.
  • [24:55-22:08] Some are experiencing issues in access regions via teleport or vehicle where the region allows object entry / is not full, but can access it on foot or by logging directly into the region (e.g. BUG-202885 “Unable to cross into full region in a vehicle despite being a Premium user and able to cross on foot.”). This is a known issue and the Lab are looking at it, although issues such as the general TP failures,  group chat, etc., have a higher priority at this point in time.

2020 SUG meeting week #46: uplift

Still Waters, September 2020 – blog post

The following notes were taken from the November 12th Simulator User Group meeting.

Server Updates and Cloud Uplift

Please reference to the server deployment thread for the latest updates.

  • On Tuesday, November 10th, the uplifted AWS RC channels were updated with simulator release 551942. This version includes some cloud configuration changes that these may improve some of the performance metrics, but otherwise should not be anything user visible.
  • On Wednesday, November 11th, around 50% of the SLS channel will be transitioned to AWS services, also running simulator version 551942.

SL Viewer

The Start of the week has seen no change to the current crop of official viewers, leaving the as follows:

  • Current release viewer version 6.4.10.549686, formerly the Mesh Uploader RC promoted on October 14 – No Change.
  • Release channel cohorts:
    • Cachaça Maintenance RC viewer updated to version 6.4.11.551711,on November 6.
  • Project viewers:
    • Project Jelly project viewer (Jellydoll updates), version 6.4.11.551213, November 2.
    • Custom Key Mappings project viewer, version 6.4.10.549685, November 2.
    • Legacy Profiles viewer, version 6.4.11.550519, October 26.
    • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

Teleport Issues

The teleport issues – particularly AWS-to-AWS regions – are still proving problematic for some.

There are reports from some AWS-hosted regions of issues with TPs manifesting with other issues – rezzing problems, errors trying to add items to object contents. All seem to be rectified by a region restart (hardly the best solution), before things start going awry once more.

The Lab have added more logging to the simulator so they can further analyse the problem(s).

2020 CCUG meeting week #45 summary

Where Our Journey Begins, September 2020 – blog post

The following notes were taken from my audio recording and chat log of the Content Creation User Group (CCUG) meeting held on Thursday, November 5th 2020 at 13:00 SLT. These meetings are chaired by Vir Linden, and agenda notes, meeting SLurl, etc, are are available on the Content Creation User Group wiki page.

SL Viewer

At the time of writing, the current selection of official viewers (release, RC and projects) was as follows:

  • Current release viewer version 6.4.10.549686, formerly the Mesh Uploader RC promoted on October 14 – No Change.
  • Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
    • Cachaça Maintenance RC viewer, version 6.4.11.551139, issued October 27.
  • Project viewers:
    • Project Jelly project viewer (Jellydoll updates), version 6.4.11.551213, November 2.
    • Custom Key Mappings project viewer, version 6.4.10.549685, November 2.
    • Legacy Profiles viewer, version 6.4.11.550519, October 26.
    • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

ARCTan / Jellydoll / Imposter Avatars

  • Bug fixes for Jellydolls include:
    • Fixing an issue where jellydolled avatars could suddenly vanish with viewed on Mac systems.
    • Ensuring Amimesh attachments on imposter avatars update in sync with the avatar.
    • Impostered avatars and their Animesh attachments are currently rendered entirely separately to one another, so the code might be looked at to unify their rendering.
    • A general clean-up on the imposter code, which has led to the discovery that while code was added to the viewer to check to see if impostered avatars had been updated, it has never been hooked up to updating the imposter avatar as rendered by the viewer, leaving them frozen. This is now being fixed.
  • Once the Mac issue is sorted, the play is to merge the Jellydoll updates back into the ARCTan project (from which they were split earlier in the year).
    • However, the current Jellydoll work will progress as is through the currently project viewer continued through to RC and, ultimately, release.
  • As a reminder:
    • The current ARCTan work is focused on the viewer-side updates to avatar complexity calculations.
    • Work on providing in-world object rendering costs (LOD models, etc.) which might affect Land Impact will be handled as a later tranche of project work, after the avatar work.

Graphics

  • No news on work for replacing OpenGL.
  • Ptolemy Linden from the Graphics team has been working on performance improvements, notably related to Linden Water rendering. This work is currently focused on providing a means for those on low-end systems to completely disable water reflection rendering entirely and / or to make water opaque, both to reduce the rendering load.
  • The entire water rendering issue is complicated for a range of reasons (e.g. the fact the water plane is still drawn at altitude even though its appearance is occluded by the sky sphere, the fact that any changes made for some users could impact the “shared experience” / expected behaviour for others, etc.).
  • Ptolemy also noted that the viewer collects a lot of static rendering information that could potentially be used to assist with better drawing / rendering of scenes, and this could be something that might be looked at in more detail once the Project Uplift work is completed.

Date of Next Meeting

  • Thursday, November 19th.