2020 CCUG meeting week #44 summary

Poughkeepsie, September 2020 – blog post

The following notes were taken from my audio recording and chat log of the Content Creation User Group (CCUG) meeting held on Thursday, October 29th 2020 at 13:00 SLT. These meetings are chaired by Vir Linden, and agenda notes, meeting SLurl, etc, are are available on the Content Creation User Group wiki page.

There was a lot of general discussion about animation synchronisation, cloud uplift (the majority of which I’ve already reported in my SUG and TPVD meetings of late), etc., none of which should out as really reportable.

Jellydoll / ARCTan

  • Vir has been working on some bug fixes to the Project Jelly viewer (version 6.4.10.549690 at the time of writing), and this work is now with the Lab’s QA team. When issues, this will see the viewer also based on the current release viewer code base.
  • As a reminder:
    • The current ARCTan work is focused on the viewer-side updates to avatar complexity calculations.
    • Work on providing in-world object rendering costs (LOD models, etc.) which might affect Land Impact will be handled as a later tranche of project work, after the avatar work.

Graphics

  • The Graphics team is still looking at options for replacing OpenGL, particularly as a result of Apple’s plans to deprecate all support for it on their systems.
  • The OpenGL project is large-scale, so the Graphic team is also looking at other shorter-term projects related to performance improvements. These may include:
    • Possible optimisations for people on low-end systems (as noted in my previous CCUG summary, roughly 1/3 of all users are running on systems that do not support Vulkan and many of these systems are more than 5 years old).
    • Linden Water: since the last TPV meeting (see: 2020 TPVD meeting week #42: summary) the Lab believe they have determined that the FPS loss seen when rendering Linden Water appears to be related to a combination of the actual rendering and with water reflections. This is still being investigated. One idea to deal with the latter is to completely disable water reflections in the viewer; however, doing this at present affects the sky and the look of the Sun, so again,further investigation is needed.
  • In addition, the graphics team is looking to see if the viewer UI rendering can be divorced from other aspects of viewer rendering to see if any performance improvements can be obtained by keeping the two separate – currently, a lot of cycles are taken up in the drawing of things like UI panels and buttons.

In  Brief / General

  • Teleport failures are still under investigation. Beq Janus from Firestorm is also investigation, and has encountered a situation where the failure may be related to the manner in which regions are queried bases on draw distance, and the system potentially becomes confused as to the actual destination.
  • Megaprims and relaxing linking constraints (BUG-229551):
    • As  expressed by the feature request, the Lab sees this as being two issues: allowing the use of prims larger than 64m on a size / diameter, and handling region surrounds.
    • In particular, the Lab is looking at the potential of braking out region surrounds (“sim surrounds”) as a supported feature in their own right, simply because of the size of the use case. One suggestion for this is to have what is seen beyond a the edges of an individual region to be definable as texturing, rather than just the default of Linden Water.
    • Increasing prim sizing beyond the current maximum is seen as problematic as the size constraint touches on multiple aspects of the simulator code (e.g. Interest Lists being one) that would have to be overhauled – which is not something the Lab wants to undertake in the near future, although this does do preclude the potential for very large prim sizes at some point.
  • The cloud migration work has left the Lab with a certain amount of “technical deficit” – work  deferred in favour of simply getting systems and services transitioned to running on AWS infrastructure and hardware. This means that once the Uplift Project is completed, there will be a number of areas of simulator and back-end services that will need to be re-visited.
  • Potential projects to follow-on projects outside of the graphics performance updates mentioned above could include:
    • Updating the default terrain texture.
    • Further work on easing the new user experience,
    • Further work on viewer UI improvements.
    • Streaming changing an avatar’s appearance.
  • Bone uploads: the question was asked that why, given Bento provides 120+ bones, single avatar sub-mesh uploads are limited to a maximum of 110 bones. The reason is that 110 bones is the maximum some graphics systems can handle in a single upload before things go wrong.

Date of Next Meeting

2020 SUG meeting week #44: further uplift update

Boulder, September 2020 – blog post

The following notes were taken from the October 20th Simulator User Group meeting.

Server Updates and Cloud Uplift

  • There are no planned deployments to the regions running within the Lab’s co-lo facilities.
  • Starting on Tuesday, October 27th, all regions on the “AWS channel” will be updated with a simulator release  – version 551155 – that includes extended logging capabilities and a range of fixes, as indicated in the release notes.

For details of the deployment of main grid regions running via AWS cloud services, please see the following:

Issues

The grid has seen a range of issues coincident to the uplift work that may or may not be related to that work (as they are occurring across regions that are both now running on AWS and those still running at the Lab’s co-lo facility, although some appear to happen more frequently within AWS region.) The latest of these is object failing to rez when pulled from inventory. However, Mazidox Linden indicated that LL believe they have an handle on this issue, and the AWS region simulator version may have address it.

The most significant  issues (again which may or may not be related to the cloud uplift) being experienced comprise:

  • Group chat failures.
  • Teleport failures.
  • Rezzing issues.
  • Slow opening of legacy profiles.
  • Slow opening on the World Map (hopefully fixed with the 551155 release).

A further issue that has been noted, but yet to be confirmed, is that idle scripts appear to be taking up to to between 16% and 26% more script timing on regions running on AWS when compared to regions still operating out of the Labs co-lo facility.

Uplift Status

  • As per April Linden’s blog post, around 1,000 region on the main grid that are running on AWS.
    • Thus far, LL have only seen minor issues from their perspective, with those regions are running with “only minor hiccups”.
  • The deployment this week will see the total number of regions on AWS increase to between 2,000-2,500.
  • The updates required to allow the simulator to run on AWS systems represents – in Mazidox Linden’s words – “”the largest change to the simulator [software]  ever.”
  • Maxidox also confirmed there is an upcoming change that will see simulators running on AWS given a synthetic hostname rather than the viewer just giving the IP address of the AWS compute node in URI form.  The synthetic hostname will map to how LL describes simhosts internally.

SL Viewer

The Start of the week has seen the following viewer updates:

  • The Cachaça Maintenance RC viewer updated to version 6.4.11.551139 on Tuesday, October 27th, 2020.
  • The Legacy Profiles Viewer updated to version 6.4.11.550519 on Monday, October 26th, 2020.

The rest of the official viewers in the pipelines remain as follows:

  • Current release viewer version 6.4.10.549686, formerly the Mesh Uploader RC released October t and promoted on October 14 – No Change.
  • Project viewers:
    • Project Jelly project viewer (Jellydoll updates), version 6.4.10.549690, October 1.
    • Custom Key Mappings project viewer, version 6.4.5.544079, June 30.
    • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

In Brief

  • Some users (notably Firestorm users) are claiming that uplift is “breaking” region crossings, and that Firestorm should offer a higher default bandwidth allowance. However:
    •  The bandwidth setting applies to UDP only – the majority of data related to avatars, attachments and objects are handled via HTTP through the CDN, over which the UDP bandwidth has no influence.
    • While UDP messaging is involved in region crossings, changing the bandwidth default is viewed as potentially unwise whilst the uplift for is in progress, as it would largely be impossible to objectively tell if the change has improved matters.

2020 SUG meeting week #43: further uplift update

Bungenäs, Binemust – blog post

The following notes were taken from the October 20th Simulator User Group meeting.

Cloud Uplift Update

There are no formal simulator deployments for week #43. This is to make way for selected regions (both RC and Main channel) to be migrated to AWS services (the cloud). This means those regions that are transitioned will be restarted.

For details of the initial announce on the uplift, please refer to Lab expanding number of regions on AWS.

Speaking at the simulator User Group meeting on October 20th, Maxidox Linden provided the following update:

This morning we started our medium-size (at least by the size of a world) movement of regions to cloud based simhosts. Specifically we now have over 300 simulators running in the cloud. We’re looking at how quickly we can move more, as well.
For this round we went with a mixture of extremely high impact regions and extremely low impact “filler” regions, to stress various parts of our systems like the region allocation process and how we pack regions into a host.
In future rounds it’s likely we’ll be focusing on specific Release Candidate channels.
Basically: My *goal* (I’m not saying it’s what we’re doing, but I’m saying it’s what I’m trying to do) is to make this the worst it gets, and it only gets better from here.

– Mazidox Linden

Bullet Notes on Uplift

  • As of the October 20th SUG meeting, some 300 regions have been transitioned to the cloud, representing a mix of region in terms of use.
  • Performance issues have been warned about because there are a number of back-end systems that have yet to be uplifted (and presumably need the simulators in place before they can be), which may have some impact.
  • The hope is that some of these additional services will be transitioned when they can, rather than being “left until last”.
    • One of those earmarked for transition that has been causing some angst are the servers managing the KVP database for experiences.  There have already bee reports of degraded experience performance on AWS-hosted regions, so the hope is to get the KVP database moved, with “quiet a bit of effort” going into it.
  • Those who have had regions uplifted and feel they are suffering adversely from the move can request a roll back to being hosted in the Lab’s co-lo. However:
    • As all regions will be running on AWS services by year’s end, LL would rather region holders bear with any issues they may encounter whilst things are in a state of flux, and if they are encountering specific issues, to work with the Lab to try and identity and rectify matters.
    • Those who feel they need a roll-back to the lab’s co-lo facility should file a support ticket, providing a clear indication of why they would like their region(s) rolled back, and the issues they are encountering.
During the transition (as now) either the uplifted or data centre regions will have some penalty … we expect that will mostly disappear by the time we’re done

– Oz Linden on possible simulator (and other) performance impacts
during region migration

  • The Lab’s aim is to continue to make the transitioning of services from their co-lo to AWS services as invisible to users as possible (that is, you shouldn’t really be able to tell the difference between a service now running on AWS and when it was running via the Lab’s co-lo).

SL Viewer

There have been no updates to the current list of official viewers, leaving the pipelines as follows:

  • Current release viewer version 6.4.10.549686, formerly the Mesh Uploader RC released October t and promoted on October 14 – No Change.
  • Release channel cohorts:
    • Cachaça Maintenance RC viewer, version 6.4.10.549752, issued October 1.
  • Project viewers:
    • Project Jelly project viewer (Jellydoll updates), version 6.4.10.549690, October 1.
    • Custom Key Mappings project viewer, version 6.4.5.544079, June 30.
    • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
    • Legacy Profiles viewer, version 6.3.2.530836, September 17, 2019. Covers the re-integration of Viewer Profiles.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

In Brief

  • Group chat issues are being worked on, but is pending the uplift work.
  • The long-promised increase in sound loop support from 10 seconds to 30seconds, first promised a fair while ago now, is … pending the uplift work – although it is rumoured (and subject to confirmation) to be a Premium Plus benefit.

2020 TPVD meeting week #42: summary

Isolation’s Passengers – blog post

The following notes are taken from the TPV Developer meeting held on Friday, October 16th, 2020. These meetings are generally held every other week, unless otherwise noted in any given summary. The embedded video is provided to Pantera – my thanks to her for recording and providing it. Time stamps are included with the notes will open the video at the point(s) where a specific topic is discussed. Note these summaries are not intended to be a full reporting on all topics discussed, but focus on those items that are more directly user-facing.

Another exceptionally brief meeting, with some discussion in chat, so please refer to the video as well.

SL Viewer News

[0:15-2:18]

As per my CCUG meeting notes, the Mesh Uploader RC was promoted this week to de facto release status – refer to the release notes (link below) for details of the cosmetic and informational changes made to the uploader (which are in part based on contributions by Beq Janus of the Firestorm team). This leaves the current official viewer release list as follows:

  • Current release viewer version 6.4.10.549686, formerly the Mesh Uploader RC released October t and promoted on October 14 – NEW.
  • Release channel cohorts:
    • Cachaça Maintenance RC viewer, version 6.4.10.549752, issued October 1.
  • Project viewers:
    • Project Jelly project viewer (Jellydoll updates), version 6.4.10.549690, October 1.
    • Custom Key Mappings project viewer, version 6.4.5.544079, June 30.
    • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
    • Legacy Profiles viewer, version 6.3.2.530836, September 17, 2019. Covers the re-integration of Viewer Profiles.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

General Viewer Notes

  • As per this week’s CCUG meeting, Vir is hopeful that the Legacy Profiles project viewer may be able to move forward soon using a workaround to get it past the current blocker of needing some web-side updates.
  • The Custom Key Mappings project viewer is being considered from promotion to RC status with its next update.
  • There is an upcoming RC viewer featuring some UI changes – possibly including the update to give HUD items in inventory their own inventory icon, as discussed by Steeltoe Linden in the CCUG week #40 meeting. Thus might be appearing in the next week or so.
The proposal UI change to distinguish HUD items from other objects in inventory.Left: how things are now, with HUDs using the same cube icon as other objects. Right: the proposed new HUD cog icon.
  • [5:24-6:54] There have been some issues with viewer documentation not appearing on the relevant web pages, leading to concerns that such documentation might be “going away”.
    • The problem has actually been caused by the mechanism used to generate the information not having been updated for some time (and also not even using current information), so when it was migrated to AWS services, it “fell apart”.
    • Rather than try to fix the issue, the decision has been made so that once uplift has been completed, the mechanism will be replaced with an easier to maintain process. However, it does mean that for the time being, some viewer related documentation may not be available.

In Brief

  • [9:20-10:00] BUG-228679: Media-on-a-Prim doesn’t register keystrokes correctly on Mac – unfortunately, no progress to report.
  • [12:12-13:40] A couple of TPVs have been holding off making a formal release of the Environment Enhancement Project (EEP) updates beyond their current beta releases in case LL make further EEP update releases like the recent Love Me Render #4 viewer release. However, the advice from the Lab to these TPVs is:
    • To consider the high-priority EEP fixes as being “done”, as the Graphics team is now focused on priority long-term architecture work (including moving the viewer beyond OpenGL).
    • That future EEP fixes, when made, will form a part of the regular Maintenance RC viewer releases, rather than having any dedicated RC viewer updates.
  • [13:49-16:58] EEP performance issues:
    • Some TPVs have reported significant performance / FPS drop-off with EEP (around 15%). However, the Lab is not seeing anything close to this with the official viewer.
    • Unless there are detailed reports filed on the issue – using the official release viewer – that provide information on where / when / how the issue is being witnessed (e.g. the kind of test carried out, where in-world, the hardware configuration and viewer settings used, the results obtained, etc.), that allow LL to reproduce the problems, they do not see what can be done by way of investigation / fixes.
      • It was pointed out that comparable testing on the official viewer is difficult due to the limited amount of video memory available (512 MB compared to a minimum of 1 GB on most TPVs), which also affects performance, and so might skew results / how results are seen (e.g. are they EEP or are they due to texture load issues?).
      • Vir indicated that the amount of assigned texture memory is something the Lab will be looking at in the future.

2020 CCUG meeting week #42 summary

La Vie, September 2020 – blog post

The following notes were taken from my audio recording and chat log of the Content Creation User Group (CCUG) meeting held on Thursday, October 15th 2020 at 13:00 SLT. These meetings are chaired by Vir Linden, and agenda notes, meeting SLurl, etc, are are available on the Content Creation User Group wiki page.

There was a lot of general discussion about animation synchronisation, cloud uplift (the majority of which I’ve already reported in my SUG and TPVD meetings of late), etc., none of which should out as really reportable.

SL Viewers

Default Viewer

The Mesh Uploader viewer, version 6.4.10.549686, was promoted to de facto release status on Wednesday, October 14th. This viewer provides additional information on mesh models prior to upload, with contributions from Beq Janus from the Firestorm team and also revisions by the Lab’s viewer team. Cosmetic changes with this viewer include:

  • Upgrade to preview resolution to 1024×1024.
  • Scalable preview window.
  • Fixed display of colours in preview.
  • Adjustment of colours to better correlate to in world (yellow frame of mesh, blue tint physics).
  • Rearrange UI elements to give more space for the preview even when not scaled up.

Informational changes  in this viewer include two new boxes: cost breakdown and physics breakdown – these provide access to information that has always been available to the viewer form the upload costs update message but were not previously shown to the user.

Legacy Profiles

The Legacy Profiles viewer (version 6.3.2.530836 at the time of writing) has been stuck awaiting a web-side update, but as the web team are all focused on Uplift work, attempts are being made to work around the delay in order to get the viewer moving again.

Graphics

  • Work has resumed at looking towards a replacement for OpenGL, particularly as a result of Apple’s plans to deprecate all support for it on their systems.
    • Vulkan had been an option under examination, and additional logging was added to the viewer to help identify how many users are on systems capable of running it.
    • It now turns out that a rather high number of users (up to a 1/3 of all users) are running Second Life systems that do not have support for Vulkan (e.g. those 5+years old, and notably systems with Intel integrated graphics).
    • As a result other options are also being looked at, as is the potential for supporting people on older hardware separately to those using systems capable of running with more recent graphics APIs – although this is an option the Lab would approach with caution, due to the added complexities in maintenance.
  • In addition, the graphics team is looking to see if the viewer UI rendering can be divorced from other aspects of viewer rendering to see if any performance improvements can be obtained by keeping the two separate – currently, a lot of cycles are taken up in the drawing of things like UI panels and buttons.

Jellydoll / ARCTan

  • Vir has been working on some updates to the Project Jelly viewer (version 6.4.10.549690 at the time of writing).
  • This viewer is a series of performance improvements related to Jellydoll avatars, and the additional work should be appearing in an update to the project viewer soon. After that, Vir is hoping to completely re-focus on the ARCTan work, which will eventually fold-in the Jellydoll work as well.
  • Should he be able to switch back to ARCTan, the work will remain split between avatar rendering (to be tackled first), and then in-world object rendering to follow along at some point after.
  • The work on avatar rendering will initially be focused on the data gathering aspect and better refining the avatar complexity calculations, followed by UI work to deliver more meaningful and consistent information to users about the rendering impact of their own avatar and those around them.

Date of Next Meeting

  • Thursday, October 29th, 2020.

2020 Simulator User Group week #42 summary

Souls of Dreams, September 2020 – blog post

The following notes were taken from the October 13th Simulator User Group meeting.

Simulator Deployments

Please refer to the server deployment thread for news and updates.

  • There was no deployment to the grid on Tuesday, October 13th.
  • On Wednesday, October 14th there should be an RC deployment, server maintenance release 550305, comprising further logging updates for the cloud migration work,together with additional logging that it is hoped will assist with locating the causes of teleport time-outs that have been occurring.

SL Viewer

There have been no updates to the current list of official viewers, leaving the pipelines as follows:

  • Current release viewer :Love Me Render #4 (EEP fixes), version 6.4.9.549455, released September 24, promoted September 28 – No Change.
  • Release channel cohorts:
    • Cachaça Maintenance RC viewer, version 6.4.10.549752, issued October 1.
    • Mesh uploader RC viewer, version 6.4.10.549686, October 1.
  • Project viewers:
    • Project Jelly project viewer (Jellydoll updates), version 6.4.10.549690, October 1.
    • Custom Key Mappings project viewer, version 6.4.5.544079, June 30.
    • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
    • Legacy Profiles viewer, version 6.3.2.530836, September 17, 2019. Covers the re-integration of Viewer Profiles.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

Cloud Uplift

  • While the Lab hasn’t as yet got to uplifting the “big three” simulator release candidate (RC) channels – BlueSteel,  LeTigre and Magnum – it was confirmed that as these are eventually migrated to AWS services, all things being equal, updates will continue to be deployed to them on Wednesdays.
  • People are reporting few or no region crossing issues when moving between adjacent regions that are running within the Lab’s co-lo facility and those now running on AWS.
  • The London City regions obtained permission from LL to confirm they are one of the private estates that has been assisting the Lab with cloud migration – see London City Uplifted.
  • Scripters confused about obtaining host names via scripted means as a result of the uplift work should refer to this forum thread.

Group Chat Issues

There is some confusion around the problems of group chat failures (in particular, although IMs can all suffer). these are routed in a September 1st Grid Status report that indicated the cause of the problem was known. Responding to requests made during the meeting for specifics on what the Lab had found, Oz Linden replied:

Sorry if that text got your hopes up. … We understand that that problem has a lot of impact on some people, which is why it’s on the status board. We’ve had some difficulty with getting the time to attack it given the demands of uplift. We have freed up some time, though, so we should start making progress on it soon.

As the current bout of group chat failures seems to be in part linked to teleports, there is a hope that the additional teleport-related logging being deployed to one (or more) of the RC channels on Wednesday, October 14th might provide some information on why this is the case.

In Brief

  • Some running services with external management to SL are still experiencing HTTP 499 connection loss errors. The request is that while the issue can be intermittent and irregular, if those experiencing could file a Jira specifying time, location, and object ID when they see the issue, it would be helpful.