2020 Simulator User Group week #2 summary

Last Dove, November 2019 – blog post

Simulator Deployments

At the time of writing. the server deployment thread appears at odds with reality:

  • The thread refers to the SLS Main channel being on simulator release 533895. However:
    • Following restarts, it appears that simulators on the Main channel (and some on RC channels) have remained on 2019-12-04T20:29:26.533447, which according to previous deployment threads, was the last formal deployment to the Main channel.
    • According to the release note page, 533895 was deployed on December 19th, but has channel names are now obfuscated, it is not clear which RC received the update.
  • The thread refer to restarts on the RC channels on Wednesday, January 8th, some RC servers should be restarted, but again without any deployment – although the 533895 version number is given for some when it should perhaps be 53447.

Simulator Issues

Restart Issues

The Tuesday, January 7th restarts were repeated a number of times, for reasons Simon Linden explained:

So we haven’t had any server updates since mid-December. Today we restarted a bunch because that can help performance, and in the middle of that we uncovered a bug that’s been lurking there for months, so it got a bit exciting.

Mazidox Linden added that – as of the time of writing this update – further restarts could not be ruled out.

The bug itself was related to at least one directory required by the Mono compiler not being created correctly, preventing scripts from being saved. It is apparently a bug that has been around for some time, but only surfaced as a result of changes to the way simulator restarts are run. Ironically, the changes were intended to make simulator restarts faster and smoother

Holiday Issues

Numerous region holders reported significant performance issues over the holiday period. According to Grumpity Linden, part of the issue was a failure with an automated tool, itself something of a workaround for dealing with simulator performance issues:

Hi all! I was really hoping my first post of the new year would be more jolly, but here we are.  Happy New Year, though! 

For a couple of years now, we’ve had automated tools, aptly named “Grid Poking Bot” (GPB for short) responsible for doing region restarts, and this has been working quite well – most of the time. Very unfortunately, there was a problem with the GPB over the holidays, and due to a combination of events, it took us much too long to notice – and we finally caught it in part thanks to this very forum thread and a certain vigilant “Spray Can”.  We’re now actively pursuing the least disruptive ways to address this problem as quickly as possible.  We’ll have a more detailed post-mortem blog in a couple of days as well.  

We’re very sorry about souring your holidays.  

It’s possible there is more than one issue causing problems, and investigations are still in progress, including the distribution of poorly-performing regions (e.g. are they on their own on the grid, do they have neighbours) as well as looking at distribution by server.

SL Viewer

There have been no viewer updates to mark the start of the week, leaving the current pipelines as follows:

  • Current Release version 6.3.5.533275, formerly the Wassail RC viewer, dated December 4th, promoted December 12th – No change.
  • Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
    • Xanté Maintenance RC viewer, version 6.3.6.533748, December 19th.
    • Love Me Render RC viewer, version 6.3.5.533347, December 5th.
    • EEP RC viewer, version 6.4.0.532771, November 20th.
  • Project viewers:
    • Copy / Paste viewer, version 6.3.5.533365, December 9th.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22nd.
    • Legacy Profiles viewer, version 6.3.2.530836, September 17th. Covers the re-integration of Viewer Profiles.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16th.

Destination Guide

Not strictly part of the engineering team’s remit, but it has been reported the Destination Guide is giving errors when trying to submit new destinations. This has been seen by the Web Team, and is apparently being looked at.

2019 Content Creation User Group week #51 summary

Last Dove, November 2019 – blog post

The following notes are taken from my audio recording of the Content Creation User Group (CCUG) meeting held on Thursday, December 19th 2019 at 13:00 SLT. These meetings are chaired by Vir Linden, and agenda notes, meeting SLurl, etc, are available on the Content Creation User Group wiki page.

The majority of this meeting was a generic conversation of ideas such as moving Second Life to support PBR, what might be done to improve Pathfinding, etc., none of which are on the road map for Second Life at present; as such these notes keep the the current projects that are in progress at the Lab.

SL Viewer

A new Maintenance viewer, code named Xanté, was released on Thursday, December 19th. Version 6.3.6.533748 contains around 30 fixes for reported issues and bugs. All other viewer remain as per my Current Viewer Release List.

With regards to viewers:

  • The Lab’s focus has been on transitioning their Bitbucket viewer build repositories from Mercurial to Git – see my week #50 TPVD meeting notes for more.
  • As well as the current pipelines of viewers, work is also in hand to ensure the viewer is ready to manage Name Changes when that capability is deployed in early 2020.

Environment Enhancement Project

Project Summary

A set of environmental enhancements (e.g. the sky, sun, moon, clouds, and water settings) to be set region or parcel level, with support for up to 7 days per cycle and sky environments set by altitude. It uses a new set of inventory assets (Sky, Water, Day), and includes the ability to use custom Sun, Moon and cloud textures. The assets can be stored in inventory and traded through the Marketplace / exchanged with others, and can additionally be used in experiences.

Due to performance issues, the initial implementation of EEP will now likely not include certain atmospherics such as crepuscular rays (“God rays”).

Resources

Current Status

  • Bug fixing continues, notably around alpha rendering issues.
  • The hope is that of the remaining issues, some my be related, and so solving one will help to solve others of a similar nature.

ARCTan

Project Summary

An attempt to re-evaluate object and avatar rendering costs to make them more reflective of the actual impact of rendering both. The overall aim is to try to correct some inherent negative incentives for creating optimised content (e.g. with regards to generating LOD models with mesh), and to update the calculations to reflect current resource constraints, rather than basing them on outdated constraints (e.g. graphics systems, network capabilities, etc).

Current Status

  • Vir is working on getting things to a state where he can do so practical testing over the holiday period to ensure the relevant data is being collected. This is dependent on whether he has the time to confirm the internal version of the viewer is logging everything it needs to be logging.
  • The work is still very much focused on the data collection aspect, rather than doing anything with the data that is gathered.
    • The kind of data being gathered includes: what are the graphics and geometric properties of the objects in a scene, what rendering settings are being used, poly count for different LODs with a model, what are the graphics properties in use (materials, texture + texture size, etc.), plus the time required to generate a frame successfully given the work required to render the scene.
  • Once the data has been gathered, the idea is to run the viewer on multiple hardware configurations (GPU, CPU, etc.), and gather data on the the impacts of changes those various properties.
  • The aim is to get a more accurate feel for how performance is impacted, and how significantly changes impact performance (e.g. what’s the impact of enabling Full Bright compared to enabling materials? Which is genuinely better: properly optimised mesh or plain faces with materials or a combination of low-resolution mesh + materials?
  • As well as allowing the complexity calculations for avatar attachments and in-world objects to be better refined, the data gathered might, further down the line in the project, enable LL to make plausible forecasts of what might be seen by way of performance improvements in relation to suggested constraints being put on objects as a part of the creation process.
  • Textures are still proving a problem in terms of measuring impact (e.g. is it more a total threshold limit being hit, rather than the number of textures used within an individual object?).
  • Anther limiting aspect is the number of different bottlenecks users can experience quite outside of the Lab’s control (e.g. their network connection, what else is going on across that connection at the same time, etc)., and bottlenecks within individual systems that can vary.
  • One attempt to improve things that has been made in Firestorm is for the matrix calculations for worn mesh to be cached the the bones to which the mesh has been rigged hasn’t moved between frames. This can save up to 7 sets of calculations for a mesh with 8 faces that the viewer may not actually need to make. This may be contributed to LL for evaluation.

** The next Content Creation User Group Meeting should be on Thursday, January 9th, 2020, but check the wiki page for confirmation **

 

2019 Simulator User Group week #51 summary

:oxygen:, October 2019 – blog post

No major news again this week; the meeting was given over to live music and way too much gesturbating to follow conversations.

Simulator Deployments

Please refer to the server deployment thread for news and updates:

  • There was no deployment on Tuesday, December 17th, leaving the SLS Main channel and a portion of the RC channel servers on simulator version 2019-12-04T20:29:26.533447, originally deployed on Thursday, December 5th, and comprising:
    • A build of release 2019-11-15T21:13:13.532828 using new build technology.
    • Addresses some cases of scripts erroneously stopping.
    • Fixes a crash.
  • On Wednesday, December 18th, some RC servers should be updated with server release 2019-12-06T21:03:45.533558, comprising internal fixes. This is a re-deployment of the release originally made on December 9th, 2019, containing “a difference under the hood”.

No Change Window

These deployments are the last scheduled for 2020. Due to the end of year No Change Window (December 20th through to January 2nd, 2020, the next simulator updates will not likely be deployed before Tuesday, January 7th, 2020.

SL Viewer

There have been no viewer updates to mark the start of the week, leaving the current pipelines as follows:

  • Current Release version 6.3.5.533275, formerly the Wassail RC viewer, dated December 4th, promoted December 1th2 – NEW.
  • Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
  • Project viewers:
    • Copy / Paste viewer, version 6.3.5.533365, December 9th.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22nd.
    • Legacy Profiles viewer, version 6.3.2.530836, September 17th. Covers the re-integration of Viewer Profiles.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16th.

 

2019 TPVD meetings week #50 summary

Nostalgia Falls, October 2019 – blog post

The following notes are taken from the TPV Developer meeting held on December 13th, 2019. A video of the meeting is embedded below, my thanks to Pantera for recording and providing it. As always:

  • Time stamps are given with links that will open the video at the appropriate point in a separate browser tab for reference.
  • Core points of the meeting are listed below. Other subjects of lesser import may have been discussed, please refer to the video.

SL Viewer News

Note: the comments below also include updates given at the Content Creation User Group meeting of December 12th.

[4:29-6:52]

  • Copy / Paste viewer project vewer updated to version 6.3.5.533365 on December 9th.
  • The Wassail RC viewer, version 6.3.5.533275, was promoted to de-facto release status on Thursday, December 12th. This was likely the last viewer promotion to release status for 2019.

The rest of the current viewer pipelines remain as follows:

  • Release channel cohorts:
    • EEP RC viewer, version 6.4.0.532771, dated  November 20th.
  • Project viewers:
    • Copy / Paste viewer, version 6.3.5.533365, dated December 9th.
    • Legacy Profiles viewer, version 6.3.2.530836, dated September 17th. Covers the re-integration of Viewer Profiles.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.530473, dated September 11th.
    • 360 Snapshot project viewer, version 6.2.4.529111, dated July 16th.

General Viewer Notes

  • It is thought the Legacy Profiles viewer could be next in line for promotion to defacto release status. However, this viewer is still awaiting the release of an update that integrates the profile feed into the viewer.
  • There are reportedly multiple issues with the Copy / Paste project viewer, some of which are UI related – no specific bug report reference to relay here at present, though.
  • In addition, and awaiting surfacing / further work internally, are further Maintenance viewers and improvements to the mesh uploader.

Mercurial to Github Migration

[0:05-2:51]

Bitbucket, used to manage viewer repositories) will be sunsetting support for Mercurial in early 2020. Because of this, Linden Lab has now started migrating their viewer repositories from Mercurial to Git on Bitbucket.

  • This means the pace of viewer updates might slow down while this work is in progress.
  • A document outlining the steps in migration has been provided specifically for TPVs to allow them to keep in sync with how LL main the viewer code and its branches. See: Viewer Repository Migration (Google Docs).
    • This is not the only way for TPVs to manage their own repositories, but it is the one that LL recommend for those wishing to reflect how LL manages things.
  • [36:48-41:15] Thus far, the Lab have not run into problems in making the migration. It was also pointed out that:
    • There are a number of cheat sheets out on the web for TPVs and self-compilers to make / understand the switch from Mercurial to Git.
    • There will be fewer repositories used by LL for the viewer, as Git provides better support for branching and better matches the LL build process.

Viewer Updates for 2020

[24:27-27:05]

  • Two viewer updates that will be surfacing in 2020 are Name Changes and Premium Plus.
    • With regards to Name Changes, TPVs many what to consider migrating people’s settings and chat histories when they change their names, as all of this information is stored in a local folder  / directory based on the avatar name.
    • This could take the form of switching to using the avatar key UUID, although this isn’t necessarily user-friendly when looking at log files, etc.

Deprecating Windows 7 Support

[27:54-31:54]

  • Windows 7 officially reaches its end of life on January 14th, 2020. After this date, patches and security updates, etc., will no longer be provided, and the company is unlikely to provide any support.
  • As a result of this, it is very Linden Lab will cease officially supporting Windows 7 after that date. While users will still be able to log-in to Second Life on PCs running Windows 7:
    • They will not receive assistance from LL support should they encounter problems.
    • Viewer updates from the Lab will no longer be tested against Windows 7 prior to release.
    • Bugs reported against Windows 7 that cannot be reproduced using Windows 8 or Windows 10 will not be investigated.
    • Around 12% of SL users are running Windows 7, and this applies to them.
  • Obviously, given Microsoft is ending Windows 7, the recommendation is for users to upgrade to Windows 10.

Linden Lab No Change Window

[2:52-4:00]

The 2019 end-of year No Change window is from Friday, December 20th 2019 through Thursday, January 2nd 2020. During this time there will be no simulator or official viewer updates, and TPVs are asked also not to make releases during this period in case they accidentally cause support or operational issues for LL.

In Brief

  • [7:15-9:02] The updates to the viewer build process (to support Visual Studio 2017 and Xcode 10.3) are in a bug hunting mode at the moment, and so will see light of day in early 2020.
    • It is possible that work on updating to VS 2019 might commence later in 2020.
  • [9:07-11:23] One of the projects awaiting the deployment of the new build process is updates to the Chrome Embedded Framework (CEF) responsible for handling media in the viewer. It is hoped that when these updates can be provided within the viewer they will help with media support playback and eliminate at least some problems.
    • These updates should help with MP4 playback as more codecs are being added – although again, MP4 is a container with flavours rather than a specific format, so all flavours may not play back equally.
  • [12:08-19:00] Discussion on Bugsplat, the crash reporting tool. This is probably outside the interest of most users, so is not listed below – please refer to the video.
  • [22:40-24:00] Discussion on a Firestorm OpenSim release. Again, this probably not relevant to the majority of readers of the blog (and I’ll be covering it separately as and when it appears anyway), so again – please refer to the video.
  • [32:04-32:40] BUG-227179 – All off-line inventory offers from scripted objects are STILL lost: this bug is still awaiting work.

 

2019 Content Creation User Group week #50 summary

Shadowlands Retreat, October 2019 – blog post

The following notes are taken from my audio recording of the Content Creation User Group (CCUG) meeting held on Thursday, December 12th 2019 at 13:00 SLT. These meetings are chaired by Vir Linden, and agenda notes, meeting SLurl, etc, are available on the Content Creation User Group wiki page.

Environment Enhancement Project

Project Summary

A set of environmental enhancements (e.g. the sky, sun, moon, clouds, and water settings) to be set region or parcel level, with support for up to 7 days per cycle and sky environments set by altitude. It uses a new set of inventory assets (Sky, Water, Day), and includes the ability to use custom Sun, Moon and cloud textures. The assets can be stored in inventory and traded through the Marketplace / exchanged with others, and can additionally be used in experiences.

Due to performance issues, the initial implementation of EEP will now likely not include certain atmospherics such as crepuscular rays (“God rays”).

Resources

Current Status

  • Bug fixing continues – the estimate is around 18 or so bugs the Lab would like to resolve prior to any potential release.

Project Muscadine

Project Summary

Currently: offering the means to change an Animesh size parameters via LSL.

Current Status

  • Still on hold due to the focus on ARCTan.
  • There are still requests to allow attachments on Animesh items.
    • This is something Vir hopes to look at in detail later in Muscadine.
    • It may require attachments to be handled differently to how they are managed with avatars.

ARCTan

Project Summary

An attempt to re-evaluate object and avatar rendering costs to make them more reflective of the actual impact of rendering both. The overall aim is to try to correct some inherent negative incentives for creating optimised content (e.g. with regards to generating LOD models with mesh), and to update the calculations to reflect current resource constraints, rather than basing them on outdated constraints (e.g. graphics systems, network capabilities, etc).

Current Status

  • Vir is working on getting things to a state where he can do so practical testing over the holiday period to ensure the relevant data is being collected. This is dependent on whether he has the time to confirm the internal version of the viewer is logging everything it needs to be logging.
  • The work is still very much focused on the data collection aspect, rather than doing anything with the data that is gathered.
  • It is not currently clear whether the ARCTan work will appear in a dedicated project viewer or will form a part of a Maintenance viewer update.

Texture Caching and Loading

  • LL is working on a viewer intended to improve texture loading and texture caching (the latter as part of a general overall of how the viewer caches data).
  • This will hopefully include a rethinking of the order in which textures are loaded (e.g. objects  / faces that all use the same texture may all have that texture loaded together/in sequence, rather than the texture having to be re-loaded each time it is encountered).
  • The improvements should see textures load faster in general. In particular, there is a re-examination of some of the “optimisation” work previously done with textures, as this might actually now be slowing things down, so the hope is the new viewer will streamline how textures are handled and loaded in general, so bringing about improvements.
    • An example of this is switching the viewer from downloading a texture (or grabbing it from cache) and rendering it incrementally to just letting it grab the entire texture, particularly now that most broadband connections will allow this without it becoming a significant bottleneck.
    • This will allow a significant reduction in the amount of checking and re-checking the viewer has to carry out when obtaining and loading textures, which can have an impact.
  • Hopefully, the viewer will also improve the texture load order (e.g. those textures nearest to you or filling your immediate field of view, such as a vendor board on a wall, will be loaded and rendered first, rather than waiting for other textures loading first).
  • There is currently no date on when this viewer might surface for public use.

2019 Simulator User Group week #50 summary

The Boho Refuge, October 2019 – blog post

No major news again this week.

Simulator Deployments

Please refer to the server deployment thread for news and updates:

  • On Tuesday, December 10th, the SLS Main channel was updated to simulator version 2019-12-04T20:29:26.533447, originally deployed on Thursday, December 5th, and comprising:
    • A build of release 2019-11-15T21:13:13.532828 using new build technology.
    • Addresses some cases of scripts erroneously stopping.
    • Fixes a crash.
  • On Wednesday, December 11th, there should be an RC deployment to the BlueSteel RC. Simulator version 2019-12-06T21:03:45.533558 comprises internal fixes.

Deployment Notes

  • The Tuesday deployment started a little later than usual.
  • Further, as some of the processes managing deployments are being modified to increase stability, some deployments may run longer than usual. Once this work has finished, it is hoped deployments will be faster and more stable.

SL Viewer

At the end of week #49, the following viewers were updated:

  • On December 4th, the Maintenance RC viewer updated to version 6.3.5.533275.
  • On December 5th, the Love Me Render RC viewer updated to version 6.3.5.533347.

On Monday, December 9th the Copy / Paste project viewer updated to version 6.3.5.533365. The rest of the official viewer pipelines are as follows:

  • Current Release version 6.3.4.532299, formerly the Ordered Shutdown RC viewer, dated November 4th, – No change.
  • Release channel cohorts:
  • Project viewers:
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22nd.
    • Legacy Profiles viewer, version 6.3.2.530836, September 17th. Covers the re-integration of Viewer Profiles.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16th.