Tag: Server Updates

The following notes were taken from the Tuesday, October 26th, 2021 Simulator User Group (SUG) meeting. The meeting was recorded by Pantera Północy, and the video is embedded at the end of this summary. Note this summary focuses on the key points of the meeting; where there is something to report, the video should be referred to should full details of the meeting wish to be reviewed.

Server Deployments

• Tuesday, October 26th saw the simhosts on the SLS main channel restarted without any update.
• Wednesday, October 27th should see the deployment of  simulator version that includes a revised implementation of PRIM_PROJECTION.
  • For the time being, it will be write only, meaning it can used in llSetPrimitiveParams but not in llGetPP. The associated wiki documentation has yet to be updated.
  • This update will also include BUG-231158 Allow llGetNotecardLine to return more than 255 bytes (to a maximum of 1023 characters).

SL Viewer

There have been no updates to the current crop on official viewers to mark the start of the week, leaving the pipelines as:

• Release viewer: version version 6.4.23.564172, formerly the Apple Notarisation Fix RC viewer, issued September 24 and promoted October 15 – see notes below.
• Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
  • 360 Snapshot RC viewer, version 6.5.0.564863, issued October 21.
  • Maintenance RC viewer updated to version 6.5.0.564805, on October 20.
  • Simplified Cache RC viewer, version 6.4.23.562623, dated September 17, issued September 20.
• Project viewers:
  • Performance Improvements project viewer, version 6.4.23.564530, dated October 12.
  • Performance Floater project viewer, version 6.4.23.562625, issued September 2.
  • Mesh Optimizer project viewer, version 6.4.23.562614, issued September 1.
  • Legacy Profiles viewer, version 6.4.11.550519, dated October 26, 2020.
  • Copy / Paste viewer, version 6.3.5.533365, dated December 9, 2019.

Apple Notarisation Viewer Issue

With the release of the Apple Notarisation Viewer there were updates to many of the viewer’s third party libraries, and some of these updates have be found to cause issues related to playback of certain media types in-world including, but not possibly limited to MP3s and MP4s. LL are working towards a fix, but in the meantime, workaround are offered:

• Windows:
  • Uninstall the Second Life viewer the usual way.
  • Navigate to your Program Files folder (Win 64-bit) or Program Files (x86) (Win 32-bit); locate and delete the “SecondLifeViewer” folder.
  • Download and install the Simplified Cache viewer (the previous release viewer).
• Apple Mac:
  • Remove the Apple Notarisation viewer from your system.
  • Download and install the Simplified Cache viewer (the previous release viewer).
  • Read the instructions on this page to work through any occurrences of unwanted notarisation warnings.

In Brief

• With the on-going work to analyse viewer performance using the Tracy debugger / system analyser (see my TPV Developer summaries for more), the simulator team are looking to try and leverage that work on the simulator side, once the current tools upgrade on the server-side has been completed.
• There was a 10-minutes discussion around animated particles / revising the particle systems, but there is nothing on the card for immediate work.
• The upcoming simhost operating system upgrade (also to follow the completion of the tools upgrade) was also discussed. Essentially there is a lot of work to be done, and the potential for regressions or other issues to occur. However, LL are planning things as carefully as possible; no overall decision as to how it will be deployed to the main grid come time for it to do so – mainly because the work has yet to start.

The following notes were taken from the Tuesday, October 19th, 2021 Simulator User Group (SUG) meeting. The meeting was recorded by Pantera Północy, and the video is embedded at the end of this summary. Note this summary focuses on the key points of the meeting; where there is something to report, the video should be referred to should full details of the meeting wish to be reviewed.

Server Deployments

See the current server deployment thread for updates

• Tuesday, October 19th saw the simhosts on the SLS main channel restarted without any update. for some this may have seen the server release number increment to 564759. However, there were some issues, so some simulators on the channel were refreshed with release 564394.
  • LL is working to get to a state where version numbers only increment when there is a deployment and rolling restart, rather than incrementing as a result of a rolling restart alone.
• Wednesday, October 20th should see the deployment of  simulator version 564610. This includes the following feature requests:
  • SL-15994 Add “force” option to estate_environment command (non-public Jira).
  • BUG-231158 Allow llGetNotecardLine to return more than 255 bytes.
  • BUG-229639 [FEATURE REQUEST] PRIM_PROJECTION.

Operating System Upgrade

As I’ve recently noted, following the current server tools update, the Lab plans to update the underpinning operating system. This work is now on the horizon. Or, to put in in the words of Mazidox Linden:

Do you all remember back in the Before Times when we were still in the colo and offices were still a thing and we did simulator operating system upgrades? Well, we’re in the cloud, I’m still working from a cramped desk in a corner of my guest room, but guess what we’re doing!

I know an OS upgrade is not the most attractive thing nor fun to hear about, so I’m willing to offer you a carrot. We’ve seen at least 10% more scripts running per frame on the same hardware with the same starting state for a simulator. [SO] the sooner we get this os upgrade tested the sooner it gets out the door and into your hands.

Those wishing to test their scripts, etc., on the new OS version can currently do so on the following regions on Aditi (the beta grid): Cale, Gothlauth, Jigglypuff, Leafeon, London City, London City Gateway, Mauve, Moonberry, Riccione, Sapas, Smithereens, Snark, Testylvania Sandbox, and Woods of Heaven.

Mazidox also stated:

If you have a region on Agni that has some significant script load, or unusual characteristics like lots of physics utilization, pathfinding… Regions that really stress aspects of the server without people being on them, please feel free to IM me here on Agni and ask for the region to be copied to Aditi on a first come first serve basis.

SL Viewer

There have been no updates to the current crop on official viewers to mark the start of the week, leaving the pipelines as:

• Release viewer: version version 6.4.23.564172, formerly the Apple Notarisation Fix RC viewer, issued September 24 and promoted October 15.
• Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
  • Maintenance RC viewer updated to version 6.4.23.564063, on September 21.
  • Simplified Cache RC viewer, version 6.4.23.562623, dated September 17, issued September 20.
• Project viewers:
  • Performance Improvements project viewer, version 6.4.23.564530, dated October 12.
  • 360 Snapshot project viewer, version 6.4.23.563579, issued September 3.
  • Performance Floater project viewer, version 6.4.23.562625, issued September 2.
  • Mesh Optimizer project viewer, version 6.4.23.562614, issued September 1.
  • Legacy Profiles viewer, version 6.4.11.550519, dated October 26, 2020.
  • Copy / Paste viewer, version 6.3.5.533365, dated December 9, 2019.

In Brief

• There has been a lot of discussion about the viewer’s bandwidth slider on the forums again recently, which has also touched on default cache sizes
  • As noted in that thread, the bandwidth slider in particular is less useful nowadays than once was the case, prompting the question on whether or not it should be retained within the viewer.
  • During the meeting, the question was asked if the simulators are liable to become defensive if the setting were to be fixed (presumably relatively high) within a viewer like Firestorm, which could provoke “microburst contention” due to the volume of UDP message requests hitting the servers.  The suggestion has been to go ahead and try, which is easier said than done inasmuch as increasing the default is easy – rolling it back perhaps not so (given it would require a further update, which people might opt to ignore unless part of a larger release – which then might not appear for several months after the original change).
  • In terms of cache, that is also a problem, as the limits do tend to be low (based on older computer specs), plus the viewer tends to have multiple and distinct caching structures, none of which are particularly efficient – although the Lab is trying to address some of them (hence the Simplified Cache RC viewer).
• There is a very short discussion on Voice options, given Vivox will be ending the Vivox 4 API. As this in more a viewer question, no conclusions could be drawn.

The following notes were taken from the Tuesday, October 12th, 2021 Simulator User Group (SUG) meeting. The meeting was recorded by Pantera Północy, and the video is embedded at the end of this summary. Note this summary focuses on the key points of the meeting; where there is something to report, the video should be referred to should full details of the meeting wish to be reviewed.

Server Deployments

The planned deployment for this week has been postponed due to late-breaking issues, and so will not be deployed for a week. However, regions that have not been restarted in 10 or more days will be restarted. See the (“lack of”) deploy plans notes for more.

SL Viewer

There have been no updates to the current crop on official viewers to mark the start of the week, leaving the pipelines as:

• Release viewer: version version 6.4.22.561752, formerly the CEF Update RC viewer, issued July 24 and promoted August 10.
• Release channel cohorts:
  • Apple Notarisation Fix RC viewer, version 6.4.23.564172, issued September 24 – this should remove the warning messages which are currently popping up.
  • Maintenance RC viewer updated to version 6.4.23.564063, on September 21.
  • Simplified Cache RC viewer, version 6.4.23.562623, dated September 17, issued September 20.
• Project viewers:
  • 360 Snapshot project viewer, version 6.4.23.563579, issued September 3.
  • Performance Floater project viewer, version 6.4.23.562625, issued September 2.
  • Mesh Optimizer project viewer, version 6.4.23.562614, issued September 1.
  • Legacy Profiles viewer, version 6.4.11.550519, dated October 26, 2020.
  • Copy / Paste viewer, version 6.3.5.533365, dated December 9, 2019.

Region Crossings

• The subject of region crossings came up again, specifically in reference to multiple sequential crossings via vehicle, and the problems that can occur (passing from one region to the next, and then on to the next before all relevant data between the first two has been fully transferred). There should be code in place to handle this, but the Lab acknowledges there may be cases there this doesn’t work well.
• Rider Linden acknowledged that more work is required on the entire physical region crossing protocol, but that, “it may involve starting from scratch and rethinking how the entire protocol works. That’s going to be a big job.”
  • The question here is, how best to delay interpolation to ensure all information on a vehicle and its passengers is received by one region so all of it can be passed on to the next. User Animats submitted a code contribution to Firestorm in 2018 (which has since been further revised) to help with this, but it is not perfect. The problem here is, too much delay  – say more than around 1/2 a second – is noticeable and can impact immersion; a second problem is, what may help ease some types of region crossing may make others more noticeable.
• Another problem is that physical / vehicle region crossings are such that there is little opportunity for any kind of “pre-transfer” of vehicle and avatar data, until the vehicle is on top of / crossing the actual region boundary (the 1m boundary). This is because there is no guarantee that a vehicle will turn away from a crossing without actually moving between regions – so simulator time (on both sides) is taken up in handling the pre-transfer without no point to the exercise.

Another option (again, to stress **as an example**) would be to always have up to date data on all adjacent regions. But that would cost us real money. How do we recoup that increased cost in a way that is fair to the people who actually make use of the increased data availability? I’m just trying to give examples of why none of these solutions are “free” or “simple”.

– Mazidox Linden on region crossings + potential solutions

• The suggestion was made to run regions on virtual machines, such that adjacent regions are on the “same machine”, removing the need for transferring data between different physical simhosts. The problems here are:
  • a) The number of “adjacent” regions can be huge (e.g. Blake Sea and the surrounding private estates + Mainland continents, and the Mainland continents as a whole).
  • Even if broken down into more manageable groups of regions all on the same hardware, but this again doesn’t entirely eliminate problems, an will result in some region crossings appearing smoother, and others (where they remain between different hardware) appearing “worse” by comparison.
  • Also, if the hardware fails fails running a batch of virtual machines, that’s potentially a larger number of regions that go with it than is currently the case. And while hot shadowing is possible so that if a server does fail, it’s shadow can automatically take over, that’s doubling overall hardware requirements + associated costs, which would have to be met somehow.
• As it is, the move to AWS has seen an overall improvement in region crossings, primarily because the hardware and infrastructure available via AWS is a lost more recent, and so more powerful (hardware) and faster (network) than the Lab’s old infrastructure.
• Whilst not just related to region crossings, an experiment on the Lab’s to-do list is to try to group clusters of regions by hardware, something that has not been tried in some time.

In Brief

• A major focus on the server  / simulator side of Second Life remains the work in updating the tools the Lab has at its disposal, which is to be followed by / overlap with a major operating system upgrade (not to 64-bit, which is viewed as a “humongous” piece of work, but one that will eventually need to be addressed, depending on the platform’s continued longevity).
• There is a brief discussion at the end of the meeting concerning mesh decimation, avatar meshes, rendering, and possible improvement, much of which is a subject of CCUG meeting discussions.

The following notes were taken from the Tuesday, October 5th, 2021 Simulator User Group (SUG) meeting. The meeting was recorded by Pantera Północy, and the video is embedded at the end of this summary. Note this summary focuses on the key points of the meeting; where there is something to report, the video should be referred to should full details of the meeting wish to be reviewed.

Server Deployments

There are no planned deployments or restarts this week, due to the rolling restarts made on Friday, October 1st to correct an expiration issue related to Let’s Encrypt certificates that resulted in failures for in-world scripts making HTTP calls to websites secured with a Let’s Encrypt certificates.

SL Viewer

There have been no updates to the current crop on official viewers to mark the start of the week, leaving the pipelines as:

• Release viewer: version version 6.4.22.561752, formerly the CEF Update RC viewer, issued July 24 and promoted August 10.
• Release channel cohorts:
  • Apple Notarisation Fix RC viewer, version 6.4.23.564172, issued September 24 – this should remove the warning messages which are currently popping up.
  • Maintenance RC viewer updated to version 6.4.23.564063, on September 21.
  • Simplified Cache RC viewer, version 6.4.23.562623, dated September 17, issued September 20.
• Project viewers:
  • 360 Snapshot project viewer, version 6.4.23.563579, issued September 3.
  • Performance Floater project viewer, version 6.4.23.562625, issued September 2.
  • Mesh Optimizer project viewer, version 6.4.23.562614, issued September 1.
  • Legacy Profiles viewer, version 6.4.11.550519, dated October 26, 2020.
  • Copy / Paste viewer, version 6.3.5.533365, dated December 9, 2019.

In Brief

• The next maintenance update for the simulator will include a longer read for notecard lines (1024 bytes) that will be retroactive.
• PRIM_PROJECTOR is a new parameter for llSet/GetPrimitiveParams. Documentation is already on the wiki, but the simulator code isn’t ready to deploy for using testing on Aditi yet.
• It still has not been indicated when the Land team will flip the switch to ease the issue of some Mainland regions having darken default EEP settings.
• The simulator engineering team is “finishing up” some retooling, and will be moving on to starting to address performance improvements “in the next few months”.
• BUG-229205 “Re-enable PRIM_CAST_SHADOWS” – this has been seeing some discussion within the Lab, but no inform as to when it might be actioned.
• An interesting question was asked about using EEP to create an aurora effect that would be seen across the sky when used. The suggestion is to try using a texture in place of clouds, which would give a degree of motion (but whether this could mimic the “motion” of aurora streams would be interesting).
• Monday, October 4th saw a log-in update that “caused some issues for Firestorm”. The main fix is in place, but the issues is still being worked on for Aditi (the beta grid).

The following notes were taken from the Tuesday, September 28th, 2021 Simulator User Group (SUG) meeting. The meeting was recorded by Pantera Północy, and the video is embedded at the end of this summary. Note this summary focuses on the key points of the meeting; where there is something to report, the video should be referred to should full details of the meeting wish to be reviewed.

Server Deployments

See the deployment thread for updates.

• There are no planned deployments for the SLS Main channel.
• The RC channels will be subject to a rolling restart on Wednesday, September 29th. Due to technical constraints this, will update the simulator version number to 564195, however there are no changes in code or configuration from the currently deployed version 563375.

SL Viewer

There have been no updates to the current crop on official viewers to mark the start of the week, leaving the pipelines as:

• Release viewer: version version 6.4.22.561752, formerly the CEF Update RC viewer, issued July 24 and promoted August 10.
• Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
  • Apple Notification Fix RC viewer, version 6.4.23.564172, issued September 24 – this should remove the warning messages which are currently popping up.
  • Maintenance RC viewer updated to version 6.4.23.564063, on September 21.
  • Simplified Cache RC viewer, version 6.4.23.562623, dated September 17, issued September 20
• Project viewers:
  • 360 Snapshot project viewer, version 6.4.23.563579, issued September 3.
  • Performance Floater project viewer, version 6.4.23.562625, issued September 2.
  • Mesh Optimizer project viewer, version 6.4.23.562614, issued September 1.
  • Legacy Profiles viewer, version 6.4.11.550519, dated October 26.
  • Copy / Paste viewer, version 6.3.5.533365, dated December 9, 2019.

In Brief

• Some are reporting an uptick in 502 responses to outgoing HTTP requests that is described as “not common, but common enough” to cause concern, although the requests usually go through as expected on retry. The uptick appears to coincide with the updates to HTTP-out proxies. The usual Jiras on target URL, source regions/objects, owners, etc., have been requested from those experiencing these issues.
• Some are also reporting issues with texture loading taking longer of late, including textures that should be locally cached by the viewer. The simulator and messaging is being pointed to as a potential culprit, although a) this shouldn’t affect cached textures; b) aside from the initial object and texture information, textures (as with all assets) come via the CDN, not through the simulator.
  • It’s been suggested that changes to the Interest List in support of the 360º Capture project viewer could be the cause, and this hasn’t been ruled out.
  • Similar issues had been noted with Firestorm following its release. However these have tended to be in terms of initial viewer start-up at the start of a session (when VFS load seems to take abnormally long), but these do not appear to be related to this issue, as this appears to occur across all viewer flavours – official and TPV.
  • Again, Jira reports on the issue giving viewer details, locations and actual timing snapshots (using ALT-CTRL-3 to display the texture queue) would be useful to the Lab. Also tests using the TextureTest2 and MeshTest2 regions on Aditi (which are structured to test exactly this sort of thing) that can be reported would also be useful.
• BUG-231113 “PRIM_POINT_LIGHT sets the wrong colour (not a duplicate)” has been accepted (and pending more detailed investigation). It is not clear if the viewer is amending the script-supplied light parameters prior to sending them to the simulator, or whether the change is on the simulator side of things.
• The last part of the meeting comprised grumbling about blog reports apparently not being addressed (or possibly more correctly, not being addressed in a time frame those experiencing an issue – which may not be everyone or even a majority or users – want it to be addressed). In response, it was pointed out that there is a lot of working going on under-the-hood that is not user-visible, but nevertheless is of a high priority.

The following notes were taken from the Tuesday, September 21st, 2021 Simulator User Group (SUG) meeting. The meeting was recorded by Pantera Północy, and the video is embedded at the end of this summary. Note this summary focuses on the key points of the meeting; where there is something to report, the video should be referred to should full details of the meeting wish to be reviewed.

There is little to report, given the meeting was also another Solstice party.

Server Deployments

At the time of writing, the server deployment thread had not been published.

• All remaining simhosts were updated to simulator release  563375 on Tuesday, September 21st, 2021, the maintenance release containing changes related to llChar(), llUnsit() and extended internal logging.
• There are no planned RC deployments during the week.

HTTP-Out Proxy

Monty Linden deployed the new HTTP-out proxies in week #36. It wasn’t entirely glitch-free (but not as bad as the August attempt), with issues occurring in a part of the configuration that didn’t allow for immediate correction. Monty hopes that the lessons learnt with make future deployments smoother.

SL Viewer

• The Simplified Cache viewer updated to version 6.4.23.562623 on Friday, September 17th (issued Monday, September 20th).

The rest of the pipelines remain as:

• Release viewer: version version 6.4.22.561752, formerly the CEF Update RC viewer, issued July 24 and promoted August 10.
• Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
  • Maintenance RC viewer version 6.4.23.563789, issued on September 16 – combines the Grappa and Happy Hours RCs.
• Project viewers:
  • 360 Snapshot project viewer, version 6.4.23.563579, issued September 3.
  • Performance Floater project viewer, version 6.4.23.562625, issued September 2.
  • Mesh Optimizer project viewer, version 6.4.23.562614, issued September 1.
  • Legacy Profiles viewer, version 6.4.11.550519, dated October 26.
  • Copy / Paste viewer, version 6.3.5.533365, dated December 9, 2019.

In Brief

• Some are reporting issues of not being able to easily teleport back to a location previously visited in a session, a similar problems that manifested for some using Malwarebytes. In this case, the issue seems to be exceptionally slow teleport when trying to go back, with some incidence of the viewer shutting down.