SL project updates 27/3: TPV Developer meeting and JIRA

Stony Hill Vineyard, Stony Hill; Inara Pey, June 2017, on Flickr Stony Hill Vineyardblog post

Friday, July 7th saw a truncated TPV Developer meeting take place in lieu of the planned meeting for Friday, June 30th. As always, the video of the meeting is embedded at the end of this report, and my thanks to North for recording and providing it. Timestamps in the text below refer to that video, and will run the video from that point in a separate browser tab.

SL Viewers

[3:30] The Alex Ivy 64-bit release candidate viewer (version 5.1.0.507006 at the time of writing) is due to be updated in week #28 (commencing Monday, July 10th). The viewer is liable to receive further updates prior to any promotion to release status, as the Lab further refines the Windows SL Launcher.

As previously noted in these pages, the launcher is designed to detect which version of the viewer a Windows system can run. If it detects the system can run the 64-bit version of the viewer, it downloads and installs that, if not, it defaults to downloading the 32-bit version. An upcoming update to the viewer will include a debug setting to override this and force one or other version to be downloaded and installed (obviously, this will not work trying to run the 64-bit viewer on 32-bit Windows, but will allow the 32 viewer to be run on 64-bit Windows).

Crash rates for the viewer are elevated, but the Lab suspects this is down to the number of 32-bit versions being run.

[5:35] This leaves the current viewer pipelines as:

  • Current Release version 5.0.6.326593, released on May 26th, promoted June 20th – formerly the AssetHTTP RC viewer – overview
  • Release channel cohorts:
  • Project viewers:
  • Obsolete platform viewer version 3.7.28.300847 dated May 8th, 2015 – provided for users on Windows XP and OS X versions below 10.7.

The 360 snapshot viewer should be updated with some fixes “soon”.

Server Updates

[6:05] The simulator version using the new Linux operating system update, currently deployed to the Magnum RC channel (server version 17#17.06.29.327400) should be promoted to the Main (SLS) channel on Tuesday, July 11th.

[12:07] This simulator version also includes the fix for DJ boards (see BUG-10073). Some people may need to update their board scripts.

JIRA Temporarily Closed to Comments

The Second Life JIRA has been temporarily closed to comments, because: idiots. Or to allow the Lab to explain it:

Recently, our bug reporting system (Jira) was hit with some spam reports and inappropriate comments, including offensive language and attempts at impersonating Lindens. The Jira system can email bug reporters when new comments are added to their reports, and so unfortunately the inappropriate comments also ended up in some Residents’ inboxes.

We have cleaned up these messages, and continue to investigate ways to prevent this kind of spam in the future. We appreciate your understanding as we work to manage an open forum and mitigate incidents like this.

In the short-term, we have disabled some commenting features to prevent this from recurring. This means that you will not be able to comment on Jiras created by other Residents. We apologize for this inconvenience as we look into long-term solutions to help prevent this type of event from occurring.

Other Items

  • The next Firestorm release is liable to be in September
  • [8:51] The Lab is looking at turning off the remaining UDP messaging for asset delivery on the servers in February 2018, although this is TBC. This means that viewers not running the latest HTTP updates will not be able to receive any asset data.
  • [17:41] The next scheduled TPV Developer meeting is Friday, July 28th

 

Advertisements

SL project updates 24/3: TPV Developer meeting

Le Sixième Sens, Les Reves Perdus; Inara Pey, June 2017, on Flickr Les Reves Perdusblog post

The majority of the notes in this update are taken from the TPV Developer meeting held on Friday, June 16th, 2017. The video of that meeting is embedded at the end of this update, my thanks as always to North for recording and providing it. Timestamps in the text below will open the video in a separate window at the relevant point for those wishing to listen to the discussions. Note that the timestamps may not be in chronological order, reflecting the fact that some topics were discussed more than once during the course of the meeting.

Server Deployments, Week 24 – Recap

As always, please refer to the server deployment thread for the latest information.

There was no deployment to the main (SLS) channel on Tuesday, June 13th. Nor, as the channel was updated in week #23, was there a restart.

On Wednesday, June 14th, two of the server RC channels were be updated as follows:

  • LeTigre received a new server maintenance package (#17.06.12.327066), comprising additional internal logging and features and improvements to region start
  • BlueSteel received a new server maintenance package (17#17.06.13.327122) containing internal fixes

[15:20] The Magnum RC was initially updated with a newer version of the new operating system update (#17.06.12.327060), which included a fix for BUG-100737 “Shoutcast receivers unable to relay on RC Magnum” (see part 1 of this report for more on this issue). However, this deployment had to be subsequently rolled-back as the corrective intent of the BUG-100737 didn’t work as expected. This update will likely be re-deployed to Magnum ion week #25 (commencing Monday, June 19th).

SL Viewer Pipelines

Asset HTTP Viewer

[1:39] The Asset HTTP viewer should be promoted to release status at the start of week #25 (week commencing Monday, June 19th). The promotion has been delayed while the viewer goes through a complete regression test (something the Lab does every X number of viewer releases).  This viewer sees delivery of all remaining asset types (wearables, gestures, animations, sounds, etc) over HTTP via the CDN.

[11:39] This viewer should hopefully see faster first-time playback of sounds and animations, as these will be obtained via the CDN, which should be faster than being obtained through the simulator. It also means obtaining assets should also be a lot more reliable when you’re in a busy region, because – again – the assets are not coming via the simulator, but through a CDN node.

The Lab will – several months hence from now – remove the server-side UDP messaging support for these asset types. This will in turn mean that any viewers not updated to the HTTP support at the time the messaging is removed from the simulator will no longer be able to receive these asset types.

Maintenance RC Viewer

[5:25] The Maintenance RC viewer updated on Thursday, June 15th to version 5.0.6.327125. This includes an update to prevent the viewer crashing if it receives a UDP message from the simulator that it doesn’t recognise, by having the viewer ignore all unrecognised messages.

Voice RC Viewer

[5:04] The Voice RC viewer has been updated, but the update has a high crash rate and so the update is unlikely to see the light of day.

Alex Ivy 64-bit Project Viewer

[2:23] The next version of the 64-bit project viewer is completing testing. This includes the new Windows SL Launcher and updater, together with a 64-bit version of the Havok sub-libraries. As noted in my last TPV Developer meeting update, the launcher is essentially a 32-bit executable that checks a Windows system to see if it is 32-bit or 64-bit, and then endeavours to download the correct version (32- or 64-bit) of the viewer if an update is available, install it and then launch it. SL Launcher is only required for Windows as the Mac version of the viewer will only be provided in 64-bit once the Alex Ivy viewer reaches release status.

A follow-up build for RC release has apparently been built, and this should appear soon after the project update, and work has commenced on updating the wiki build instructions for building the viewer to match the 64-bit build process.

[38:26] The wiki instructions are being updates to reflect the requirements of the 64-bit build, so care should be taken when following them for other builds.

360-degree Snapshot Project Viewer

[6:21] The 360-snapshot viewer is now up-to-date and includes code to generate a 360 equirectangular images and their metadata, which can then be uploaded to suitable websites supporting 360-images. The update will appear once it has cleared the Lab’s QA testing.

There is still further work to be done on this viewer – the UI is going to be updated to allow integrated uploads of 360-images to SL Place Pages (and this may be done for Flickr, etc), and SL Place Pages will be updated to accept 360-degree images from the viewer.

TP Throttle

[13:28] The Lab is still looking at throttling the speed at which teleport requests can be re-tried when trying to access a busy region. An initial change is currently on the LeTigre RC, and further changes are liable to be made. As previously noted, these updates shouldn’t impact manual teleports, but may affect teleport HUDs which are scripted to repeatedly re-try teleports in rapid succession until one is  successful (requiring the scripts running them to be modified so they don’t exceed the throttle).

This change is being made because a high incidence of failed teleport requests hitting a busy region places an additional load on the region’s simulator, adversely affecting performance for those already in the region.

Other Items

Uploading Meshes Rigged to Attachment Points

[17:48] This subject came up at the Content Creation User Group meeting as a part of the discussion on animating weapons to follow hands. There was some confusion on whether mesh objects rigged to attachment points could be uploaded, after it was reported that the LL viewer supported it, and Firestorm didn’t (see FIRE-21000 – which now has a fix).  While there is a server-side validation error which can cause some issues when uploading meshes (fix in progress) which might cause upload problems, it is believed that the current behaviour here should be that new objects rigged to attachment points should be blocked from upload, but existing items rigged to attachment points previously uploaded to SL will still work.

Supplemental Animations and Animation Priorities

[24:17] The question was asked if there was any historic reason for not being able to change the priority of an animation post upload (see SVC-8094). It is thought this might be because the priority is set within the animation asset, which cannot be edited. However, it is hoped the forthcoming server-side supplemental animation updates will help eliminate some of the conflicts created by priority clashes.

Providing a Means to Compile Experience Scripts in the User’s Inventory

[35:21] Some people working collaboratively on experiences are finding it problematic when having to update scripts used by the experience, but which are contained in another user’s objects for that experience, as it requires a lot of swapping and changing, rather than simply editing the script in question (see BUG-8180).

While the Lab understands these difficulties, it was a conscious decision to have experience management work as it does, and while at some point in the future they might revisit things, doing so isn’t on the short-term roadmap.

Resetting Scripts in No-Mod Objects

[36:47] This is a request the Lab is unlikely to implement, because it would violate the expectations of the script authors.

SL project updates 22/2: TPV Developer meeting

Chess Wonderland, Egypt; Inara Pey, May 2017, on Flickr Chess Wonderlandblog post

The majority of the notes in this update are taken from the TPV Developer meeting held on Friday, June 2nd, 2017. The video of that meeting is embedded at the end of this update, my thanks as always to North for recording and providing it. Timestamps in the text below will open the video in a separate window at the relevant point for those wishing to listen to the discussions.

Server Deployment – Recap

  • On Tuesday, May 23rd, the Main (SLS) channel was updated with a server maintenance package (#17.05.22.326523), containing a fix for BUG-100704, “[Server] If Anyone Can visit is selected after Allow Group was set only group members can enter”, related to the parcel overrides update.
  • On Wednesday, May 31st, the RC channels were updated as follows:
    • BlueSteel and LeTigre received the same server maintenance package (#17.05.26.326655), comprising “Tweaks to help with capability loss”.
    • Magnum received a server maintenance package (#17.05.26.326659) for the simulator operating system update, which does not contain and functionality changes.

SL Viewer

[0:20] The Voice RC cohort updated to version 5.0.6.326589 on Wednesday, May 31st.

[0:31] The Asset HTTP viewer is doing reasonably well in its latest RC update (version 5.0.6.326593, dated May 26th), and this might be the next viewer to gain promotion to release status.

[28:22] Once the asset HTTP viewer has been promoted, the Lab plan to develop a schedule for removing simulator support for the old UDP messaging the HTTP updates have replaced. The actual removal of the messaging paths might not occur until 2018.

Maintenance RC

[0:40] A new Maintenance viewer was released as a RC cohort on Thursday, June 1st, 2017. Version 5.0.6.326731 includes a range of bugfixes and updates, including further updates to Trash management:

  • Fix for number of items in Trash calculation
  • Trash is full floater wasn’t showing in some conditions when it should have been. It will do as it should now.
  • Fix for when Trash version gets out of synch
  • A delete warning will now show up once every session to cut down on accidents
  • Trash purge message will include number of items deleted.

Alex Ivy (64-bit viewer)

[0:55] A new update for the 64-bit viewer will be appearing soon, it will include the new viewer update code, to help ensure the correct version of the viewer (32-bit or 64-bit) is delivered to the user.  Additional changes to the server-side viewer update manager (the code which tells the viewer there is a newer version available) have delayed this update from appearing sooner.

[2:10] When this version of the viewer is issued, it is anticipated that work will commence on building and QAing a Release Candidate version of the viewer. At this point the Lab will also start work on updating the SL wiki instructions for building the viewer.

[5:35] The Lab’s reocmmendation is that if Windows users are using the 32-bit version of the viewer on a 64-bit system, they should swap to a 64-bit version of the viewer for a less crash-prone experience.

[6:16] A caveat to this is 64-bit systems using the Intel HD and Intel HD4000 GPU chipsets. These systems cannot run the 64-bit version of the viewer and require the 32-bit version.

360-degree Viewer

[3:43] An update to the 360-snapshot viewer is expected soon, pending the fix for an issue with the Mac OS X build.

When the update appears, it will not require any post-processing of a ZIP file (as is currently the case). Instead, it should produce an equirectangular image with all the necessary metadata required for displaying on sites which support 360-degree images.

OTR and Chat Text Encryption

[7:14-18:56] Questions have again been raised on the use of end-to-end encryption of IM conversations by TPVs. This was a thing several years ago, largely derived out of misplaced fears that LL staff where reading people’s IM conversations willy-nilly. However, as it used viewer code never intended for that purpose, it was “broken” when the underpinning code was updated / removed.

At the time the encryption was used, it was problematic for the Lab, as it interfered with things like verifying abuse reports by checking IM logs (if the conversation is encrypted, LL cannot verify what was said, and so cannot take any action, even if warranted), and could also be an issue assisting in other types of necessary investigation (e.g. possible fraud / money laundering acts).

The circumstances under which someone from LL can view IM files are strictly limited and controlled, both in terms of who can view them, when then can view them and how much of them can be viewed. As such, there is no risk in IM logs just being casually read at the Lab. Given this, and the issues it can cause, use of any form of encrypted IM exchange would likely be frowned upon or be regarded as a “normal” function of SL (and so wouldn’t be safeguarded from potential “breakage” during future changes to the viewer). Instead, those feeling they need secure communications with other are encouraged to use other means of doing so.

Shoutcast / Streaming Scripts Issues

[23:30-26:33] The Magnum channel RC has resulted in breakages to scripts used by various streaming service DJ boards (see BUG-10073). The issue has been diagnosed, and a means to allow these services to keep operating is being developed. However, when deployed it will mean that the scripts themselves will also need to be revised, and it will not be backwards compatible with systems that are not updated by their creators, which will remain broken. As the change causing the issue is integral to the HTTP updates being made to improve SL’s performance, it will not be rolled back.

Griefing Following Ban

[30:02-35:39] Some estate managers are finding themselves specifically targeted for griefing after banning someone from their land (identified by the griefer as ban notifications carry the banner’s name). In some cases the griefing can extend to groups associated with the land. One solution is to use a bot for banning purposes, however, the request has been made for better anonymity for those carrying out the ban. A JIRA has been requested, providing the fullest possible description of the issue, what happens, and how, and what is believed to be needed to counter it.

General

  • [19:17] BUG-40824, “Missing Offline Group Notice (Unreliable Delivery)” is still proving a problem for some. The Lab haven’t looked at the issue recently, but will get back to it.
  • [21:25] Local and group chat has been lagging – again, it’s been noted, but no specifics on possible causes.
  • [21:49] Increasing limits (physics, etc) – currently the focus is on the simulator OS update, which includes, software tools and libraries. Once this work has been completed, the Lab hopes to look at all the limits and see what can be adjusted, either directly or with some up-front work.

SL project updates 20/3: TPV Developer meeting

Nitroglobus Roof Gallery: Black and White Women – blog post

The majority of the notes in this update are taken from the TPV Developer meeting held on Friday, May 19th, 2017. The video of that meeting is embedded at the end of this update, my thanks as always to North for recording and providing it. Timestamps in the text below will open the video in a separate window at the relevant point for those wishing to listen to the discussions.

Server Deployments Re-cap

  • There was no Main (SLS) channel deployment or restart on Tuesday, May 16th.
  • On Wednesday, May 17th, the three RC channels were updated as follows:

SL Viewer

[1:00] The Voice RC viewer has an elevated crash rate, and the Lab currently haven’t determined why.

The Maintenance RC viewer updated to version  5.0.5.326444 on Thursday, May 18th. This viewer currently has a lower crash rate that the other RC viewers (although it has not been out that long), so might be a candidate for promotion. I have an overview of this viewer for those interested.

64-bit Viewer

[2:23] The last major functional addition for the 64-bit Alex Ivy viewer is currently with the Lab’s QA. If all goes well, a further project viewer update should arrive in week #21 (commencing Monday, 22nd May).

This introduces a new executable to the viewer – SL Launcher – which runs an update check at start-up. If there is a new version of the viewer available, the Launcher manages the download and installation – including ensuring Windows users get the right version for their operating system (32-, or 64-bit).  If there is no new version to install, or once the viewer installation has completed, the Launcher will launch the viewer as a child process, and will shut down when the viewer exits at the end of a session.

The plan is to move the crash data capture package to the Launcher in the future, which will give full end-to-end monitoring of the viewer in the event of a crash.

360 Snapshot Viewer

[6:07] The work on the 360 snapshot viewer is once again progressing. A new library has been added, which provides the appropriate meta data so that websites supporting 360-degree viewing can correctly such images taken by the viewer on upload, eliminating the need to process them separately via the web service currently supplied by the Lab.

This work is currently being tested, and should find its way into a project viewer update some time in the next two weeks or so, with a release candidate hopefully not too far behind that.

Region Crossing Hand-off / Caps Router Issues

[7:43] Fantasy Faire experienced very high levels of region crossing hand off problems with avatars trying to move between the various regions. A similar issue has surfaced at the just-opened Home and Garden Expo.

While it issue isn’t new, the Lab found a cause is the Caps Router running out of connections due to the number of avatars it is attempting to serve. New monitoring has been put in place which will determine how many connections the Caps Router is using, and when it is approaching its limits. The data gathered will be used to help better determine how many connections are needed, allowing the Lab to adjust the number supported.

This work is going to be carried out incrementally, starting with an initial RC deployment in week #21 containing conservative adjustments in the hope of avoiding creating additional bottlenecks in changing things too radically at one time. However, the hope is that the changes will in time result in two improvements:

  • It could result in an increase in the number of avatars a region can comfortably support
  • As this is an issue at the SERVER level (not the simulator), the changes should help reduce people on regions with few avatars on them experiencing issues as a result of the region being hosted on the same server as one (or more) regions with a lot of avatars on them.

As a result of understanding the problem, the Lab was aware the issue was impacting the Home and Garden Expo even before it had been reported.

Unsuccessful Teleports Impacting Region Performance

[14:20] During investigations into the region issues at Fantasy Faire, the Lab noted that a simulator running a busy region has to carry out a lot of work to determine whether or not someone can teleport into it, which can degrade overall simulator performance.

To combat this, the Lab is going to change the teleport re-try throttle following a failed TP. As viewer-initiated teleports are already somewhat throttled, the change should not affect them. However, it will likely mean that the very rapid retry TP HUDs (aka “TP hammers”) will break or degrade in their performance unless adjusted.

The hope is that by reducing the load placed on a simulator as it tries to deal with too rapid a succession of TP requests which cannot be granted as the region is full, overall performance will be improved and those already in the region will enjoy a better experience.

This change should be appearing in a server RC update soon.

Additional comments on teleport failures:

  • A queuing system will not be added, as this is deemed to be too difficult to implement and manage.
  • There is no relationship between the size of an avatar’s inventory and the frequency with which that avatar may experience teleport failures. However, the amount of items attached to an avatar, the scripts they are running, etc.
  • The Lab can monitor teleport failures in real-time.

Automatic Additional Logging after Region Crashes

[29:27] It was asked if additional logging could be automatically enabled on a region crash. This is something that cannot be done, and Oz’s belief is that doing so would result in an additional load on the simulator during recovery, and so not be a good idea.

Avatar and Object Rendering Cost Investigations

[31:00] The Lab is continuing work in reviewing the rendering cost calculations for in-world objects and avatars, work I first reported on in September 2016. However, the numbers aren’t at a point where any adjustments can be made to the calculations.

Fun Fact

Oz Linden marked his seventh anniversary at the Lab this week – so a belated happy rezday to him! Some of us can likely remember his 2010 appearance at the SLCC, when Esbee Linden introduced him to the audience in Boston 🙂 .

Oz at one of the viewer / open-source panels at SLCC 2010, with Esbee Linden just visible to the right