SL project updates week 36/2: TPV Dev meeting + SL in the cloud

Brand New Colony; Inara Pey, September 2017, on FlickrBrand New Colonyblog post

Updated to reflect the release of the Wolfpack RC, which appeared after this article had been uploaded for publishing.

The majority of the notes in this update are taken from the TPV Developer meeting held on Friday, September 8th 2017. The video of that meeting is embedded at the end of this update, my thanks as always to North for recording and providing it. Timestamps in the text below will open the video in a separate window at the relevant point for those wishing to listen to the discussions.

Server Deployments Week #36 – Recap

Please refer to the deployment notice for the week for latest updates and news.

Region Return Issues

Whether connected to the Wednesday deployment or the grid-wide issues experienced later that day, but some regions reported significant returns of in-world objects (Mother Road, on the Main (SLS) channel, which I blogged about here, suffered the problem, as Heavenly Views (Magnum RC) also apparently did). Most of these problems seem to have been rectified by a roll-back of the affected regions.

SL Viewer

A new RC viewer, codenamed Wolfpack – version 5.0.8.328879 – was released late on September 8th. This viewer is functionally identical to the release viewer, but includes additional back-end logging to “help catch some squirrelly issues”.

[1:15-4:15] Otherwise have been no further viewer updates since the start of the week, leaving the current pipeline as follows:

  • Current Release version 5.0.7.328060, dated August 9, promoted August 23 – formerly the Maintenance RC
  • RC viewers:
    • Alex Ivy 64-bit RC viewer version 5.1.0.508209, dated September 5th
    • Voice RC viewer updated version 5.0.8.328552, dated September 1st
    • Maintenance RC viewer version 5.0.8.328812, dated August 31st
  • Project viewers:
  • Obsolete platform viewer version 3.7.28.300847, dated May 8, 2015 – provided for users on Windows XP and OS X versions below 10.7.

The Alex Ivy and Voice updates bring these RCs into parity with the current release viewer. These updates have lowered the overall crash rates for both viewers, although the Voice viewer’s crash rate is still somewhat elevated compared to the release viewer. This elevated crash rate is something the Lab has seen in the various viewers for the last few months, and are working to lower.

Alex Ivy will likely have two more RC releases before it is ready for promotion to de facto release status, as there are a couple of issues the Lab wants to clear up before any promotion of this viewer.

It is hoped work will shortly be resuming on the 360-degree snapshot viewer, part of which will also be bringing it up to parity with more recent releases.

New Viewer Splash / Log-in Screen

[4:25-8:07] The Lab is in the process of revamping the official viewer’s splash / log-in screen. This should be inherited automatically by those TPVs using the Lab’s splash screen. This will see the screen have a different look and feel, with the information arranged a little differently, with the grid status made somewhat more prominent. Some of the information widgets associated with the splash screen are also being updated as a part of this work, and will require testing by TPVs when available (RSS feeds should not be affected).

This work is being carried out by Phronimos Linden, one of the more recent (3 months ago at the time of writing) Lab staff recruited to work on Second Life.

Moving to the Cloud

[15:52-25:25] As confirmed in a recent Lab blog post, Second Life services are moving to the cloud. There are no specific details on this as yet, but in short:

  • The work will be carried out in stages.
  • It will include providing regions from the cloud (rather than the Lab’s own co-located servers).
  • As many know, the asset services has been in S3 cloud servers for several years.
  • The order of moving services hasn’t been determined as yet, but the aim will be to move services, then test for reliability and performance before opening them up / moving to the next service(s).
  • Some testing has already begun, using services only accessed indirectly by users.
  • The Lab has been working on the infrastructure for this for a while, but none of the major services have been moved.

It is unlikely this will lead to regions of a different size to those presently on the grid, or which can be varied in size (a-la the OpenSim varregions), as region size is very deeply baked into the simulator and viewer code (and probably elsewhere in the overall SL code base); although some at the Lab would love to be able to have such a region capability. However – as indicated in the Lab’s blog post – it may allow the Lab to introduce new region products, possibly ones capable of handling greater loads.

It is hoped that this work will result in lower tier over time, but whether this will be the case of not is still and open question at this time, as it is unclear as to what costs will be involved in terms of cloud instance types., etc., that will be required to support simulators.

Overall, there is a lot the Lab hopes to achieve with the move, but the precise benefits are likely to only become clear once things have been tried and found to work / over time as the work progresses and beyond. However, at this point time scales are TBD, and it is liable to be some time before user-visible aspects of the service move to the cloud (although non-visible services – such as the log-in service, for example – could be moved sooner).

A precursor to this project is the continuing work to update the servers to newer version of Linux, work which is making “good progress”.

Other Items

Asset HTTP Messaging and Asset HTTP Issues

[10:11-10:45 and 12:10-15:23] With the promotion of the Asset HTTP viewer to release status in June 2017, the majority of assets are now delivered to the viewer using HTTP via the Content Delivery Network(s) used by the Lab. This means that UDP messaging for all of the affected asset types will be turned off at the server end at some point.  Currently no date has been set for this, and it looks like it may not occur much before February 2018.

There is still an issue related to fetching assets over HTTP in general which is experienced by some users some of the time   whereby something causes the pipeline textures to get out of sync and things to go awry. As the problem happens for some and not others, the Lab is still trying to determine the root cause for the problem, but it is a matter the Lab is trying to resolve.

Catznip viewer reports issues with textures over HTTP “stalling” for between 5-10 minutes on their pre-release viewer with the latest HTTP updates, but this isn’t something the Lab is aware of as a more general issue.

Estate Tool Ban List Improvements

[10:48-11:43] The Lab did some initial work to improve ban lists at the estate / region level a while ago, but the intended work to improvement to overall layout for the lists, etc., has been delayed do to work on dealing with viewer crashes, etc. This work is apparently now “next in line” once some of the existing viewer projects are shipped.

The conversation from 26 minutes to the end is more general chat on viewer stats, speculation on the reason for the texture load stalls, and an ad for the Firestorm birthday party.

Advertisements

SL project updates week 34/2: TPV Developer Meeting

Yamagata; Inara Pey, August 2017, on Flickr Yamagatablog post

The majority of the notes in this update are taken from the TPV Developer meeting held on Friday, August  25th 2017. The video of that meeting is embedded at the end of this update, my thanks as always to North for recording and providing it. Timestamps in the text below will open the video in a separate window at the relevant point for those wishing to listen to the discussions.

Server Deployments Week #34 – Recap

Please refer to the deployment notice for the week for latest updates and news.

  • On Tuesday, August 22nd, the  Main (SLS) channel was updated with server maintenance package, 17#17.08.11.328159, comprising internal fixes and the following feature requests:
    • BUG-5398: llGetObjectDetails() constants OBJECT_SELECTED & OBJECT_SAT_UPON. This sees the addition of two new parameters:
      • OBJECT_SELECTION_COUNT – returns how many agents are selecting any link in a linkset
      • OBJECT_SITTER_COUNT – returns how many agents are sitting on any links in a linkset.
    • BUG-9666: llGetObjectDetails() constants OBJECT_REZ_TIME, OBJECT_CREATION_TIME and OBJECT_RETURN_TIME.
    • BUG-134057 OBJECT_CREATION_TIME output precision possibly clamped – this sees a shift to 6-digit precision.
  • On Wednesday, August 23rd, the three RC channels all received a new server maintenance package, 17#17.08.11.328152 comprising the MIME type changes for HTTP.

SL Viewer

[2:35] The Maintenance RC viewer, version  5.0.7.328060 and dated August 9th was promotion as the de facto release viewer on Wednesday, August 23rd.  A new Maintenance viewer is expected to appear around the middle of week #35 (commencing Monday, August 28th).

[1:23] The Alex Ivy 64-bit viewer failed a QA test as a result of issues being introducing in refectoring how the SL Luncher code (used to determine whether the 32-bit or 64-bit version of the viewer should be installed on Windows systems). The issues aren’t serious, so it is hoped this viewer will appear some time in week #35.

[10:50] A further delay in promoting this version of the viewer, which currently remains at version 5.1.0.507412, is that accurate crash reporting data isn’t being gathers, due to issues with the back-end crash analyser, which are in the process of being addressed.  This problem isn’t uinque to the 64-bit build, but does seem to impact it the most.

[2:13] The Voice viewer, currently version 5.0.7.327253, dated June 21st, has been updated for parity with the release viewer, but has yet to pass QA testing at the time of writing.

This leaves the overall pipeline as follows:

  • Current Release version 5.0.7.328060, dated August 9, promoted August 23 – formerly the Maintenance RC
  • Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
  • Project viewers:
  • Obsolete platform viewer version 3.7.28.300847, dated May 8, 2015 – provided for users on Windows XP and OS X versions below 10.7.

[2:50] It is likely the the Lab will, at some point in the future, move to block some of the older and more outdated versions of the official viewer from accessing Second Life.

Texture Memory Increase and Cache Improvements

[13:58] The 64-bit viewer build will see an increase in the texture memory size, although this isn’t part of the current Alex Ivy build. This work includes:

  • [14:23] Changes to texture caching, including some re-organising and optimisation, and improvements to  statistics gathering to assist in understanding how well it is performing.
  • [15:25] Changes to format of the stored texture data, using the post-decoding raw data, rather than the JPEG2000 data, which should “significantly” improve the amount of textures which can be loaded per second by the viewer, when visited a location previously cached.
  • [16:18] An expansion of the cache version guards so that if a TPV changes versions of their texture decoding software (e.g. KDU) or change from 32- to 64-bit, the cache will get wiped, and force the start of a new one (this is something Firestorm already does).
  • An increase in cache size.

Further changes will follow from these, including possibly changing how the static VFS cache is handled, but this work has yet to be fully characterised.

BUG-10515: Unable to Rez Due To Invalid Mesh Data

[24:42] The have been increased report of mesh rezzing failures with the message “Unable to rez object because its mesh date is invalid.” It has been noted across different objects, but multiple reports have been made for it occurring with the Maitreya Lara 4.1 body.

The problems seem to occur randomly on different simulators, and the Lab has had problems getting the issue to reproduce, as often a simulator restart may clear it, so Grumpity Linden is seeking reports of the problem which can be made as they occur (object, region name, date / time, etc.), so that an attempt to reproduce the error can be be made and log files immediately gathered before any restart takes place (and/or log files are lost).

Other Items

[3:32] Infrastructure work:  the events of Tuesday, August 22nd (see April Linden’s blog post) became an accidental way of testing some of the new infrastructure changes (in this case, the log-in servers) the Lab is implementing, and will continue to implement over the next several months. This encompasses Second Life and the Lab’s various web properties related to it. This includes further operating system updates as well as various component updates.

[24:09] Part of this work involves inventory, although whether it will lead to user-visible improvements to inventory handling is unclear at this point.

[4:19] Place Pages: this have received a number of fixes recently, for those who wish to use them (see here for general information on Place Pages if you are unfamiliar with them).

[5:32] Estate Tool ban list improvements: The lab did some initial work to improve ban lists at the estate / region level a while ago, but the intended work to improvement to overall layout for the lists, etc., has been delayed do to work on dealing with viewer crashes, etc. However, work will be resuming before the end of the year, and possibly sooner. However, this work is unlikely to see an increase in the number held by the ban list, but will focus on usability improvements.

[28:44 McAfee Total Protection Issue: This is a problem being experienced by some users on TPVs (e.g. Alchemy and Firestorm) where the viewer is being flagged by McAfee Total Protection. This might be down to a code signing issue, however, to assist in further investigations, the problem needs to be reproduced and reported using the LL viewer.

[33:35] BUG-6925, HUDs and Attachments randomly detaching on region crossings / teleports: This has been investigated and determined to most likely be the result of a race condition, but a fixed has yet to be implemented.

[39:56-end] General discussion on documentation (e.g Axon animation; Objectllsd) and avatar physics calculations.

 

SL project updates week 32/3: TPV Developer Meeting

Whimberly, Whimberly; Inara Pey, August 2017, on Flickr Whimberlyblog post

The majority of the notes in this update are taken from the TPV Developer meeting held on Friday, August  11th 2017. The video of that meeting is embedded at the end of this update, my thanks as always to North for recording and providing it. Timestamps in the text below will open the video in a separate window at the relevant point for those wishing to listen to the discussions.

Server Deployments Week #32 – Recap

Please refer to the deployment notice for the week for latest updates and news.

  • On Tuesday, August 8th, the Main (SLS) channel was updated with a new server maintenance package (#17.07.27.327933), comprising “additional internal fixes”,
  • The scheduled deployment to the RC channels for Wednesday, August 9th has been cancelled due to a back-end data issue which meant the region channel names weren’t being set correctly, so they didn’t start.

SL Viewer

[00:54] The Maintenance RC viewer was updated on Wednesday, August 9th to version 5.0.7.328060. This is currently the most likely candidate for promotion to de facto release status. The rest of the LL viewer pipeline remains as at the start of the week:

  • Current Release version 5.0.6.326593, released on May 26, promoted June 20 – formerly the AssetHTTP RC viewer – overviewdownload and release notes
  • Release channel cohorts:
  • Project viewers:
  • Obsolete platform viewer version 3.7.28.300847, dated May 8th, 2015 – provided for users on Windows XP and OS X versions below 10.7.

[01:15-01:40 and 4:51-5:33] It is hoped the Alex Ivy 64-bit viewer will be updated in the early part on week #33 (commencing Monday, August 14th). Currently, the viewer is still showing an elevated crash rate for some, for reasons yet to be determined. In the meantime, the Lab is adding further code that will hopefully capture more crash information.

[01:56 and 06:49] The 360-degree snapshot viewer is currently on hold due to people being on vacation at the Lab. This may be a slightly extended delay while resources are diverted to the 64-bit viewer and CEF work.

[14:22-14:34] The Voice viewer is also suffering a high crash rate, also for unknown reasons.

[14:35-14:46] One of the next viewer projects for the Lab is a major re-vamping of the crash reporting and analysis system, which will hopefully make it easier to trace down probable causes for viewer crashes.

llHttpRequest Issues

[02:15-04:21] The least couple of weeks have seen server-side issues with llHttpRequest. Some of this has been due to low-level library changes that were inherited in updating the simulator code to some of the new components, including the OS itself. Other issues were the result of changes the Lab made at around the same time.

Most of these issues have now been rectified, although there is still some incompatibility issues still to be sorted. There is one further significant update related to this which will be going out on Aditi (the Beta grid), which will be put to the test with the intention of seeing if anything else breaks. A blog post / forum notice will precede this.

Simulator Infrastructure Updates

[08:52-09:28] The Lab is going to be carrying out a lot of simulator infrastructure work over the next two quarters. Efforts will be made to make user-visible updates during this work, but most of the work will be transparent to users, and may result in some performance improvements.

Other Items

[08:18-08:46] Group notice failures: No work has been put into looking at why some group notices are being dropped. This is currently seen as a lower priority than dealing with issues which can crash simulators.

[10:29-11:37] Dynamic mirrors (again): (see STORM-2055). The Lab has no plans to work on this at all, but Oz offered a *possible* compromise:

If an open-source developer implements mirrors that work, and are acceptable from a performance point of view, without unreasonable restrictions on where you can put them and what you can put them on, I at least will be happy to consider the contribution. But I don’t believe that’s possible, and I am assured by graphics developers who know a great deal more about rendering than I do, that it isn’t possible; and that’s why we’re not doing in. So, if you have a miracle worker out there who wants to do a contribution to the viewer – go for it! But since I’d like to get other work done, I’m not going to devote our developers to doing it.

[13:37-14:01] EEP – Environment Enhancement Project: a re-iteration of this project see the outline Google doc or my overview for more) will allow for things like phases of the Moon.

[18:20-18:32] Viewer UI Region / Estate ban list improvements: this work is still pending a developer resource.

[19:10-24:12] Abuse of Setting Home:  it has been reported that griefers are able to circumvent being kicked out of a region (TP Home) by moderators if they have home set to that region. The alternative to this is to ban them from the land – but this requires the rights to do so. There are also alleged means to bypass estate bans if the home position is set to the banned region – which should not be possible. If there is a bug allowing this to happen, it needs to be reported with repro steps.

PacMan Stars and Black Stars: the issue of “Pacman stars” (irregularly shaped stars) has been fixed via a contribution by Drake Aconis (Sovereign Engineer). However, the issue of stars appearing as black dots in some daylight windlight settings has yet to be fixed. This may be looked at as a part of the EEP.

 

SL project updates 30/3: TPV Developer meeting

Follow Your Bliss, Sea Foam; Inara Pey, July 2017, on Flickr Follow Your Blissblog post

The majority of the notes in this update are taken from the TPV Developer meeting held on Friday, July 28th 2017. The video of that meeting is embedded at the end of this update, my thanks as always to North for recording and providing it. Timestamps in the text below will open the video in a separate window at the relevant point for those wishing to listen to the discussions.

Server Deployments Week #30 – Recap

Please refer to the deployment notice for the week for latest updates and news.

Main (SLS) Channel

There was no deployment / re-start of the Main (SLS) channel on Tuesday, July 25th, the Main (SLS), which remains on server maintenance package #17.07.11.327548. This update included a fix to allow DJ boards to work, however, any scripts which have not been updated to meet the new requirements may not work. for details, please refer to this forum thread.

RC  Channels

The deployment to the RC channels did not go as planned, after it was discovered the release had a creeping crash bug that could not be reproduced on Aditi, but would be a bad experience for all regions on the RCs. A roll-back was therefore performed, leaving all three RCs on the same server maintenance package as the Main (SLS) channel (#17.07.11.327548).

SL Viewer

[0:10-3:33] The Maintenance RC viewer updated to version 5.0.7.327940 on Friday, July 28th. This update includes two crash fixes, and an update for a pull request fix, which Oz has asked third-party viewers to include as soon as possible. Essentially, some of the very low-level code in the log-in process has a very aggressive approach to retries if it encounters log in delays, and this needs rectifying.

This only became apparent when a main network cable in the Lab’s data centre was severed (by a backhoe!). This shifted log-in attempts to an alternative network connection where the combination of high user numbers and this aggressive retry loop in the code resulted in what was essentially a denial of service attack which overwhelmed the log-in servers. Given the depth of the code in the viewer, it is believed it is present in the vast majority (if not all) viewers. This also means the Maintenance viewer will get promoted relatively soon.

[3:59-5:16] The Alex Ivy 64-bit viewer still has an elevated crash rate and the Lab is making changes to obtain better data on the crashes. The 32-bit version of the viewer in particular is suffering “really high” crash rates.

The remaining viewers in the pipelines remain unchanged:

Web Property, Attachments and ToS Issues

[5:48-7:01] Some users have been experiencing some or all of the following issues during the week:

  • Inability to access various LL web properties (e.g. their own dashboard, the JIRA system, etc). As of the TPV Developer meeting, all web services were believed to be back up and running, although the Lab weren’t sure as a to the cause of the issues.
  • Attachment failures – either on attach or with scripts. These are still being investigated, and a definitive cause has not yet been found.
  • The Terms of Service update, which officially comes into force on Monday, July 31st but which currently requires acceptance by users, has been causing log-in issues for some. Again, the cause isn’t as yet known.

Other Items

Parcel Sounds

[7:55-13:12] There is a discussion on parcel sounds (from objects and avatars). In short: parcel sounds can be restricted from within a parcel going out, but not for sounds from other parcels coming in. As parcel privacy is two-way, this has caused some confusion for some users. The Lab is hesitant to change the scope of the setting (e.g. make it apply to both sounds leaving and entering a parcel), they are possibly amenable to changing the setting to sounds are restricted to parcels by default, or receiving a code contribution / feature request to separately stop sounds entering a parcel.

Sansar Public Beta

[17:58-18:50] During the meeting, Oz Linden indicated that the Sansar public Creator Beta will be launching on Monday, July 31st / during the week commencing Monday, July 31st.

However, during the week of July 24th through 28th, the Lab hosted a series of meetings involving staff, those of us fortunate enough to be in Sansar and creators and bloggers from Second Life who have not been a part of the Creator Preview in which it was indicated the company wasn’t ready to provide a date for the public Beta, other than it being “soon”.

We’ll obviously find out what is happening on Monday, but either way, I’ll have at least one article on Sansar and Second Life coming out quite soon which will delve into the meat of the meetings – so to speak!

Lab’s Second Life Development Summit

[18:00-18:42] Week 31 (commencing Monday, July 31st) marks the next Second Life summit in which Lab staff directly involved in running and maintaining SL (developer, product team, operations, etc), get together to plan out the next several months of work and projects for the platform.

In Brief

[19:51-20:36Further Estate Tool Improvements: the work to improve the Region / Estate floater in the viewer to make it easier to manage ban lists, etc., is still in-hand at the Lab, it’s just been delayed due to other work being carried out / completed.

[21:26-22:57] Premium “Goodies”: Grumpity Linden indicated there are some “awesome goodies” in the pipeline for Premium members, which sparked a lot of text-based suggestions.

[22:57-25:46] Group notices being dropped: there are issues around group notices being received by users in certain groups. The Lab has made adjustments to handling the UDP packets which handle group notices to try to improve things, but the issue of dropped notices is hard to reproduce, and therefore hard to diagnose and fix.

[26:51] Place Pages: have been updated to include events.