As always, please refer to the server deployment thread for the latest information and feedback.
On Tuesday, November 3rd, the Main (SLS) channel received the server maintenance package previously deployed to BlueSteel and LeTigre, comprising a simulator crash fix
On Wednesday, November 4th, all three RC channel received a new server maintenance package comprising a fix for group invite throttle notifications, and an internal server code clean-up.
Part of the code-clean up for the RC channels deployment appears to involve a fix for the region edge issue I reported on in my week 44 report, with Simon Linden commenting at the Simulator User Group meeting on Tuesday, November 3rd, “The new code going into RC has one interesting tool in it for our support staff … there’s a sim console command to reset the land edges to match what’s visible.”
SL Viewer Updates
The HTTP release candidate viewer (Project Azumarill) updated to version 3.8.7.306796 on Monday, November 2nd, possibly the last update before this viewer gets promoted to the de facto viewer in week #46.
Group Issues
There are a number of issues with groups the Lab is aware of, including those who are ejected from a group whilst in a group chat session still being able to continue using the chat until they close the window (see SVC-32), an issue not fully resolved when group bans were introduced.
This issue is one that the Lab is hoping to address, however, it is dependent upon some infrastructure changes being made, so may now happen in the immediate future, although the hope is to get the underpinning work done “pretty soon”.
On Tuesday, October 27th, the Main (SLS) channel received the same server maintenance package as deployed to the three RC channel in week #43. This comprised server-side improvements for delivering group notice attachments, together with some simulator crash fixes and code to prevent the uploading of “hacked” mesh content (e.g. mesh items which show a LI of 1 no matter what their actual complexity, or which spoof a creator’s name).
On Wednesday, October 28th, the RC channels should be updated as follows:
BlueSteel and LeTigre will each receive the same new server maintenance package, comprising a simulator crash fix
Magnum will also receive a new server maintenance package, comprising the same simulator crash fix as BlueSteel and LeTigre, together with a fix for a security issue.
SL Viewer
On Monday, October 26th, the Notifications viewer was promoted to the de facto release viewer. Version 3.8.6.305981 comprises the new floater for managing incoming notifications (Group, Transactions, Invitations, System).
The notifications viewer became the de facto release viewer on Monday, October 26th.
Also on Monday, October 26th the HTTP release candidate viewer updated to version 3.8.6.306549, which includes fixes for the following issues:
MAINT-4952 Replace old http client usage with llcorehttp
In my last project update, I reported on asset upload validation checks, and the Lab’s work to move them from the viewer to the simulator (e.g. checks to ensure animations are what they say they are, further checking of mesh uploads to ensure they can be rendered correctly, etc.). the overall aim is to improve reliability in SL. However, one impact of these changes is that once implemented, it will no longer be possible to upload mesh items weighted to attach points.
While the Lab has never regarded this as a supported means of handle mesh attachments, it has nevertheless been widely used by creators in SL. So much so that when Firestorm removed the ability to upload such meshes from their viewer, it caused some consternation (see FIRE-17144).
Once implemented, the change should only affect new uploads of mesh weighted to attachment points, and not affected any items already uploaded and in use within SL. However, a JIRA on the matter has now been raised directly with the Lab on the issue (see BUG-10543), in which creators who use the technique can provide input and use-cases to the Lab on why the approach is used (if you wish to leave a comment, please try to make it constructive and provide examples / reasoned argument, as these work better than rants).
Commenting on the JIRA, Troy Linden has said, “We realise that there will be a lot of residents/assets affected by this and currently looking into it. Sorry for the inconvenience and worries it caused. Please stay tuned.”
Adding to this at the Simulator User Group meeting on Tuesday, October 27th, Oz Linden said:
Up until a couple of days ago, we didn’t know anyone was doing it. You’d be amazed how often that happens. So please allow us a little time to digest what we’ve been learning. It’s actually one of the great things about working on SL. While we don’t always love how people apply their creativity, we all just love seeing how creative our users are.
Region Edge Issues
There has been an occasional problem encountered along the edges of simulators in which, when terraforrming and using the Smooth option, a very thin “cliff” can be raised into the air (see BUG-5979), or where when an avatar gets close to a region, it falls through a “gap” (see BUG-242).
Various reasons have been identified for this – as noted in the bug reports – and at the Simulator User Group meeting, Simon Linden said, “I think I found the cause of those odd region edges you sometimes see, where you can’t move in the areas you think you should at the edge of the land. It’s actually not the navmesh [which had been thought to be one of the possible causes] but it is the physics shape of the land.”
Essentially, the simulator, where no longer connected to a region which might once have been next to it, gets confused as to what it thinks should be alongside it. It then attempts to match the physics and land to what it thinks is in the “neighbouring” region, rather than simply resetting itself.
The following notes are primarily taken from the TPV Developer (TPVD) meeting held on Friday, October 23rd, 2015. A video of the meeting is included at the end of this report, and time stamps to it are provided. My thanks as always to North for the video recording and providing it for embedding.
Deploys for Week #43 – Recap
There was no deployment to the Main (SLS) channel on Tuesday, October 20th. On Wednesday, October 21st, all three RCs should receive the same new server maintenance package, which include server-side improvements for delivering group notice attachments, together with some simulator crash fixes and code to prevent the uploading of “hacked” mesh content (e.g. mesh items which show a LI of 1 no matter what their actual complexity, or which spoof a creator’s name).
SL Viewers
[00:00] There have been no further updates to the SL viewer currently at RC or project status since the update to the Quick Graphics RC, which I covered in part 1 of this week’s report.
It still appears as if the next RC that will be promoted to a de facto release status will be the Notifications RC viewer (currently version 3.8.6.305981), although this won’t be confirmed until the start of week #44.
[00:37] As previously noted, the Quick Graphics and HTTP viewers still have some problems which need to be addressed before either one can proceed forward to release status. So except to see further RC updates to both of these in due course.
[00:54] The Chromium Embedded Framework project viewer is getting some good public testing, and is expected to get some “fairly frequent” updates.
[01:05] The Oculus Rift viewer currently isn’t being worked on, as attention is focused on the other viewers in the various pipelines. However, it is hoped work will resume “pretty soon”.
[01:20] There are further bug fixes and updates in progress, including some fixes for some minor regressions in the current release viewer, and these are getting fast-tracked and will be appearing in a further Maintenance RC in due course.
[15:00] It is likely that when the next release viewer promotion takes place (Notifications viewer), the Lab will block a series of much older versions of their viewer, but not the Obsolete Platforms viewer, version 3.7.28.300847, which they still plan to have available until such time as it simply stops working – although it should be remembered this viewer is already regarded as unsupported by the Lab.
Grid Status Page RSS Feed
[01:58] In week #31 I reported that the Lab are working to update the Grid Status page RSS feed from RSS version 1 to RSS version 2. As existing web pages, etc., using the feed may need to be adjusted to accept the updated feed format, a proxy URL (http://beta.status.secondlifegrid.net/feed) has been available for testing purposes. It now appears that the switch will be thrown very soon to make the new version 2 RSS feed the default.
Inventory Updates
[02:15] As per the Lab’s recent SL improvements blog post, the first set of viewer-side updates aimed at improving inventory reliability are being queued up ready to appear in the project viewer. however, these are unlikely to appear before the HTTP updates currently in RC reach a release status. Once the viewer-side changes appear, the Lab will start work on deprecating the outdated inventory messages which are no longer required on the simulator side.
Server-side Upload Checks for Assets
[02:50] As a part of their overall work in trying to improve reliability within second Life, the Lab is looking to add further server-side checks to ensure viewers aren’t attempting to do anything the simulators aren’t equipped to handle, or which may cause unintended issues or outcomes, etc., for users.
This work will see the introduction of further validation checking on various asset types at upload. An example of this might be checks to ensure animations are what they say they are, further checking of mesh uploads to ensure they can be rendered correctly, etc.
[08:00] In particular with mesh, a simulator check is to be introduced to prevent the upload of mesh items weighted to all attachment points. There has been a viewer-side check designed to prevent such mesh items from being uploaded, which will in future be enforced simulator side. Existing content weighted in this way should not be “broken”, appearance-wise; but the simulator check will ensure such meshes cannot be uploaded (see FIRE-17144 for further details).
Attachment Point Validation
[04:20] Alongside the above, the Lab will be looking to introduce simulator-side attachment point validation checks. Right now, it is possible to have the viewer attach items to an avatar using points with are not correctly defined on the avatar skeleton. This can result in other viewers showing the item randomly attached to the avatar (or possibly floating nearby – for those with long memories, think of the Emerald “attachment tail” which multiple attachments were introduced on that viewer without adequate simulator support).
To prevent this, the Lab are looking to enforce the recognised attachment points simulator-side, so that any attachments using an incorrect attach point will be mapped to a defined attachment point, and moved there. This work will be carried out incrementally, and will initially offer the changes to Aditi to allow for testing.
Other Items
Firestorm is now in Feature Freeze in preparation for the next release, which is now undergoing testing. Assuming no major blockers are uncovered, the next release should be available in November. This will be up-to-date with the Lab’s current release viewer (3.8.5.305531 – Maintenance release), and may included some additional items cherry-picked from upstream of that release. However, it is unlikely to have the new notifications updates, HTTP, Quick Graphics, etc.
There was no deployment to the Main (SLS) channel on Tuesday, October 20th.
On Wednesday, October 21st, all three RCs should receive the same new server maintenance package, which include server-side improvements for delivering group notice attachments, which is hoped will provide some improvements to issues of trying to open such attachments from the group notice notifications.
The update also includes some simulator crash fixes, and while not advertised in the release notes, should also include simulator-side code to prevent the uploading of “hacked” mesh content (e.g. mesh items which show a LI of 1 no matter what their actual complexity, or which spoof a creator’s name).
The Quick Preferences RC viewer was updated to version 3.8.6.305942 on October 19th, most likely bringing it up to parity with the current release viewer.
Commenting on the status of the current RC viewers during the open Source Developer’s meeting on Monday, October 19th and in relation to any possible viewer promotion in week #44, Oz Linden said, “The Quick Graphics viewer has too many bugs to promote; mostly minor, but there’s no reason to go forward with it yet. HTTP has a couple of significant bugs to fix. So if I had to bet on the next promotion I’d guess the Notifications viewer.”
Expect more news on viewers following the TPVD meeting on Friday, October 23rd.
Other Items
A question was asked during the Simulator User Group meeting on Tuesday, October 20th about making script time for in-world objects in a parcel available to the parcel holder (e.g. an extension of the region-wide Top Scripts functionality available to region / estate owners / managers). The idea here being to help parcel owners better understand how in-world objects within their parcel might be impacting simulator resources (see this forum thread and SCR-60 as well).
Commenting on the idea, Simon Linden said:
Since we send it to the viewer already, you have access to the info … You can probably get some of that info by scanning but I can see it would be a lot easier to get that in a different format. One of the limits (I’m sure you all know) of LSL is it’s not good with large data sets, so I’m not sure how you can expose that info for a large number of items … so maybe the top 10 or 25 objects on a given parcel? I can totally see the need for that to manage land.
… HUDs and attachments add a twist to it … they would probably be lumped together with the AV as they are now. I wonder if it would be easy to add something like PARCEL_SCRIPT_TIME to llGetParcelDetials()? that doesn’t give you what you want, but is a higher level for finding the parcels with the most scripts.
Those looking for such a capability are advised to raise a JIRA feature request.
As noted in part 1 of this week’s update, there were no deployments to either the Main (SLS) or RC channels in week #42. This means there will be no deployment to the Main channel in week #43 (week commencing Monday, October 19th).
The RC deployment planned for week #42 was ultimately postponed pending further work.
Week #43 RC Deployment
The deployment originally planned for week #42 is now targeted for deployment on Wednesday, October 21st. It will include the server-side updates intended to help prevent attachment links in group notifications from going stale. However, as per the first part of this week’s report (see link above), because it is also believed that there are some viewer-side issues causing these problems, the update might not fix all situation where an attachment to a group notice fails to open when viewed in the incoming notification.
The reason for the late postponement of the release in week #42 was because the Lab wasn’t entirely happy with aspects of the update, and as a result of recent issues in attempting to deploy some server-side improvements to inventory handling which lead to unanticipated side-effects, caution won the day.
This RC release may also incorporate a server-side update to prevent the uploading of “hacked” mesh content (e.g. mesh items which show a LI of 1 no matter what their actual complexity, or which spoof a creator’s name).
SL Viewer
The Notification RC viewer updated to version 3.8.6.305981 on Wednesday, October 14th, following the promotion of the Maintenance viewer to the de facto release viewer. This release adds a further set of fixes to the viewer:
MAINT-5370 [Notice] Hard to operate with notification window control
MAINT-5694 [Project Notice] Links on the 1st line of a group notice message are sometimes not clickable.
MAINT-5708 [Project Notice] Images shouldn’t be displayed in notification titles
MAINT-5730 [Project Notice] Text overlapping in group invites with non-default group role.
The notifications viewer update corrects four issues within the viewer relating to layout and control of notifications
European Connection Issues
Thursday, October 15th saw a DNS issue upstream of Second Life impact users in Europe, and notably The Netherlands, preventing people from logging-in to SL and from accessing web properties such as secondlife.com and the Marketplace, etc. The situation occurred as the Lab were also carrying out some unscheduled maintenance on the grid, prompting them to update the original Grid Status report with news on the DNS issue:
[Updated 4:27 PM PDT, 15 October 2015] We’ve received confirmation from our upstream provider that a widespread DNS outage is impacting parts of Europe, including the Netherlands, and possibly other regions. We’re monitoring the situation and will report on developments.
There was no deployment to the Main (SLS) channel on Tuesday, October 13th. Also, the server maintenance package planned for deployment to the three RC channels in week #42 did not pass final testing, and the deployment has been postponed as of LL CoB on Tuesday, October 13th.
SL Viewer Updates
The Maintenance viewer, version 3.8.5.305531, was promoted to the de facto release viewer on Tuesday, October 13th. The viewer has over 90 fixes, improvements and feature requests – please refer to the release notes (link above) for details.
Group Notice Attachments
The server maintenance package postponed from deployment in week #42 comprises a partial fix for group notice attachments failing to respond when clicked, forcing recipients to open the Group floater, search the Notices tab for the relevant notice and retrieve the attachment from there.
This problem most frequently occurs as a result of some of the data relating to the notification and the original notification users open to access the attachment being lost to the simulator after about an hour, and the fix in question is geared to correcting this. However, there can be other circumstances where the attachment fails to respond, which might be either server-side issues, or viewer issues.
“I’m pretty sure there’s a viewer issue too,” Simon Linden said, discussing the matter during the Simulator User Group meeting on Tuesday, October 13th. “If you notice when you click on notice attachments, sometimes when it doesn’t work you see nothing at all; no UI feedback where that attachment shading changes or anything. I think in those cases the viewer is dropping the click and not doing anything at all, and it might work if it actually sent a request to the server.” He added that he hopes the RC deployment, once available, will at least improve matters.
Other Items
Accessing Crowded Regions
A question was asked at the Simulator group about whether repeated attempts to access a crowded region by repeatedly trying to cross to it from neighbouring region or constantly trying to teleport into the region, causes it any significant issues in terms of performance. There are already some HUDs on the marketplace which will automatically re-try teleports to a specific location, and the question was prompted by a feature request to Firestorm to make a similar capability within the viewer.
Simon Linden was understandably cautious in his response, pointing out that any attempts to move between regions will use up time and network traffic, but suggested that a retry system with a delay of around 5 seconds or so shouldn’t put any stress on the simulator (some of the available HUDs actually work on a faster retry cycle than this, so could be having more of an impact). It was also suggested – although not by a Linden – that moving such a capability client-side might actually reduce a small load from the simulators when compared to using a scripted HUD.
This question prompted the idea of some kind of list managed queuing system for crowded regions, so that people attempting to reach it would be automatically teleported when there is room, and in the order they were added to the queue.
However, as was pointed out in the meeting, such an approach would require a considerable amount of work in terms of just managing the list (keeping track of who is on the list, who joins the list, who opts to log-out before a space becomes available, forcing an update to the entire list, removing people who are teleported from the list, and so on). As such, Simon indicated he’d rather work on improving simulator performance to allow more avatars into a region, and have the viewer improved so it can better handle large numbers of avatars.
Special Events Regions
In a similar vein, there have been rumours of large events getting dedicated servers to handle the load more effectively. Commenting on this, Simon said, “We’ve talked about that, and I don’t know what might happen; but we’ve discussed the short-term need people have for regions or better performance. It’s now more in the product folks lap … they need to figure out how we would offer something like that.”
Temp Attach from Object Inventory
A question was asked at the meeting about whether anyone at the Lab was working on a means to allow a temporary attachment directly from an object’s inventory (contents). Simon noted that while no-one is currently working on this, it is on the Lab’s feature list, adding, “it’s just an idea; but the current attach process used by experiences is rez the thing, then attach. It would be better to just directly attach.”
Sorrow, Paper Dinosaurs (Flickr) – blog post
Inara Pey: Living in a Modemworld 