SL project updates week 24/1: server; VMM; group list changes

Goatswood; Inara Pey, June 2015, on Flickr Goatswood (Flickr) – blog post – visit soon, closes June 19th, 2015

Server Deployments Week 24

As always, please refer to the server deployment thread for the latest updates / news.

  • On Tuesday, June 9th, the Main (SLS) channel received the server maintenance package previously deployed to the three RC channels, comprising:
    • Change logic on accessing group member lists for large groups – please see more below
    • Internal server logging changes
  • On Wednesday, June 10th, the three RC channels should all receive a new server maintenance package comprising further Internal server logging changes.

Group Member List Changes

The “Change logic on accessing group member lists for large groups” refers to how the members list for groups with more than 5,000 members are now handled. A full explanation of the change and the reasons behind it can be found in my blog post on the matter. however, in short:

  • Groups with 5,000 or more members will no longer display the list of members unless:
    • You are assigned the Owner or Officer role within the group
    • You are assigned an ability within the group which requires the members list to be displayed (e.g. you are able to assign members to assigners roles, or are able to eject / ban people from the group, etc.)
  • Instead, and until corresponding changes are made to the viewer, all you will see on opening the members list as a message stating “Retrieving member list (0 / XXXXX)” – where XXXXX is the total number in the group,

The has already caused concern over how the change may be perceived as a functional breakage – see BUG-9393. In addition, two further issues resulting from the change have been reported:

  • BUG-9404: “Group members of large groups in a role which has “Invite people to this group” ability are not able to send group invites” (initially filed against the RC regions when the change was deployed in week #23, and now applicable to the grid as a whole)
  • BUG-9428: “Users using older viewers are unable to leave groups with >5k members on regions running 15.05.28.302161”

Scripting Memory Limits

A request was made for the Lab to consider allowing llSetMemoryLimit to request up to 128kb or 256kb of memory (whichever is more feasible), with a performance penalty for scripts using less than 50% of the memory requested – see BUG-9382.

The arguments for such an increase are not new; many coders run into problems with utilising memory for both code storage and code operation, resulting in having to write inefficient scripts and additional operations to communicate between scripts. A similar request has also been put forward (see BUG-8761), but which limits the additional memory allocation purely to Experiences on the ground that offering increased memory to all could lead to performance and other issues.

Commenting on the request at the simulator User Group meeting on Tuesday, June 9th, Simon Linden said:

I don’t think we’d want to limit performance … that seems like it would get into odd rules and conditions. Plus that’s likely to be in a place where we don’t want to add more code. FWIW, when you have lots and lots of scripts in a region, the time spent rotating through all the scripts becomes significant, so your script time isn’t just running scripts.

Oz Linden then added:

One of our frequent themes this year has been to look at various limits and consider making them better … perhaps we can look at the memory limit at some point too. One of the things I hope will happen as Experiences are adopted is that some of the code that’s being used to manage saving state and communicating can be replaced by simpler code to use the key/value store.

This drew agreement from Simon, who then continued:

I suspect larger script size has been limited by not having script memory limits. But at a smaller scale, it’s easy to add more scripts, so perhaps doubling or a bit more won’t really make it any easier to hog memory.

This doesn’t automatically mean that script limits will change in the future; as Simon also pointed out, script memory is the largest part of each avatar load, and can have an impact on things like physical region crossings and teleporting, which would likely have to be be considered. However, script memory is likely to get added to “The List” of things the Lab is looking at.

Viewer-Managed Marketplace

Some confusion has been evidenced over the use of the term “archived” in recent communications from the Commerce Team regarding what will happen during the auto migration process, and particularly with reference to items that have not seen sales in over a year.

  • The first point to remember is that any “archiving” will only occur for those merchants who have more than 5,000 items on the Marketplace when the auto-migration process reaches them. As noted in my last VMM update, all such affected merchants have already been notified
  • From information made available by those merchants so affected, it would appear that “archived” means “items will be returned to the merchant’s Received Items folder”. Firstly, any items the Merchant has stored on the Marketplace without any associated listing will be returned. If this fails to reduce their total count to below 5,000, then those items which have not seen sales for over a year will be unlisted and the items again returned to the merchant’s Received items folder.  From this it would seem that there will not be any actual “archiving” of listing information or items on the part of the Lab.

 

Second Life project updates 22/2; server and viewer

Obedience, LEA 1 - blog post
Obedience, LEA 1blog post

Server Deployments, Week 22 – Recap

The planned RC deployment scheduled for Wednesday, May 27th was rolled back as a result of a back-end issue. This currently leaves grid as a whole on the same server release.

Commenting on the roll-back at the Server Beta User Group (SBUG) meeting on Thursday, May 28th, Simon Linden said, “there was a minor issue but it was worth reverting; some internal tools weren’t running right and sending postcards was broken. [However] that code will likely be back next week, [as] I’ve already fixed the bug.”

These issues aren’t related to the region restart issues / caps failure people have noticed with some regions following a rolling restart, and as reported in my week 21/2 report, and which Simon indicates have yet to be looked into in-depth.

SL Viewer

Thursday, May 28th saw the Avatar Layer Limits viewer, version 3.7.29.301305, updated to the de facto release viewer. This viewer removed the limit of only being able to wear a maximum of 5 items per clothing layer (e.g. a maximum of 5 jackets and 5 shirts and 5 pants, etc), with a global limit of 60 layers which can be worn in any combination (e.g. you can wear 58 jacket layers, a tattoo layer and a pants layer if you wish).

This leaves two RCs in the release channel at present: the Avatar Attachment fixes RC (aka Project Big Bird and currently version 3.7.29.301943), and the Experience Keys viewer (currently version 3.8.0.300963, and which is awaiting the completion of back-end updates to the Experience Keys services). Both of these viewers will be updated to match the new release viewer, and it is anticipated that they will be joined by a new Snowstorm RC viewer in the near future (see below), which is currently awaiting some fixes prior to release.

General

Project news coming out of the Lab is a little light at the moment. This shouldn’t be taken to mean there isn’t a lot happening with Second Life. There are several projects that are in the pipeline – Viewer-Managed Marketplace and Experience Keys (/ Tools) being two that people are aware of.

The Lab don’t talk too much ahead of time as to what is going on, but it’s clear to see from Simon’s back-end work around avatar counts in regions, that there are various things which are being looked at. Again, we only recently had it confirmed that the Lab have, as a part of continuing work on improving the CDN services, shifted to another provider – and they are looking to move the delivery of more asset types to the CDN in the coming months.

In the meantime, we can expect to see more RC viewers appearing  – notably the next Snowstorm RC viewer with Avatar Complexity, and which should include STORM-2082, the ability to save and load graphics settings to assist with viewer performance, depending on the environment you’re in.

Jonathan Yap is working on the ability to various graphics settings in the official viewer, allowing users to quickly change between saved settings depending on their performance needs - this should be appearing in an upcoming Snowstorm contributions viewer (note the finished panel may not resemble the one shown left, above)
Jonathan Yap is working on the ability to various graphics settings in the official viewer, allowing users to quickly change between saved settings depending on their performance needs – this should be appearing in an upcoming Snowstorm contributions viewer (note the finished panel may not resemble the one shown left, above)

 

Second Life project updates 22/1; server, viewer, avatar rendering

Stand: Relay D'Alliez
Stand, Relay D’Alliez – Relay for Life Exhibit – blog post

Server Deployments, Week 22

Update, May 28th: a back-end issue with the RC deployment has meant that all three RC channels have been rolled-back to the their previous release, leaving the grid as a whole on the same server release.

As always, please refer to the server deployment thread for the latest updates / news.

There was no deployment to the Main (SLS) channel on Tuesday, May 26th, due to there having been no RC deployment in week #21.

On Wednesday, May 27th, all thee RCs should receive a new server maintenance package, comprising:

  • A change logic on accessing group member lists for large groups
  • Internal server logging changes.

SL Viewer

Due to Monday being Memorial Day in the United States, the Lab was closed for normal office business, and there was no meeting to discuss potential RC viewer promotions. However, the most recent update to the Attachment Fixes RC viewer (Project Big Bird, currently version 3.7.29.301943) is showing a must reduced crash rate compared to the previous release (and which prevented it from being promoted to the de facto release viewer).

The crash rate is still slightly high than for the current release viewer, but speaking at the Simulator User Group meeting on Tuesday, May 26th, Oz Linden described it as “probably not a statistically significant difference”. Whether this means the viewer will be promoted to release status later in the week or not, remains to be seen.

Increasing the Number of Avatars Per Region

Simon Linden: looking at ways an means to make it easier for the simulator and a viewer to better handle large numbers of avatars
Simon Linden: looking at ways an means to make it easier for the simulator and a viewer to better handle large numbers of avatars

“There’s one change that I will follow up on … I added a way so I can adjust the ‘max avatars in a region’ setting.  I’d like to do an experiment soon and see what falls apart if we can get over 100 people into a region,” Simon Linden said at the simulator UG when discussing the upcoming RC deployment.

“This is purely experimental and there are no plans for changing the SL limits,” he went on. “But sometimes regions hit 100 [and] it would be nice if the viewer and simulator handled that better.”

There is already an additional means en route to the viewer by which users can have greater control over how avatars around them in a region are rendered by the viewer, Avatar Complexity, when will draw avatars above a rendering limit set by the user as a solid colour (the so-called “Jelly Baby” avatars).  The will work alongside the existing Avatar Imposters capability already in the viewer.

However, in terms of his experiment, Simon suggested that one way to improve things might be for the viewer to simply not draw everyone within a region; although how this would work, and the criteria used to determine what avatars are drawn and which aren’t, does require careful consideration. Simon suggested the viewer might simply skip drawing those avatars that are furthest away once a threshold number of avatars in the region has been reached. Another (suggest by a meeting attendee) would be for the control to be via the Max Number of Avatars settings within the viewer – so that once exceeded, avatars are again simply not rendered.

As noted, Simon’s work is purely experimental, and primarily aimed at helping the Lab understand what might be done to improve things where there are large gatherings of avatars, and to perhaps try out one or two ideas based on what they learn.

Simon’s Rendering Tricks

As a part of the discussion on avatar rendering, Simon handed out a note card of tips and trick for improving your performance when dealing with complex avatars. While this includes the debugs which will form a part of the new Avatar Complexity functionality, which will be appearing in a a Snowstorm RC viewer soon, as well as suggestions which may already be known, I’m including his suggestions in full here for reference:

From Advanced > Show Debug Settings, set:

  • RenderAutoHideSurfaceAreaLimit   0
  • RenderAutoMuteByteLimit  0
  • RenderAutoMuteFunctions  7
  • RenderAutoMuteLogging  False
  • RenderAutoMuteRenderWeightLimit  350000
  • RenderAutoMuteSurfaceAreaLimit  150

In preferences / graphics, change “Max # of non-imposter avatars” to something like 8. Also try ctrl-alt-shift-4 to hide avatars, or ctrl-alt-shift-2 for alphas.

Note the two debugs shown in green are those related directly to Avatar Complexity and drawing avatars as “Jelly Babies”. Note that RenderAutoMuteFunctions must be set to 7 in order for this to work. Also note that the RenderAutoMuteRenderWeightLimit of 350,000 is purely an advisory starting point. The Lab estimate that this will reduce the very top 3% of very rendering-intensive avatars as solid colours. You may find you have to set the value somewhat lower in certain environments  – such as night clubs and dance venues – in order for it to be effective. I’ve personally found that 150-200K tends to be required in very busy ballrooms, etc.

Second Life project updates 21/2: general notes

Living in a Bowl
Living in a Bowl, May 2015 – blog post

Server Deployments, Week 21

As always, please refer to the server deployment thread for the latest updates / news.

On Tuesday, May 19th the Main (SLS) channel received the server maintenance package previously deployed to the three RC channel, comprising Internal server logging changes, back-end system bug fixes and a change to Reply-To e-mail addressing on snapshots. There were no RC deployments on Wednesday, May 20th.

SL Viewer

The Attachments Viewer RC (Project Big Bird) was updated to version 3.7.29.301943 on Thursday May 21st. As noted in part 1 of this week’s report, the initial RC release of this viewer had an elevated crash rate compared to the current release viewer, including a crash-on-exit bug, so this release will hopefully address those issues.

Group Chat

A fix for issues around BUG-9130, where some people were unable to see any posts in some or all of there group chats, including their own posts, while everyone else in the same group could see their posts, has started to be deployed across the chat servers, and should be completed on Friday, May 22nd.

“The chat servers got stuck with bad info about where the sender was, so the messages never reached them,” Simon Linden said at the Server Beta User Group meeting on Thursday, May 21st, reiterating an explanation given at a recent Simulator UG meeting. “And unfortunately it wouldn’t fix with relogging or even a chat server restart.”

“Loading…” Issue with Names in Group Chats

This is a viewer-side problem which causes avatar names to appear as “Loading” under certain circumstances in group chat (see BUG-3829 and STORM-2114). A contribution by Ansariel Hiller is currently with the Lab and is expected to be released as a part of the next Snowstorm contributions viewer, which is expected to appear soon.

Other Items

Region Restart Glitch

There has been something of a rise in reports of regions experiencing issues following recent following restarts – most noticeably caps failures. This is something the Lab is looking into, and Simon commented, “we have a suspicion that after rolls, as that server host starts up regions, it’s doing enough of them at about the same time that things get overloaded.   It’s still a theory but makes some sense why we’d get cap failures like that.”

SL project updates Week 21/1: server, viewer CDN change, SL network update

WindWept, Dolly; Inara Pey, May 2015, on Flickr Windwept (General) May 2015 (Flickr) – blog post

Server Deployments, Week 21

As always, please refer to the server deployment thread for the latest updates / news.

On Tuesday, May 19th the Main (SLS) channel received the server maintenance package previously deployed to the three RC channel, comprising:

  • Internal server logging changes
  • Back-end system bug fixes
  • Reply-To email changed in postcard sends

As previously noted in these pages, the “reply-to email changed in postcard sends” relates to changing the way snapshots forwards to e-mail are handled. Until now, the Lab has substituted the user’s e-mail address in the “from” field of snapshots sent to e-mail, rather than displaying the “secondlife.com” address.

However, this added to issues of e-mail originating from “secondlife.com” being treated as spam by a/v software and ISPs. With the new format employed with this change, the sender’s e-mail address is given as the “reply to” address in the snapshot, and the “from” is “no-reply@secondlife.com”, thus avoiding the issue of LL looking like spammers who are forging invalid addresses.

There will be no RC deployment on Wednesday, May 20th.

SL Viewer

The week has not so far seen an RC viewer promoted to release status. If there is any promotion, it would most likely be the Layer Limits RC (currently version 3.7.29.301305). The Experience Tools RC viewer is still awaiting the completion of back-end work, while the Attachment Fixes RC (Project big Bird) currently has an elevated crash rate compared to the current release viewer, which includes a crash-on-exit bug, so further work is required on that RC.

CDN Provider Move

The Lab has been moving between CDN providers, and as a result, some people may have been experiencing particular texture / mesh / avatar rendering delays of late. Commenting on the process at the Simulator User Group meeting on Tuesday, May 19th, Oz Linden said:

We’ve just finished moving from one CDN provider to another, and it may take the caches a little while to catch up. We tried to do it gradually in a way that would be minimally disruptive, but when you’re dealing with as much data as we are, there are no perfect solutions.

One of the cases it is hoped the move will assist is with SL users in Florida (and neighbouring states) in the US who use Mediacom as their ISP, and who have found that there have been what appear to have been issues with Mediacom throttling the service at certain times of the day. Preliminary feedback from users so affected who have been involved in testing with the new CDN provider has been positive.

What Goes Through the CDN, And How

During the CDN conversation Oz reinterated the data that is currently delivered to the viewer through the CDN: textures, world map tiles, avatar baking data, and mesh data. In terms of in-world objects, two distinct operations are taking place:

  • Where an object is, how big it is, and so on, comes to the viewer via the simulator, together with the UUIDs fr the relevant objects / textures
  • The viewer then uses the UUIDs to fetch the mesh and texture data directly from the CDN.

As previously noted on these pages, this should mean faster loading of things like textures and mesh in-world, as the data is coming from a CDN node that is “local” relative to you, rather than coming to you from the Lab and through the simulator itself. However, experience is showing that for a small number of people, this isn’t always the case, and there can be situations where mesh and texture loading aren’t what might be expected. However, the Lab continues to try to improve things.

Second Life Network Architecture

Writing on the forums, noted SL photographer Jackson Redstar recently asked meshmaxconcurrentrequests – does anybody know the real setting? In the ensuing debate, Monty Linden offered an updated overview of the SL network architecture.

Monty Linden's updated SL network diagram
Monty Linden’s updated SL network diagram

To borrow from Monty’s explanation:

  • On the left, in red, are pieces of the viewer; on the right, in blue are simhost/simulators and other backend services; at the bottom (green) are new CDN services
  • Solid lines with arrowheads are communication paths, either UDP or TCP/HTTP; dashed lines are legacy communication paths that are now or soon will be deprecated, obsoleted and/or deleted
  • Sold ball-and-stick indicators (e.g. TextureFetchConcurrency) indicate a viewer debug setting and the communication path or paths that setting influences; dashed ball-and-stick indicators (e.g. MeshMaxConcurrentRequests) indicate obsolete debug settings.

Monty goes on to say:

Generally, things are moving in the direction of simplification and less resource conflict.  The mesh and texture HTTP traffic, which is usually the greatest load, tends to part ways with the UDP traffic a few network hops after a user’s router or modem.  Lacking TCP’s throttling mechanism, UDP often wins in a fight (give-or-take the efforts of fairness algorithms along the path).  Allowing UDP to overrun the path between viewer and simulator does still degrade the experience and the bandwidth setting remains an effective tool for avoiding this problem.

Other settings should generally be left alone.  A lot of bad advice was spread around in the community in an effort to work around throughput problems.  We’re trying to undo that history and get back on track with more typical (albeit aggressive) HTTP patterns.

 Viewer Caching

During the Simulator UG meeting, Oz repeated a call he originally made at a TPV Developer meeting recently, asking that if there is developer wishing to volunteer for a “deep dive” into viewer caching, he’d like to hear from them.

While interest list updates made key changes to how the viewer’s cache is used, there are numerous issues which appear to be viewer-side caching related, so a deep investigation into the code could go further towards improving things.

One long-standing issue, which is thought to be caching related, is If someone uses a texture rezzed in-world same texture for a group profile image or their avatar profile image or in a profile pick, the object will never fully load the texture.

So, if you’re a developer willing to looking into viewer-side caching, Oz would like to hear from you.

SL project updates week 20/1: Server and viewer; outfits

Join Hands: raising money to help the WFP's aid work in earthquake-struck Nepal
Join Hands: raising money to help the WFP’s aid work in earthquake-struck Nepal with Fashion for Food – May 13th through May 16th inclusive

Server Deployments, Week 20

As always, please refer to the server deployment thread for the latest updates / news.

There was no main (SLS) channel deployment on Tuesday, May 11th. On Wednesday, May 12th, the three RC channels were all updated with a new server maintenance package, comprising:

  • Internal server logging changes
  • Back-end system bug fixes
  • Reply-To email changed in postcard sends

As noted in my TPV Developer meeting report for week #17, snapshots are sent via the “secondlife.com” domain, but use the sender’s own e-mail address as the originating address in the “from” field. This, and other ways ways in which e-mails flowing out from “secondlife.com” are handled has resulted in some ISPs regarding the domain as a spam domain, and have been pro-actively blocking it.

The “reply-to email changed in postcard sends” refer to a change made to how the “from” field in snapshots sent to e-mail (which the Lab refers to as “postcards”)  is addressed in an attempt to alleviate the problem.

At the time this issue was raised at the TPV Developer meeting, it was indicated that the Lab was considering removing the the snapshot to e-mail capability server-side. However, as was indicated in the meeting, doing so would break a number of wardrobe HUD systems which are popular among users. Whether or not this fix is an attempt to address the spam issue without having to remove the snapshot or e-mail functionality is currently unclear.

SL Viewer

Attachments Viewer (Project Big Bird)

The attachments viewer was promoted to release candidate status with version 3.7.29.301361, release on Wednesday, May 13th. This viewer provides a series of fixes for attachment-related issues, particularly when multiple attachments are added or removed at the same time. It also has enhanced logging, so the SecondLife.log file will have some additional lines related to avatar state in general and attachments in particular.

Linux

As noted in a separate report, the Lab have now blogged about seeking assistance from open source developers in the continued development and maintenance of the Linux flavour of the viewer.

 Outfit Folder Changes

A recent change to viewer functionality means that it is no longer possible to drag and drop sub-folders of items into the My Outfits  / Outfits folder – see BUG 9209 (FIRE-15603). This change, which is in the official release viewer, is filtering out into various TPVs, and has been causing some consternation.

While it is still possible to create sub-folders within My Outfits / Outfits and drag and drop items into them, many people have tended to simply unpack new clothing items into a default folder and drag that folder (or the Direct Delivery folder for the clothing) into My Outfits / Outfits, and then sort the contents into suitable outfits from there. Others have used the Appearance floater to create outfits, save them, and then drop them into My Outfit / Outfits – which also is no longer possible.

It’s unclear precisely what problems can occur in allowing drag-and-drop in My Outfits, although it appears drag-and-drop into My Oufits was never intended to be allowed. The change itself was made by Vir Linden, shoe has most recently been working on a range of improvements to try to reduce issues of inventory loss; he is now involved in the JIRA discussion, seeking to understand use cases relating to dragging-and-dropping folders into  My Outfits.