Category: SL User Group Meetings

SL project updates 23/1: server, TPV Developer meeting

Posted on by Inara Pey

Crestwick Island; Inara Pey, June 2015, on Flickr Crestwick Island (Flickr) – blog post

The following notes are primarily taken from the TPV Developer (TPVD) meeting held on Friday, June 5th. A video of the TPVD meeting is included below, with any time stamps in the following text referring to it. My thanks as always to North for the recording and providing it for embedding.

Server Deployments Week 23 – Recap

There was no deployment to the Main (SLS) channel on Tuesday, June 2nd, due to the week #22 RC deployment being rolled-back.

On Wednesday, June 3rd, all thee RCs received the same server maintenance package, which was the same update which had been attempted in week #22, but with additional back-end fixes to prevent a repeat of the earlier problems. The update comprises:

  • A change logic on accessing group member lists for large groups
  • Internal server logging changes.

SL Viewer

On Thursday, May 3rd, the attachment fixes RC viewer (Project Big Bird) updated to version 3.7.30.302190.  Presumably, this update is to bring the viewer up to parity with the current release viewer (formerly the avatar layers update), and so will hopefully clear the way for it to be promoted to the de facto release viewer in the near future.

[02:40] A new maintenance fixes RC viewer should be appearing in the release channel soon, the should also be an update to the Mesh Importer project viewer (currently version 3.7.28.300878) appearing soon; and work is progressing on updating the Oculus Rift project viewer.

Experience Keys (/ Tools)

[03:35] Work is continuing on the back-end of the Experience Keys systems prior to the capabilities being formally released across the grid as a whole.

“We are making progress on the back-end issues that have been delaying that,” Oz Linden informed the TPV Developer meeting. “But as far as I’m aware, we have not yet  … uncovered a viewer-related problem with that. so far these are all back-end related issues.” He went on:

To be a little bit more transparent, it’s really scaling issues. Experience actually work just fine at the scale we’re using them right now … but what we’re concerned about is what will happen when we turn them loose on a much, much larger population and there are lots more experiences running and there are a lot more simulators with experiences running on them. 

So what we’ve doing … is fairly intensive scaling and performance testing and we’re solving the problems that the testing uncovers. So, we’re making progress on it, and we’re fairly confident the problems are solvable, but we have run into a whole host of assorted issues with that.

In the meantime, it is expected that the current RC release of the Experiences viewer (currently version 3.8.0.300963) will be brought up to parity with the release viewer during week #24.

Viewer-Managed Marketplace

[05:53] For ease of reference, please refer to my update on VMM, available here.

Land Bans

[17:58] Oz Linden has issues an invitation to open-source contributors to assist the Lab in trying to improve the management of land ban lists. Again, for ease of reference, please refer to my separate report on this, available here.

Group Membership Changes

[22:15]  The update to the server RC channels referred to as “a change logic on accessing group member lists for large groups”, as noted in the server deployments recap at the top of this article, refers to a new way in which the members lists for larger groups (5,000 members and over) are handled.

In brief, the members lists for such groups will no longer load in the group floater in the viewer, unless the person attempting to see the list is in a role which requires they need to be able to do so. While this change is discussed within the meeting, I have been specifically asked not to blog on the change until it is deployed to the Main (SLS) simulator channel on Tuesday, June 9th, when I’ll have the details in full.

Unified Snapshot Floater

[33:12] As I’ve recently reviewed, NiranV Dean has updated the unified snapshot floater in his Black Dragon viewer. As he originally contributed the code for this floater to the Lab, he has also raised a JIRA (see BUG-9325) listing improvements and fixes and has contributed his updated code. This has been under review at the Lab, and commenting at the TPV Developer meeting, Oz indicated that the Lab are “fine” with the majority of Niran’s suggestions, although there are a couple that appear to be subject to further consideration, which might take a little while to sort through.

NiranV Dean's unified snapshot floater improvements, including the separate, resizeable preview panel, are now with the Lab, although it is not yet entirely clear which of the updates will be adopted, or when they will appear in the official viewer.
NiranV Dean’s unified snapshot floater improvements, including the separate, resizeable preview panel, are now with the Lab, although it is not yet entirely clear which of the updates will be adopted, or when they will appear in the official viewer.

Continue reading “SL project updates 23/1: server, TPV Developer meeting” →

Second Life project updates 22/2; server and viewer

Posted on by Inara Pey
Obedience, LEA 1 - blog post
Obedience, LEA 1 – blog post

Server Deployments, Week 22 – Recap

The planned RC deployment scheduled for Wednesday, May 27th was rolled back as a result of a back-end issue. This currently leaves grid as a whole on the same server release.

Commenting on the roll-back at the Server Beta User Group (SBUG) meeting on Thursday, May 28th, Simon Linden said, “there was a minor issue but it was worth reverting; some internal tools weren’t running right and sending postcards was broken. [However] that code will likely be back next week, [as] I’ve already fixed the bug.”

These issues aren’t related to the region restart issues / caps failure people have noticed with some regions following a rolling restart, and as reported in my week 21/2 report, and which Simon indicates have yet to be looked into in-depth.

SL Viewer

Thursday, May 28th saw the Avatar Layer Limits viewer, version 3.7.29.301305, updated to the de facto release viewer. This viewer removed the limit of only being able to wear a maximum of 5 items per clothing layer (e.g. a maximum of 5 jackets and 5 shirts and 5 pants, etc), with a global limit of 60 layers which can be worn in any combination (e.g. you can wear 58 jacket layers, a tattoo layer and a pants layer if you wish).

This leaves two RCs in the release channel at present: the Avatar Attachment fixes RC (aka Project Big Bird and currently version 3.7.29.301943), and the Experience Keys viewer (currently version 3.8.0.300963, and which is awaiting the completion of back-end updates to the Experience Keys services). Both of these viewers will be updated to match the new release viewer, and it is anticipated that they will be joined by a new Snowstorm RC viewer in the near future (see below), which is currently awaiting some fixes prior to release.

General

Project news coming out of the Lab is a little light at the moment. This shouldn’t be taken to mean there isn’t a lot happening with Second Life. There are several projects that are in the pipeline – Viewer-Managed Marketplace and Experience Keys (/ Tools) being two that people are aware of.

The Lab don’t talk too much ahead of time as to what is going on, but it’s clear to see from Simon’s back-end work around avatar counts in regions, that there are various things which are being looked at. Again, we only recently had it confirmed that the Lab have, as a part of continuing work on improving the CDN services, shifted to another provider – and they are looking to move the delivery of more asset types to the CDN in the coming months.

In the meantime, we can expect to see more RC viewers appearing  – notably the next Snowstorm RC viewer with Avatar Complexity, and which should include STORM-2082, the ability to save and load graphics settings to assist with viewer performance, depending on the environment you’re in.

Jonathan Yap is working on the ability to various graphics settings in the official viewer, allowing users to quickly change between saved settings depending on their performance needs - this should be appearing in an upcoming Snowstorm contributions viewer (note the finished panel may not resemble the one shown left, above)
Jonathan Yap is working on the ability to various graphics settings in the official viewer, allowing users to quickly change between saved settings depending on their performance needs – this should be appearing in an upcoming Snowstorm contributions viewer (note the finished panel may not resemble the one shown left, above)

 

Second Life project updates 22/1; server, viewer, avatar rendering

Posted on by Inara Pey
Stand: Relay D'Alliez
Stand, Relay D’Alliez – Relay for Life Exhibit – blog post

Server Deployments, Week 22

Update, May 28th: a back-end issue with the RC deployment has meant that all three RC channels have been rolled-back to the their previous release, leaving the grid as a whole on the same server release.

As always, please refer to the server deployment thread for the latest updates / news.

There was no deployment to the Main (SLS) channel on Tuesday, May 26th, due to there having been no RC deployment in week #21.

On Wednesday, May 27th, all thee RCs should receive a new server maintenance package, comprising:

  • A change logic on accessing group member lists for large groups
  • Internal server logging changes.

SL Viewer

Due to Monday being Memorial Day in the United States, the Lab was closed for normal office business, and there was no meeting to discuss potential RC viewer promotions. However, the most recent update to the Attachment Fixes RC viewer (Project Big Bird, currently version 3.7.29.301943) is showing a must reduced crash rate compared to the previous release (and which prevented it from being promoted to the de facto release viewer).

The crash rate is still slightly high than for the current release viewer, but speaking at the Simulator User Group meeting on Tuesday, May 26th, Oz Linden described it as “probably not a statistically significant difference”. Whether this means the viewer will be promoted to release status later in the week or not, remains to be seen.

Increasing the Number of Avatars Per Region

Simon Linden: looking at ways an means to make it easier for the simulator and a viewer to better handle large numbers of avatars
Simon Linden: looking at ways an means to make it easier for the simulator and a viewer to better handle large numbers of avatars

“There’s one change that I will follow up on … I added a way so I can adjust the ‘max avatars in a region’ setting.  I’d like to do an experiment soon and see what falls apart if we can get over 100 people into a region,” Simon Linden said at the simulator UG when discussing the upcoming RC deployment.

“This is purely experimental and there are no plans for changing the SL limits,” he went on. “But sometimes regions hit 100 [and] it would be nice if the viewer and simulator handled that better.”

There is already an additional means en route to the viewer by which users can have greater control over how avatars around them in a region are rendered by the viewer, Avatar Complexity, when will draw avatars above a rendering limit set by the user as a solid colour (the so-called “Jelly Baby” avatars).  The will work alongside the existing Avatar Imposters capability already in the viewer.

However, in terms of his experiment, Simon suggested that one way to improve things might be for the viewer to simply not draw everyone within a region; although how this would work, and the criteria used to determine what avatars are drawn and which aren’t, does require careful consideration. Simon suggested the viewer might simply skip drawing those avatars that are furthest away once a threshold number of avatars in the region has been reached. Another (suggest by a meeting attendee) would be for the control to be via the Max Number of Avatars settings within the viewer – so that once exceeded, avatars are again simply not rendered.

As noted, Simon’s work is purely experimental, and primarily aimed at helping the Lab understand what might be done to improve things where there are large gatherings of avatars, and to perhaps try out one or two ideas based on what they learn.

Simon’s Rendering Tricks

As a part of the discussion on avatar rendering, Simon handed out a note card of tips and trick for improving your performance when dealing with complex avatars. While this includes the debugs which will form a part of the new Avatar Complexity functionality, which will be appearing in a a Snowstorm RC viewer soon, as well as suggestions which may already be known, I’m including his suggestions in full here for reference:

From Advanced > Show Debug Settings, set:

  • RenderAutoHideSurfaceAreaLimit   0
  • RenderAutoMuteByteLimit  0
  • RenderAutoMuteFunctions  7
  • RenderAutoMuteLogging  False
  • RenderAutoMuteRenderWeightLimit  350000
  • RenderAutoMuteSurfaceAreaLimit  150

In preferences / graphics, change “Max # of non-imposter avatars” to something like 8. Also try ctrl-alt-shift-4 to hide avatars, or ctrl-alt-shift-2 for alphas.

Note the two debugs shown in green are those related directly to Avatar Complexity and drawing avatars as “Jelly Babies”. Note that RenderAutoMuteFunctions must be set to 7 in order for this to work. Also note that the RenderAutoMuteRenderWeightLimit of 350,000 is purely an advisory starting point. The Lab estimate that this will reduce the very top 3% of very rendering-intensive avatars as solid colours. You may find you have to set the value somewhat lower in certain environments  – such as night clubs and dance venues – in order for it to be effective. I’ve personally found that 150-200K tends to be required in very busy ballrooms, etc.

Second Life project updates 21/2: general notes

Posted on by Inara Pey
Living in a Bowl
Living in a Bowl, May 2015 – blog post

Server Deployments, Week 21

As always, please refer to the server deployment thread for the latest updates / news.

On Tuesday, May 19th the Main (SLS) channel received the server maintenance package previously deployed to the three RC channel, comprising Internal server logging changes, back-end system bug fixes and a change to Reply-To e-mail addressing on snapshots. There were no RC deployments on Wednesday, May 20th.

SL Viewer

The Attachments Viewer RC (Project Big Bird) was updated to version 3.7.29.301943 on Thursday May 21st. As noted in part 1 of this week’s report, the initial RC release of this viewer had an elevated crash rate compared to the current release viewer, including a crash-on-exit bug, so this release will hopefully address those issues.

Group Chat

A fix for issues around BUG-9130, where some people were unable to see any posts in some or all of there group chats, including their own posts, while everyone else in the same group could see their posts, has started to be deployed across the chat servers, and should be completed on Friday, May 22nd.

“The chat servers got stuck with bad info about where the sender was, so the messages never reached them,” Simon Linden said at the Server Beta User Group meeting on Thursday, May 21st, reiterating an explanation given at a recent Simulator UG meeting. “And unfortunately it wouldn’t fix with relogging or even a chat server restart.”

“Loading…” Issue with Names in Group Chats

This is a viewer-side problem which causes avatar names to appear as “Loading” under certain circumstances in group chat (see BUG-3829 and STORM-2114). A contribution by Ansariel Hiller is currently with the Lab and is expected to be released as a part of the next Snowstorm contributions viewer, which is expected to appear soon.

Other Items

Region Restart Glitch

There has been something of a rise in reports of regions experiencing issues following recent following restarts – most noticeably caps failures. This is something the Lab is looking into, and Simon commented, “we have a suspicion that after rolls, as that server host starts up regions, it’s doing enough of them at about the same time that things get overloaded.   It’s still a theory but makes some sense why we’d get cap failures like that.”

SL project updates Week 21/1: server, viewer CDN change, SL network update

Posted on by Inara Pey

WindWept, Dolly; Inara Pey, May 2015, on Flickr Windwept (General) May 2015 (Flickr) – blog post

Server Deployments, Week 21

As always, please refer to the server deployment thread for the latest updates / news.

On Tuesday, May 19th the Main (SLS) channel received the server maintenance package previously deployed to the three RC channel, comprising:

  • Internal server logging changes
  • Back-end system bug fixes
  • Reply-To email changed in postcard sends

As previously noted in these pages, the “reply-to email changed in postcard sends” relates to changing the way snapshots forwards to e-mail are handled. Until now, the Lab has substituted the user’s e-mail address in the “from” field of snapshots sent to e-mail, rather than displaying the “secondlife.com” address.

However, this added to issues of e-mail originating from “secondlife.com” being treated as spam by a/v software and ISPs. With the new format employed with this change, the sender’s e-mail address is given as the “reply to” address in the snapshot, and the “from” is “no-reply@secondlife.com”, thus avoiding the issue of LL looking like spammers who are forging invalid addresses.

There will be no RC deployment on Wednesday, May 20th.

SL Viewer

The week has not so far seen an RC viewer promoted to release status. If there is any promotion, it would most likely be the Layer Limits RC (currently version 3.7.29.301305). The Experience Tools RC viewer is still awaiting the completion of back-end work, while the Attachment Fixes RC (Project big Bird) currently has an elevated crash rate compared to the current release viewer, which includes a crash-on-exit bug, so further work is required on that RC.

CDN Provider Move

The Lab has been moving between CDN providers, and as a result, some people may have been experiencing particular texture / mesh / avatar rendering delays of late. Commenting on the process at the Simulator User Group meeting on Tuesday, May 19th, Oz Linden said:

We’ve just finished moving from one CDN provider to another, and it may take the caches a little while to catch up. We tried to do it gradually in a way that would be minimally disruptive, but when you’re dealing with as much data as we are, there are no perfect solutions.

One of the cases it is hoped the move will assist is with SL users in Florida (and neighbouring states) in the US who use Mediacom as their ISP, and who have found that there have been what appear to have been issues with Mediacom throttling the service at certain times of the day. Preliminary feedback from users so affected who have been involved in testing with the new CDN provider has been positive.

What Goes Through the CDN, And How

During the CDN conversation Oz reinterated the data that is currently delivered to the viewer through the CDN: textures, world map tiles, avatar baking data, and mesh data. In terms of in-world objects, two distinct operations are taking place:

  • Where an object is, how big it is, and so on, comes to the viewer via the simulator, together with the UUIDs fr the relevant objects / textures
  • The viewer then uses the UUIDs to fetch the mesh and texture data directly from the CDN.

As previously noted on these pages, this should mean faster loading of things like textures and mesh in-world, as the data is coming from a CDN node that is “local” relative to you, rather than coming to you from the Lab and through the simulator itself. However, experience is showing that for a small number of people, this isn’t always the case, and there can be situations where mesh and texture loading aren’t what might be expected. However, the Lab continues to try to improve things.

Second Life Network Architecture

Writing on the forums, noted SL photographer Jackson Redstar recently asked meshmaxconcurrentrequests – does anybody know the real setting? In the ensuing debate, Monty Linden offered an updated overview of the SL network architecture.

Monty Linden's updated SL network diagram
Monty Linden’s updated SL network diagram

To borrow from Monty’s explanation:

  • On the left, in red, are pieces of the viewer; on the right, in blue are simhost/simulators and other backend services; at the bottom (green) are new CDN services
  • Solid lines with arrowheads are communication paths, either UDP or TCP/HTTP; dashed lines are legacy communication paths that are now or soon will be deprecated, obsoleted and/or deleted
  • Sold ball-and-stick indicators (e.g. TextureFetchConcurrency) indicate a viewer debug setting and the communication path or paths that setting influences; dashed ball-and-stick indicators (e.g. MeshMaxConcurrentRequests) indicate obsolete debug settings.

Monty goes on to say:

Generally, things are moving in the direction of simplification and less resource conflict.  The mesh and texture HTTP traffic, which is usually the greatest load, tends to part ways with the UDP traffic a few network hops after a user’s router or modem.  Lacking TCP’s throttling mechanism, UDP often wins in a fight (give-or-take the efforts of fairness algorithms along the path).  Allowing UDP to overrun the path between viewer and simulator does still degrade the experience and the bandwidth setting remains an effective tool for avoiding this problem.

Other settings should generally be left alone.  A lot of bad advice was spread around in the community in an effort to work around throughput problems.  We’re trying to undo that history and get back on track with more typical (albeit aggressive) HTTP patterns.

 Viewer Caching

During the Simulator UG meeting, Oz repeated a call he originally made at a TPV Developer meeting recently, asking that if there is developer wishing to volunteer for a “deep dive” into viewer caching, he’d like to hear from them.

While interest list updates made key changes to how the viewer’s cache is used, there are numerous issues which appear to be viewer-side caching related, so a deep investigation into the code could go further towards improving things.

One long-standing issue, which is thought to be caching related, is If someone uses a texture rezzed in-world same texture for a group profile image or their avatar profile image or in a profile pick, the object will never fully load the texture.

So, if you’re a developer willing to looking into viewer-side caching, Oz would like to hear from you.

SL project updates week 20/2: TPV Developer meeting, VMM

Posted on by Inara Pey

Miyagi; Inara Pey, May 2015, on Flickr Miyagi (General), May 2015 (Flickr) – blog post

The following notes are primarily taken from the TPV Developer (TPVD) meeting held on Friday, May 15th, and from the Server Beta meeting held on Thursday, May 14th. A video of the TPVD meeting is included below, with any time stamps in the following text referring to it. My thanks as always to North for the recording and providing it for embedding,

Server Deployments, Week 20 – Recap

As always, please refer to the server deployment thread for the latest updates / news.

There was no Main (SLS) channel deployment on Tuesday, May 11th. On Wednesday, May 12th, the three RC channels were all updated with a new server maintenance package, comprising internal server logging changes, back-end system bug fixes, reply-to email changed in postcard sends (see below for more).

Group Chat

The Server Beta User Group meeting on Thursday, May 14th, saw a test of a group chat update it is hoped will fix the issue of some people not seeing all or some chat when the group chat window is open (see BUG-9130). The test appeared to yield positive results, with Simon reporting no unusual event logging. The problem here is that the instances of the problem seem to be so rare, it’s hard to guarantee a small sampling of testers will catch any problems which might still exist.

SL Viewer

[00:15] It has been a quiet week on the viewer front. As noted in part 1 of this week’s report, the attachment viewer (Project big Bird) reached RC status, other than that there has been not additional movement with either the current selection of RC and project viewers.

A further update to the mesh importer project viewer (currently at version 3.7.28.300878) is with LL’s QA team and should be released relatively soon. Updates are also being made to the Viewer-Managed Marketplace project viewer (which is likely to go to RC status once through LL’s QA process) and the Oculus Rift project viewer.

Snapshots to E-mail

[02:22] Commenting at the TPV Developer meeting, Oz Linden gave a little more information on the “reply-to email changed in postcard sends” update deployed to the RCs, indicating this was indeed a fix aimed at preventing snapshot to e-mail being tagged as spam by ISPs and a/v software due to the way they handle the “from” field (see my TPV Developer meeting report for week #17), which had caused the Lab to consider removing the snapshot to e-mail capability.

“Instead,” Oz told the meeting, “we found we could get around that by sending the e-mail differently … So the way it’s changed now is that instead of sending the ‘from’ address as the sender’s address, we send the ‘reply-to’ address; and the ‘from’ address is ‘no-reply@secondlife.com’ … so that ducks the problem of us looking like spammers who are forging invalid addresses.”

This should hopefully negate any need to remove the snapshot to e-mail capability, and retain compatibility with sending snapshots to the likes of Snapzilla and the SLU forums.

Unified Snapshot Floater

[04:40] NiranV Dean, who submitted the unified snapshot floater to LL (and which has most recently been integrated into Firestorm among the TPVs) asked if there had been any feedback on it. Both Oz and Grumpity Linden indicated that overall, feedback has been positive, although some have complained at the amount of screen real estate it takes up with the preview panel open. As this allows the snapshot preview to match the aspect ratio of the user’s screen, there’s not a lot that can be done about it – and the preview window can always be closed / the panel minimised when initially setting-up shots.

The unified snapshot floater - further work is being carried out by TPV devs for contribution to LL, including the possible full integration of the Facebook, Flickr and Twitter upload options
The unified snapshot floater – further work is being carried out by TPV devs for contribution to LL, including the possible full integration of the Facebook, Flickr and Twitter upload options

Niran is proposing a further set of updates (one of which, a fix for auto snapshot, is in the works at the Lab), including possibly making the preview screen detachable from the main floater.  Cinder Roxley also indicated she is working on fully integrating the Facebook, Twitter and Flickr options into the main snapshot floater (they currently retain their own floaters due to the authentication workflow required for each. This work will be contributed to the Lab for consideration / integration when complete.

Viewer-Managed Marketplace (VMM)

[08:09] Over the last two weeks, as a part of the on-going beta of VMM, around 15-20 volunteers have had their stores migrated by the Lab from Direct Delivery to VMM. Brooke linden reports that the exercise has uncovered “pretty much minor issues” which the Lab can address. A further batch of volunteer migrations is planned to help further test the robustness of the process in the next week or so.

As noted above, the VMM viewer is now heading for an RC release once it has cleared LL’s QA testing. However,  the time frame on when this might happen is a little vague; it might be in the next week or so, or it might be longer.

Avatar Complexity

[17:34]  The next Snowstorm contributions viewer is progressing internally at the Lab. This is the viewer which includes the new Avatar Complexity (aka “Jelly Babies” or “rainbow avatars”) functionality which allows users to define a level of complexity (a weighting number) which will render any avatar exceeding that value as a solid colour, rather than a full avatar. The aim of this is to help reduce the rendering load placed on people’s computers, particularly in very busy locations. The value is adjustable, as so can of course be varied to suit your current needs.

Avatar complexity is intended to help those who may hit performance issues as a result of their GPU struggling to render complex (hight render cost) avatars, by rendering such avatars as solid colours.
Avatar complexity is intended to help those who may hit performance issues as a result of their GPU struggling to render complex (hight render cost) avatars, by rendering such avatars as solid colours.

A slight hiccup has occurred in that in making some changes to the code, Oz accidentally broke the code such that instead of rendering as a solid colour, avatars exceeding the limit are currently rendering as transparent, and this is yet to be fixed. Code has been added to the viewer to report how many people around you are rendering your avatar as a solid colour (should your avatar be complex enough to be rendered thus), but this has yet to be made visible through the viewer UI, and simulator support for this is now in place on the RC channels and will be rolled to the Main channel in the coming week.

Mac Updates

[19:36] Cinder Roxley has a set of contributions for using the viewer with Mac Retina displays ready to go to the Lab, and it seems likely these will flow into a further Snowstorm contributions viewer in development alongside the one containing the Avatar Complexity updates.

A question was also asked whether there were plans to update the Mac viewer to use a newer OpenGL core profile. The Lab is not working on this, as their rendering team believe there is little or no benefit to be gained from it. However, they would accept any contributions offered for consideration (subject to a “long, terrible QA process”). However, a good part of this would require working through some eight years of OpenGL code.