Category: SL User Group Meetings

SL project updates week 41/3: TPV Developer meeting

Posted on by Inara Pey

The following notes are drawn from the TPV Developer meeting held on Friday October 10th, and shown in the video above. Time stamps, where relevant, have been included for ease of reference to the video. Note that items are listed according to subject matter, rather than chronologically, so time stamps may appear out-of-sequence in places. My thanks as always to North for the recording.

SL Viewer

HTTP Pipeline Viewer

[0:26:35] As noted in part 2 of this week’s report, the HTTP pipelining viewer was looking close to being ready for deployment as an RC viewer, possibly in the next week. However, it hit a final QA snag, and in Oz’s words, “Monty is busily correcting the problem and getting it back in the queue.” Or as Monty put it in chat “:(“. Apparently a workaround for a CURL bug “tends to disable pipelining”.

Benchmark and Experience Tools Viewers

[0:27:20] It is hoped that these two project viewers will be updating and moving into the viewer release channel as release candidates “pretty soon”.

Viewer Build Tool Chain

[0:41:51] The tools upgrade project for building the viewer has been subject to a few delays. Currently, the Lab has just about finished putting together all the build prerequisites for building the Mac version of the viewer directly on OS X 10.9, and are about to commence test builds of the viewer using the new tool chain. This has also led to some progress being made on updating the Linux build process as well. The windows environment will require further work, so it will be a little longer before things are fully in place.

Group Chat

[0:28:58] Again, as noted in part 2 of this week’s report, the latest updates for group chat are being deployed to various back-end chat servers by the Lab, and may be deployed to all of the chat servers in the next week.

While investigating group chat, the Lab has noticed that in general it is “unbelievably spikey”, with chat sessions having peaks of really good performance followed by troughs of really bad performance regardless as to whether the groups are regarded as being “good” or “bad” in handling group chat. There is no real pattern to these peaks and troughs, other than the larger the group, the more it seems to swing between the two extremes, and nothing to correlate them with anything in particular beyond the bad times occurring when a lot of people in a group are online.

Z-offet height Adjustment

[0:31:52] Work is continuing on the z-offset height proposal, which Vir Linden has been engaged upon. However, it appears the Lab has encountered some issues which have made it “a little more complicated” than had initially be thought. hopefully, these will be overcome, and they won’t bring the work to a halt.

CDN

[0:32:28] As noted in part 2 of this week’s report the number of regions on Snack and utilising  the Highwinds CDN for mesh and texture data servicing had reached around the 270 mark, but has since come down a little as a result of the Lab overloading the Snack sim hosts with regions running high volumes of users.

Metrics gathered by the Lab have been positive, and even though the Snack sim hosts were initially overloaded, the Lab feel they performed significantly better thanks to texture and mesh fetching being off-loaded to the CDN than would have been the case had the “old” method of texture / mesh fetching still been in use.

One aspect of the move to using the CDN is that until now, the sim host Apache service was being used for texture and mesh data handling and “lots” of other things which are timing critical to operations such as region crossings. With the move to the CDN, much (if not all) of the texture and mesh data handling is removed from the Apache service, making it easier for it to better handle time-critical activities.

If all goes according to plan, the CDN support will be expanded to the BlueSteel release candidate channel in week 42 (commencing Monday October 13th). This will allow the Lab to gain data on performance using the CDN support which can be directly compared with historical data available for region / sim host performance in BlueSteel. Those regions already running on Snack will continue to do so alongside of BlueSteel, so that around 5% of the main grid will be using the CDN service.

Continue reading “SL project updates week 41/3: TPV Developer meeting” →

SL project updates 41/2: server, HTTP, Group Chat

Posted on by Inara Pey

Pigeon Island, Neverending; Inara Pey, June 2014, on FlickrPigeon Island, Neverending (Flickr) – blog post

The following notes are taken from the Server Beta meeting held on Thursday October 9th, 2014. The transcript is available here, and the support agenda notes here.

Server Deployments Week 41 – Recap

  • Server deployment thread
  • The Main (SLS) channel and the Snack RC received the same server maintenance package as had been deployed to the three primary RCs in week 40,which fixes a bug related to viewing parcel details in gaming regions
  • There was no deployment to any of the primary RC channels (LeTigre, BlueSteel and Magnum).

CDN and the Snack RC

There are now some 270 regions on the Snack RC and which use the CDN for texture and mesh data fetching. The majority of these are mainland regions, although there is still no public list of all of them. Those who would like their region added to Snack are asked to contact the Lab at cdn-test@lindenlab.com.

If you’re using a TPV that can display region details in a dialogue box following a region crossing, you can use it to identify regions using the CDN, or you can check the About Land floater for the region you’re in. In both cases look for “Second Life RC Snack” in the region’s descriptive text. If you see it, the region is using the CDN.

Group Chat

Simon Linden’s work to both reduce the volume of additional group chat information messages (people joining leaving group chat sessions, updates to the group members list as people log-in or out of SL), and the frequency with which they are sent appears to have satisfied the Lab that they will produce some measure of improvement. There has already been a deployment of the updates to some of the back-end chat servers, and according to Maestro Linden, it is anticipated they should be on all of the chat servers some time in the next week. There may be more to add to this following the TPV Developer meeting.

HTTP Pipelining

Monty Linden: HTTP guru
Monty Linden: HTTP guru

The new HTTP pipelining viewer (referred to by the Lab’s QA team as the “weaponized” viewer, it is apparently so slick), is due to hit RC status. “Project Drano”, as Monty Linden, who has been spearheading the HTTP work over the last 2+ years, jokingly calls it, offers both improvements of its own and should further assist in data downloads via the CDN. Commenting on the viewer, Monty said:

The next viewer adds HTTP pipelining support to mesh and texture fetches. This allows multiple asset requests to be issued without waiting for intervening responses from the server, and it goes very far towards making client-to-server ping time irrelevant in throughput metrics. It’s a huge step on its own making our services perform as well as they can. Combined with CDN it’s even better. I won’t mention specific numbers but a region can show up in a very few seconds … Right now, we plan on skipping the project viewer stage. We’re going out the door [as an RC] once it passes QA (maestro heading that up).

Mesh fetching via the CDN should see an improvement because the viewer-side throttle of 100 meshes/second has been removed (this won’t affect non-CDN regions, as the throttle is also applied server-side on those).

As well as the core HTTP work, Monty has also done some work on inventory fetching within this viewer, particularly the code which is used to initially populate the inventory floater. This has been converted to use HTTP, and reduces the number of connections it uses. Again, no precise figures were given, but it should lead to improvements in inventory loading in situations where inventory data has been removed from cache (see the Firestorm wiki for information on removing inventory data files from cache). In one test with no inventory data, Monty saw a 100K inventory load around 10 times faster with the new viewer.

Whirly Fizzle (based in the UK) carried out inventory load tests for an 105K inventory between what was at the time the SL release viewer and a pre-release version of the HTTP pipelining viewer, both starting from a clear cache, and achieved some impressive results:

  • Second Life 3.7.16 (294015) Sep 10 2014 11:08:26 (Second Life Release)
    • Session 1: 16 mins 28 secs
    • Session 2: 17 mins 53 secs
    • Session 3: 17 mins 18 secs
    • Session 4: 17 mins 51 secs
  • Second Life 3.7.17 (294571) Sep 26 2014 12:32:36 (HTTP pipelining viewer)
    • Session 1: 2 mins 29 secs
    • Session 2: 2 mins 17 secs
    • Session 3: 2 mins 11 secs
    • Session 4: 2 mins 27 secs

I tend to have a very organised inventory, old / no longer used items are boxed, and anything not used in 6 months is similarly boxed (see: zdrop). Thus I only have an “active” inventory of around 10K (9,140 items at the moment), out of a total of around 70-100K. However, even with a “small” inventory like that, and working with a cleared cache, my results were impressive. The latest project version of the HTTP pipelining viewer (3.7.18.295146) averaged 9-10 seconds for inventory to download compared with an average of 2 minutes 50 seconds to 3 minutes for the current release viewer (3.7.17.294959).

Again, overall mileage on inventory downloads will vary, hence why Monty is hesitant to mention figures. However, it is thought that those with excessively “flat” inventories should see particular benefit (although they’d also be better off nesting their inventory into groups of folders).

SL project updates week 41/1: server, viewer, CDN

Posted on by Inara Pey
Sailing through Columbia River - blog post
Sailing through Columbia River – blog post

Server Deployments

As always, please refer to the server deployment thread for the latest updates and news.

  • On Tuesday October 7th, the Main (SLS) channel was updated with the server maintenance package deployed to the three primary RC channels in week 40, which fixes a bug related to viewing parcel details in gaming regions
  • There are no planned deployments / restarts scheduled for the three primary RC channels of BlueSteel, Magum and LeTigre
  • On Wednesday October 8th, the Snack RC, which is currently being used with the CDN project, should receive the same server maintenance package deployed the Main (SLS) channel.

SL Viewer

On Monday October 6th, the new log-in screen viewer was promoted to the de facto release viewer, version 3.7.17.294959. This viewer offers a revised log-in / splash screen for both new and returning users, based on feedback gathered during A/B testing (release notes).

 CDN Update

There is still no list of all the regions now running on the Snack RC and using the CDN for mesh and texture data retrieval, although Bay City – Sconset is one to add to any unofficial lists out there. It also appears that a number of Blake Sea regions may have been added to Snack, and are being blamed for region crossing issues.

The CDN service itself shouldn’t result in any worsening of region crossing issues, but Simon indicated that in moving regions onto the Snack Rc, the Lab, “inadvertently overloaded the servers on the Snack channel…. In this case, we picked a bunch of popular areas and put them all on the same machine, which was bad” (this overloading also might be related to BUG-7444 – note the comment from Maestro Linden).  “We did get some very good info on how things get overloaded, however,” Simon added. “…And possible future work to keep us busy.”

In the meantime, it is still not clear when a wider deployment of CDN support might take place. However, there do not appear to be any major blockers to a possible wider deployment to one (or more) of the primary RCs. It will be interesting to see if anything is announced for week 42 (week commencing Monday October 13th).

SL project updates week 40/2: server, viewer, CDN news, group chat update

Posted on by Inara Pey

Square Pegs, Round Holes, Kashmir Dreams; Inara Pey, September 2014, on FlickrSquare Pegs, Round Holes, Kashmir Dreams (Flickr) – blog post

Server Deployments Week 40 – recap

As always, please refer to the server deployment forum thread for the latest news and updates.

  • On Tuesday September 30th, the Main (SLS) channel received the server maintenance package previously deployed to the three primary RC channels (BlueSteel, Magnum and LeTigre), and which focuses on the Experience Tools project
  • On Wednesday October 1st, the three primary RC channels all received the same new server maintenance package which fixes a bug related to viewing parcel details in gaming regions.

The RC update was to fix BUG-7329 “RemoteParcelRequest cap returned by a skill gaming region does not exist (returns HTTP error code 404)”.  In the official viewer, this capability is mainly used to show parcel script info, but some TPVs use it to show the parcel_id of parcels as well.

SL Viewer

The new log-in screen RC viewer was updated to version 3.7.17.294762 on September 29th, although it didn’t appear on the Alternate Viewers wiki page until either September 30th or October 1st.

A new Maintenance RC entered the viewer release channel on Thursday October 2nd. Version 3.7.17.294943 contains around 40 updates focused on voice, privacy, rendering, texture animation, avatar distortion, inventory management, sounds, and mouselook in Mac, together with a scripting crash fix and multiple UI fixes in script editor, Pay flow, chat, stats floater, edit menu etc. See the release notes (linked-to above) for details.

CDN News

Speaking at the Server Beta Meeting on Thursday October 2nd, Maestro Linden confirmed that there are now 28 regions using the CDN service for texture and mesh asset data. A full list of regions is still not available, but among the new additions are Morris, Dore, Ahern, and Bonifacio.

These regions are all running on the Snack RC. There is unlike to be any wider deployment to a larger RC in week 41 (commencing Monday October 6th), as the Lab is still fine-tuning their gathering data. However, there are still a few slots left on Snack for those wishing to have their regions added to it. Requests should be sent to cdn-test@lindenlab.com. Note that regions should be those with a relatively high texture / mesh count,

The CDN service being used by the Lab is operated by Highwinds Network Group, which the Lab has also been using in support of server-side appearance.

Highwinds supply their CDN service to a broad range of businesses, including a number of games companies such as Valve (Steam), Funcom, Meteor Entertainment, GameFly and Virgin Gaming. They operate 25 centres around the world, with 11 in North America, seven in Europe, 4 in Asia, one in Australia and two in South America, over their own network infrastructure, which they call “RollingThunder”, which  peers with more than 1,600 provider networks worldwide and over 14,000 ASNs, and uses the anycast network addressing and routing methodology.

Highwinds data centres - click for full size, or see additional information on the centres and Highwinds PoPs here (image courtesy of Highwinds)
Highwinds data centres – click for full size, or see additional information via Highwinds here (image courtesy of Highwinds)

Group Chat

As I’ve previously reported, one of the biggest issues of chat delays in group chat sessions is to do with the numbers of updates the chat server has to send as people join / leave session and log-in / out of SL (which causes an update to their status in the group member’s list).

Some work has already been carried out in ways to reduce the volume of update messages being sent with the aim of lessening the impact they have on the flow of actually text messages, and Simon Linden is attempting to further refine this work, again with the agin of reducing the volume of update messages and their impact on group chat test messages.

The Server Beta User Group meeting on October 2nd saw a further test of his work in order for him to gather data on the effectiveness of these changes, and feedback on his findings will likely be given at one of the user group meetings next week.

SL project updates week 40/1: server, CDN

Posted on by Inara Pey
Aakriti Art Gallery - blog post
Aakriti Art Gallery – blog post

Server Deployments Week 40

As always, please refer to the server deployment forum thread for the latest news and updates.

On Tuesday September 30th, the Main (SLS) channel received the server maintenance package previously deployed to the three primary RC channels (BlueSteel, Magnum and LeTigre). This updated focuses on the Experience Tools project and includes the following updates:

  • llGetExperienceDetails(), now includes group_id in return list. In addition,
  •  llUpdateKeyValue() now correctly creates a key if it did not previously exist
  • Objects using experience permissions and llAttachToAvatar() are no longer automatically detached when leaving an area where the Experience is allowed.

On Wednesday October 1st, the three primary RC channels should all receive the same new server maintenance package which fixes a bug related to viewing parcel details in gaming regions.

SL Viewer

The SL viewer list remains as per my Current Releases page, namely:

  • Release viewer: version 3.7.16.294015 (formerly the Maintenance RC, promoted on September 22nd – release notes
  • RC viewerS:
    • Log-in release candidate viewer 3.7.17.294762 (September 25th), with the updated viewer log-in splash screens – download and release notes
  • Project viewers:
    • Benchmark project viewer version 3.7.17.294710 (September 10th) featuring the removal of the GPU table – download and release notes
    • Experience Keys project viewer version 3.7.16.293901(September 10th) – managing Experiences and for contributing content for Experiences – download and release notes
    • Oculus Rift project viewer version 3.7.12.292141 (July 22nd) support for Oculus Rift – download.

CDN Work

As noted in a separate report, the Lab has issued an update on various projects, including viewer changes and the CDN. At the Open-source User Group meeting on Monday September 29th,, it was intimated that Highwinds Network Group are the CDN operator, who supply CDN services to the likes of Valve (Steam), Funcom, Meteor Entertainment, GameFly, and others.

At present, there are 10 regions known to be using the CDN service, although additional regions may have been added to the Snack channel on Monday September 29th – we just don’t have an updated list of regions available.

SL project updates 39/3: TPV Developer meeting

Posted on by Inara Pey

The following notes are drawn from the TPV Developer meeting held on Friday September 26th, and shown in the video above. Time stamps, where relevant, have been included for ease of reference to the video. Note that items are listed according to subject matter, rather than chronologically, so time stamps may appear out-of-sequence in places. My thanks as always to North for the recording.

Benchmark Viewer & GPU Table

[01:00] As noted in part 2 of this report, a new GPU Benchmark project viewer is available (version 3.7.17.294710), designed to put an end to the need for a dedicated GPU graphic table as the mean by which the viewer determines a computer’s initial graphic settings.

Instead, if there is no settings file for the viewer (such as after a clean install),  the viewer will measure how quickly data can be copied back and forth between GPU memory and your computer’s main memory. This, combined with a couple of other benchmarks, determines the initial graphics settings in the viewer. It may not always pick the most preferred settings (it might still set things a little high or a little low), but testing has shown it to be reasonably accurate,  and it does prevent the viewer opting for the lowest settings simply because a card isn’t listed on the GPU table. As is currently the case, any subsequent adjustments you make to the graphics settings should be saved within the viewer and take precedence.

Feedback on the viewer is encouraged (a wipe of any SL viewer setting files on your computer will be required), particularly if you encounter issues such as finding the viewer “sticks” with the settings it has determined, rather than allowing you to adjust them. When filing JIRA, the Lab requests that log files are attached.

HTTP and CDN

[09:39] The anticipated HTTP pipelining viewer should be appearing as a release candidate viewer in the early part of week 40 (week commencing Monday 29th September). This is the viewer that the QA team in LL have been referring to as QA,  “weaponized viewer”, it is so fast as a result of leveraging the HTTP streaming.

This viewer works with the CDN, with Oz Linden indicating a personal experience of logging-in to a CDN-enabled region with an empty cache and having the textures and meshes for the region loaded by the time the log-in process had finished, so it will be interesting to see how the viewer performs under more widespread use.

TPVs are being encouraged to adopt the HTTP updates as soon as their integration / release cycles allow. In the meantime, those wishing to test this viewer, when it appears, with the CDN can do so via one of the following regions: Denby, Hippo Hollow, Hippotropolis,Testsylvania, Brasil Rio, Brocade, Fluffy, Freedom City, Rocket City or Whippersnapper. It is anticipated  further regions will be added to the CDN channel (Snack) in the next week or so, prior to CDN support rolling to one of the server RC channels.

 Voice Updates

[17:16] Another batch of viewer updates due out, and which TPVs are being urged to adopt as soon as they can, are for voice. These mostly relate to managing voice sessions rather than voice improvements, and are aimed at helping Vivox with problems at their end, and should make troubleshooting genuine issues within voice a lot easier. However, this update should plug the hole where stalkers can track where someone using voice has teleported to just by monitoring their voice channels.

Z-offset Height Adjustment

Jessica Lyon demonstrated part of the avatar height offset issue at the last TPV Developer meeting: when seated using her preferred sitting pose, her avatar floats above a chair, and she has no means of adjusting the height so that she appears to be sitting in the chair
The z-offset hegiht adjustment option should help in situations where the current Hover option is unusable – such as trying to adjust you avatar’s height when using a preferred AO sitting pose

[18:42] Vir Linden is now working on the z-offset height proposal. The work is in the early stages, so no date on when it will appear in a viewer.

The current plan is for a new option to be added to the right-click avatar context menu which will access an adjustment slider. However, at present, any adjustments made using it will not be persistent across log-ins, although it will work alongside the existing Edit Appearance > Hover option (allowing for the No Mod shape limitation of the latter).

It has been suggested the offset setting could be made persistent by tying it to a debug setting. This is something the Lab has said they’ll think about; should they opt not to go that route, there will hopefully be no reason why TPVs should not go that route if persistence was deemed vital to their users’ experience.

[48:13] Adjustments made using the slider will occur locally until such time as the mouse button is released; only then is an update message sent to the server & relayed to other viewers, to prevent multiple messages spamming a server as people make adjustments. It is hoped that this approach will also allow z-offset adjustments to interact with other active animations relatively smoothly (e.g. adjusting your height to prevent appearances of dancing on air when using couples dance poseballs).

Continue reading “SL project updates 39/3: TPV Developer meeting” →