Tag: CDN

SL project updates 41/2: server, HTTP, Group Chat

Posted on by Inara Pey

Pigeon Island, Neverending; Inara Pey, June 2014, on FlickrPigeon Island, Neverending (Flickr) – blog post

The following notes are taken from the Server Beta meeting held on Thursday October 9th, 2014. The transcript is available here, and the support agenda notes here.

Server Deployments Week 41 – Recap

  • Server deployment thread
  • The Main (SLS) channel and the Snack RC received the same server maintenance package as had been deployed to the three primary RCs in week 40,which fixes a bug related to viewing parcel details in gaming regions
  • There was no deployment to any of the primary RC channels (LeTigre, BlueSteel and Magnum).

CDN and the Snack RC

There are now some 270 regions on the Snack RC and which use the CDN for texture and mesh data fetching. The majority of these are mainland regions, although there is still no public list of all of them. Those who would like their region added to Snack are asked to contact the Lab at cdn-test@lindenlab.com.

If you’re using a TPV that can display region details in a dialogue box following a region crossing, you can use it to identify regions using the CDN, or you can check the About Land floater for the region you’re in. In both cases look for “Second Life RC Snack” in the region’s descriptive text. If you see it, the region is using the CDN.

Group Chat

Simon Linden’s work to both reduce the volume of additional group chat information messages (people joining leaving group chat sessions, updates to the group members list as people log-in or out of SL), and the frequency with which they are sent appears to have satisfied the Lab that they will produce some measure of improvement. There has already been a deployment of the updates to some of the back-end chat servers, and according to Maestro Linden, it is anticipated they should be on all of the chat servers some time in the next week. There may be more to add to this following the TPV Developer meeting.

HTTP Pipelining

Monty Linden: HTTP guru
Monty Linden: HTTP guru

The new HTTP pipelining viewer (referred to by the Lab’s QA team as the “weaponized” viewer, it is apparently so slick), is due to hit RC status. “Project Drano”, as Monty Linden, who has been spearheading the HTTP work over the last 2+ years, jokingly calls it, offers both improvements of its own and should further assist in data downloads via the CDN. Commenting on the viewer, Monty said:

The next viewer adds HTTP pipelining support to mesh and texture fetches. This allows multiple asset requests to be issued without waiting for intervening responses from the server, and it goes very far towards making client-to-server ping time irrelevant in throughput metrics. It’s a huge step on its own making our services perform as well as they can. Combined with CDN it’s even better. I won’t mention specific numbers but a region can show up in a very few seconds … Right now, we plan on skipping the project viewer stage. We’re going out the door [as an RC] once it passes QA (maestro heading that up).

Mesh fetching via the CDN should see an improvement because the viewer-side throttle of 100 meshes/second has been removed (this won’t affect non-CDN regions, as the throttle is also applied server-side on those).

As well as the core HTTP work, Monty has also done some work on inventory fetching within this viewer, particularly the code which is used to initially populate the inventory floater. This has been converted to use HTTP, and reduces the number of connections it uses. Again, no precise figures were given, but it should lead to improvements in inventory loading in situations where inventory data has been removed from cache (see the Firestorm wiki for information on removing inventory data files from cache). In one test with no inventory data, Monty saw a 100K inventory load around 10 times faster with the new viewer.

Whirly Fizzle (based in the UK) carried out inventory load tests for an 105K inventory between what was at the time the SL release viewer and a pre-release version of the HTTP pipelining viewer, both starting from a clear cache, and achieved some impressive results:

  • Second Life 3.7.16 (294015) Sep 10 2014 11:08:26 (Second Life Release)
    • Session 1: 16 mins 28 secs
    • Session 2: 17 mins 53 secs
    • Session 3: 17 mins 18 secs
    • Session 4: 17 mins 51 secs
  • Second Life 3.7.17 (294571) Sep 26 2014 12:32:36 (HTTP pipelining viewer)
    • Session 1: 2 mins 29 secs
    • Session 2: 2 mins 17 secs
    • Session 3: 2 mins 11 secs
    • Session 4: 2 mins 27 secs

I tend to have a very organised inventory, old / no longer used items are boxed, and anything not used in 6 months is similarly boxed (see: zdrop). Thus I only have an “active” inventory of around 10K (9,140 items at the moment), out of a total of around 70-100K. However, even with a “small” inventory like that, and working with a cleared cache, my results were impressive. The latest project version of the HTTP pipelining viewer (3.7.18.295146) averaged 9-10 seconds for inventory to download compared with an average of 2 minutes 50 seconds to 3 minutes for the current release viewer (3.7.17.294959).

Again, overall mileage on inventory downloads will vary, hence why Monty is hesitant to mention figures. However, it is thought that those with excessively “flat” inventories should see particular benefit (although they’d also be better off nesting their inventory into groups of folders).

SL project updates week 41/1: server, viewer, CDN

Posted on by Inara Pey
Sailing through Columbia River - blog post
Sailing through Columbia River – blog post

Server Deployments

As always, please refer to the server deployment thread for the latest updates and news.

  • On Tuesday October 7th, the Main (SLS) channel was updated with the server maintenance package deployed to the three primary RC channels in week 40, which fixes a bug related to viewing parcel details in gaming regions
  • There are no planned deployments / restarts scheduled for the three primary RC channels of BlueSteel, Magum and LeTigre
  • On Wednesday October 8th, the Snack RC, which is currently being used with the CDN project, should receive the same server maintenance package deployed the Main (SLS) channel.

SL Viewer

On Monday October 6th, the new log-in screen viewer was promoted to the de facto release viewer, version 3.7.17.294959. This viewer offers a revised log-in / splash screen for both new and returning users, based on feedback gathered during A/B testing (release notes).

 CDN Update

There is still no list of all the regions now running on the Snack RC and using the CDN for mesh and texture data retrieval, although Bay City – Sconset is one to add to any unofficial lists out there. It also appears that a number of Blake Sea regions may have been added to Snack, and are being blamed for region crossing issues.

The CDN service itself shouldn’t result in any worsening of region crossing issues, but Simon indicated that in moving regions onto the Snack Rc, the Lab, “inadvertently overloaded the servers on the Snack channel…. In this case, we picked a bunch of popular areas and put them all on the same machine, which was bad” (this overloading also might be related to BUG-7444 – note the comment from Maestro Linden).  “We did get some very good info on how things get overloaded, however,” Simon added. “…And possible future work to keep us busy.”

In the meantime, it is still not clear when a wider deployment of CDN support might take place. However, there do not appear to be any major blockers to a possible wider deployment to one (or more) of the primary RCs. It will be interesting to see if anything is announced for week 42 (week commencing Monday October 13th).

SL project updates week 40/2: server, viewer, CDN news, group chat update

Posted on by Inara Pey

Square Pegs, Round Holes, Kashmir Dreams; Inara Pey, September 2014, on FlickrSquare Pegs, Round Holes, Kashmir Dreams (Flickr) – blog post

Server Deployments Week 40 – recap

As always, please refer to the server deployment forum thread for the latest news and updates.

  • On Tuesday September 30th, the Main (SLS) channel received the server maintenance package previously deployed to the three primary RC channels (BlueSteel, Magnum and LeTigre), and which focuses on the Experience Tools project
  • On Wednesday October 1st, the three primary RC channels all received the same new server maintenance package which fixes a bug related to viewing parcel details in gaming regions.

The RC update was to fix BUG-7329 “RemoteParcelRequest cap returned by a skill gaming region does not exist (returns HTTP error code 404)”.  In the official viewer, this capability is mainly used to show parcel script info, but some TPVs use it to show the parcel_id of parcels as well.

SL Viewer

The new log-in screen RC viewer was updated to version 3.7.17.294762 on September 29th, although it didn’t appear on the Alternate Viewers wiki page until either September 30th or October 1st.

A new Maintenance RC entered the viewer release channel on Thursday October 2nd. Version 3.7.17.294943 contains around 40 updates focused on voice, privacy, rendering, texture animation, avatar distortion, inventory management, sounds, and mouselook in Mac, together with a scripting crash fix and multiple UI fixes in script editor, Pay flow, chat, stats floater, edit menu etc. See the release notes (linked-to above) for details.

CDN News

Speaking at the Server Beta Meeting on Thursday October 2nd, Maestro Linden confirmed that there are now 28 regions using the CDN service for texture and mesh asset data. A full list of regions is still not available, but among the new additions are Morris, Dore, Ahern, and Bonifacio.

These regions are all running on the Snack RC. There is unlike to be any wider deployment to a larger RC in week 41 (commencing Monday October 6th), as the Lab is still fine-tuning their gathering data. However, there are still a few slots left on Snack for those wishing to have their regions added to it. Requests should be sent to cdn-test@lindenlab.com. Note that regions should be those with a relatively high texture / mesh count,

The CDN service being used by the Lab is operated by Highwinds Network Group, which the Lab has also been using in support of server-side appearance.

Highwinds supply their CDN service to a broad range of businesses, including a number of games companies such as Valve (Steam), Funcom, Meteor Entertainment, GameFly and Virgin Gaming. They operate 25 centres around the world, with 11 in North America, seven in Europe, 4 in Asia, one in Australia and two in South America, over their own network infrastructure, which they call “RollingThunder”, which  peers with more than 1,600 provider networks worldwide and over 14,000 ASNs, and uses the anycast network addressing and routing methodology.

Highwinds data centres - click for full size, or see additional information on the centres and Highwinds PoPs here (image courtesy of Highwinds)
Highwinds data centres – click for full size, or see additional information via Highwinds here (image courtesy of Highwinds)

Group Chat

As I’ve previously reported, one of the biggest issues of chat delays in group chat sessions is to do with the numbers of updates the chat server has to send as people join / leave session and log-in / out of SL (which causes an update to their status in the group member’s list).

Some work has already been carried out in ways to reduce the volume of update messages being sent with the aim of lessening the impact they have on the flow of actually text messages, and Simon Linden is attempting to further refine this work, again with the agin of reducing the volume of update messages and their impact on group chat test messages.

The Server Beta User Group meeting on October 2nd saw a further test of his work in order for him to gather data on the effectiveness of these changes, and feedback on his findings will likely be given at one of the user group meetings next week.

SL project updates week 40/1: server, CDN

Posted on by Inara Pey
Aakriti Art Gallery - blog post
Aakriti Art Gallery – blog post

Server Deployments Week 40

As always, please refer to the server deployment forum thread for the latest news and updates.

On Tuesday September 30th, the Main (SLS) channel received the server maintenance package previously deployed to the three primary RC channels (BlueSteel, Magnum and LeTigre). This updated focuses on the Experience Tools project and includes the following updates:

  • llGetExperienceDetails(), now includes group_id in return list. In addition,
  •  llUpdateKeyValue() now correctly creates a key if it did not previously exist
  • Objects using experience permissions and llAttachToAvatar() are no longer automatically detached when leaving an area where the Experience is allowed.

On Wednesday October 1st, the three primary RC channels should all receive the same new server maintenance package which fixes a bug related to viewing parcel details in gaming regions.

SL Viewer

The SL viewer list remains as per my Current Releases page, namely:

  • Release viewer: version 3.7.16.294015 (formerly the Maintenance RC, promoted on September 22nd – release notes
  • RC viewerS:
    • Log-in release candidate viewer 3.7.17.294762 (September 25th), with the updated viewer log-in splash screens – download and release notes
  • Project viewers:
    • Benchmark project viewer version 3.7.17.294710 (September 10th) featuring the removal of the GPU table – download and release notes
    • Experience Keys project viewer version 3.7.16.293901(September 10th) – managing Experiences and for contributing content for Experiences – download and release notes
    • Oculus Rift project viewer version 3.7.12.292141 (July 22nd) support for Oculus Rift – download.

CDN Work

As noted in a separate report, the Lab has issued an update on various projects, including viewer changes and the CDN. At the Open-source User Group meeting on Monday September 29th,, it was intimated that Highwinds Network Group are the CDN operator, who supply CDN services to the likes of Valve (Steam), Funcom, Meteor Entertainment, GameFly, and others.

At present, there are 10 regions known to be using the CDN service, although additional regions may have been added to the Snack channel on Monday September 29th – we just don’t have an updated list of regions available.

Lab updates on viewer changes and CDN

Posted on by Inara Pey

secondlifeThe Lab has issued a blog post outlining some of the current improvements being made to Second Life.

Regular readers of my weekly SL project updates will already be familiar with the work referenced in the blog post, which focus on the changes being made to the viewer’s log-in screen, the removal of the viewer’s reliance on the GPU table when initially setting graphics preferences, the ongoing deployment of support for using a Content Delivery Network (CDN) for texture and mesh fetching, and an announcement of the upcoming HTTP pipelining viewer, which should offer some significant improvements in people’s SL experience, as well as including further adjustments to leverage the CDN.

Commenting on the new benchmark viewer, which will eliminate the need for the GPU table, the Lab’s blog post states:

This is a new way of figuring out the best default graphics settings. Maybe this has happened to you: you got an awesome new graphics card, fired up SL… only to discover your graphics settings are set to Low, and can’t be changed? No more! This Viewer does away with the old GPU table and instead uses a quick benchmark measurement to detect your GPU to assign appropriate default graphics settings on startup. The settings on shiny powerful hardware should really let that hardware shine. Get a Project Benchmark Viewer today and help us gather metrics!  Please file bugs in JIRA if you find them.

The new log-in viewer is currently the only release candidate viewer sitting in the viewer release channel. As such, it is liable to be promoted to the de facto release viewer in the near future – probably in week 41 (week commencing Monday October 6th), assuming the statistics for it haven’t shown up any issues.

As the Lab’s blog-post indicates, this viewer is being introduced as a result of several months of A/B testing with the current viewer log-in screen. This testing appears to show that new user retention is some 3-5% better when incoming users are presented with the updated viewer’s log-in / splash screens than when compared with those for the current version.

For those interested in finding out how the new viewer differs from the current version, I have an overview of the new version already posted.

The log-in / splash screen in the login RC viewer seen by users who have previously logged-in to SL
The log-in / splash screen in the login RC viewer seen by users who have previously logged-in to SL

A point to note with the log-in screen changes is that they do not impact the widgets, etc., used by TPVs. Therefore, these changes shouldn’t force those TPVs using their own log-in splash screens to replace them with the Lab’s updates.

The final two aspects of the Lab’s blog post are the deployment of the CDN, which is currently for texture and mesh fetching, and which I’ve also extensively documented through my week SL project updates. At the time of writing, the CDN is available in ten regions across the main grid: Denby, Hippo Hollow, Hippotropolis, Testsylvania, Brasil Rio, Brocade, Fluffy, Freedom City, Rocket City or Whippersnapper. However, more regions will be added as time goes on.

There is no requirement for any special viewer in order to get an idea of the faster downloading of textures and meshes users should witness on entering any of these regions (there may be some rare instances where things are a little slower if you happen to reside closer to one of the Lab’s data centres than to your local CDN node, but these instances are likely to be very rare). However, once the CDN service is available across the grid, it may see a final viewer-side update as a part of final fine-tuning, and well as potentially being extended to include the delivery of other viewer-consumable assets.

The HTTP work, which has been ongoing for the last couple of years and very much a focus of Monty Linden’s work, is something I’ve also reported upon through my weekly SL project updates. This should have some general improvements on performance, both with texture and mesh downloads through the CDN, and with other HTTP-specific SL services. This viewer code is allegedly so fast, the Lab refer to it internally as the “weaponized viewer”.

The benefit of the CDN and the HTTP viewer code – which TPVs are being encouraged to adopt as quickly as their merge / test / release cycles allow – is summed-up in the closing comments on the Lab’s post:

Separately, each of these will improve texture and mesh loading performance, but put together, you should really see some exciting improvements in how long it takes to load new areas and objects – making touring the many fabulous places in Second Life you have not yet visited even better!

Those who have been independently testing both the CDN and the pipelining viewer (in a pre-project viewer release state) have been reporting that results with either / both are impressive. Check Shug Maitland’s comment on this blog, for example, after she tried the CDN regions with a current viewer.

SL project updates 39/3: TPV Developer meeting

Posted on by Inara Pey

The following notes are drawn from the TPV Developer meeting held on Friday September 26th, and shown in the video above. Time stamps, where relevant, have been included for ease of reference to the video. Note that items are listed according to subject matter, rather than chronologically, so time stamps may appear out-of-sequence in places. My thanks as always to North for the recording.

Benchmark Viewer & GPU Table

[01:00] As noted in part 2 of this report, a new GPU Benchmark project viewer is available (version 3.7.17.294710), designed to put an end to the need for a dedicated GPU graphic table as the mean by which the viewer determines a computer’s initial graphic settings.

Instead, if there is no settings file for the viewer (such as after a clean install),  the viewer will measure how quickly data can be copied back and forth between GPU memory and your computer’s main memory. This, combined with a couple of other benchmarks, determines the initial graphics settings in the viewer. It may not always pick the most preferred settings (it might still set things a little high or a little low), but testing has shown it to be reasonably accurate,  and it does prevent the viewer opting for the lowest settings simply because a card isn’t listed on the GPU table. As is currently the case, any subsequent adjustments you make to the graphics settings should be saved within the viewer and take precedence.

Feedback on the viewer is encouraged (a wipe of any SL viewer setting files on your computer will be required), particularly if you encounter issues such as finding the viewer “sticks” with the settings it has determined, rather than allowing you to adjust them. When filing JIRA, the Lab requests that log files are attached.

HTTP and CDN

[09:39] The anticipated HTTP pipelining viewer should be appearing as a release candidate viewer in the early part of week 40 (week commencing Monday 29th September). This is the viewer that the QA team in LL have been referring to as QA,  “weaponized viewer”, it is so fast as a result of leveraging the HTTP streaming.

This viewer works with the CDN, with Oz Linden indicating a personal experience of logging-in to a CDN-enabled region with an empty cache and having the textures and meshes for the region loaded by the time the log-in process had finished, so it will be interesting to see how the viewer performs under more widespread use.

TPVs are being encouraged to adopt the HTTP updates as soon as their integration / release cycles allow. In the meantime, those wishing to test this viewer, when it appears, with the CDN can do so via one of the following regions: Denby, Hippo Hollow, Hippotropolis,Testsylvania, Brasil Rio, Brocade, Fluffy, Freedom City, Rocket City or Whippersnapper. It is anticipated  further regions will be added to the CDN channel (Snack) in the next week or so, prior to CDN support rolling to one of the server RC channels.

 Voice Updates

[17:16] Another batch of viewer updates due out, and which TPVs are being urged to adopt as soon as they can, are for voice. These mostly relate to managing voice sessions rather than voice improvements, and are aimed at helping Vivox with problems at their end, and should make troubleshooting genuine issues within voice a lot easier. However, this update should plug the hole where stalkers can track where someone using voice has teleported to just by monitoring their voice channels.

Z-offset Height Adjustment

Jessica Lyon demonstrated part of the avatar height offset issue at the last TPV Developer meeting: when seated using her preferred sitting pose, her avatar floats above a chair, and she has no means of adjusting the height so that she appears to be sitting in the chair
The z-offset hegiht adjustment option should help in situations where the current Hover option is unusable – such as trying to adjust you avatar’s height when using a preferred AO sitting pose

[18:42] Vir Linden is now working on the z-offset height proposal. The work is in the early stages, so no date on when it will appear in a viewer.

The current plan is for a new option to be added to the right-click avatar context menu which will access an adjustment slider. However, at present, any adjustments made using it will not be persistent across log-ins, although it will work alongside the existing Edit Appearance > Hover option (allowing for the No Mod shape limitation of the latter).

It has been suggested the offset setting could be made persistent by tying it to a debug setting. This is something the Lab has said they’ll think about; should they opt not to go that route, there will hopefully be no reason why TPVs should not go that route if persistence was deemed vital to their users’ experience.

[48:13] Adjustments made using the slider will occur locally until such time as the mouse button is released; only then is an update message sent to the server & relayed to other viewers, to prevent multiple messages spamming a server as people make adjustments. It is hoped that this approach will also allow z-offset adjustments to interact with other active animations relatively smoothly (e.g. adjusting your height to prevent appearances of dancing on air when using couples dance poseballs).

Continue reading “SL project updates 39/3: TPV Developer meeting” →