Tag: Server Updates

SL project updates 41/2: server, HTTP, Group Chat

Posted on by Inara Pey

Pigeon Island, Neverending; Inara Pey, June 2014, on FlickrPigeon Island, Neverending (Flickr) – blog post

The following notes are taken from the Server Beta meeting held on Thursday October 9th, 2014. The transcript is available here, and the support agenda notes here.

Server Deployments Week 41 – Recap

  • Server deployment thread
  • The Main (SLS) channel and the Snack RC received the same server maintenance package as had been deployed to the three primary RCs in week 40,which fixes a bug related to viewing parcel details in gaming regions
  • There was no deployment to any of the primary RC channels (LeTigre, BlueSteel and Magnum).

CDN and the Snack RC

There are now some 270 regions on the Snack RC and which use the CDN for texture and mesh data fetching. The majority of these are mainland regions, although there is still no public list of all of them. Those who would like their region added to Snack are asked to contact the Lab at cdn-test@lindenlab.com.

If you’re using a TPV that can display region details in a dialogue box following a region crossing, you can use it to identify regions using the CDN, or you can check the About Land floater for the region you’re in. In both cases look for “Second Life RC Snack” in the region’s descriptive text. If you see it, the region is using the CDN.

Group Chat

Simon Linden’s work to both reduce the volume of additional group chat information messages (people joining leaving group chat sessions, updates to the group members list as people log-in or out of SL), and the frequency with which they are sent appears to have satisfied the Lab that they will produce some measure of improvement. There has already been a deployment of the updates to some of the back-end chat servers, and according to Maestro Linden, it is anticipated they should be on all of the chat servers some time in the next week. There may be more to add to this following the TPV Developer meeting.

HTTP Pipelining

Monty Linden: HTTP guru
Monty Linden: HTTP guru

The new HTTP pipelining viewer (referred to by the Lab’s QA team as the “weaponized” viewer, it is apparently so slick), is due to hit RC status. “Project Drano”, as Monty Linden, who has been spearheading the HTTP work over the last 2+ years, jokingly calls it, offers both improvements of its own and should further assist in data downloads via the CDN. Commenting on the viewer, Monty said:

The next viewer adds HTTP pipelining support to mesh and texture fetches. This allows multiple asset requests to be issued without waiting for intervening responses from the server, and it goes very far towards making client-to-server ping time irrelevant in throughput metrics. It’s a huge step on its own making our services perform as well as they can. Combined with CDN it’s even better. I won’t mention specific numbers but a region can show up in a very few seconds … Right now, we plan on skipping the project viewer stage. We’re going out the door [as an RC] once it passes QA (maestro heading that up).

Mesh fetching via the CDN should see an improvement because the viewer-side throttle of 100 meshes/second has been removed (this won’t affect non-CDN regions, as the throttle is also applied server-side on those).

As well as the core HTTP work, Monty has also done some work on inventory fetching within this viewer, particularly the code which is used to initially populate the inventory floater. This has been converted to use HTTP, and reduces the number of connections it uses. Again, no precise figures were given, but it should lead to improvements in inventory loading in situations where inventory data has been removed from cache (see the Firestorm wiki for information on removing inventory data files from cache). In one test with no inventory data, Monty saw a 100K inventory load around 10 times faster with the new viewer.

Whirly Fizzle (based in the UK) carried out inventory load tests for an 105K inventory between what was at the time the SL release viewer and a pre-release version of the HTTP pipelining viewer, both starting from a clear cache, and achieved some impressive results:

  • Second Life 3.7.16 (294015) Sep 10 2014 11:08:26 (Second Life Release)
    • Session 1: 16 mins 28 secs
    • Session 2: 17 mins 53 secs
    • Session 3: 17 mins 18 secs
    • Session 4: 17 mins 51 secs
  • Second Life 3.7.17 (294571) Sep 26 2014 12:32:36 (HTTP pipelining viewer)
    • Session 1: 2 mins 29 secs
    • Session 2: 2 mins 17 secs
    • Session 3: 2 mins 11 secs
    • Session 4: 2 mins 27 secs

I tend to have a very organised inventory, old / no longer used items are boxed, and anything not used in 6 months is similarly boxed (see: zdrop). Thus I only have an “active” inventory of around 10K (9,140 items at the moment), out of a total of around 70-100K. However, even with a “small” inventory like that, and working with a cleared cache, my results were impressive. The latest project version of the HTTP pipelining viewer (3.7.18.295146) averaged 9-10 seconds for inventory to download compared with an average of 2 minutes 50 seconds to 3 minutes for the current release viewer (3.7.17.294959).

Again, overall mileage on inventory downloads will vary, hence why Monty is hesitant to mention figures. However, it is thought that those with excessively “flat” inventories should see particular benefit (although they’d also be better off nesting their inventory into groups of folders).

SL project updates week 41/1: server, viewer, CDN

Posted on by Inara Pey
Sailing through Columbia River - blog post
Sailing through Columbia River – blog post

Server Deployments

As always, please refer to the server deployment thread for the latest updates and news.

  • On Tuesday October 7th, the Main (SLS) channel was updated with the server maintenance package deployed to the three primary RC channels in week 40, which fixes a bug related to viewing parcel details in gaming regions
  • There are no planned deployments / restarts scheduled for the three primary RC channels of BlueSteel, Magum and LeTigre
  • On Wednesday October 8th, the Snack RC, which is currently being used with the CDN project, should receive the same server maintenance package deployed the Main (SLS) channel.

SL Viewer

On Monday October 6th, the new log-in screen viewer was promoted to the de facto release viewer, version 3.7.17.294959. This viewer offers a revised log-in / splash screen for both new and returning users, based on feedback gathered during A/B testing (release notes).

 CDN Update

There is still no list of all the regions now running on the Snack RC and using the CDN for mesh and texture data retrieval, although Bay City – Sconset is one to add to any unofficial lists out there. It also appears that a number of Blake Sea regions may have been added to Snack, and are being blamed for region crossing issues.

The CDN service itself shouldn’t result in any worsening of region crossing issues, but Simon indicated that in moving regions onto the Snack Rc, the Lab, “inadvertently overloaded the servers on the Snack channel…. In this case, we picked a bunch of popular areas and put them all on the same machine, which was bad” (this overloading also might be related to BUG-7444 – note the comment from Maestro Linden).  “We did get some very good info on how things get overloaded, however,” Simon added. “…And possible future work to keep us busy.”

In the meantime, it is still not clear when a wider deployment of CDN support might take place. However, there do not appear to be any major blockers to a possible wider deployment to one (or more) of the primary RCs. It will be interesting to see if anything is announced for week 42 (week commencing Monday October 13th).

SL project updates week 40/2: server, viewer, CDN news, group chat update

Posted on by Inara Pey

Square Pegs, Round Holes, Kashmir Dreams; Inara Pey, September 2014, on FlickrSquare Pegs, Round Holes, Kashmir Dreams (Flickr) – blog post

Server Deployments Week 40 – recap

As always, please refer to the server deployment forum thread for the latest news and updates.

  • On Tuesday September 30th, the Main (SLS) channel received the server maintenance package previously deployed to the three primary RC channels (BlueSteel, Magnum and LeTigre), and which focuses on the Experience Tools project
  • On Wednesday October 1st, the three primary RC channels all received the same new server maintenance package which fixes a bug related to viewing parcel details in gaming regions.

The RC update was to fix BUG-7329 “RemoteParcelRequest cap returned by a skill gaming region does not exist (returns HTTP error code 404)”.  In the official viewer, this capability is mainly used to show parcel script info, but some TPVs use it to show the parcel_id of parcels as well.

SL Viewer

The new log-in screen RC viewer was updated to version 3.7.17.294762 on September 29th, although it didn’t appear on the Alternate Viewers wiki page until either September 30th or October 1st.

A new Maintenance RC entered the viewer release channel on Thursday October 2nd. Version 3.7.17.294943 contains around 40 updates focused on voice, privacy, rendering, texture animation, avatar distortion, inventory management, sounds, and mouselook in Mac, together with a scripting crash fix and multiple UI fixes in script editor, Pay flow, chat, stats floater, edit menu etc. See the release notes (linked-to above) for details.

CDN News

Speaking at the Server Beta Meeting on Thursday October 2nd, Maestro Linden confirmed that there are now 28 regions using the CDN service for texture and mesh asset data. A full list of regions is still not available, but among the new additions are Morris, Dore, Ahern, and Bonifacio.

These regions are all running on the Snack RC. There is unlike to be any wider deployment to a larger RC in week 41 (commencing Monday October 6th), as the Lab is still fine-tuning their gathering data. However, there are still a few slots left on Snack for those wishing to have their regions added to it. Requests should be sent to cdn-test@lindenlab.com. Note that regions should be those with a relatively high texture / mesh count,

The CDN service being used by the Lab is operated by Highwinds Network Group, which the Lab has also been using in support of server-side appearance.

Highwinds supply their CDN service to a broad range of businesses, including a number of games companies such as Valve (Steam), Funcom, Meteor Entertainment, GameFly and Virgin Gaming. They operate 25 centres around the world, with 11 in North America, seven in Europe, 4 in Asia, one in Australia and two in South America, over their own network infrastructure, which they call “RollingThunder”, which  peers with more than 1,600 provider networks worldwide and over 14,000 ASNs, and uses the anycast network addressing and routing methodology.

Highwinds data centres - click for full size, or see additional information on the centres and Highwinds PoPs here (image courtesy of Highwinds)
Highwinds data centres – click for full size, or see additional information via Highwinds here (image courtesy of Highwinds)

Group Chat

As I’ve previously reported, one of the biggest issues of chat delays in group chat sessions is to do with the numbers of updates the chat server has to send as people join / leave session and log-in / out of SL (which causes an update to their status in the group member’s list).

Some work has already been carried out in ways to reduce the volume of update messages being sent with the aim of lessening the impact they have on the flow of actually text messages, and Simon Linden is attempting to further refine this work, again with the agin of reducing the volume of update messages and their impact on group chat test messages.

The Server Beta User Group meeting on October 2nd saw a further test of his work in order for him to gather data on the effectiveness of these changes, and feedback on his findings will likely be given at one of the user group meetings next week.

SL project updates week 40/1: server, CDN

Posted on by Inara Pey
Aakriti Art Gallery - blog post
Aakriti Art Gallery – blog post

Server Deployments Week 40

As always, please refer to the server deployment forum thread for the latest news and updates.

On Tuesday September 30th, the Main (SLS) channel received the server maintenance package previously deployed to the three primary RC channels (BlueSteel, Magnum and LeTigre). This updated focuses on the Experience Tools project and includes the following updates:

  • llGetExperienceDetails(), now includes group_id in return list. In addition,
  •  llUpdateKeyValue() now correctly creates a key if it did not previously exist
  • Objects using experience permissions and llAttachToAvatar() are no longer automatically detached when leaving an area where the Experience is allowed.

On Wednesday October 1st, the three primary RC channels should all receive the same new server maintenance package which fixes a bug related to viewing parcel details in gaming regions.

SL Viewer

The SL viewer list remains as per my Current Releases page, namely:

  • Release viewer: version 3.7.16.294015 (formerly the Maintenance RC, promoted on September 22nd – release notes
  • RC viewerS:
    • Log-in release candidate viewer 3.7.17.294762 (September 25th), with the updated viewer log-in splash screens – download and release notes
  • Project viewers:
    • Benchmark project viewer version 3.7.17.294710 (September 10th) featuring the removal of the GPU table – download and release notes
    • Experience Keys project viewer version 3.7.16.293901(September 10th) – managing Experiences and for contributing content for Experiences – download and release notes
    • Oculus Rift project viewer version 3.7.12.292141 (July 22nd) support for Oculus Rift – download.

CDN Work

As noted in a separate report, the Lab has issued an update on various projects, including viewer changes and the CDN. At the Open-source User Group meeting on Monday September 29th,, it was intimated that Highwinds Network Group are the CDN operator, who supply CDN services to the likes of Valve (Steam), Funcom, Meteor Entertainment, GameFly, and others.

At present, there are 10 regions known to be using the CDN service, although additional regions may have been added to the Snack channel on Monday September 29th – we just don’t have an updated list of regions available.

SL project updates week 39/2: server releases, viewer, misc

Posted on by Inara Pey

The Trace, April 2014 by Inara Pey, on FlickrThe Trace, April 2014 (Flickr) – blog post

Server Deployments Week 39 – Recap

As always, please refer to the forum discussion thread for the latest updates and information.

  • There was no scheduled deployment to the Main (SLS) channel this week.
  • On Wednesday 24th September, all three RC channel received a further update to the Experience Tool maintenance release deployed in week 38, which includes:
    • llGetExperienceDetails(), now includes group_id in return list. In addition,
    •  llUpdateKeyValue() now correctly creates a key if it did not previously exist
    • Objects using experience permissions and llAttachToAvatar() are no longer automatically detached when leaving an area where the Experience is allowed.

SL Viewer

The promised viewer using the new GPU benchmark system appeared as a project viewer on Thursday September 25th. Version 3.7.17.294710, includes MAINT-3131, which is simply described as “Death to the GPU Table”, which pretty much sums the viewer up!

Rather than using the GPU table as a means of determining default graphics level for a graphics card, the viewer measures the memory bandwidth of the card, and sets the default based on that (plus a couple of other parameters. The release notes add, “Initial settings on shiny powerful hardware should now let that hardware shine. Initial settings on low-end machines should not degrade performance from current experience.”

CDN and Regions

The Snack RC is currently being used for CDN testing on Agni. As noted in my week 38 updates, Denby, Hippo Hollow, Hippotropolis and Testsylvania Sandbox were initially placed on the RC, and they have since been joined by Brasil Rio, Brocade, Fluffy, Freedom City, Rocket City and Whippersnapper.

Group Ban Trivia

OK, so not exactly an update, but Baker Linden indicates that some 4500 group bans have occurred since the arrival of the group ban list functionality in the official viewer (and a number of TPVs). Will be interesting to hear how that number increases once Firestorm includes the capability.

 

SL project updates: week 39/1: server, viewer, iCloud and other issues

Posted on by Inara Pey

Matoluta Sanctuary and Bay; Inara Pey, September 2014, on FlickrSunrise, Matoluta Sanctuary and Bay (Flickr) – blog post

The following notes were taken from the Open Source Dev meeting on Monday 22nd September 22nd, and the Simulator User Group meeting on Tuesday 23rd September.

Server Deployments Week 39

As always, please refer to the forum discussion thread for the latest updates and information.

  • There will be no scheduled deployment to the Main (SLS) channel this week.
  • On Wednesday 24th September, all three RC channel should receive a further update to the Experience Tool maintenance release deployed in week 38, which includes a fix for an issue with llGetExperienceDetails().

SL Viewer

The most recent Maintenance release viewer, version 3.7.16.294015, was promoted to the de facto release viewer on Monday 22nd September. This viewer includes fixes for inventory and outfit management; appearance editing; group & group ban management; camera controls; multi-grid support for favourites; notifications management; stability, bug and crash fixes – see the release notes for further information.

On Friday September 19th, the New Log-in Screen RC viewer reached release candidate status when 3.7.16.294345 arrvied in the release channel. This viewer brings a simple and clean login screen for new users, and a corresponding update for returning users. (download and release notes, my overview).

GPU Table Retirement

An ongoing project at the Lab is to remove the need for the GPU table within the viewer. This is currently used to set the default graphics level for a user’s graphics card, and requires constant checking and update as new GPUs and cards are produced. Recent work has seen the GPU table massively updated, with the Lab working towards an alternative strategy of determining the capabilities of a graphics system.  This is primarily done by  measuring the memory bandwidth of a card and setting the default based on that (plus a couple of other parameters.

A viewer utilising this approach is currently with LL’s QA team and should be making an appearance soon. This strategy has already shown sufficient promise that new GPUs are no longer being added to the GPU table in preparation for it to be phased out.

Other Items

iCloud Conflict

A recent update to Apple’s iCloud service aimed at users of windows system using the service has had an unexpected impact on various aspects of the SL including killing mesh uploads, snapshots (saving to inventory fails with “Error encoding snapshot”) and textures (upload fail with “Couldn’t convert the image to jpeg2000”), and UI elements can turn completely black.

Full details of the issue can be found on BUG-7343,  and the problems have been particularly noted in both the official viewer and Firestorm, and Catznip. investigations are underway by both the Lab and the Firestorm team, and one line of thinking is that it might be some DLL injection poisoning issue.

The iCloud update, which was apparently deployed over the weekend of the 20th /21st September has, at the time of writing, yet to be deployed for Mac systems. There is some speculation that it may not result in similar issues for Mac users due to the way iCloud is implemented for each OS. One potential work-around is to roll-back to an earlier version of the service’s client, making sure that any auto-update option is disabled.

Group Tags

We’re probably all aware how changing group tags can often be a cure-all for a number of problems, even when logically it should be the case. One possible explanation as to why this is the case is that changing your group tag may trigger a full update of your avatar.

However, possibly as a result of interest list changes, there is now one situation where changing your group tag is not a good idea – and that is when a scene is still loading, as doing so can cause the scene load to fail, and the only means of resuming it is to relog – see BUG-6299. So, if you arrive in a location that sends you a request to join a group you’d like to join, wait a couple of minutes in order to give the scene the chance to fully load before you do so.

This issue is known to the Lab, but a fix has yet to be determined.

llSetlinkAlpha Update Issue

This is an issue that is getting a little long in the tooth – see BUG-1786 – which sees llSetLinkAlpha failing to correctly update a percentage of prims when a large(ish) number are updated simultaneously. Weapons users are liable to be familiar with this, as it can occur in “holstering”  or “slinging” a weapon which should cause the “held” version of the weapon to turn transparent and the “slung” / “holstered” version rendered, but often results in elements of the “held” version of the weapon remaining visible.

This issue appears to be related to UDP packets being lost between the server and the viewer, with Simon Linden commenting, “I remember digging into this and it seemed like lost packets.  It’s really hard to predict when they’ll get lost, but it seems it’s not slowing down updates quite right when there’s a sudden flood.” He promised to pass the issue with LL’s product team, but wasn’t optimistic it might move higher up the “fix” chain due to the current volume of work.