Tag: CDN

Lab asks: how is SL for you?

Posted on by Inara Pey

Just Another Tequilla Sunrise, Isle of Love; Inara Pey, October 2014, on FlickrSL should be looking and feeling a lot better for many of us as a result of recent work by the Lab – how’s it going for you? The Lab asks you drop them a line in the forums or via Twitter (image: Just Another Tequilla Sunrise, Isle of Love (Flickr) – blog post)

Following-on from the grid-wide deployment of CDN support and the promotion of the HTTP pipelining viewer as the de facto release viewer, the Lab has blogged about recent improvements to Second Life, finishing with the question “how is it for you?”

The blog post, entitled, Performance, Performance, Performance, opens thus:

Has Second Life seemed a bit faster for you lately? Improving performance for all Second Life users has been an important focus for us at Linden Lab, and we’ve recently seen some great results from several projects that should make your Second Life experiences faster, smoother, and more reliable.

It goes on to make fair mention of the CDN / HTTP work, noting:

Faster Texture & Mesh Loading
The entire grid is now using a CDN service for textures and meshes. This change means that textures and meshes should load more quickly, particularly for those who login to Second Life from places that are far from our US data centers. Our testing showed dramatic improvements: average download times for textures and meshes have been reduced by more than 50% on average, and the improvement is even more dramatic outside of North America.

Quicker Viewer-Server Communications
Another way we’re enhancing Second Life performance is through our HTTP project, which improves the way your Viewer communicates with grid services. With the HTTP Project Viewer out now, the faster content download times you’ll see thanks to the CDN change get even better – we’re talking 80% faster!

For those who may have missed news on the HTTP pipelining viewer and the CDN support, you can catch-up with things via a couple of posts on this blog: SL project updates week 42/2: Monty’s HTTP update and the HTTP pipelining viewer, and HTTP pipelining viewer reaches release status as CDN support is grid-wide.

Mention is also made of the recent Group Chat updates (work is still continuing on this, and you can get updates via the Group Chat tag in this blog).

However, perhaps the most interesting aspect of the blog post is the news that there have been some significant infrastructure updates with the SL Marketplace, which appear to have slipped mention through other mediums such as the var UG meetings. Here the blog post notes:

Speeding Up the Marketplace
If you visit the Marketplace today, you should be seeing a much snappier experience than in the past. We recently deployed infrastructure upgrades for the Marketplace, and the site has since shown some of the best performance we’ve ever seen from it. Even during peak usage periods over the weekend, when in the past performance would degrade, we’re seeing response times that average 70% faster and page load times that are 30% faster than before the changes.

I’ve not used the Marketplace of late – although I have been covering the upcoming Viewer-managed Marketplace (VMM) changes that will be occurring in 2015, and which the Lab is currently gearing-up for, along with merchants and TPVs.

So have you noticed changes and improvement in your Second Life experience? If so, then why not follow the Lab’s request:

So, How’s Second Life Performing for You?
Performance improvements are generally behind-the-scenes work, and we know it’s not always as exciting as rolling out a new feature, but these changes directly impact all our Second Life experiences and our daily lives in-world. We hope you’re starting to notice the effects of these improvements – if you are, please let us know in the Forums, on Twitter, or however you prefer.

Note that is let the Lab know – not me (although general views are always welcome here, they might not be picked-up by the Lab)!

SL project updates week 44/1: Server, CDN update

Posted on by Inara Pey

The Pines at Jacob's Pond, Jacob; Inara Pey, October 2014, on FlickrThe Pines at Jacob’s Pond, Jacob (Flickr) – blog post

Server Deployments Week 44

As always, please refer to the server deployment thread for the latest information and updates.

Main (SLS) Channel

On Tuesday, October 28th, CDN support was deployed across the Main channel, meaning that the entire grid now utilises the Highwinds CDN for texture and mesh fetching.

As the 130 regions deployed to the Snack channel were all originally from the Main channel, they have been / will be reabsorbed into that channel, and Snack will once again be dissolved.

Release Candidate Channels

On Wednesday, October 29th, all three RC channels should receive the same server maintenance project. which includes some minor improvements.

SL Viewer

There are currently two RC viewers possibly vying for promotion to the de facto release viewer. These are the HTTP Pipelining RC (version 3.7.19.295700) and the Benchmark viewer (version 3.7.19.295759), which should put an end to the use of a manually maintained list of GPUs in order to initially set the graphics defaults in the viewer.

Both of these were updated on Friday, October 24th, which has delayed any promotion to the de facto release viewer while the Lab gathers performance statistics on both of them. commenting on the status of both during the Simulator User Group meeting on Tuesday, October 27th, Oz Linden indicated that all things being equal, the HTTP Pipelining viewer should be promoted in the next 24-48 hours. He also indicated that there may be a further round of updates to come to Benchmark viewer in the offing as the Lab continued to tweak it.

CDN: Next Steps

In the vast majority of cases, the CDN is working as expected for users. There is a very small minority who, possibly because of their geographical closeness to the Lab’s servers or possibly due to issues between their ISP and the Highwinds services, are experiencing slightly worse ping times to their nearest CDN nodes when compared with pinging the Lab’s servers directly.

Even with the deployment to the Main channel, the Lab is continuing to monitor reports from CDN closely. However, as previously mentioned in my CDN coverage, it is likely the scope of CDN usage will be expanding in the future to handle other asset data – sounds, animations etc.

Also, and as noted in my week 43 TPV Developer meeting report, an offshoot of the CDN work is that there is a belief within the Lab that viewer caching may not be working as well as it might be. Internal discussions have held on possibly validating whether or not this is the case, and it is likely some work will be carried out in this area – and may well involve TPVs.

However, where both the viewer cache and extending the use of the CDN to cover other asset data are concerned, there are no time frames currently in mind. At the moment, the focus is very much on get the new tool chain and build process for the viewer finalised and into production, and in dealing with bugs and issues. As such, it might be a while before specific work on the viewer cache and / or work on extending the use of the CDN gets underway.

On a wider front, as well as monitoring the direct effectiveness of the CDN service, the Lab will be “spending quite a bit of time and effort assessing just what the effect of this change has been on operations from a number of perspectives”, to quote Oz.

Other Items

LI Issues?

There are reports circulating of unexpected changes to Land Impact (LI) values. While the Lab hasn’t altered land accounting, there have apparently been incidents of LI suddenly increasing in builds which have gone unchanged; one instance quoted during the Simulator User Group meeting referenced a door which apparently increased from 0.5 LI to 3 LI.

There have been instances in the past of the viewer incorrectly reporting the LI for an item when it is pulled from inventory. Corrections can generally be made either by relogging or by returning the item to inventory and rezzing it again. Altering the physics shape of linksets (from prim to convex and back again) can sometimes lead to problems, particularly if a prim in the linkset is contains torturing (such as a hollow or advance cut / twisting) or a script. However, this issue appears to be new, and a cause is proving hard to identify.

As always, if you have encountered the problem, and it is both persistent and reproducible, please raise a bug report.

Graphics Profiles in the Viewer

A suggestion was put forward during the Simulator User Group meeting for the Lab to allow the saving of graphics profiles. This would mean, for example that you could have a graphics profile where various options – the quality slider, shadows, occlusion, draw distance, etc., could be pushed towards their upper limits; and another where the setting are more conservative and less taxing on your GPU / system.

Then, where you are in a region (your own, or somewhere you know), where you know you can use the higher settings, you can quickly enable that profile, but when you move on to a region where (say) there are a lot of avatars and a lot going on, you can select the more conservative profile and thus reduce the potential performance issue on your system without actually having to go through and manually adjust all your settings.

Responding to the idea, Oz suggested the idea might be best suited to being a code contribution – and there was some potential interest in taking the idea on. However, this does not guarantee the idea will be carried forward – but it will be interesting to see if the idea does move forward at all in the coming months.

SL project updates week 43/3: TPV dev meeting, inc. viewer-managed Marketplace

Posted on by Inara Pey

The following notes are drawn from the TPV Developer meeting held on Friday, October 24th, and shown in the video above. Time stamps, where relevant, have been included for ease of reference to the video. Note that items are listed according to subject matter, rather than chronologically, so time stamps may appear out-of-sequence in places. My thanks as always to North for the recording.

SL Viewer

[00:10] There have been a couple of end-of-week viewer updates:

  • The HTTP Pipelining viewer was updated on October 24th to version 3.7.19.295700, incorporating the most recent viewer releases
  • The Benchmarking viewer (which removes dependencies on the GPU table) was also updated on October 24th, to version 3.7.19.295759, incorporating the most recent viewer updates.

Both of these viewers have performed well both in the RC role (HTTP Pipelining) and project viewer (in the case of the Benchmark viewer), and it is likely one or the other will be promoted in week 44 (week commencing Monday, October 27th), although the promotion may not occur until later in the week, depending on how these new releases perform.

[01:58] The Experience Keys project viewer is in the “final throes” of bug fixing. However, it is unlikely to appear as an RC viewer until after the next set of server-side Experience Keys have been deployed. When this will be is unclear. It is not “imminent”, but by the same measure, it is not “distant”.

[02:33] The Oculus Rift work continues, but again, this is unlikely to move beyond being a project viewer until such time as the Oculus hardware is more generally available. The Lab will continue to keep the viewer code base current with releases, and will doubtless continue to test and refine as further SDK updates appear (such as the just-released 0.4.3).

Viewer-managed Marketplace

[03:16] Progress is continuing to be made with the viewer-managed Marketplace updates (VMM). There is now a technical wiki page for the web API available, and Skylar Linden was on-hand to provide and overview of the page, which by his own admission, is “pretty dry stuff”, but includes things like:

  • Figuring out if a user is a merchant or not, and how the system will respond (loading the correct page or not)
  • A data dictionary that gets transmitted back and forth between the viewer and the Marketplace
  • Links to obtain listings, create listings, modify listings, associate inventory with listings, and delete listings (which are the Marketplace functions specific to the new capability).

[08:40] The ability to delete listings is a feature that is new to the Marketplace as a result of VMM, although it is somewhat confusing in that  – as Brooke Linden explained in the meeting – it doesn’t actually delete anything, but gives a means for merchants to remove listing information by making it unavailable. As such, listings deleted in this way will be non-recoverable.

VMM Testing

[10:03]  The Lab is now “very close” to getting the VMM updates available on Aditi’s Marketplace. When this happens, testing will initially be with a “small group” of merchants and TPV devs using a project viewer, the code for which will be made available as the testing starts in order to allow TPVs to integrate it into test versions of their viewers. If all goes according to plan, this initial test could start in the next couple of weeks, with the aim of getting initial feedback.

Once this has been done, testing will by opened-up to allow wider involvement. This is expected to happen “within the next couple of months”, and will be aimed at allowing anyone who would like to try VMM. There will be a beta testing application form made available for those interested.

Group Chat

[13:45] The Lab have been continuing to roll changes to group chat, and the thinking is that the results of the changes are “pretty good”, and the Firestorm support team are reporting they’ve seen an improvement as well. As noted in part 2 of this week’s update, the work isn’t finished – Simon is looking at the possibility to improve code elsewhere in the system; there is also liable to be a round of code clean-up as well. However, Oz cautioned that the Lab may have implemented the changes which may have the biggest noticeable impact in terms of improving the group chat experience.

CDN Progress

[14:49] Again, as noted in part 2 of this week’s report, the CDN is now supporting all the primary Server RC channels. so far, the results have been “excellent”, but the Lab is waiting to see how things fair over the weekend, when the grid is liable to see its heaviest use in terms of numbers of users on-line since the RC deployment was completed this week.

The load on the sim host Apache services is described as “way, way, way, down”, which is a good thing, as it means that all the other things the Apache services have to do (such as participating in region crossings) are no longer being impacted by the simulators handing the texture and mesh fetching loads.

One offshoot of the CDN work is that the Lab is likely to spend time validating whether the viewer cache is working as well as it could be, even allowing for the interest list updates. There are apparently differences in how well the cache works across different code-bases, so this is something that TPVs may well be involved in as well.

Z-offset height Adjustment

There has been no further progress on this since the last TPV meeting.

Continue reading “SL project updates week 43/3: TPV dev meeting, inc. viewer-managed Marketplace” →

SL project updates week 43/1: server, viewer, group chat

Posted on by Inara Pey
Asalia House (November 2013) - blog post
Asalia House (November 2013) – blog post

Server Deployments Week 43

As always, please refer to the server deployment forum thread for the latest news and updates.

Main (SLS) Channel

On Tuesday, October 21st, the Main channel was updated with the server maintenance release previously deployed to the LeTigre and Magnum RCs in week 42. This update includes a crash fix and improves the delivery pipeline for abuse reports.

RC Channels

On Wednesday, October 22nd, the release candidate channels should be updated as follows:

  • BlueSteel and Snack will remain on the CDN project, but should also be updated with the same server maintenance package being deployed to the Main channel
  • LeTigre and Magnum should both be updated to the CDN for texture and mesh fetching.

SL Viewer

On Wednesday, October 15th, the Lab issued a Grid Status update on potential problems accessing external websites using the viewer’s internal browser.  The notice was issued as a result of the Padding Oracle On Downgraded Legacy Encryption (Poodle) vulnerability reported by Google.

At the time the notice was released, the Lab indicated they were working to fix the issue (via the removal of SSL v3.0 support within the viewer’s browser). This work was completed at the end of week 42, and resulted in the release of the Browser Fix RC viewer, version 3.7.18.295539.

On Monday, October 20th, the Browser Fix viewer was promoted to the de facto release viewer.  Further information on it and the POODLE vulnerability can be found in a separate article in this blog.

MAC Yosemite Issues

There have been some mixed results for those Mac users updating to the 10.10 Yosemite version of OS X, which reached a consumer release status on October 16th.

While most of those updating have found it relatively smooth, there has been a report of keyboard shortcuts borking, although there is a workaround should others encounter the same problem – see here.

A potentially more widespread issue people may encounter is that with Yosemite, Apple have apparently reverted to older nVidia drivers (as reported by Cinder Roxley at the Open-source Developer’s meeting on Monday, October 20th). One JIRA on the matter has already been filed (see BUG-7575), although if Apple have chosen to roll back some / all of their default drivers, there’s not a lot the Lab can do about matters. Cinder also reports that Yosemite makes greater use of OpenGL, which she indicates can tax GPUs further.

CDN

Following the scheduled Wednesday deployment, all of the RCs should be using the CDN. This represents around 30% of the total grid, possibly a little more, given the existence of Snack. The BlueSteel deployment does not seem to have resulted in the kind of performance issues experienced during the expansion of Snack to 250 regions, suggesting the latter problems were most probably down to overloading the sim hosts with too many high volume (in terms of avatar numbers) regions. If all goes well in the coming week, it is likely that the Main channel will gain CDN support in week 44.

Group Chat

“Today we also finished an update to the group chat servers,” Simon Linden announced at the Simulator User Group meeting on Tuesday, October 21st. He continued:

Many of you have helped us test that at Thursday’s beta user group, and that’s now on the newest version. Those are back-end servers … when you send something to group chat, it goes from the viewer to the region you’re on, then to the (new) back-end chat servers, and from there it’s distributed to everyone in the group.

While no promises are being made as to the overall improvement – or how visible improvements may be to users, Simon went on to note, “I know [from] looking at the metrics and data we can gather … the systems are running much nicer than before.”

As well as working on the update issues (see my previous group chat updates), the Lab has been brainstorming ideas on how groups might be configured differently. There are a lot of groups that have very heavy chat usage, and others where chat is actively discouraged in favour of outgoing notices; however the latter are handled exactly the same as those where chatting goes on, generating updates as people log-in and out, etc., even though these aren’t actually required. Exactly what might be done is unclear at this time, but it would appear ideas are being put forward and discussed.

 

SL project updates week 42/3: server, HTTP, CDN, issues

Posted on by Inara Pey
No Signal, MIC Imagin@rium, May 2014 - blog post
No Signal, MIC Imagin@rium, May 2014 – blog post

Unless indicated otherwise, the notes below are taken from the Server Beta meeting held on Thursday October 16th, 2014. The transcript is available here, and the support agenda notes here.

Server Deployments Week 42 – Recap

As always, please refer to the server deployment thread for updates and the latest information.

  • There was no code promotion to the Main (SLS) channel on Tuesday, October 14th – this follows from there having been no deployments to the primary RC channels in week 41
  • On Wednesday, October 15th, the primary RC channels were updated as follows:
    • Bluesteel received the CDN texture & mesh fetching capabilities – release notes
    •   LeTigre and Magnum both received a new server maintenance package,  which includes a crash fix and improves the delivery pipeline for abuse reports.

SL Viewer

HTTP Pipelining RC

On Wednesday October 15th, the Lab released the HTTP Pipelining RC viewer. Version 3.7.18.295372 brings with it:

  • Pipelined HTTP Operations for Mesh and Texture Fetches: this feature allows the viewer to issue multiple asset fetches on a connection without waiting for responses to earlier requests. This reduces the impact of a user’s physical location on scene loading and generally improves the experience for everyone
  • Inventory Fetch Improvements: Inventory folder and item fetches are getting some of the same treatment that textures and meshes did in previous releases. Initial inventory load should be quicker and more robust for all users.

A blog post from Monty Linden accompanies the release, which provides further information on the HTTP project alongside of the viewer release notes.

A blog post from monty Linden arrived with the Pipelining RC viewer, and explain how the viewer (and CDN) should generate further improvements in texture and mesh fetching operations
A blog post from Monty Linden arrived with the Pipelining RC viewer, and explain how the viewer (and CDN) should generate further improvements in texture and mesh fetching operations

Maintenance RC Viewer Removal

The latest Maintenance RC viewer,  version 3.7.17.294943,  has been withdrawn from the release channel due to significant issues and a range of attachment bugs which affect it (see my week 41 update from the TPV developer meeting, under “AIS v3 Issues”).

CDN

As noted above, use of the CDN service for texture and mesh data was extended to the BlueSteel RC in week 42, and there is some anticipation within LL that it could be deployed to all three of the primary RCs sooner rather than later.

An issue has been noted on some regions running on the Snack RC where people are seeing very slow parcel information updates (e.g. the name of the parcel taking up to a full minute to appear in the status bar, for music stream changes to occur, etc.).

The thinking from the Lab is that these problems are a result of the Snack sim hosts receiving a top-heavy load of regions with high traffic counts, resulting in possible resource contentions with certain data lookups as a result – contentions which may have actually brought the regions down had it not been for the CDN, as Maestro Linden commented:

In the cases I’ve seen so far, the slowness was due to a rapid rate of requests from all the users logged into the regions.  If the sim were serving all the texture and mesh requests as it traditionally would, the regions would have ‘fallen over’ before becoming that populated.

So this would seem to be an unintentional cloud with its own silver(ish) lining (at least the regions didn’t fall over). Nevertheless, the issue isn’t welcome. Given that BlueSteel has a more even distribution of regions across sim hosts (e.g. the servers are not all packed to the gills with high volume traffic regions),  It is hoped that the issue will not exhibit itself there. Checks (either by the Lab or residents or both) are likely to be carried out on busy regions on the RC to see if the problem does raise its head, which might indicate some further investigations might be required.

Assuming all goes well with the BlueSteel deployment, it is thought that CDN support will likely extended to the remaining RCs sooner rather than later. In the meantime, for those interested in seeing how use of the CDN works for them, a list of regions by RC can be obtained via Tyche Shepherd’s Grid Survey website, allowing BlueSteel regions to be easily located. Just click the RC Server Regions button (note the list will require a little cleaning-up, post download).

Continue reading “SL project updates week 42/3: server, HTTP, CDN, issues” →

SL project updates week 41/3: TPV Developer meeting

Posted on by Inara Pey

The following notes are drawn from the TPV Developer meeting held on Friday October 10th, and shown in the video above. Time stamps, where relevant, have been included for ease of reference to the video. Note that items are listed according to subject matter, rather than chronologically, so time stamps may appear out-of-sequence in places. My thanks as always to North for the recording.

SL Viewer

HTTP Pipeline Viewer

[0:26:35] As noted in part 2 of this week’s report, the HTTP pipelining viewer was looking close to being ready for deployment as an RC viewer, possibly in the next week. However, it hit a final QA snag, and in Oz’s words, “Monty is busily correcting the problem and getting it back in the queue.” Or as Monty put it in chat “:(“. Apparently a workaround for a CURL bug “tends to disable pipelining”.

Benchmark and Experience Tools Viewers

[0:27:20] It is hoped that these two project viewers will be updating and moving into the viewer release channel as release candidates “pretty soon”.

Viewer Build Tool Chain

[0:41:51] The tools upgrade project for building the viewer has been subject to a few delays. Currently, the Lab has just about finished putting together all the build prerequisites for building the Mac version of the viewer directly on OS X 10.9, and are about to commence test builds of the viewer using the new tool chain. This has also led to some progress being made on updating the Linux build process as well. The windows environment will require further work, so it will be a little longer before things are fully in place.

Group Chat

[0:28:58] Again, as noted in part 2 of this week’s report, the latest updates for group chat are being deployed to various back-end chat servers by the Lab, and may be deployed to all of the chat servers in the next week.

While investigating group chat, the Lab has noticed that in general it is “unbelievably spikey”, with chat sessions having peaks of really good performance followed by troughs of really bad performance regardless as to whether the groups are regarded as being “good” or “bad” in handling group chat. There is no real pattern to these peaks and troughs, other than the larger the group, the more it seems to swing between the two extremes, and nothing to correlate them with anything in particular beyond the bad times occurring when a lot of people in a group are online.

Z-offet height Adjustment

[0:31:52] Work is continuing on the z-offset height proposal, which Vir Linden has been engaged upon. However, it appears the Lab has encountered some issues which have made it “a little more complicated” than had initially be thought. hopefully, these will be overcome, and they won’t bring the work to a halt.

CDN

[0:32:28] As noted in part 2 of this week’s report the number of regions on Snack and utilising  the Highwinds CDN for mesh and texture data servicing had reached around the 270 mark, but has since come down a little as a result of the Lab overloading the Snack sim hosts with regions running high volumes of users.

Metrics gathered by the Lab have been positive, and even though the Snack sim hosts were initially overloaded, the Lab feel they performed significantly better thanks to texture and mesh fetching being off-loaded to the CDN than would have been the case had the “old” method of texture / mesh fetching still been in use.

One aspect of the move to using the CDN is that until now, the sim host Apache service was being used for texture and mesh data handling and “lots” of other things which are timing critical to operations such as region crossings. With the move to the CDN, much (if not all) of the texture and mesh data handling is removed from the Apache service, making it easier for it to better handle time-critical activities.

If all goes according to plan, the CDN support will be expanded to the BlueSteel release candidate channel in week 42 (commencing Monday October 13th). This will allow the Lab to gain data on performance using the CDN support which can be directly compared with historical data available for region / sim host performance in BlueSteel. Those regions already running on Snack will continue to do so alongside of BlueSteel, so that around 5% of the main grid will be using the CDN service.

Continue reading “SL project updates week 41/3: TPV Developer meeting” →