Category: News

Lab issues further CDN update – more improvements coming

Posted on by Inara Pey

On November 1st, 2014, the Lab blogged about improvements seen from their side of things as a result of CDN support deployment.  At the time the updates were being issued, the Lab was also asking for feedback from users as to how things were going for them.

As a result of this request for feedback, the Lab issued a further update on the improvements on Friday, November 7th,, and it is a tale of two halves.

The first part of the blog post re-states the core benefits that have been seen as a result of the CDN deployment for mesh and texture data, which is again split into two key areas: a considerable reduction in the load on some key systems on the simulator hosts, and a big performance improvement in texture and mesh data loading, resulting in users seeing faster rez times in new areas they’re visiting.

In the graphic released with the Lab's November 7th update on the CDN deployment,
From: An update on the CDN project, linden Lab, November 7th, 2014

However, the experience of some users hasn’t been so good, as reported in the forum thread, and it could not be put down to matters of distance from the CDN nodes vs. the Lab’s simulators, or to people experiencing slower load times as a result of being the very first to enter a region which had not been cached at the local CDN node.

This feedback encouraged the Lab into further investigation and data-gathering of specific situations, allowing them to engage with CDN supplier Highwinds in order to try to determine possible reasons for the poorer experiences. The second part of the blog post notes the outcome of these efforts:

We believe that the problems are the result of a combination of the considerable additional load we added to the CDN, and a coincidental additional large load on the CDN from another source. Exacerbating matters, flaws in both our viewer code and the CDN caused recovery from these load spikes to be much slower than it should have been. We are working with our CDN provider to increase capacity and to configure the CDN so that Second Life data availability will not be as affected by outside load. We are also making changes to our code and in the CDN to make recovery quicker and more robust.

The blog post also points out some of the risks involved when trying to deploy large-scale changes to a complex and dynamic environment such as Second Life:

Making any change to a system at the scale of Second Life has some element of unavoidable risk; no matter how carefully we simulate and test in advance, once you deploy at scale in live systems there’s always something to be learned. This change has had some problems for a small percentage of users; unfortunately, for those users the problems were quite serious for at least part of the time.

The post concludes by thanking all those who contributed to helping the Lab understand the nature of the problems being experienced and in taking the time to help provide data on their particular circumstances which helped with further investigations, and with a note that it is hoped that the changes that are to be made as a result of this work will reduce such problems, allowing more people to enjoy the benefits offered through the use of the CDN for asset data delivery.

Lab divests itself of Desura

Posted on by Inara Pey

LL logoIn a press release made on Wednesday, November 5th, Linden Lab announced the sale of Desura, the on-line digital distribution service which they acquired under Rod Humble’s leadership in July 2013.

Desura has been sold to Bad Juju Games, the Indie Game and Middleware Tools Developer for the Mobile, Console and PC Gaming markets, and based in Aliso Viejo, California. In the brief press release, the Lab state:

As has just been announced, Bad Juju Games has acquired Desura from Linden Lab.

Bad Juju has taken over all day-to-day support, maintenance, and ongoing enhancements to the Desura website and service platform. The Bad Juju team will be reaching out directly to developers with games on Desura and are happy to respond to any questions they may have.

Transitioning Desura to a new owner is great for Linden Lab and our customers, as it allows us to further enhance our focus on creating the ambitious next-generation virtual world, while continuing to improve Second Life and growing Blocksworld.

Desura is a fantastic platform for game developers and players, and we look forward to seeing it continue to evolve and grow, now as part of Bad Juju Games.

Desura: just 16 months in the Lab's portfolio
Desura: just 16 months in the Lab’s portfolio

A press release from Bad Juju themselves indicates that negotiations over the future of Desura have been underway for some months:

Indie Game and Middleware Tools Developer Bad Juju Games®, today announced that it has officially acquired Desura™, a comprehensive digital distribution service for PC, Mac and Linux gamers from its former owner and operator Linden Lab®. The move comes after several months of coordinated planning by the companies to ensure uninterrupted operation of the Desura service as well as a comprehensive roadmap of new features that will significantly benefit both its users and game developers during the upcoming months and beyond.

It is unclear what the sale of Desura may mean for Scott Reismanis, if indeed it means anything at all. Reismanis, who founded the digital distribution service, and who joined Linden Lab as “Director of Digital” not long after the acquisition. With his experience in empowering creators to benefit from their digital content, Mr. Reismanis may already be involved in the development of the company’s “next generation” virtual worlds platform.

While this may sound a little like 20/20 hindsight, I confess to being unsurprised by the sale of Deura; I’ve actually been expecting news of this kind for the past couple of months, and particularly since the Designing Worlds interview with Ebbe Altberg. During that programme, Mr. Altberg referred to the Lab being in a process of cleaning up their product portfolio (some 3 minutes into the show), with the use of the present tense suggesting to me that the process was still going on, rather than him simply referring to the company’s removal of Versu, Creatoverse and dio from their portfolio. Given the statements of support that have been repeatedly given about Blocksworld, the comment in the DW show seemed to indicate something would be happening to Desura and / or Patterns.

To be totally honest, at the time I felt it more likely that Desura would be let-go than Patterns, as the latter at least seemed to fit with the Lab’s sandbox creativity philosophy, while Desura always seemed more of an awkward fit, and something more likely to generate a decent return if offered for sale. Of course, as it turned out, Patterns went first.

Confirmation also seemed to come when an enquiry on an unrelated matter led to a response from the Lab that all remaining advertising including Desura, would be removed from their remaining web properties (my.secondlife.com and search.secondlife.com). The reference to Desura chiming as odd, given it was ostensibly a Lab product.

I was broadly supportive of the Lab’s attempts to diversify their product portfolio, even iff the effort always did seem half-hearted. Companies with all their eggs in one basket tend to by very vulnerable to any number of circumstances not always of their own making – although equally, they can also sit very comfortably in a niche and enjoy a long life. Even with Blocksworld still on the books – and another repeated statement that the company will be standing beside that product – it now seems that for better or worse, the Lab has opted to keep only their golden egg, Second Life, and focus on perhaps producing another they can nurture alongside it.

Skill Gaming enforcement in play; CapEx suspends activities

Posted on by Inara Pey

Saturday, November 1st saw the Skill Gaming Policy from the Lab coming into full effect. Notice of the enforcement was given in a blog post issued by the Lab on Monday, September 29th, which in part read:

As our FAQs explained, applicants to the program who submitted their applications prior to the September 1 deadline have been permitted to continue their skill gaming activity while their applications are reviewed.

Beginning November 1, 2014, the enforcement of our Policy will apply to all Residents, including those with pending Skill Gaming applications that await Linden Lab review and approval. As of that date, any objects or regions found in violation of our Policy will be taken down. We strongly encourage all applicants to submit any outstanding materials for our review as soon as possible so that their applications may be processed before the deadline.

At that time, there had already been a delay in the introduction of the new policy (from August 1st through until September 1st) to allow additional time for applications and applications processing.

The end of October saw further additions to the list of Approved Participants, such that some 94 regions have been reclassified at Skill Gaming regions, each of them offering a mix of games from a pool of 56 games from five approved Skill Gaming Creators. Please refer to the Approved Participants list for an idea of the spread of actual Skill Gaming Operators.

However, the enforcement deadline has brought with it one casuality, although hopefully only temporarily.

Capital Exchange (CapEx) is a fictional stock market simulation game, and although it does not offer any opportunity for direct real-world investment or profit, it does operate on what amounts to a “pay to play basis” through the trading of L$-valued “securities” in the SL-based companies listed with the exchange. As such, it falls under the remit of the Skill Gaming Policy.

Capital Exchange’s application to become a Skill Gaming activity lodged with Linden Lab since just after the new policy was originally announced July 2014, including a reasoned legal opinion from their legal counsel outlining why CapEx is a game of skill under federal and New York State laws. However, at the time the deadline was reached, they had not received formal approval as a Skilled Gaming Operator from the Lab, which has forced CapEx to suspend market activities activity until further notice.

The suspension was announced via statement from CapEx’s CEO Skip Oceanlane published on the CapEx website and via in-world note card givers at the CapEx headquarters. This announcement reads in part:

On July 16, 2014, I filled out the first form for the Skill Gaming Application for Capital Exchange Stock Market Simulation Game. From that point until today, there have been numerous correspondences between me, my law firm, and Linden Lab. As of today we still do not have an official determination, positive or negative, on our application.

I was hoping that because much of the delay has been on the part of Linden Lab, that we would be allowed to operate past the November 1, 2014 deadline set forth in a recent post by Linden Lab. Unfortunately I was informed by Rowan Linden that we cannot. So therefore effective immediately, I am suspending all trading at Capital Exchange until our application is approved or denied by Linden Lab.

Capital Exchange has been hit - hopefully temporarily - by the November 1sr, 2014 enforcement of the new Skill Gaming Policy
Capital Exchange has been hit – hopefully temporarily – by the November 1st, 2014 enforcement of the new Skill Gaming Policy

As a result of this move, as the blog post goes on to state, all trading, including person-to-person transfers (unless they were authorised before the trading halt), payment of dividends is suspended, as is any other activity carried out through Capital Exchange which would violate the Lab’s Skill Gaming policy. However, all other aspects of CapEx, such as posting to the forums (including company news etc.), continues. The announcement also notes that CapEx ATMs will still be in operation for those wishing to deposit / withdraw L$, as the Lab has not indicated that doing so would be violating the new policy.

Please refer to the CapEx blog post for detailed information on the suspension and planned steps, and to the CapEx website for further updates on the situation.

It is not currently clear if other gaming operators within Second Life have been similarly hit by the deadline being reached prior to their application being approved or turned down. It is also not clear whether the deadline for enforcement was set as a result of external pressure to comply with legal requirements, or as a result of an arbitrary decision on the part of the Lab.

If it is the latter, it is hard to fathom why the Lab opted for a blanket enforcement of the policy, rather than allowing further dispensation for those operators / creators who have submitted applications, particularly where the delays in approval appears to lie with the Lab’s own handling of applications (as seems to be the case with CapEx).

Related Links

With thanks to Nalates Urriah for the pointer to the CapEx blog post.

CDN – Lab issues data on improvements

Posted on by Inara Pey

On top of their feature blog post on recent improvements to SL, on which I also blogged, the Lab has also issued a Tools and Technology update with data on the initial deployment of the CDN.

Entitled CDN Unleashed, the post specifically examines the percentage of simulator servers experiencing high load conditions (and therefore potentially a drop in performance) on the (presumably) BlueSteel RC both before and after deployment of the CDN service to that channel – and the difference even caught the Lab off-guard.

ss
Charting servers on a production release candidate channel with high HTTP load conditions before and after we rolled the CDN code onto them (image via Linden Lab)

While a drop in load had been expected prior to the deployment, no-one at the Lab had apparently expected it to be so dramatic that it almost vanishes. Such were the figures that, as the blog post notes, at first those looking at them thought there was something wrong, spending two days investigating and checking and trying to figure out where the error in data came from – only it wasn’t an error; the loads really have been dramatically reduced.

Elsewhere, the blog post notes:

Second Life was originally designed for nearly all data and Viewer interactions to go through the Simulator server. That is, the Viewer would talk almost exclusively to the specific server hosting the region the Resident was in. This architecture had the advantage of giving a single point of control for any session. It also had the disadvantage of making it difficult to address region resource problems or otherwise scale out busy areas.

Over the years we’ve implemented techniques to get around these problems, but one pain point proved difficult to fix: asset delivery, specifically textures and meshes. Recently we implemented the ability to move texture and mesh traffic off the simulator server onto a Content Delivery Network (CDN), dramatically improving download times for Residents while significantly reducing the load on busy servers.

Download times for textures and meshes have been reduced by more than 50% on average, but outside of North America those the improvements are even more dramatic.

Quite how dramatic for those outside North America isn’t clear, quite possibly because the Lab is still gathering data and monitoring things. However, the post does go on to note that in combination with the HTTP pipelining updates now available in the current release viewer (version 3.7.19.295700 at the time of writing), the CDN deployment is leading to as much as an 80% reduction in download times for mesh and texture data. Hence why the Lab is keen to see TPVs adopt the HTTP code as soon as their release cycles permit, so that their users can enjoy the additional boost providing the code on top of enjoying the benefits offered by the CDN.

Again, at the time of writing, the following TPVs already have the HTTP pipelining code updates:

As per the Performance, Performance, Performance blog post, the Lab want to hear back from users on the improvements. Comments can be left on the Performance Improvements forum thread, where Ebbe and Oz has been responding to questions and misconceptions, and Whirly Fizzle has been providing valuable additional information.

Lab asks: how is SL for you?

Posted on by Inara Pey

Just Another Tequilla Sunrise, Isle of Love; Inara Pey, October 2014, on FlickrSL should be looking and feeling a lot better for many of us as a result of recent work by the Lab – how’s it going for you? The Lab asks you drop them a line in the forums or via Twitter (image: Just Another Tequilla Sunrise, Isle of Love (Flickr) – blog post)

Following-on from the grid-wide deployment of CDN support and the promotion of the HTTP pipelining viewer as the de facto release viewer, the Lab has blogged about recent improvements to Second Life, finishing with the question “how is it for you?”

The blog post, entitled, Performance, Performance, Performance, opens thus:

Has Second Life seemed a bit faster for you lately? Improving performance for all Second Life users has been an important focus for us at Linden Lab, and we’ve recently seen some great results from several projects that should make your Second Life experiences faster, smoother, and more reliable.

It goes on to make fair mention of the CDN / HTTP work, noting:

Faster Texture & Mesh Loading
The entire grid is now using a CDN service for textures and meshes. This change means that textures and meshes should load more quickly, particularly for those who login to Second Life from places that are far from our US data centers. Our testing showed dramatic improvements: average download times for textures and meshes have been reduced by more than 50% on average, and the improvement is even more dramatic outside of North America.

Quicker Viewer-Server Communications
Another way we’re enhancing Second Life performance is through our HTTP project, which improves the way your Viewer communicates with grid services. With the HTTP Project Viewer out now, the faster content download times you’ll see thanks to the CDN change get even better – we’re talking 80% faster!

For those who may have missed news on the HTTP pipelining viewer and the CDN support, you can catch-up with things via a couple of posts on this blog: SL project updates week 42/2: Monty’s HTTP update and the HTTP pipelining viewer, and HTTP pipelining viewer reaches release status as CDN support is grid-wide.

Mention is also made of the recent Group Chat updates (work is still continuing on this, and you can get updates via the Group Chat tag in this blog).

However, perhaps the most interesting aspect of the blog post is the news that there have been some significant infrastructure updates with the SL Marketplace, which appear to have slipped mention through other mediums such as the var UG meetings. Here the blog post notes:

Speeding Up the Marketplace
If you visit the Marketplace today, you should be seeing a much snappier experience than in the past. We recently deployed infrastructure upgrades for the Marketplace, and the site has since shown some of the best performance we’ve ever seen from it. Even during peak usage periods over the weekend, when in the past performance would degrade, we’re seeing response times that average 70% faster and page load times that are 30% faster than before the changes.

I’ve not used the Marketplace of late – although I have been covering the upcoming Viewer-managed Marketplace (VMM) changes that will be occurring in 2015, and which the Lab is currently gearing-up for, along with merchants and TPVs.

So have you noticed changes and improvement in your Second Life experience? If so, then why not follow the Lab’s request:

So, How’s Second Life Performing for You?
Performance improvements are generally behind-the-scenes work, and we know it’s not always as exciting as rolling out a new feature, but these changes directly impact all our Second Life experiences and our daily lives in-world. We hope you’re starting to notice the effects of these improvements – if you are, please let us know in the Forums, on Twitter, or however you prefer.

Note that is let the Lab know – not me (although general views are always welcome here, they might not be picked-up by the Lab)!

HTTP pipelining viewer reaches release status as CDN support is grid-wide

Posted on by Inara Pey

On Wednesday, October 29th, the Lab promoted the HTTP pipelining viewer to the de facto release viewer, a move that came just after the grid-wide deployment of CDN support on Tuesday, October 28th. While the two are complementary rather than reliant upon one another, both should help improve the majority of users’ Second Life experience to some degree.

Monty Linden: the HTTP pipelining viewer marks the culmination of over 2 years of work inproving SL's HTTP capabilities
Monty Linden: the HTTP pipelining viewer marks the culmination of over 2 years of work improving SL’s HTTP capabilities

The HTTP pipelining viewer is the latest phase of over two years of work on Second Life by Monty Linden, and which has involved both the viewer and the servers and back-end services which support SL.

The work, originally a part of Project Shining, which was itself heralded as complete in June 2014, initially focused on texture handling between the servers and the viewer. Since then, Monty has gone on to tackle a number aspects of improving the use of HTTP in Second Life, such as making connections more robust and reliable, improving throughout to the viewer via HTTP, and so on.

The HTTP pipelining viewer, as the name suggests, leverages HTTP pipelining, a technique in which multiple HTTP requests are sent on a single TCP connection without waiting for the corresponding responses, which significantly improves the download of data (currently avatar baking information, texture data, and mesh data) to the viewer. The upshot of this is that the impact of a user’s physical location on scene loading is reduced, improving their overall experience.

As well as this, the HTTP viewer includes significant improvements to inventory folder and item fetches, which can markedly decrease the time taken for inventory to load, particularly if a user’s local inventory files have been flushed as a part of a cache clearing (or similar) exercise.

These inventory updates alone are liable to be appreciated by users as the viewer-side HTTP code gains wider adoption by TPVs. Tests have shown that a decently structured inventory (e.g. one that uses a folder hierarchy, rather than everything dumped into just a handful of top-level folders) of 100K can have a “clean” load time of 16-18 minutes reduced to around 3 minutes.

Earlier in October 2014, Monty blogged on his work, showing how both the CDN and the HTTP pipelining viewer, coupled with his earlier HTTP improvements have benefited texture and mesh fetching in SL. If you’ve not read that blog post, I recommend that you do.

Monty Linden's recent blog post shows how the HTTP work has improved texture and mesh fexture within SL
Monty Linden’s recent blog post shows how the HTTP work has improved texture and mesh texture fetching within SL

As well as working on HTTP, Monty has also been engaged on rebuilding and cleaning-up many of the third-party libraries used in the building of the viewer. This work should not only improve the viewer build process and such third-party libraries are consistently used in the build process, it may also help pave the way toward the Lab producing 64-bit versions of their viewer in the future.

Continue reading “HTTP pipelining viewer reaches release status as CDN support is grid-wide” →