SL projects updates week 18/1: server, viewer

UNIA launches at 12:00 noon on Monday, April 27th
MadPea’s UNIA is now open for those of a brave disposition, and uses Experience Keys / Tools

Server Deployments Week 18

As always, please refer to the server deployment thread for the latest news.

  • There was no Main (SLS) deployment on Tuesday, April 28th.
  • On Wednesday, April 29th the three RC channels all received the same sever maintenance package. This comprises Internal server logging changes  and a new flag for llGetObjectDetails()  – OBJECT_LAST_OWNER_ID; plus new data which can be requested  via llGetEnv(). These are:
    • “agent_limit”- get the maximum number of avatars normally allowed on the region (teleport home, and login to last location, are allowed to exceed this).
    • “estate_name”- returns the name of the estate (e,g, “mainland”, “Linden Homes”, “My Happy Estate”, etc. )
    • “region_cpu_ratio”- returns the number of regions per CPU for this region type (i.e. “1” or “4”)
    • “region_product_name” – returns the type of region this is: “Estate / Full Region”, “Mainland / Homestead”, “Estate / Openspace”, “Estate / Full Region – Skill Gaming” etc.
    • “region_product_sku” – returns the region’s product number as a string
    • “region_start_time” – returns the time the region was last (re)started, in llGetUnixTime format
    • “simulator_hostname”  – returns the simhost running this region. Same as llGetSimulatorHostname but without the script delay.

Commenting on the llGetEnv() updates at the simulator User Group meeting on Tuesday, April 28th, Simon Linden, who made the updates, said, “these are all pretty simple ones … I went for the easy pickings.  Basically, information we already  sent to the viewer, or was readily available, and thus not a privacy issue.”

He continued, “There was one [further option] for the max number of agents that was in the original list but that one got skipped … not part of a sinister plot but I overloooked it.  want to do some other things with that limit sometime soon as well 🙂 … I’d like to see how the region and viewer performs with bigger numbers. Things go bad with many AVs for a variety of reasons … the server has more updates to send to more people, all wearing more scripts and AOs and HUDS [and] the viewer gets overwhelmed with too many complex avatars and too many textures in the download and graphics pipeline.”

SL Viewer

The Avatar Layer Limits RC viewer updated to version 3.7.29.301305 on April 28th, bringing it to code parity with the current release viewer. This RC allows users to wear up to 60 wearable layers (jackets, shirts, tattoo, alpha, etc.) in any combination – so you can wear 60 tattoo layers with it an nothing else, if you want – rather than being restricted to wearing a maximum of 5 of each type of layer.

Other Items

Online / Offline Indicators

People are noticing that the group chat list (the list of group members in the Group panel), is now much slower to update as people’s status changes (i.e. whether they are on-line or off-line). This is intentional, and comes as a result of the recent improvements made to group chat.

In particular, and as I reported in these pages as work on group chat commenced in 2014, the volume of people logging-in to and out of SL can generate a huge amount of updates for the group chat service (given your status has to be sent to every group of which you’re a member, and to over member of that group who is online to update the group list in their viewer with your new status), meant that more time was being consumed by the group chat servers in handling these update messages than in handling actual messages.  The fix for this problem means there is a natural delay in group list updates, as they are now processed differently to reduce the impact they have on message handling.

However, some people have started noticing that some group chat lists with 20+ members seem to take a very long time to update – times of 5-10 minutes have been mentioned, and this is causing some confusion when seeking things like assistance from group owners / moderators (as they can appear to be logged-in long after they have logged-off). It’s also bee reported that at times the list seems to get stuck with no updates until the group chat itself is closed and re-opened, although this appears to be somewhat intermittent.

SL project updates week 17/1: server, viewer, Avatar Complexity

221B Baker Street; Inara Pey, April 2015, on Flickr 221B Baker Street, circa 2012-2015, as seen in the BBC’s series Sherlock – and in Second Lifeblog post

Server Deployments, Week 17

As always, please refer to the server deployment thread for the latest updates.

  • On Tuesday, April 20th, the Main (SLS) channel received the server maintenance package deployed to all three RC channel in week #16, which comprises internal server logging changes and new flags for llGetObjectDetails()
    • OBJECT_BODY_SHAPE_TYPE – returned list entry is a float between 0.0 and 1.0. Anything > 0.5 is male, otherwise female; -1.0 if the avatar is not found
    • OBJECT_HOVER_HEIGHT – returned list entry is a float, -1.0 if the avatar is not found.
  • There will be no deployment or restart on the three RC channels on Wednesday, April 22nd.

This means there will be no Main channel roll in week #18, but there should be a new RC update, although this is still being worked on.

SL Viewer

The Avatar Layer Limits RC viewer updated to version 3.7.28.301019 on April 20th. This viewer allows users to wear up to 60 wearable layers (jackets, shirts, tattoo, alpha, etc.) in any combination and any number per layer up to the overall maximum of 60, rather than each individual layer being limited to a maximum of 5 items.

The Tools Update RC viewer has been performing very well since the last update (April 15th), and there has been something of a debate in the Lab as to whether or not to promote it to the de facto release viewer. While there is no hard-and-fast rule about when an RC is promoted to release status, very often the Lab prefers to leave two weeks between releases unless something is urgently required. Sticking to this would mean the viewer won’t be promoted until week #18 (week commencing Monday, 27th April); however we’re still early in the week, and things might change.

Viewer Managed Marketplace Beta

The Viewer-Managed Marketplace (VMM) officially started an open beta test on the main grid, which is scheduled to last for about a month for details see:

Avatar Complexity

Avatar Complexity is the term the Lab has settled upon for the upcoming functionality which provides greater control to user to define how other avatars are rendered in their world-view.

The idea is that as avatars can often be the single biggest impact on the viewer in terms of rendering, particularly in crowded places, so  Avatar Complexity will present a means by which avatars which require a load of render processing by your GPU can be rendered as a solid colour instead, which should help with performance on lower specification systems. Due to their solid colours, avatars rendered in this way have already been dubbed Jelly Babies or Rainbow People.

At the Open-source Developer’s meeting on Monday, April 20th, Oz Linden explained that “Avatar Complexity” has been chosen for the name of the capability to distinguish it from avatar imposter rendering, which will remain in the viewer alongside Avatar Complexity when it arrives. The difference between the two can be summarised as:

  • Avatar impostor rendering is a simplified and less frequent rendering of avatars further away from you, while those close to you remain fully rendered
  • Avatar Complexity renders any avatar exceeding the value set within your viewer as a single, solid colour, regardless of the avatar’s distance from you.
Avatar complexity is intended to help those who may hit performance issues as a result of their GPU struggling to render complex (hight render cost) avatars, by rendering such avatars as solid colours.
Avatar complexity is intended to help those who may hit performance issues as a result of their GPU struggling to render complex (hight render cost) avatars, by rendering such avatars as solid colours.

Oz further indicated that Avatar Complexity will be managed via the Advanced panel in Preferences > Graphics, and will initially be enabled / disabled in the official viewer based on your GPU’s benchmark (the value use to determine the viewer’s default graphics settings when first installed). Some TPVs may opt to leave the capability disabled by default (once the code is available for inclusion in TPVs), and allow users determine whether they wish to use it or not.

Currently, work at the Lab is focusing on a couple of aspects of the functionality:

  • Toning down the colours used by the viewer when rendering avatars in this way – as the functionality can currently be tested via two debug settings within the viewer, there have already been strong criticisms of that “Jelly Baby” rendering on account of the brightness of the colours
  • Server support is being added to pass on the counts of avatars that are and are not rendering to those using Avatar Complexity.

It is also probable that before the capability appears in a project viewer, it will also be set to  display notifications when you change your own complexity, and when the number of avatars not rendering you changes.

If you wish to experiment with the settings are they are at the moment, go to Advanced > Debug Settings and type-in RenderAutoMute. Select RENDERAUTOMUTEFUNCTIONS and set it to 7, then experiment with values under RENDERAUTOMUTERENDERWEIGHTLIMIT (start with 100,000, and increase or decrease it to alter the number of avatars around you rendered as solid colours (lower values = more avatars rendered as colours).

SL project updates week 16/1: server, viewer updates, misc

The City Skyline - Remnants of Earth
The city – Remnants of Earthblog post

Server Deployments Week 16

As always, please refer to the server deployment thread in the forums for the latest information and updates.

On Tuesday, April 14th the Main (SLS) channel was updated with the server maintenance package previously deployed to all three RC channels. This comprises a crash fix, minor CDN configuration updates and an internal server configuration update.

On Wednesday, April 15th, all three RC channels should receive a new server maintenance package, which comprises internal server logging changes and new flags for llGetObjectDetails()

  • OBJECT_BODY_SHAPE_TYPE – returned list entry is a float between 0.0 and 1.0. Anything > 0.5 is male, otherwise female; -1.0 if the avatar is not found
  • OBJECT_HOVER_HEIGHT – returned list entry is a float, -1.0 if the avatar is not found.

SL Viewer Updates

The Maintenance RC viewer, version 3.7.27.300636 was promoted to the de facto release viewer on April 13th. The viewer contains multiple fixes and improvements, as detailed in the release notes.

This release also includes the fix for the URI parsing error, which was originally issued in the HeatWave RC viewer (formally version 3.7.27.300424, which has been withdrawn from the release channel as a result.

Webkit Replacement, Flash and Quicktime

As I’ve reported on a number of occasions, Webkit is a third-party library which has been used within the viewer for a number of media-related tasks (powering the built-in web browser, displaying profiles, and is used with MOAP  and many in-world TVs). However, it has been something of a problem for the Lab,  with out-of-date libraries and other issues.

Because of this, there is a project under-way in the Lab to replace webkit with the Chrome Embedded Framework (CEF). Work on this within the Lab has been progressing, and they now have CEF working with the windows version of the viewer, and are now focusing on getting it working on the Mac version.  There are no plans to release a test or project viewer with CEF support until it is running on both platforms (it is thought that Linux will be able to use the Mac version).

Avatar Complexity (RenderAutoMute Functions)

The new rendering controls will allow users to set a level above which avatars will be rendered as a solid colour
The new rendering controls will allow users to set a level above which avatars will be rendered as a solid colour “jelly baby”

In week #47. 2014, I reported on how the Lab is working to give greater control to users over how other avatars are rendered in their own view.

Avatars can frequently have very high render costs associated with them which, even in modestly populated areas, can have a detrimental impact on viewer performance on lower specification hardware.

The idea with the new, still-to-be-released functionality is that users will be able to define a render weight for their viewer when drawing avatars. Any avatar that exceeds this limit will be rendered as a solid colour “imposter”, regardless as to how near / far they are from a person’s viewpoint.  Thus, the rendering load is reduced, improving overall performance.  Because of the solid colour aspect of the avatars when rendered in this way, they were somewhat quickly dubbed “Jelly Babies” after the sweets of that name. note they are only rendered like this in your own view, it doesn’t affect how others see them.

This work has been going on for some time, now, and is approaching maturity. Commenting on it at the Open-source Developer’s meeting on Monday, April 12th, Oz Linden indicated that things are currently waiting server side updates. Included in the functionality is a means by which someone can see the number of other people who are rendering their avatar as a “jelly baby”.

The capability can actually be experimented with at the moment, although it is a case of trial and error until the new UI controls are added to the viewer. Should you wish to try, go to Advanced > Debug Settings and type-in RenderAutoMute. This will list a series of options, of which RENDERAUTOMUTEFUNCTIONS and RENDERAUTOMUTERENDERWEIGHTLIMIT are the two you need:

  • RENDERAUTOMUTEFUNCTIONS is essentially the “on / off” option for enabling the other options, and must be set to 7 in order for any of them to work
  • RENDERAUTOMUTERENDERWEIGHTLIMIT is the function that determines how avatars are rendered. Try starting with a value of around 100,000 and experimenting from there.

Group Chat

BUG-9020 reports issues with people being unable to see anything typed in certain group chats they belong to – either their own messages, or anything typed by anyone else. The problem appears to possibly be more widespread than the report indicates – if you are experiencing a problem, please consider adding the details to the report: the specific groups, etc., and specific issues. The Lab is currently looking into this and checking through the additional logging / diagnostic tools they’ve added to the group chat services to see if anything is showing-up as causing the problem.

Other Items

In-viewer Translation Tool Fix

As noted in my week #12 update, the built-in viewer translation tools are now pretty much broken (Google and Bing). Nalates Urriah filed a bug report on the Bing situation recently (see: BUG-8794 “The Bing API used by the viewer is depreciated [sic]”).

Commenting on the situation at the Open-source Developer’s meeting on Monday, April 13th, Cinder Roxley indicated that the Alchemy TPV team are working to get the viewer translation tool working again, although there is currently no ETA on this. The fix is liable to appear in the Alchemy viewer, but the code will be contributed to Linden Lab.

Forum Log-in Issue

As noted in BUG-8953, there is currently an issue with signing-in to the the SL forums, and staying logged-in. the problems are broadly two-fold. In short, people are finding they are being randomly logged-out of the forums for no apparent reason, or are being redirected to the top-level community page when logging-in, rather than being redirected back to the page in the forums they had displayed prior to the log-in request being displayed (e.g. when replying to a post).

SL project updates week 15/1: server, viewer, HTTP Inventory reminder

... and don't miss out on the merfolk's beach, complete with pier and fun fair!
Don#t forget you can plunge into learning about SL’s extensive merfolk and undersea community this week, thanks to the folk at Fanci’s Deep and the Safe Waters Foundation. There’s undersea tours, dances, dolphin rides, shopping opportunities, freebies and a whole lot more. You can even visit the mer beach and fun fair (above)! To find out more, read the blog post on the event, which runs through until April 11th

Server Deployments Week 15

As always, please refer to the server deployment thread in the forums for the latest information and updates.

  • On Tuesday, April 7th the Main (SLS) channel will receive the server maintenance update previously deployed to the three RC channels. This is primarily focused on trying to prevent  inventory loss issues, and sees UDP inventory messaging deprecated (see HTTP Inventory, below, for more important information)
  • On Wednesday, April 8th all three RC channels should receive a new server maintenance package comprising:
    • A fix for a server crash when rezzing an object
    • A minor change for CDN configuration
    • Adjusted internal server configuration.

SL Viewer Updates

A new release candidate viewer was released towards the end of last week. The HeatWave viewer, version 3.7.27.300424. This is essentially the maintenance RC viewer with an additional URI parser fix to prevent a viewer crash bug, but has retained a project name to differentiate the two RCs from one another.

 HTTP Inventory

With the Tuesday deployment (noted above), the main grid now only supports HTTP Inventory fetching. This means you must have the HTTP Inventory option enabled in the viewer (it can be found under the Develop(er) menu).

Should you disable it for any reason, you will encounter two issues:

  • Your avatar will not render, but will remain a cloud
  • Should you refresh your inventory for any reason (clear cache), your viewer will never complete the process of inventory fetching.

Unfortunately, and coincidentally, the Main channel deployment on Tuesday, April 7th came at a time when asset server / inventory issues were being experienced across the grid, and inventory database maintenance was carried out as a result.

Note that from Tuesday, April 6th, you must ensure HTTP Inventory is enabled in the Develop menu (sometimes called the Developer menu in TPVs) in order to help avoid inventory and / or avatar rendering problems
Note that from Tuesday, April 6th, you must ensure HTTP Inventory is enabled in the Develop menu (sometimes called the Developer menu in TPVs) in order to help avoid inventory and / or avatar rendering problems

These issues and the maintenance may have masked any problems some people may have been having purely as a result of HTTP Inventory being disabled in their viewer.

Therefore, if you are encountering problems with your avatar remaining a cloud, or your inventory failing to load, please try the following steps to see if they resolve your situation:

  1. Make sure you have the Develop(er) menu enabled in your menu bar at the top of the viewer. Press CTRL-ALT-Q if you cannot see it.
  2. Click on Develop(er) to list the menu options.
  3. Make sure there is a tick in front of the HTTP Inventory option.
  4. If HTTP Inventory does not have a tick in front of it, then it is disabled. Click on it to enable it (and display the tick).
  5. Closed the Develop menu and re-log.
  6. Hopefully, following your re-log, your avatar will render / your inventory load properly.

UDP Inventory Messaging Deprecated

The reason for this is that the Lab has now deprecated the “old” method of inventory messaging (referred to as UDP messaging). However, if you disable the HTTP Inventory option in your viewer, the viewer will still attempt to use the “old” method, and thus you’ll have problems.

There are plans in hand for the Lab to remove the HTTP Inventory option from the viewer, and some TPVs may opt to remove it ahead of any update from the Lab. Until that time, it is essential you keep the option enabled to assist with the smooth functioning of your inventory.

Experience Keys / Tools

Not a lot to report on this project. Simon Linden has been working on the Key Value (KVP) database store used by Experiences. This work appears to be related to the Lab working to ensure the when deployed Experience Keys / Tools can be properly scaled to meet the anticipated demand for them. Commenting in general terms on the work, Oz Linden said during the Simulator User Group meeting on Tuesday, April 7th, “if we are as successful as we’d like to be with Experiences being adopted, it would run into problems. So we’re trying to solve them before we get to that point.”

SL project updates week 14/2: server, viewer, CDN

The Trace Too; Inara Pey, March 2015, on Flickr The Trace Too (Flickr) – blog post

Server Deployments Week 14 – Recap

As always, please refer to the server deployment thread in the forums for the latest information and updates.

  • There was no deployment to the Main SLS channel on Tuesday, March 31st, due to the inventory issues arising from the week #13 RC deployment – see my update here for details.
  • On Wednesday, April 1st, all three RCs received the same update to the current server maintenance package to fix the issues with Trash failing to purge in non-AIS v3 viewers (see BUG-8877. and my coverage of the recent issues here). Those suffering from inventory fetch failures on RC regions are advised to re-enable HTTP Inventory in their viewers, if disabled (found under the Develop menu).

SL Viewer

Wednesday, April 1st saw the release of the Project BigBird viewer (yes, seriously!), version 3.7.27.300377, which contains the various fixes for attachment issues which the Vir Linden has been working on. Specific fixes offered are listed as (note the MAINT designations are for the Lab’s internal JIRA, and thus non-viewable):

  • MAINT-4351 HUDs and attachments intermittently and randomly detach after teleports, sometimes reattaching on their own shortly after, sometimes staying detached completely, or showing as “worn on Invalid Attachment Point” while still detached
  • MAINT-4653 [Attachment-RC] When using “Add” or “Attach to” to attach multiple attachments at the same time, some attachments fall off and some get attached to the wrong attachment point
  • MAINT-4917 Attaching multiple objects generates multiple bake requests
  • MAINT-4918 Removing multiple attachments generates redundant detach requests
  • MAINT-4919 Attempting to wear an outfit with more than 40 attachments will fail

UDP Paths: HTTP Inventory, Textures and More

As noted at the top of this report, the week #13 RC deployments have been causing some inventory-related issues, one of which –  the Trash purging problem – has been fixed with this week’s RC RC deployment.

The second issue  – failures in inventory fetching following clearing cache on RCs regions – has been caused by a combination of the Lab deprecating the UDP message path for inventory updates and users having the HTTP Inventory option in the viewer (found under the Develop menu – CTRL-ALT-Q) disabled (unchecked).

Given this path has been deprecated, it is essential you keep HTTP Inventory enabled (the Lab will be removing the option from the Develop menu in the future to prevent it being unwittingly disabled).

Speaking at the Server Beta Meeting on Thursday, April 2nd, Oz Linden indicated that the Lab would be taking steps in the future to deprecate UDP messaging is “high on the list” for being deprecated in the future, given that textures have now moved to the CDN.

The CDN and Switching Further Services

While discussing the issue of UDP messaging, Oz again re-iterated the desire to pivot things like fetching animations and sounds away from UDP and onto HTTP, with the aim of provisioning them through the CDN, further lifting the load the simulators currently carry. However, he caveated this with two important points:

  • While this is something he’d like to see done, and is in the plans for SL’s future, the work hasn’t actually be scheduled yet, must less started; therefore it is not something that will be happening in the short-term (or perhaps even the medium term)
  • The Lab is working on a further round of CDN improvements – again, no time scale is available for their implementation – but there won’t be any additions to the data delivered via the CDN until after such improvements have been deployed.

One aspect here is that, in terms of the simulator load and in terms of the vast majority of users, the switch-over to avatar, mesh and texture data to CDN-based services has been a success for the Lab. However, as we’ve also seen, it has resulted in issues for some users, up to and including what is a degraded service due to the actions of at least one ISP.  While the latter is not something the Lab or their CDN provider can directly tackle, it does point to the fact that while off-loading the heavy lifting from the Lab’s servers can make for improvements, it can affect users in other ways.

Hence why the Lab is being cautious in approach, and is continuing to work with its CDN providers to try to improve the service as far as can be done, in the hope of reducing the number of ways in which users might find SL a poorer experience as a result of the CDN implementation. However, exactly what can be achieved and issues mitigated, remains to be seen.

In the meantime, as as per part 1 of this week’s update, if you do feel mesh and texture rendering isn’t what it once was, try following Monty Linden’s interim ideas  for easing things.

SL project updates week 14/1: server and viewer updates, misc items

Piony Hideout, Lions Hill; Inara Pey, March 2015, on Flickr Piony Hideout, Lions Hill (Flickr) – blog post

Server Deployments

As always, please refer to the server deployment thread in the forums for the latest information and updates.

There was no deployment to the Main SLS channel on Tuesday, March 31st, due to the inventory issues arising from the week #13 RC deployment – see my update here for details.

On Wednesday, April 1st, all three RCs will receive the same update to the current server maintenance package. This update is specifically aimed at correcting the trash purging issue reported with BUG-8877. However, the fix does not address the issue of inventory fetching hanging if HTTP Inventory has been disabled within the viewer.

This is because the Lab regards UDP inventory fetching as a deprecated protocol path, as indicated by the release notes, and non-HTTP based inventory fetching is now being phased out. As such, it is anticipated the option to disable HTTP Inventory within the viewer is likely to be hidden / removed at some point in the future.

SL Viewer Updates

The Tools update RC viewer updates to version 3.7.27.300242 on Monday, March 30th, bringing it into line with the current release viewer (3.7.26 with Avatar Hover Height).

The Maintenance RC viewer updated to version 3.7.27.300323 on Tuesday, March 31st. also bringing it into line with the current release viewer,  and includes fixes for many of the bugs and issues encountered with the initial release of the RC.

Other Items

Diagonal Region Rendering Issues

A fair while ago now (late 2012 / early 2013 in fact), I reported on issues that had been noted with regions seeming to be “missing” when seen from other regions.The problem was originally reported in SVC-8130, which is still marked as “unresolved”, and it had been hoped that fix for SVC-8019 would address the problem as well as dealing with other issues. However, the problem has continued intermittently ever since, with numerous issues marked as duplicates of SVC-8130 being reported, with the issue most recently being seen when looking at regions diagonally opposite Brocade on the Mainland.

The return of the missing regions issue (if it ever really went away): looking north-east from Brocade towards where Mullein isn't on the Mainland. Not, as well, the region kitty-corner beyond Mullein is also absent the view, although both appear on the map
The return of the missing regions issue (if it ever really went away): looking north-east from Brocade towards where Mullein isn’t on the Mainland. Not, as well, the region kitty-corner beyond Mullein is also absent the view, although both appear on the map

The problems are regarded as handshaking / communications issues between region, and are generally resolved through a region restart; although understandably, some holding regions on Mainland are reluctant to request a restart as this can affect multiple other region as well.

Multiple Calling Cards

Most of us are familiar with calling cards in the viewer. They can be obtained by friending someone, or by someone giving you their card (or you giving them your card, and are useful for things like  opening people’s profiles from inventory (particularly handy in cases where you haven’t friended someone, and so don’t have to use search to locate them), or initiating an IM conference call.

One of the many calling cards issues - spawning multiple copies of the same card (images via Jessica Lyon)
One of the many calling cards issues – spawning multiple copies of the same card (images via Jessica Lyon)

However, within v3-style viewer they can also be annoying, as they have a tendency to multiple for no readily apparent reason. People can often have a set of calling cards under the Calling Cards folder, which can be partially replicated in the Friends sub-folder, and then fully replicated in the All sub-folder beneath that, for example. Or individual cards can get spawned multiple times across one or more folders for no readily apparent reason.

The problem here is that having a high load of calling cards can generate problems logging-in to SL, where they run out of curl multihandles, causing log-in to hang or for them to disconnect on logging-in. This can usually be solved through … wait for it … disabling HTTP Inventory and then logging-in and deleting them, but this may not be an option in the future (see the notes at the start of this report). Given that calling cards get re-spawned following a re-log after deletion, and can start multiplying again, Oz noted in the SUG meeting that they are now an area “worthy of some study”.

Rendering / Rezzing Failures and the CDN

In week #13 I reported on rendering / rezzing issues being experienced by some people in the Florida / Alabama region of the USA. Since that time, the Lab’s investigations through the CDN provider have suggested the the ISP in question (Mediacom) has degraded the service, possibly due to the volume of traffic coming from the CDN. This was a concern voiced early-on during the CDN implementation, but this appears to be the first time such a move has been confirmed. In the forum thread on the matter, which has seen more input on the situation, Monty Linden has offered some interim ideas that may help users experiencing problems, while also emphasising the Lab is still working with the CDN providers to further refine the service.