Tag: Server Updates

SL project updates week 44/2: server, viewer, issues

Posted on by Inara Pey
Darkwood, for a Calas Galadhon Halloween
Darkwood, for a Calas Galadhon Halloween! (Blog post)

Server Deployments Week 44 – Recap

As always, please refer to the server deployment thread for the latest information and updates.

  • On Tuesday, October 28th, CDN support was deployed across the Main channel, meaning that the entire grid now utilises the Highwinds CDN for texture and mesh fetching. As the 130 regions deployed to the Snack channel were all originally from the Main channel, they have been reabsorbed into that channel, and Snack has once again be dissolved
  • On Wednesday, October 29th, all three RC channels received the same server maintenance project. which includes some minor improvements, which were described by Maestro Linden at the Server Beta meeting on Thursday, October 30th, as, “just some code cleanup and some extra optional debug logging.”

SL Viewer

On Wednesday, October 29th, the Lab promoted the HTTP Pipelining RC viewer (version 3.7.19.295700) to the de facto release viewer. As I’ve noted in previous blog posts, this viewer:

  • Can issue multiple asset fetches on a connection without waiting for responses to earlier requests. This reduces the impact of a user’s physical location on scene loading and generally improves the experience for everyone
  • Includes significant improvements to inventory folder and item fetches, which can markedly decrease the time taken for inventory to load.

All other SL viewer remain as per my Current Viewer Releases page.

Issues and Problems

Mesh Loading Issues

There have been mixed reports of mesh loading issues, which may or may be linked to some long-standing issues with mesh attachments. During the Server Beta meeting on Thursday, October 30th, the issues of mesh items which should be cached by the viewer being slow to load, don’t show up at all or show up with a low LOD, requiring a relog. Interestingly, the meshes can appear correctly when using one account, but not with another, either though both accounts are drawing from the same viewer cache.

Whirly Fizzle noted that there is a potential (and equally weird) fix for odd mesh loading issues, commenting:

There’s a weird mesh loading issue (not new) that’s “fixed” by deleting a certain per account file. I don’t even know why it should even work but it does. Sometimes a certain mesh gets stuck and will not render for a certain account. Relog doesn’t fix it & neither does a cache clear. You have to delete texture_list_last.xml in the per account settings files.

Texture Load Issues

There have been a couple of reports of texture loading issues which are still under investigation.

In the first, there have been  some reports that textures are getting stuck when “half loaded”, so that they remain blurred or may fail to render. It’s not clear how widespread this issue is, or whether it might be related to BUG-6382, although this is also unclear.

In the second, one content creator has reported customers are suddenly having issues with a HUD-based system for changing the colour of their shoes. When used, the textures rez very slowly, or the shoes get stuck as grey for up to 5 minutes at a time, or get stuck at a 1st level load until the wearer teleports or relogs. This is apparently a completely new behaviour with the shoes in question, and it appears to be exacerbated when using the HTTP viewer.

AMD Catalyst™ 14.9.2 Beta Woes

The latest AMD Catalyst™ 14.9.2 Beta drivers will not render rigged mesh unless hardware skinning is disabled. The current workaround is to either disable hardware skinning (which is not ideal when it comes to rendering mesh in general, and ALM features like materials will not render) or to remain on / roll-back to the 14.9.1 beta drivers if the problem is encountered. See BUG-7653. As was mentioned in the Server Beta UG meeting, this could be particularly confusing to new users running AMD GPUs and who are using the Lab’s default mesh avatars.

The latest AMD Catalyst™ 14.9.2 Beta driver issue: with Hardware skinning enabled, rigged messhes will not render (l); disable it, and they'll render OK (r) - click for full size; image courtesy of Maestro Linden
The latest AMD Catalyst™ 14.9.2 Beta driver issue: with Hardware skinning enabled, rigged meshes will not render (l); disable it, and they’ll render OK (r) – click for full size; image courtesy of Maestro Linden

E-mail Communications Issues for In-world Objects

A further JIRA has been filed in respect of e-mail communications with in-world objects hanging for no readily apparent reason (see: BUG-7554). similar issues have been reported in the past (although this one has a publicly viewable report), and Caleb Linden is currently poking at things.

Continue reading “SL project updates week 44/2: server, viewer, issues” →

SL project updates week 44/1: Server, CDN update

Posted on by Inara Pey

The Pines at Jacob's Pond, Jacob; Inara Pey, October 2014, on FlickrThe Pines at Jacob’s Pond, Jacob (Flickr) – blog post

Server Deployments Week 44

As always, please refer to the server deployment thread for the latest information and updates.

Main (SLS) Channel

On Tuesday, October 28th, CDN support was deployed across the Main channel, meaning that the entire grid now utilises the Highwinds CDN for texture and mesh fetching.

As the 130 regions deployed to the Snack channel were all originally from the Main channel, they have been / will be reabsorbed into that channel, and Snack will once again be dissolved.

Release Candidate Channels

On Wednesday, October 29th, all three RC channels should receive the same server maintenance project. which includes some minor improvements.

SL Viewer

There are currently two RC viewers possibly vying for promotion to the de facto release viewer. These are the HTTP Pipelining RC (version 3.7.19.295700) and the Benchmark viewer (version 3.7.19.295759), which should put an end to the use of a manually maintained list of GPUs in order to initially set the graphics defaults in the viewer.

Both of these were updated on Friday, October 24th, which has delayed any promotion to the de facto release viewer while the Lab gathers performance statistics on both of them. commenting on the status of both during the Simulator User Group meeting on Tuesday, October 27th, Oz Linden indicated that all things being equal, the HTTP Pipelining viewer should be promoted in the next 24-48 hours. He also indicated that there may be a further round of updates to come to Benchmark viewer in the offing as the Lab continued to tweak it.

CDN: Next Steps

In the vast majority of cases, the CDN is working as expected for users. There is a very small minority who, possibly because of their geographical closeness to the Lab’s servers or possibly due to issues between their ISP and the Highwinds services, are experiencing slightly worse ping times to their nearest CDN nodes when compared with pinging the Lab’s servers directly.

Even with the deployment to the Main channel, the Lab is continuing to monitor reports from CDN closely. However, as previously mentioned in my CDN coverage, it is likely the scope of CDN usage will be expanding in the future to handle other asset data – sounds, animations etc.

Also, and as noted in my week 43 TPV Developer meeting report, an offshoot of the CDN work is that there is a belief within the Lab that viewer caching may not be working as well as it might be. Internal discussions have held on possibly validating whether or not this is the case, and it is likely some work will be carried out in this area – and may well involve TPVs.

However, where both the viewer cache and extending the use of the CDN to cover other asset data are concerned, there are no time frames currently in mind. At the moment, the focus is very much on get the new tool chain and build process for the viewer finalised and into production, and in dealing with bugs and issues. As such, it might be a while before specific work on the viewer cache and / or work on extending the use of the CDN gets underway.

On a wider front, as well as monitoring the direct effectiveness of the CDN service, the Lab will be “spending quite a bit of time and effort assessing just what the effect of this change has been on operations from a number of perspectives”, to quote Oz.

Other Items

LI Issues?

There are reports circulating of unexpected changes to Land Impact (LI) values. While the Lab hasn’t altered land accounting, there have apparently been incidents of LI suddenly increasing in builds which have gone unchanged; one instance quoted during the Simulator User Group meeting referenced a door which apparently increased from 0.5 LI to 3 LI.

There have been instances in the past of the viewer incorrectly reporting the LI for an item when it is pulled from inventory. Corrections can generally be made either by relogging or by returning the item to inventory and rezzing it again. Altering the physics shape of linksets (from prim to convex and back again) can sometimes lead to problems, particularly if a prim in the linkset is contains torturing (such as a hollow or advance cut / twisting) or a script. However, this issue appears to be new, and a cause is proving hard to identify.

As always, if you have encountered the problem, and it is both persistent and reproducible, please raise a bug report.

Graphics Profiles in the Viewer

A suggestion was put forward during the Simulator User Group meeting for the Lab to allow the saving of graphics profiles. This would mean, for example that you could have a graphics profile where various options – the quality slider, shadows, occlusion, draw distance, etc., could be pushed towards their upper limits; and another where the setting are more conservative and less taxing on your GPU / system.

Then, where you are in a region (your own, or somewhere you know), where you know you can use the higher settings, you can quickly enable that profile, but when you move on to a region where (say) there are a lot of avatars and a lot going on, you can select the more conservative profile and thus reduce the potential performance issue on your system without actually having to go through and manually adjust all your settings.

Responding to the idea, Oz suggested the idea might be best suited to being a code contribution – and there was some potential interest in taking the idea on. However, this does not guarantee the idea will be carried forward – but it will be interesting to see if the idea does move forward at all in the coming months.

SL project updates week 43/2: miscellaneous, group chat

Posted on by Inara Pey

Just Another Tequilla Sunrise, Isle of Love; Inara Pey, October 2014, on FlickrJust Another Tequilla Sunrise, Isle of Love, October 2014 (Flickr) – blog post

Server Deployments Week 43 – Recap

As always, please refer to the server deployment forum thread for the latest news and updates.

  • On Tuesday, October 21st, the Main channel was updated with the server maintenance release previously deployed to the LeTigre and Magnum RCs in week 42. This update includes a crash fix and improves the delivery pipeline for abuse reports
  • On Wednesday, October 22nd, the release candidate channels were updated as follows:
  • BlueSteel and Snack will remain on the CDN project, and were updated with the same server maintenance package being deployed to the Main channel
  • LeTigre and Magnum were both updated to the CDN for texture and mesh fetching.

Upcoming Deployments

There is a further server maintenance package being prepared for deployment. however, it is still undergoing testing, and it is not clear whether it will be ready for deployment on Wednesday.

CDN and the Future of Snack

With the CDN now supporting texture and mesh fetching across all of the primary RC channels, there is a chance it may be promoted to the Main channel in week 4, particularly given that, outside of CDN support, all the server channels are now in parity. However, the Lab is making no commitments to this, and is taking a “wait and see” approach on what may happen. In particular, the Ops team will be monitoring performance over the weekend, when SL usage is expected to be at its busiest since the RC-wide deployment of CDN support, to see how things hold up.

As it stands, there are an estimated 130 regions on the Snack RC. As these all came from the Main channel, they will likely be switched back to that channel once it has CDN support, and Snack will again be dissolved until it is once again required.

Group Chat

With the most recent changes to the back-end servers, Simon Linden is continuing to poke at the code to see what can be improved. He’s now looking at the code that routes the messages – where it goes to find where group members are on the grid in order to be able to deliver messages to them, with the aim of trying to improve things there.

At the moment, if you move between regions which active in a group chat, the first message to be sent to you after you move will be sent to the region where you were and generate a failure message which has to go back to the chat service, which they has to relocate you and then attempt a further delivery. It is hoped that this can be improved, and there was a brief test following the Server Beta User Group meeting on Thursday, October 23rd to this end, just to see how the code is performing.

It has been suggested that the Lab should bypass the region hosts altogether when handling group chat, and attempt to route it directly from the chat servers to the viewer. However, this would likely require some serious re-engineering; more to the point, it could introduce security vulnerabilities.

Other Items

Aditi L$ Balance

A question was asked on how account L$ balances are updated on Aditi. Maestro replied that account balances are checked nightly (SLT), and any falling below L$ 10K (because someone has been uploading mesh models for testing / viewing prior to paying to upload them on Agni, for example), they are automatically increased by L$5K. However, if someone resets their account password (which triggers a reset on Aditi), their L$ balance will change to match their L$ balance on Agni, although again, if this is below L$10K it should be topped-up but L$ 5K overnight.

SL project updates week 43/1: server, viewer, group chat

Posted on by Inara Pey
Asalia House (November 2013) - blog post
Asalia House (November 2013) – blog post

Server Deployments Week 43

As always, please refer to the server deployment forum thread for the latest news and updates.

Main (SLS) Channel

On Tuesday, October 21st, the Main channel was updated with the server maintenance release previously deployed to the LeTigre and Magnum RCs in week 42. This update includes a crash fix and improves the delivery pipeline for abuse reports.

RC Channels

On Wednesday, October 22nd, the release candidate channels should be updated as follows:

  • BlueSteel and Snack will remain on the CDN project, but should also be updated with the same server maintenance package being deployed to the Main channel
  • LeTigre and Magnum should both be updated to the CDN for texture and mesh fetching.

SL Viewer

On Wednesday, October 15th, the Lab issued a Grid Status update on potential problems accessing external websites using the viewer’s internal browser.  The notice was issued as a result of the Padding Oracle On Downgraded Legacy Encryption (Poodle) vulnerability reported by Google.

At the time the notice was released, the Lab indicated they were working to fix the issue (via the removal of SSL v3.0 support within the viewer’s browser). This work was completed at the end of week 42, and resulted in the release of the Browser Fix RC viewer, version 3.7.18.295539.

On Monday, October 20th, the Browser Fix viewer was promoted to the de facto release viewer.  Further information on it and the POODLE vulnerability can be found in a separate article in this blog.

MAC Yosemite Issues

There have been some mixed results for those Mac users updating to the 10.10 Yosemite version of OS X, which reached a consumer release status on October 16th.

While most of those updating have found it relatively smooth, there has been a report of keyboard shortcuts borking, although there is a workaround should others encounter the same problem – see here.

A potentially more widespread issue people may encounter is that with Yosemite, Apple have apparently reverted to older nVidia drivers (as reported by Cinder Roxley at the Open-source Developer’s meeting on Monday, October 20th). One JIRA on the matter has already been filed (see BUG-7575), although if Apple have chosen to roll back some / all of their default drivers, there’s not a lot the Lab can do about matters. Cinder also reports that Yosemite makes greater use of OpenGL, which she indicates can tax GPUs further.

CDN

Following the scheduled Wednesday deployment, all of the RCs should be using the CDN. This represents around 30% of the total grid, possibly a little more, given the existence of Snack. The BlueSteel deployment does not seem to have resulted in the kind of performance issues experienced during the expansion of Snack to 250 regions, suggesting the latter problems were most probably down to overloading the sim hosts with too many high volume (in terms of avatar numbers) regions. If all goes well in the coming week, it is likely that the Main channel will gain CDN support in week 44.

Group Chat

“Today we also finished an update to the group chat servers,” Simon Linden announced at the Simulator User Group meeting on Tuesday, October 21st. He continued:

Many of you have helped us test that at Thursday’s beta user group, and that’s now on the newest version. Those are back-end servers … when you send something to group chat, it goes from the viewer to the region you’re on, then to the (new) back-end chat servers, and from there it’s distributed to everyone in the group.

While no promises are being made as to the overall improvement – or how visible improvements may be to users, Simon went on to note, “I know [from] looking at the metrics and data we can gather … the systems are running much nicer than before.”

As well as working on the update issues (see my previous group chat updates), the Lab has been brainstorming ideas on how groups might be configured differently. There are a lot of groups that have very heavy chat usage, and others where chat is actively discouraged in favour of outgoing notices; however the latter are handled exactly the same as those where chatting goes on, generating updates as people log-in and out, etc., even though these aren’t actually required. Exactly what might be done is unclear at this time, but it would appear ideas are being put forward and discussed.

 

SL project updates week 42/3: server, HTTP, CDN, issues

Posted on by Inara Pey
No Signal, MIC Imagin@rium, May 2014 - blog post
No Signal, MIC Imagin@rium, May 2014 – blog post

Unless indicated otherwise, the notes below are taken from the Server Beta meeting held on Thursday October 16th, 2014. The transcript is available here, and the support agenda notes here.

Server Deployments Week 42 – Recap

As always, please refer to the server deployment thread for updates and the latest information.

  • There was no code promotion to the Main (SLS) channel on Tuesday, October 14th – this follows from there having been no deployments to the primary RC channels in week 41
  • On Wednesday, October 15th, the primary RC channels were updated as follows:
    • Bluesteel received the CDN texture & mesh fetching capabilities – release notes
    •   LeTigre and Magnum both received a new server maintenance package,  which includes a crash fix and improves the delivery pipeline for abuse reports.

SL Viewer

HTTP Pipelining RC

On Wednesday October 15th, the Lab released the HTTP Pipelining RC viewer. Version 3.7.18.295372 brings with it:

  • Pipelined HTTP Operations for Mesh and Texture Fetches: this feature allows the viewer to issue multiple asset fetches on a connection without waiting for responses to earlier requests. This reduces the impact of a user’s physical location on scene loading and generally improves the experience for everyone
  • Inventory Fetch Improvements: Inventory folder and item fetches are getting some of the same treatment that textures and meshes did in previous releases. Initial inventory load should be quicker and more robust for all users.

A blog post from Monty Linden accompanies the release, which provides further information on the HTTP project alongside of the viewer release notes.

A blog post from monty Linden arrived with the Pipelining RC viewer, and explain how the viewer (and CDN) should generate further improvements in texture and mesh fetching operations
A blog post from Monty Linden arrived with the Pipelining RC viewer, and explain how the viewer (and CDN) should generate further improvements in texture and mesh fetching operations

Maintenance RC Viewer Removal

The latest Maintenance RC viewer,  version 3.7.17.294943,  has been withdrawn from the release channel due to significant issues and a range of attachment bugs which affect it (see my week 41 update from the TPV developer meeting, under “AIS v3 Issues”).

CDN

As noted above, use of the CDN service for texture and mesh data was extended to the BlueSteel RC in week 42, and there is some anticipation within LL that it could be deployed to all three of the primary RCs sooner rather than later.

An issue has been noted on some regions running on the Snack RC where people are seeing very slow parcel information updates (e.g. the name of the parcel taking up to a full minute to appear in the status bar, for music stream changes to occur, etc.).

The thinking from the Lab is that these problems are a result of the Snack sim hosts receiving a top-heavy load of regions with high traffic counts, resulting in possible resource contentions with certain data lookups as a result – contentions which may have actually brought the regions down had it not been for the CDN, as Maestro Linden commented:

In the cases I’ve seen so far, the slowness was due to a rapid rate of requests from all the users logged into the regions.  If the sim were serving all the texture and mesh requests as it traditionally would, the regions would have ‘fallen over’ before becoming that populated.

So this would seem to be an unintentional cloud with its own silver(ish) lining (at least the regions didn’t fall over). Nevertheless, the issue isn’t welcome. Given that BlueSteel has a more even distribution of regions across sim hosts (e.g. the servers are not all packed to the gills with high volume traffic regions),  It is hoped that the issue will not exhibit itself there. Checks (either by the Lab or residents or both) are likely to be carried out on busy regions on the RC to see if the problem does raise its head, which might indicate some further investigations might be required.

Assuming all goes well with the BlueSteel deployment, it is thought that CDN support will likely extended to the remaining RCs sooner rather than later. In the meantime, for those interested in seeing how use of the CDN works for them, a list of regions by RC can be obtained via Tyche Shepherd’s Grid Survey website, allowing BlueSteel regions to be easily located. Just click the RC Server Regions button (note the list will require a little cleaning-up, post download).

Continue reading “SL project updates week 42/3: server, HTTP, CDN, issues” →

SL project updates 42/1: server, viewer, viewer-driven Marketplace

Posted on by Inara Pey

Square Pegs in Round Holes, Kashmir Dreams; Inara Pey, September 2014, on FlickrSquare Pegs in Round Holes, Kashmir Dreams (Flickr) – blog post

Server Deployments Week 42

As always, please refer to the server deployment thread for updates and the latest information.

  • There was no code promotion to the Main (SLS) channel on Tuesday, October 14th – this follows from there having been no deployments to the primary RC channels in week 41
  • On Wednesday, October 15th, the primary RC channels should be updated as follows:
    • Bluesteel should receive the CDN texture & mesh fetching capabilities – release notes
    •   LeTigre and Magnum should both receive a new server maintenance package,  which includes a crash fix and improves the delivery pipeline for abuse reports.

SL Viewer

As noted in my report here, Monday October 13th saw the release of the latest version of the Oculus Rift project viewer. Version 3.7.18.295296 includes support for the Oculus DK2.

Viewer-managed Marketplace (VMM)

Baker Linden sports a new look, but keeps the hair, shades and bling - as al lhamsters should!
Baker Linden sports a new look, but keeps the hair, shades and bling – as all hamsters should!

Baker Linden, who has been carrying out a lot of the back-end work on support of the forthcoming viewer-managed Marketplace, was on-hand that the Simulator User Group meeting on Tuesday October 14th to talk a little more about the project.

He reconfirmed that the new capabilities, which will allow merchants to carry out a number of Marketplace-specific tasks from within the viewer (create new listings with stock, Associate inventory to an existing listing, remove items from a listing, unlist goods entirely) will commence testing on Aditi in November, together with a new project viewer. This testing is liable to run well into the first quarter of 2015, and will involve both Marketplace merchants and TPV developers.

Various additional safeguards are being built into the updated delivery mechanism, both within the Marketplace web interface (which is the responsibility of one of the Lab’s web developers) and the back-end of things. For example, if someone leaves an item in their shopping cart and either the price for the item changes or it is unlisted in the intervening period prior to them continuing to the checkout, they will receive a notification of the change when they do so.

Baker himself is working to eliminate issues with No Copy items, such as preventing a race condition in which a region restart can be exploited to obtain more than one version of a No Copy item. He also indicated that when a No copy item runs out of stock, the listing for it will no longer be delivered to users’ browsers by the Marketplace (presumably until the merchant “restocks” the item) and the listing may even be deactivated.

He also provided more information on the ability to link more than one items to a Marketplace listing, stating:

For merchants, you will be able to choose which “version” is listed so you can have a special holiday-skinned object, and then simply right-click, say “use this version” and then it will start selling the new holiday version of your object!  When you’re done selling the holiday themed object, just switch back to the original version.

However, a problem here is that some people tend to let items accumulate in their Marketplace shopping cart for a period of time – perhaps a week or so – before proceeding to the checkout. It is therefore possible that if they add a “special” version of an item to their cart and leave it there for a while, the merchant may subsequently swap the listing back to deliver the “normal” version of the item – which the buyer will receive when then do eventually proceed to checkout and pay for the items in their shopping cart, leading to complaints and upset. This is something the Lab hadn’t actively considered, so Baker has taken the issue back to the office where it will be given further thought.

The discussion of the new Marketplace capabilities lead to a number of suggestions being put forward, some of which may be on their way to being filed as JIRA feature requests:

  • The ability to mark stores as favourites in the Marketplace
  • The ability to obtain a demo version of an item directly from the item’s listing page
  • The ability to associate different coloured versions of the same product in a single listing
  • The ability to view an item’s contents like the contents of in-world objects can be inspected.

A request was also made for some kind of in-world meeting to discuss the Marketplace. This is not the first time such a request has been made. However, the last time the Commerce Team directly responded to such requests, it was with a refusal to hold in-world meetings. However, there was a lot going on with the Marketplace at that time, which was causing a lot of angst; it’s fair to say a lot has changed since then. Certainly, with an extended period of testing for the new viewer-driven capabilities about to commence, which will involve both merchants and TPV developers, it would seem that putting a mechanism in place by which the Lab, merchants and developers can discuss things would benefit the project tremendously.