Tag: Server Updates

SL project updates 46/2: viewer updates, miscellaneous news

Posted on by Inara Pey

Collins Land; Inara Pey, September 2013, on Flickr, on FlickrCollins Land, September 2013 (Flickr) – blog post

The following notes are drawn from the Server Beta user group meeting held on Thursday, November 13th, the transcript for which can be found here.

Server Deployments Week 46 – Recap

As always, please refer to the server deployment thread in the forums for the latest information and updates.

  • There was no deployment to the Main (SLS) channel on Tuesday, November 11th.
  • On Wednesday, November 12th, all three RC channels received the same server maintenance package, which comprises “minor improvements” to help configure the texture and mesh CDN, by allowing the Lab to reconfigure the CDN URL if they need to, with the intention of the of making it a more dynamic host name in the future.

SL Viewer

Snowstorm RC Viewer

A new Snowstorm contributions RC viewer arrived on Thursday, November 13th. Version 3.7.21.296724 promotes the viewer from project status, and brings with it the following contributed updates:

  • OPEN-215 kCGLRPTextureMemory is a deprecated function as of Mac OS X 10.7, replace with kCGLRPTextureMemoryMegabytes
  • OPEN-268 FFLOAD_XML missing on linux and darwin, FFSAVE_XML missing on darwin
  • STORM-2053 applicationShouldTerminate function returns NSApplicationDeligateReply when it should return NSApplicationTerminateReply
  • STORM-2056 Projector reflections do not respect the environment intensity parameter
  • STORM-2067 Glossy Projectors
  • STORM-2069 Delete key doesn’t delete first character of marked text on OSX
  • STORM-2070 Sticky modifier keys after OS window comes up on OSX
  • STORM-2071 Unwanted InputWindow comes up when typing Japanese on OSX
  • STORM-2072 Bad behavior of Input Window
  • STORM-2078 Editing an objects rotation with the rotation rings often causes the object to jump to position <0,0,0> on the region and rotation changes to <0,0,0>
  • STORM-2080 Dresses purporting to be Fitted Mesh stretch to 0,0,0
  • STORM-2081 Second Life 3.7.18 (295539) Oct 16 2014 08:19:54 (Second Life Release) crashes every time upon viewer Quit under OS X Yosemite 10.10

Attachment Fix and Maintenance RC Updates

The current RC viewers – the Attachment fix RC viewer and the Maintenance RC viewer both updated on Wednesday, November 12th as a result of the Benchmark viewer promotion to release status. The updates were as follows:

  • Attachment Fix RC viewer updated to version 3.7.21.296729. This viewer adds some fixes to previously released changes in the way joint offsets in rigged meshes are handled & fixes some issues found with adding and removing attachments after the recent AISv3 deploy, and improves the status information shown in inventory for attached objects
  • Maintenance RC viewer updated to version 3.7.21.296734. This viewer offers a broad range of fixes for voice, privacy, rendering, texture animation, avatar distortion, inventory management, sounds, Mouselook in Mac, multiple UI fixes in script editor, Pay flow, chat, stats floater, edit menu, etc.

Release Viewer VFS Failure Issue

It’s still not clear how widespread the VFS failure issue on the current release viewer extends (see part 1 of this week’s update, and BUG-7776).  As noted in the first part of this report, I’ve managed to resolve such problems by manually deleted the viewer cache, and this worked for me in this case. Whirly Fizzle posted the same advice to the bug report, but at least one person has indicated it didn’t resolve the issue for them.

“Fast Pipe” Viewer

Monty Linden is working on further viewer-side updates to address reported problems being experienced in region rezzing times (such as reported in BUG-7698). There’s an initial release of the viewer available to those who can self-compile, and Whirly Fizzle reports that the fixes appear to work, commenting that, “Fast-pipe with pipelining enabled def works better for me on CDN regions. The long texture loading stalls have pretty much stopped.” There’s no date as to when this viewer will publicly appear as an RC or project viewer.

Other Items

Viewer Stats Ping Sim Data

There have been a few question in both meetings and in the forums about what the Ping Sim data in the viewer’s statistics bar actually represents. According to the wiki page:

Ping Sim: How long it takes data to go from your computer to the region you’re currently in. This is largely dependent on your connection to the Internet. If Ping Sim is high but Ping User is not, the server might be having problems.

However, this doesn’t paint a complete picture, as there can be a noticeable difference in the ping sim value given in the statistics bar of the viewer compared to pinging a simulator host directly. Responding to a question during the Server Beta meeting, Maestro Linden said:

As I recall, the ping number is your actual network round trip, measured by the UDP connection to the sim, *plus* some other time, like the time required to render the frame… So if you’re only getting 10FPS, that’s an automatic +100ms on top of the actual network round trip time… That ping time may also include the sim’s frame time (22.2ms if running at 45fps).

Answering a similar question through the forums in October, Oz Linden offered further clarification:

Looking at the code (I had not had occasion to look at this before), the Ping Sim measure appears to be based on a separate Ping message (our own message type transmitted over UDP, not ICMP). Those messages are mixed in periodically with the other UDP messages that are more or less constantly flowing between the viewer and the simulator. Because it’s the application that is turning that message around rather than a low-level part of the network stack, the fact that it is consistently higher than ICMP ping to the same host isn’t surprising. 

[ICMP, or Internet Control Message Protocol is the protocol generally used by ping operations.]

A couple of upshots of this is that if there is packet loss as a result of packet loss with increase ping rates as seen in the viewer, while no actual increase is occurring in the network connection itself, while increasing traffic within in region will cause higher ping rates within the viewer, simply because of the resultant UDP packet queuing in the server (hence a possible reason for BUG-7797, where the ping increase is seen as causing an increase in “lag”, rather than being indicative of an increase traffic load occurring). As such, Maestro commented that the viewer’s ping data is, “probably a more useful measure of latency than network ping.”

SL project updates 46/1: server, viewer + issues, misc items

Posted on by Inara Pey

Small Town Green; Inara Pey, February 2014, on FlickrrSmall Town Green, February 2014 (Flickr) – blog post

Server Deployments Week 46

As always, please refer to the server deployment thread in the forums for the latest information and updates.

  • There was no deployment to the Main (SLS) channel on Tuesday, November 11th.
  • On Wednesday, November 12th, all three RC channels should receive the same server maintenance package, which comprises “minor improvements” to help configure the texture and mesh CDN

We’re currently approaching the run-up to holidays, which means things are liable to be a little quiet on the server release front. There is usually a code freeze in effect during the US Thanksgiving week, and again over the Christmas / New Year period (which also affects viewer releases). With this in mind, and commenting on the next couple of weeks, Simon Linden commented that there is some hardware maintenance scheduled between now and Thanksgiving, and not a lot else.

SL Viewer

Release Viewer and Start-up Issues

On Monday, November 10th, the Lab promoted the Benchmark viewer, version 3.7.20.296094, to the de facto release viewer. This viewer removes reliance on the GPU table for determining the viewer’s initial graphics settings, and can be obtained from tha main viewer download page.

Some people have been reporting problems trying to run this viewer, specifically that it fails on reaching VFS initialisation. I encountered this problem myself in allowing the viewer to update from the HTTP pipelining release (3.7.19.295700) to the current release, and found that deleting my SL viewer cache ( C:\Users\[user name]AppData\Local\SecondLife) – a trick I’ve used with similar problems following a “dirty” install with Firestorm – cleared the issue for me; on restarting the viewer, it loaded OK and I was able to log-in.

However, for those who find this doesn’t work, or who have found a completely clean install (removing the viewer, the cache and local settings found in C:\Users\[user name]AppData\Roaming\SecondLife) hasn’t worked, a JIRA is open – see BUG-7776 – and specific problems should be reported there.

Group Chat

The recent round of changes to group chat, related to how the system looks-up avatars engaged in a group chat session as they move around the grid – see here – are expected to start rolling-out this week on the back-end. However, they are not expected to yield dramatically visible improvements in group chat.

Experience Keys / Tools

There’s not a lot to report On the Experience Keys project (see my overview and update), other than the Lab is “trying hard to wrap up the last few issues with Experiences so that we can make them generally available,” as Oz Linden said during the Simulator User Group meeting. As well as having the capability of allowing users to build more immersive experiences for people to enjoy, a particular interest in the Experience Tools is the key/value store, which is being seen as a potentially powerful additional capability to help with saving and retrieving data.

Other Items

Saving Graphics Profiles (/Preferences)

A suggestion was recently put forward during a Simulator User Group meeting for the Lab to allow the saving of graphics profiles (see my week 44/1 report under “Graphics Profile in the Viewer”. This would mean, for example that you could have a graphics profile where various options – the quality slider, shadows, occlusion, draw distance, etc., could be pushed towards their upper limits; and another where the setting are more conservative and less taxing on your GPU / system.

A feature request was subsequently submitted, after Oz suggested this might be a good open-source contribution, and Jonathan Yap is now working on the feature (STORM-2082), although it may be a “somewhat altered version” of the original suggestion. Currently, the work is focused on  Quick Preferences style of panel (as found in a number of TPVs),  as shown in the image below, accessed via the Setup tab in Preferences.

Jonathan Yap is working on a new "Quick Preferences" floater for the SL viewer
Jonathan Yap is working on a new “Quick Preferences” floater for the SL viewer (seen on the left in this image), access from the Setup tab in Preferences – this Quick Preferences tab will likely have options to save graphics settings once set through the sliders

Keep in mind this is very preliminary work, and things are liable to change as the work progresses.

Prim Selection Problems due to Interest Changes

An annoying issue which has been introduced with the interest list changes is the ability for them to interfere with object selection. Essentially, if an object or group of objects is selected, and camming then moves them off-screen, they are deselected, as updates from them to the viewer are culled – see BUG-7115. A fix for this should be available in the next round fo general bug fixes for the viewer released by the Lab.

It has been thought the interest list code had been altered since that there was a minimum distance before culling would be applied (e.g. so that there would be no culling on objects within, say, 30 metres of your camera position, regardless as to whether their were in your field of view or not). Simon Linden is going to take a further look at the code to see if this is in fact the case.

Script Sizes

The Simulator User Group meeting saw further discussion on script sizes (still 16Kb for LSL and 64Kb for Mono) and the viability of increasing them. Commenting on the discussion, Oz Linden observed, “Increasing script size is one of those things that must be considered very, very carefully. It would affect many, many things.” As such, it’s unlikely that any changes will be on the horizon.

SL project updates week 45/1: server, viewer, texture rendering

Posted on by Inara Pey

France Portnawak, Dreamland volcano; Inara Pey, October 2014, on FlickrFrance Portnawak, Dreamland Volcano (Flickr) – blog post

Server Deployments, Week 45

  • On Tuesday, November 4th, the Main (SLS) channel was updated with the server maintenance package deployed to all three RC channel in week 44. comprising some minor improvements described by Maestro Linden at the Server Beta meeting on Thursday, October 30th, as, “just some code cleanup and some extra optional debug logging.”
  • There are no planned RC channel deployments for week 45.

SL Viewer

On Friday, October 31st, the Lab issued a new Snowstorm contributions project viewer. Version 3.7.20.296071 includes a number of contribution, including Geenz Spad’s work on projectors STORM-2056, STORM-2067, a number of Mac OS X issues, a fix for editing an objects rotation with the rotation rings causing the object to jump to position <0,0,0> on the region and rotation changes to <0,0,0> (STORM-2078), and more. See the release notes in the link above or the full list of updates and known issues.

The snowstorm project viewer includes Geenz Spad's improvements to projector reflections (STORM-2067) and
The Snowstorm project viewer includes Geenz Spad’s improvements to projector reflections (STORM-2067) and issues with projected reflections failing to respect the environment intensity (STORM-2056)

 Texture Rendering Issues

There have been mixed reports of texture rendering issues since the arrival of the HTTP Pipelining viewer and the CDN. The reports can be variable in nature, but are apparently quite noticeable. The common symptom is for certain textures to remain fuzzy, as if stuck at a low resolution value (e.g. 64×64), with the higher resolution values failing to load until the mouse is placed over the object displayed the blurred texture or the object is physically clicked upon.

The issue appears to be particularly noticeable when using the HTTP pipelining viewer (I’ve encountered this myself running both the current SL release viewer version 3.7.19.295700, and the latest Alchemy viewer version 3.7.19.34077. It is also not the same issue as texture thrashing (which sees texture change from blurred to clear to blurred, etc., as a result of low video memory) nor is it the same as the interest list issue where panning the camera so that an object is off-screen and then back to it again causes the textures for the object to be re-loaded / rendered.

Gibson Firehawk has raised an JIRA on this issue, BUG-7700, which the Lab sees as a duplicate of BUG-7698, raised by Whirly Fizzle, which is being investigated. If this is a problem you are encountering on a persistent basis within one or more locations in SL, please consider raising a bug report and providing as much information as possible, including steps to reproduce. The more information the Lab has, the better position they’ll be in to poke at things.

SL project updates week 44/2: server, viewer, issues

Posted on by Inara Pey
Darkwood, for a Calas Galadhon Halloween
Darkwood, for a Calas Galadhon Halloween! (Blog post)

Server Deployments Week 44 – Recap

As always, please refer to the server deployment thread for the latest information and updates.

  • On Tuesday, October 28th, CDN support was deployed across the Main channel, meaning that the entire grid now utilises the Highwinds CDN for texture and mesh fetching. As the 130 regions deployed to the Snack channel were all originally from the Main channel, they have been reabsorbed into that channel, and Snack has once again be dissolved
  • On Wednesday, October 29th, all three RC channels received the same server maintenance project. which includes some minor improvements, which were described by Maestro Linden at the Server Beta meeting on Thursday, October 30th, as, “just some code cleanup and some extra optional debug logging.”

SL Viewer

On Wednesday, October 29th, the Lab promoted the HTTP Pipelining RC viewer (version 3.7.19.295700) to the de facto release viewer. As I’ve noted in previous blog posts, this viewer:

  • Can issue multiple asset fetches on a connection without waiting for responses to earlier requests. This reduces the impact of a user’s physical location on scene loading and generally improves the experience for everyone
  • Includes significant improvements to inventory folder and item fetches, which can markedly decrease the time taken for inventory to load.

All other SL viewer remain as per my Current Viewer Releases page.

Issues and Problems

Mesh Loading Issues

There have been mixed reports of mesh loading issues, which may or may be linked to some long-standing issues with mesh attachments. During the Server Beta meeting on Thursday, October 30th, the issues of mesh items which should be cached by the viewer being slow to load, don’t show up at all or show up with a low LOD, requiring a relog. Interestingly, the meshes can appear correctly when using one account, but not with another, either though both accounts are drawing from the same viewer cache.

Whirly Fizzle noted that there is a potential (and equally weird) fix for odd mesh loading issues, commenting:

There’s a weird mesh loading issue (not new) that’s “fixed” by deleting a certain per account file. I don’t even know why it should even work but it does. Sometimes a certain mesh gets stuck and will not render for a certain account. Relog doesn’t fix it & neither does a cache clear. You have to delete texture_list_last.xml in the per account settings files.

Texture Load Issues

There have been a couple of reports of texture loading issues which are still under investigation.

In the first, there have been  some reports that textures are getting stuck when “half loaded”, so that they remain blurred or may fail to render. It’s not clear how widespread this issue is, or whether it might be related to BUG-6382, although this is also unclear.

In the second, one content creator has reported customers are suddenly having issues with a HUD-based system for changing the colour of their shoes. When used, the textures rez very slowly, or the shoes get stuck as grey for up to 5 minutes at a time, or get stuck at a 1st level load until the wearer teleports or relogs. This is apparently a completely new behaviour with the shoes in question, and it appears to be exacerbated when using the HTTP viewer.

AMD Catalyst™ 14.9.2 Beta Woes

The latest AMD Catalyst™ 14.9.2 Beta drivers will not render rigged mesh unless hardware skinning is disabled. The current workaround is to either disable hardware skinning (which is not ideal when it comes to rendering mesh in general, and ALM features like materials will not render) or to remain on / roll-back to the 14.9.1 beta drivers if the problem is encountered. See BUG-7653. As was mentioned in the Server Beta UG meeting, this could be particularly confusing to new users running AMD GPUs and who are using the Lab’s default mesh avatars.

The latest AMD Catalyst™ 14.9.2 Beta driver issue: with Hardware skinning enabled, rigged messhes will not render (l); disable it, and they'll render OK (r) - click for full size; image courtesy of Maestro Linden
The latest AMD Catalyst™ 14.9.2 Beta driver issue: with Hardware skinning enabled, rigged meshes will not render (l); disable it, and they’ll render OK (r) – click for full size; image courtesy of Maestro Linden

E-mail Communications Issues for In-world Objects

A further JIRA has been filed in respect of e-mail communications with in-world objects hanging for no readily apparent reason (see: BUG-7554). similar issues have been reported in the past (although this one has a publicly viewable report), and Caleb Linden is currently poking at things.

Continue reading “SL project updates week 44/2: server, viewer, issues” →

SL project updates week 44/1: Server, CDN update

Posted on by Inara Pey

The Pines at Jacob's Pond, Jacob; Inara Pey, October 2014, on FlickrThe Pines at Jacob’s Pond, Jacob (Flickr) – blog post

Server Deployments Week 44

As always, please refer to the server deployment thread for the latest information and updates.

Main (SLS) Channel

On Tuesday, October 28th, CDN support was deployed across the Main channel, meaning that the entire grid now utilises the Highwinds CDN for texture and mesh fetching.

As the 130 regions deployed to the Snack channel were all originally from the Main channel, they have been / will be reabsorbed into that channel, and Snack will once again be dissolved.

Release Candidate Channels

On Wednesday, October 29th, all three RC channels should receive the same server maintenance project. which includes some minor improvements.

SL Viewer

There are currently two RC viewers possibly vying for promotion to the de facto release viewer. These are the HTTP Pipelining RC (version 3.7.19.295700) and the Benchmark viewer (version 3.7.19.295759), which should put an end to the use of a manually maintained list of GPUs in order to initially set the graphics defaults in the viewer.

Both of these were updated on Friday, October 24th, which has delayed any promotion to the de facto release viewer while the Lab gathers performance statistics on both of them. commenting on the status of both during the Simulator User Group meeting on Tuesday, October 27th, Oz Linden indicated that all things being equal, the HTTP Pipelining viewer should be promoted in the next 24-48 hours. He also indicated that there may be a further round of updates to come to Benchmark viewer in the offing as the Lab continued to tweak it.

CDN: Next Steps

In the vast majority of cases, the CDN is working as expected for users. There is a very small minority who, possibly because of their geographical closeness to the Lab’s servers or possibly due to issues between their ISP and the Highwinds services, are experiencing slightly worse ping times to their nearest CDN nodes when compared with pinging the Lab’s servers directly.

Even with the deployment to the Main channel, the Lab is continuing to monitor reports from CDN closely. However, as previously mentioned in my CDN coverage, it is likely the scope of CDN usage will be expanding in the future to handle other asset data – sounds, animations etc.

Also, and as noted in my week 43 TPV Developer meeting report, an offshoot of the CDN work is that there is a belief within the Lab that viewer caching may not be working as well as it might be. Internal discussions have held on possibly validating whether or not this is the case, and it is likely some work will be carried out in this area – and may well involve TPVs.

However, where both the viewer cache and extending the use of the CDN to cover other asset data are concerned, there are no time frames currently in mind. At the moment, the focus is very much on get the new tool chain and build process for the viewer finalised and into production, and in dealing with bugs and issues. As such, it might be a while before specific work on the viewer cache and / or work on extending the use of the CDN gets underway.

On a wider front, as well as monitoring the direct effectiveness of the CDN service, the Lab will be “spending quite a bit of time and effort assessing just what the effect of this change has been on operations from a number of perspectives”, to quote Oz.

Other Items

LI Issues?

There are reports circulating of unexpected changes to Land Impact (LI) values. While the Lab hasn’t altered land accounting, there have apparently been incidents of LI suddenly increasing in builds which have gone unchanged; one instance quoted during the Simulator User Group meeting referenced a door which apparently increased from 0.5 LI to 3 LI.

There have been instances in the past of the viewer incorrectly reporting the LI for an item when it is pulled from inventory. Corrections can generally be made either by relogging or by returning the item to inventory and rezzing it again. Altering the physics shape of linksets (from prim to convex and back again) can sometimes lead to problems, particularly if a prim in the linkset is contains torturing (such as a hollow or advance cut / twisting) or a script. However, this issue appears to be new, and a cause is proving hard to identify.

As always, if you have encountered the problem, and it is both persistent and reproducible, please raise a bug report.

Graphics Profiles in the Viewer

A suggestion was put forward during the Simulator User Group meeting for the Lab to allow the saving of graphics profiles. This would mean, for example that you could have a graphics profile where various options – the quality slider, shadows, occlusion, draw distance, etc., could be pushed towards their upper limits; and another where the setting are more conservative and less taxing on your GPU / system.

Then, where you are in a region (your own, or somewhere you know), where you know you can use the higher settings, you can quickly enable that profile, but when you move on to a region where (say) there are a lot of avatars and a lot going on, you can select the more conservative profile and thus reduce the potential performance issue on your system without actually having to go through and manually adjust all your settings.

Responding to the idea, Oz suggested the idea might be best suited to being a code contribution – and there was some potential interest in taking the idea on. However, this does not guarantee the idea will be carried forward – but it will be interesting to see if the idea does move forward at all in the coming months.

SL project updates week 43/2: miscellaneous, group chat

Posted on by Inara Pey

Just Another Tequilla Sunrise, Isle of Love; Inara Pey, October 2014, on FlickrJust Another Tequilla Sunrise, Isle of Love, October 2014 (Flickr) – blog post

Server Deployments Week 43 – Recap

As always, please refer to the server deployment forum thread for the latest news and updates.

  • On Tuesday, October 21st, the Main channel was updated with the server maintenance release previously deployed to the LeTigre and Magnum RCs in week 42. This update includes a crash fix and improves the delivery pipeline for abuse reports
  • On Wednesday, October 22nd, the release candidate channels were updated as follows:
  • BlueSteel and Snack will remain on the CDN project, and were updated with the same server maintenance package being deployed to the Main channel
  • LeTigre and Magnum were both updated to the CDN for texture and mesh fetching.

Upcoming Deployments

There is a further server maintenance package being prepared for deployment. however, it is still undergoing testing, and it is not clear whether it will be ready for deployment on Wednesday.

CDN and the Future of Snack

With the CDN now supporting texture and mesh fetching across all of the primary RC channels, there is a chance it may be promoted to the Main channel in week 4, particularly given that, outside of CDN support, all the server channels are now in parity. However, the Lab is making no commitments to this, and is taking a “wait and see” approach on what may happen. In particular, the Ops team will be monitoring performance over the weekend, when SL usage is expected to be at its busiest since the RC-wide deployment of CDN support, to see how things hold up.

As it stands, there are an estimated 130 regions on the Snack RC. As these all came from the Main channel, they will likely be switched back to that channel once it has CDN support, and Snack will again be dissolved until it is once again required.

Group Chat

With the most recent changes to the back-end servers, Simon Linden is continuing to poke at the code to see what can be improved. He’s now looking at the code that routes the messages – where it goes to find where group members are on the grid in order to be able to deliver messages to them, with the aim of trying to improve things there.

At the moment, if you move between regions which active in a group chat, the first message to be sent to you after you move will be sent to the region where you were and generate a failure message which has to go back to the chat service, which they has to relocate you and then attempt a further delivery. It is hoped that this can be improved, and there was a brief test following the Server Beta User Group meeting on Thursday, October 23rd to this end, just to see how the code is performing.

It has been suggested that the Lab should bypass the region hosts altogether when handling group chat, and attempt to route it directly from the chat servers to the viewer. However, this would likely require some serious re-engineering; more to the point, it could introduce security vulnerabilities.

Other Items

Aditi L$ Balance

A question was asked on how account L$ balances are updated on Aditi. Maestro replied that account balances are checked nightly (SLT), and any falling below L$ 10K (because someone has been uploading mesh models for testing / viewing prior to paying to upload them on Agni, for example), they are automatically increased by L$5K. However, if someone resets their account password (which triggers a reset on Aditi), their L$ balance will change to match their L$ balance on Agni, although again, if this is below L$10K it should be topped-up but L$ 5K overnight.