SL project updates 46/1: server, viewer + issues, misc items

Small Town Green; Inara Pey, February 2014, on FlickrrSmall Town Green, February 2014 (Flickr) – blog post

Server Deployments Week 46

As always, please refer to the server deployment thread in the forums for the latest information and updates.

  • There was no deployment to the Main (SLS) channel on Tuesday, November 11th.
  • On Wednesday, November 12th, all three RC channels should receive the same server maintenance package, which comprises “minor improvements” to help configure the texture and mesh CDN

We’re currently approaching the run-up to holidays, which means things are liable to be a little quiet on the server release front. There is usually a code freeze in effect during the US Thanksgiving week, and again over the Christmas / New Year period (which also affects viewer releases). With this in mind, and commenting on the next couple of weeks, Simon Linden commented that there is some hardware maintenance scheduled between now and Thanksgiving, and not a lot else.

SL Viewer

Release Viewer and Start-up Issues

On Monday, November 10th, the Lab promoted the Benchmark viewer, version 3.7.20.296094, to the de facto release viewer. This viewer removes reliance on the GPU table for determining the viewer’s initial graphics settings, and can be obtained from tha main viewer download page.

Some people have been reporting problems trying to run this viewer, specifically that it fails on reaching VFS initialisation. I encountered this problem myself in allowing the viewer to update from the HTTP pipelining release (3.7.19.295700) to the current release, and found that deleting my SL viewer cache ( C:\Users\[user name]AppData\Local\SecondLife) – a trick I’ve used with similar problems following a “dirty” install with Firestorm – cleared the issue for me; on restarting the viewer, it loaded OK and I was able to log-in.

However, for those who find this doesn’t work, or who have found a completely clean install (removing the viewer, the cache and local settings found in C:\Users\[user name]AppData\Roaming\SecondLife) hasn’t worked, a JIRA is open – see BUG-7776 – and specific problems should be reported there.

Group Chat

The recent round of changes to group chat, related to how the system looks-up avatars engaged in a group chat session as they move around the grid – see here – are expected to start rolling-out this week on the back-end. However, they are not expected to yield dramatically visible improvements in group chat.

Experience Keys / Tools

There’s not a lot to report On the Experience Keys project (see my overview and update), other than the Lab is “trying hard to wrap up the last few issues with Experiences so that we can make them generally available,” as Oz Linden said during the Simulator User Group meeting. As well as having the capability of allowing users to build more immersive experiences for people to enjoy, a particular interest in the Experience Tools is the key/value store, which is being seen as a potentially powerful additional capability to help with saving and retrieving data.

Other Items

Saving Graphics Profiles (/Preferences)

A suggestion was recently put forward during a Simulator User Group meeting for the Lab to allow the saving of graphics profiles (see my week 44/1 report under “Graphics Profile in the Viewer”. This would mean, for example that you could have a graphics profile where various options – the quality slider, shadows, occlusion, draw distance, etc., could be pushed towards their upper limits; and another where the setting are more conservative and less taxing on your GPU / system.

A feature request was subsequently submitted, after Oz suggested this might be a good open-source contribution, and Jonathan Yap is now working on the feature (STORM-2082), although it may be a “somewhat altered version” of the original suggestion. Currently, the work is focused on  Quick Preferences style of panel (as found in a number of TPVs),  as shown in the image below, accessed via the Setup tab in Preferences.

Jonathan Yap is working on a new "Quick Preferences" floater for the SL viewer
Jonathan Yap is working on a new “Quick Preferences” floater for the SL viewer (seen on the left in this image), access from the Setup tab in Preferences – this Quick Preferences tab will likely have options to save graphics settings once set through the sliders

Keep in mind this is very preliminary work, and things are liable to change as the work progresses.

Prim Selection Problems due to Interest Changes

An annoying issue which has been introduced with the interest list changes is the ability for them to interfere with object selection. Essentially, if an object or group of objects is selected, and camming then moves them off-screen, they are deselected, as updates from them to the viewer are culled – see BUG-7115. A fix for this should be available in the next round fo general bug fixes for the viewer released by the Lab.

It has been thought the interest list code had been altered since that there was a minimum distance before culling would be applied (e.g. so that there would be no culling on objects within, say, 30 metres of your camera position, regardless as to whether their were in your field of view or not). Simon Linden is going to take a further look at the code to see if this is in fact the case.

Script Sizes

The Simulator User Group meeting saw further discussion on script sizes (still 16Kb for LSL and 64Kb for Mono) and the viability of increasing them. Commenting on the discussion, Oz Linden observed, “Increasing script size is one of those things that must be considered very, very carefully. It would affect many, many things.” As such, it’s unlikely that any changes will be on the horizon.

SL project updates week 45/3: TPV Developer meeting

The following notes are drawn from the TPV Developer meeting held on Friday, November 7th, and shown in the video above. Time stamps, where relevant, have been included for ease of reference to the video. Note that items are listed according to subject matter and not necessarily chronologically, so some time stamps may appear out-of-sequence in places. My thanks as always to North for the recording.

SL Viewer

Maintenance RC Viewer

[00:06] The Lab released a further maintenance viewer on Thursday, November 6th (although it didn’t reach the release candidates section on the alternative Viewers wiki page until Friday, November 7th). Version 3.7.20.296368 is a further maintenance release RC, offering a broad range of fixes for voice, privacy, rendering, texture animation, avatar distortion, inventory management, sounds, Mouselook in Mac, multiple UI fixes in script editor, Pay flow, chat, stats floater, edit menu, and so on. Follow the link above to the release notes. This viewer also includes a couple of the AIS v3 fixes as well.

Benchmark Viewer

The Benchmark RC viewer, which eliminates the need for a manually maintained GPU table in order for the viewer to initially set graphic options, looks set to be the next viewer that will be promoted to release status.

Viewer Build Tools

[01:36] As the Lab is primarily focused on completing the updates to the viewer build chain and tools. As has been previously reported, the Mac viewer can now be built using the new build process, but it isn’t performing as well as expected, so it is being looked into. The Lab still has yet to get the new windows build process up and running.

[33:08] This work also means that the Lab currently isn’t attempting to progressing getting 64-bit versions of the Havok libraries at the present time.

Group Chat

[02:15] As noted in part 2 of this week’s update, the Lab is continuing to poke at group chat.  Oz confirmed there are still “one or two” more rounds of changes to be made, and the overall, the Lab is “pretty happy” with what has been achieved thus far. The Firestorm team, who perhaps have one of the most active of groups with a large number of users in their English Support group, have noted a substantial increase in group chat performance.

Z-offset Height Adjustment

[03:20] Vir Linden has resumed work on the z-offset height adjustment (aka, on-the-fly avatar height adjustment feature), after having been diverted to take look at the AIS v3-related attachment issues (the fixes for which are now out in an Attachments RC viewer, as noted in part 2 of this report). While there currently isn’t any significant news on progress as yet, it is hoped that there will be an update at the next TPV Developer meeting.

CDN

[03:50] The Lab issued a further update on the CDN on Friday, November 7th (my report is here). commenting on this during the TPV Developer meeting, Oz added:

Mostly, it’s going really, really well, and that contributes to figuring out what is going wrong when it isn’t going well. But we have a lot of things to work on, and we think we are making progress even on those. 

As the Lab’s blog post indicates, there have been a lot of people assisting the Lab with remote testing, which Oz described as “invaluable”, given that tying-down issues has a lot to do with where you are on the Internet, which CDN nodes / PoPs you hit, and so on.

Viewer Managed Marketplace

[06:10] Testing on Aditi for the new viewer-managed marketplace updates is due to commence “pretty soon”, although it appears to be running a little behind the schedule outlined for it when first announced. Testing will hopefully commence prior to US Thanksgiving towards the end of the month, but the overall plan remains that this will be a slow but steady implementation; there will be no quick-firing testing on Aditi with a sudden main grid deployment, particularly given the Christmas shopping season is now on the horizon (not to mention the Lab’s own code freezes over the holiday periods).

Continue reading “SL project updates week 45/3: TPV Developer meeting”

SL project updates week 45/2: viewer, group chat

The Trace, The Trace; Inara Pey, November 2014, on FlickrThe Trace, The Trace (Flickr) – blog post

This update is taken from notes gathered at the Server Beta meeting on Thursday November 6th, and is somewhat late due to the motherboard on my primary PC failing, and my having to pull out an old PC and get it running (Vista – bleah!).

Server Deployments Week 45 – Recap

  • On Tuesday, November 4th, the Main (SLS) channel was updated with the server maintenance package deployed to all three RC channel in week 44. comprising some minor improvements described by Maestro Linden at the Server Beta meeting on Thursday, October 30th, as, “just some code cleanup and some extra optional debug logging.”
  • There were no RC channel deployments for week 45.

SL Viewer

On Wednesday, November 5th, the Lab issues a new Attachments RC viewer. Version 3.7.20.296355 carries the following description from the Lab:

This release adds some fixes to previously released changes in the way we handle joint offsets in rigged meshes. Originally, if you attached multiple meshes with joint offsets, the results were unpredictable, depending on the order in which the attachments were processed. This means that you could see yourself with different joint positions at different times, and the way you saw yourself might not match up to what others saw. With these changes, the behavior is deterministic, so wearing the same set of meshes should always produce the same joint positions, and the way you see yourself should consistently match the way others will see you.The process for removing attached meshes is also more robust, so you should be able to reliably get your old joint positions back after meshes are removed. People running un-updated viewers will continue to see the original behavior with the original bugs, but will not experience any new regressions due to these changes.

This release also fixes some issues found with adding and removing attachments after the recent AISv3 deploy, and improves the status information shown in inventory for attached objects.

Specific fixes include:

  • BUG-6197 (MAINT-4158) Avatar distorted when changing outfits’
  • BUG-6487 (MAINT-4196) “Wear” Behaves Erratically For Attachments on Same Point’
  • BUG-6908 (MAINT-4350 Attachments scripted to llDetachFromAvatar on region change show as “Worn on invalid attachment point” in inventory after region change’
  • BUG-6439 (MAINT-4189) Shape distortion when wearing / taking off rigged mesh with joint offset’
  • MAINT-4572 Body crusher animations for quadruped avatars do not appear to load
  • MAINT-4605 [LION] Avatar eyes sometimes disappear after adding or removing rigged mesh
  • MAINT-4606 [LION] Avatar shape is deformed after removing some rigged meshes. Deformation is only seen by users on the Lion viewer, not default release.

Note: MAINT JIRA are internal to LL, and not available for general viewing.

Group Chat

Simon Linden is continuing to tweak the group chat code, and has most recently been looking at  the code that routes the messages – where it goes to find where group members are on the grid in order to be able to deliver messages to them, with the aim of trying to improve things there.

A further brief test was carried out during the Server Beta meeting on Thursday, November 6th to see how the changes are working out. During the test, Simon noted that any differences resulting from this round of updates are liable to be less noticeable than other recent changes. However, they should help smooth the process of how the system locates people as they move around SL.

The problem with group chat routing is that everything is run via the region you’re in; so if you move to another region, messages go to the last region you were in, which replies with a message that you’re no longer there, which triggers the chat service to perform a look-up as to where you now are. “So the system has logic (in many places, not just group chat) which is kinda like, ‘try sending to where I think they are, and if that fails, go ask and try again,'” Simon said in explaining the methodology used. It can sometimes result in messages being dropped as a result, particularly if people are moving about a lot as a result of rolling restarts, etc.

Suggestions have been made about using a central service – such as a virtual server – to handle group chat. However, such a solution isn’t favoured by the Lab. “There can be roughly 50k people online at once, all in 42 groups … it doesn’t scale well,” Simon explained.

A further problem noted (by users) with group chat is that it will often open with the original poster’s message absent – the first thing seen is a reply to what was originally said to initiate the session. This doesn’t happen all the time, but has happened enough to be noticed.

In the meantime, and depending on the results of the most recent tests, these updates may well be in an upcoming RC release.

Other Items

Aditi log-in Issues

Some people recently experienced issues when logging into Aditi which resulted in them being logged into a random region rather than their last location. The issue appears to have been linked to a specific inventory host on Aditi, and is now believed to have been fixed (see BUG-7707), so if you’ve experienced the issue yourself on the beta grid, it now should be cleared-up.

 Disconnected When Last to TP on a Region Restart

There have been a number of reports of people finding that if they are the last the leave a region prior to it restarting (either a manual restart or during a rolling restart), they are being disconnected from SL, even though they have apparently successfully arrived in their destination region (i.e. it has rezzed around them and the viewer log records their arrival). BUG-5034 outlines the issue, although it has been reported that the problem isn’t just restricted to estate managers.

Open Scripts Issue

There’s been a long-standing issue where if you have two copies of a scripted object, the script editor won’t open for scripts in one of the objects if the script in the other copy has already been opened – the script editor keeps the first script active (see VWR-27512). This issue apparently now has a fix, which should be appearing in the near future.

SL project updates week 45/1: server, viewer, texture rendering

France Portnawak, Dreamland volcano; Inara Pey, October 2014, on FlickrFrance Portnawak, Dreamland Volcano (Flickr) – blog post

Server Deployments, Week 45

  • On Tuesday, November 4th, the Main (SLS) channel was updated with the server maintenance package deployed to all three RC channel in week 44. comprising some minor improvements described by Maestro Linden at the Server Beta meeting on Thursday, October 30th, as, “just some code cleanup and some extra optional debug logging.”
  • There are no planned RC channel deployments for week 45.

SL Viewer

On Friday, October 31st, the Lab issued a new Snowstorm contributions project viewer. Version 3.7.20.296071 includes a number of contribution, including Geenz Spad’s work on projectors STORM-2056STORM-2067, a number of Mac OS X issues, a fix for editing an objects rotation with the rotation rings causing the object to jump to position <0,0,0> on the region and rotation changes to <0,0,0> (STORM-2078), and more. See the release notes in the link above or the full list of updates and known issues.

The snowstorm project viewer includes Geenz Spad's improvements to projector reflections (STORM-2067) and
The Snowstorm project viewer includes Geenz Spad’s improvements to projector reflections (STORM-2067) and issues with projected reflections failing to respect the environment intensity (STORM-2056)

 Texture Rendering Issues

There have been mixed reports of texture rendering issues since the arrival of the HTTP Pipelining viewer and the CDN. The reports can be variable in nature, but are apparently quite noticeable. The common symptom is for certain textures to remain fuzzy, as if stuck at a low resolution value (e.g. 64×64), with the higher resolution values failing to load until the mouse is placed over the object displayed the blurred texture or the object is physically clicked upon.

The issue appears to be particularly noticeable when using the HTTP pipelining viewer (I’ve encountered this myself running both the current SL release viewer version 3.7.19.295700, and the latest Alchemy viewer version 3.7.19.34077. It is also not the same issue as texture thrashing (which sees texture change from blurred to clear to blurred, etc., as a result of low video memory) nor is it the same as the interest list issue where panning the camera so that an object is off-screen and then back to it again causes the textures for the object to be re-loaded / rendered.

Gibson Firehawk has raised an JIRA on this issue, BUG-7700, which the Lab sees as a duplicate of BUG-7698, raised by Whirly Fizzle, which is being investigated. If this is a problem you are encountering on a persistent basis within one or more locations in SL, please consider raising a bug report and providing as much information as possible, including steps to reproduce. The more information the Lab has, the better position they’ll be in to poke at things.

SL project updates week 44/2: server, viewer, issues

Darkwood, for a Calas Galadhon Halloween
Darkwood, for a Calas Galadhon Halloween! (Blog post)

Server Deployments Week 44 – Recap

As always, please refer to the server deployment thread for the latest information and updates.

  • On Tuesday, October 28th, CDN support was deployed across the Main channel, meaning that the entire grid now utilises the Highwinds CDN for texture and mesh fetching. As the 130 regions deployed to the Snack channel were all originally from the Main channel, they have been reabsorbed into that channel, and Snack has once again be dissolved
  • On Wednesday, October 29th, all three RC channels received the same server maintenance project. which includes some minor improvements, which were described by Maestro Linden at the Server Beta meeting on Thursday, October 30th, as, “just some code cleanup and some extra optional debug logging.”

SL Viewer

On Wednesday, October 29th, the Lab promoted the HTTP Pipelining RC viewer (version 3.7.19.295700) to the de facto release viewer. As I’ve noted in previous blog posts, this viewer:

  • Can issue multiple asset fetches on a connection without waiting for responses to earlier requests. This reduces the impact of a user’s physical location on scene loading and generally improves the experience for everyone
  • Includes significant improvements to inventory folder and item fetches, which can markedly decrease the time taken for inventory to load.

All other SL viewer remain as per my Current Viewer Releases page.

Issues and Problems

Mesh Loading Issues

There have been mixed reports of mesh loading issues, which may or may be linked to some long-standing issues with mesh attachments. During the Server Beta meeting on Thursday, October 30th, the issues of mesh items which should be cached by the viewer being slow to load, don’t show up at all or show up with a low LOD, requiring a relog. Interestingly, the meshes can appear correctly when using one account, but not with another, either though both accounts are drawing from the same viewer cache.

Whirly Fizzle noted that there is a potential (and equally weird) fix for odd mesh loading issues, commenting:

There’s a weird mesh loading issue (not new) that’s “fixed” by deleting a certain per account file. I don’t even know why it should even work but it does. Sometimes a certain mesh gets stuck and will not render for a certain account. Relog doesn’t fix it & neither does a cache clear. You have to delete texture_list_last.xml in the per account settings files.

Texture Load Issues

There have been a couple of reports of texture loading issues which are still under investigation.

In the first, there have been  some reports that textures are getting stuck when “half loaded”, so that they remain blurred or may fail to render. It’s not clear how widespread this issue is, or whether it might be related to BUG-6382, although this is also unclear.

In the second, one content creator has reported customers are suddenly having issues with a HUD-based system for changing the colour of their shoes. When used, the textures rez very slowly, or the shoes get stuck as grey for up to 5 minutes at a time, or get stuck at a 1st level load until the wearer teleports or relogs. This is apparently a completely new behaviour with the shoes in question, and it appears to be exacerbated when using the HTTP viewer.

AMD Catalyst™ 14.9.2 Beta Woes

The latest AMD Catalyst™ 14.9.2 Beta drivers will not render rigged mesh unless hardware skinning is disabled. The current workaround is to either disable hardware skinning (which is not ideal when it comes to rendering mesh in general, and ALM features like materials will not render) or to remain on / roll-back to the 14.9.1 beta drivers if the problem is encountered. See BUG-7653. As was mentioned in the Server Beta UG meeting, this could be particularly confusing to new users running AMD GPUs and who are using the Lab’s default mesh avatars.

The latest AMD Catalyst™ 14.9.2 Beta driver issue: with Hardware skinning enabled, rigged messhes will not render (l); disable it, and they'll render OK (r) - click for full size; image courtesy of Maestro Linden
The latest AMD Catalyst™ 14.9.2 Beta driver issue: with Hardware skinning enabled, rigged meshes will not render (l); disable it, and they’ll render OK (r) – click for full size; image courtesy of Maestro Linden

E-mail Communications Issues for In-world Objects

A further JIRA has been filed in respect of e-mail communications with in-world objects hanging for no readily apparent reason (see: BUG-7554). similar issues have been reported in the past (although this one has a publicly viewable report), and Caleb Linden is currently poking at things.

Continue reading “SL project updates week 44/2: server, viewer, issues”

SL project updates week 43/1: server, viewer, group chat

Asalia House (November 2013) - blog post
Asalia House (November 2013) – blog post

Server Deployments Week 43

As always, please refer to the server deployment forum thread for the latest news and updates.

Main (SLS) Channel

On Tuesday, October 21st, the Main channel was updated with the server maintenance release previously deployed to the LeTigre and Magnum RCs in week 42. This update includes a crash fix and improves the delivery pipeline for abuse reports.

RC Channels

On Wednesday, October 22nd, the release candidate channels should be updated as follows:

  • BlueSteel and Snack will remain on the CDN project, but should also be updated with the same server maintenance package being deployed to the Main channel
  • LeTigre and Magnum should both be updated to the CDN for texture and mesh fetching.

SL Viewer

On Wednesday, October 15th, the Lab issued a Grid Status update on potential problems accessing external websites using the viewer’s internal browser.  The notice was issued as a result of the Padding Oracle On Downgraded Legacy Encryption (Poodle) vulnerability reported by Google.

At the time the notice was released, the Lab indicated they were working to fix the issue (via the removal of SSL v3.0 support within the viewer’s browser). This work was completed at the end of week 42, and resulted in the release of the Browser Fix RC viewer, version 3.7.18.295539.

On Monday, October 20th, the Browser Fix viewer was promoted to the de facto release viewer.  Further information on it and the POODLE vulnerability can be found in a separate article in this blog.

MAC Yosemite Issues

There have been some mixed results for those Mac users updating to the 10.10 Yosemite version of OS X, which reached a consumer release status on October 16th.

While most of those updating have found it relatively smooth, there has been a report of keyboard shortcuts borking, although there is a workaround should others encounter the same problem – see here.

A potentially more widespread issue people may encounter is that with Yosemite, Apple have apparently reverted to older nVidia drivers (as reported by Cinder Roxley at the Open-source Developer’s meeting on Monday, October 20th). One JIRA on the matter has already been filed (see BUG-7575), although if Apple have chosen to roll back some / all of their default drivers, there’s not a lot the Lab can do about matters. Cinder also reports that Yosemite makes greater use of OpenGL, which she indicates can tax GPUs further.

CDN

Following the scheduled Wednesday deployment, all of the RCs should be using the CDN. This represents around 30% of the total grid, possibly a little more, given the existence of Snack. The BlueSteel deployment does not seem to have resulted in the kind of performance issues experienced during the expansion of Snack to 250 regions, suggesting the latter problems were most probably down to overloading the sim hosts with too many high volume (in terms of avatar numbers) regions. If all goes well in the coming week, it is likely that the Main channel will gain CDN support in week 44.

Group Chat

“Today we also finished an update to the group chat servers,” Simon Linden announced at the Simulator User Group meeting on Tuesday, October 21st. He continued:

Many of you have helped us test that at Thursday’s beta user group, and that’s now on the newest version. Those are back-end servers … when you send something to group chat, it goes from the viewer to the region you’re on, then to the (new) back-end chat servers, and from there it’s distributed to everyone in the group.

While no promises are being made as to the overall improvement – or how visible improvements may be to users, Simon went on to note, “I know [from] looking at the metrics and data we can gather … the systems are running much nicer than before.”

As well as working on the update issues (see my previous group chat updates), the Lab has been brainstorming ideas on how groups might be configured differently. There are a lot of groups that have very heavy chat usage, and others where chat is actively discouraged in favour of outgoing notices; however the latter are handled exactly the same as those where chatting goes on, generating updates as people log-in and out, etc., even though these aren’t actually required. Exactly what might be done is unclear at this time, but it would appear ideas are being put forward and discussed.