SL project updates 50/1: Server, viewer, issues

Frisland; Inara Pey, December 2015, on FlickrFrisland (Flickr) – blog post

Server Deployments

There was no Main (SLS) channel deployment on Tuesday, December 8th, following after the update planned for release in week #49 had to be cancelled when a simulator crash bug was uncovered.

On Wednesday, December 9th, all three RC channels should receive the same new server maintenance package, which comprises simulator crash fixes (including one for the issue found during the original final testing of the package in week #49) and implements feature request  BUG-10192: adding constant OBJECT_OMEGA to llGetObjectDetails(), so that it can return a vector matching what is returned with llGetOmega(), allowing applications to determine an object’s rate and axis of rotation.

Viewer Updates

On Monday, December 7th, the Valhalla RC viewer, which comprises the Chromium embedded Framework implementation intended to replace LLQTwebkit for handling media in Second Life, was updated to version 4.0.0.308641. This update includes 13 additional fixes when compared to the previous Valhalla RC version:

  • MAINT-5846 – MOAP audio is too quiet
  • MAINT-5849 – MOAP does not run if parcel media texture is on same face
  • MAINT-5852 – Parcel media url can be hijacked from parcel to parcel
  • MAINT-5854 – Loading http://secondlife.com on parcel media does not login
  • MAINT-5855 – media navigation bars overlap all floaters in viewer
  • MAINT-5856 – toolbar search can be interrupted early get stuck on blank page
  • MAINT-5859 – Terms of Service are not loading in Linux only
  • MAINT-5896 – Add support for viewing PDF files in the viewer
  • MAINT-5901 – Click-to-Walk should work through transparent objects
  • MAINT-5902 – Qihoo 360 Anti-virus blocks SLPlugin.exe and login page web content
  • MAINT-5909 – Japanese can’t be input in CEF
  • MAINT-5911 – Pressing “return” (or “enter”) no longer performs a search
  • MAINT-5941 – Default flash to on by default.

Other Items

Interest List and “Ghost” Prims

there have been reports at the last couple of Simulator User Group meetings about “ghost prim” – objects which have been deleted / killed via llDie, continuing to render viewer-side, even though they have been removed by the simulator, requiring a right-click to remove them from the viewer’s outlook on the world.

Problems like this aren’t new, and many have encountered them, particularly since the core of the changes made to the Interest List. However, positively identifying what is going wrong where in the code, and why it is going wrong has been proving difficult, as the has not been a consistent means of reproducing the problem. However, it now appears that just such a consistent means of encountering the issue has been found, and a JIRA raised. Hopefully, this means that the Lab will be able to dig a little deeper into things and at least rectify the problem for some of the situations where “ghost prims” can be encountered.

Join / Leave Group Failures

There have been significant issues with people attempting to join or leave groups recently – see BUG-10869. The problems are apparently caused by a back-end database overload within the group services,

There are many issues in handling large groups which can be problematic: number of members, number of inactive users, impact of changes to things like established group roles (and the numbers of group members they affect), and so on. These are all largely down to the way the back-end group services were originally designed, something which is not the easiest of issues to overcome, as Simon Linden explained at the at the Simulator User Group meeting on Tuesday, December 8th:

It’s a long story, actually, but comes down to scaling issues and design. It doesn’t make sense that we basically treat a group with 100k people in it the same as 10 people . There are some things that just take more time with a large group.

However, Simon is looking into the problems, as he did with the issues of group chat earlier in the year, which so that side of things dramatically improved, but there is currently no ETA on when any fix / fixes might be issued.

No Change Window

Subject to official confirmation by the Lab, week #51 (week commencing Monday, December 14th) is liable to mark the last week in which simulator and viewer releases will be made ahead of the Christmas / New Year “no change window” coming into force, which will probably remain in place until approximately the week commencing Monday, January 4th, 2016.

The no change window is intended to ensure the grid and viewer are both relatively stable, so that the Lab can offer support, engineering and operations staff time off over the holiday period to be with their families and friends.

SL Project updates week 49

Everlong; Inara Pey, November 2015, on FlickrEverlong (Flickr) – blog post

The technical news is very light for week #49, hence my lack of updates as the week progressed.

There were no simulator deployments during the week. A new server maintenance package had been planned for deployment to the three RC channels, but this suffered a last-minute postponement do to a simulator crash situation being discovered if a simulator spins-up with a specific debug mode enabled.

As a result of this cancellation, there is unlikely to be a Main (SLS) channel roll in week 50, although the RC update should see deployment. After that, with the Christmas / New Year no change window rising over the horizon, further simulator updates are liable to be conspicuous in their absence until early 2016.

The viewer has been similarly quiet during the week, with only the Maintenance RC being updated, with version 3.8.7.308556 being released on December 3rd. There was (again) no RC promotion to release status for any viewer, thus leaving the usual suspects in the viewer release channel:

  • Maintenance RC viewer, version 3.8.7.308556, dated December 3rd, as noted above. Some 38 fixes and improvements, including updates for some regressions introduced into the viewer with the current release viewer
  • Project Azumarill (HTTP updates) RC viewer, version 3.8.7.308134, dated November 25th, which provides  a complete replacement of the under the hood HTTP infrastructure within the viewer
  • Project Valhalla (CEF media update) RC viewer, version 4.0.0.307894 dated November 18th, which replaces the LLQTWebKit functionality used in the Web media plugin with one based on the Chromium Embedded Framework (CEF)
  • Vivox RC viewer, version 3.8.7.307744, dated November 17th, which corrects a number of Voice quality and connection issues on both Windows and Mac
  • Quick Graphics RC viewer, version 3.8.7.306758, dated November 12th, which provides the new Avatar Complexity options and graphics preset capabilities for setting, saving and restoring graphic settings for use in difference environments / circumstances.

The two potential candidates for promotion to release status would seem to be either the HTTP RC viewer or the Maintenance viewer. However, the former’s promotion would appear to be dependent upon whether the remaining niggles have been ironed out to the Lab’s satisfaction, and the later on whether the most recent RC update gathers sufficient stats to indicate whether it is ripe for promotion.

There were assorted reports of issues with teleports, etc., through the first part of the week, and during the Server Beta User Group meeting, the question was asked by Caleb Linden if people had noticed an increased rate of teleport issues / failures, suggesting the Lab may have noticed something going awry.

Other than that, a light news week, as mentioned.

SL project updates: week #48/1: server, viewer

Voile; Inara Pey, November 2015, on Flickr Voile (Flickr)blog post

Server Deployments Week #48

As always, please refer to the server deployment thread for the latest updates.

  • On Tuesday, November 24th, the Main (SLS) channel was updated with the server maintenance package previously deployed to the three RC channels. This comprises the updates code for server-side attachment and upload validations (see below). The release also includes new code for a new upcoming SL feature to be revealed soon.
  • There is no RC channel deployment planned for the week, due the the US Thanksgiving weekend no change window.

Server-side Validation Enforcement

A new server maintenance package had been due for deployment on Wednesday, November 11th. This packed includes simulator-side enforcement checks against attachment point IDs (attachments using invalid attachment point ID will be moved to the chest attach point), and validation checks against animation and mesh to prevent the upload of animations and meshes weighted to invalid attachment points.

Baking Service Update

Speaking at the Simulator User Group meeting on Tuesday, November 24th, Oz Linden indicated an update to the avatar baking service (server-side appearance) has been made. This is aimed at dealing with a number of issues with the service (unfortunately Oz didn’t have a list of issues to hand).

SL Viewer Updates

After mysteriously vanishing from the viewer release channel, the HTTP project viewer (Project Azumarill) reappeared on Tuesday, November 17th, still with the same version number (3.8.7.306796, dated November 2nd) as previously. This may have been a error, and the viewer has once again been removed from the viewer release channel.

Friday, November 20th saw the current Maintenance viewer updated to version 3.8.7.308081. This viewer brings with it some 38 fixes and improvements, including updates for some regressions introduced into the viewer with the current release viewer (version 3.8.6.305981).

Note: this week is a relatively light week for technical updates from the Lab due to it being Thanksgiving week in the US, which is always marked by a no change window, as noted in the report, to allow Lab staff to enjoy a long weekend at the end of the week.

Second Life projects update 47/1: server, viewer, issues

It's a New Dawn; Inara Pey, November 2015, on Flickr It’s a New Dawn (Flickr) – blog post

Server Deployments Week #47

As always, please refer to the server deployment thread for the latest updates.

  • There was no deployment to the Main (SLS) channel on Tuesday, November 17th
  • The maintenance package deployed to all three RC channels in week #46 received a further update on Wednesday, November 18th. This was apparently due to the postponed deployment from week #46 not containing all of the desired updates.

SL Viewer

After mysteriously vanishing from the viewer release channel, the HTTP project viewer (Project Azumarill) reappeared on Tuesday, November 17th, still with the same version number (3.8.7.306796, dated November 2nd) as previously.

The Vivox viewer entered the release channel as a RC viewer on Tuesday, November 17th. Version 3.8.7.3.8.7.307744 corrects a number of Voice quality and connection issues on both Windows and the Mac.

The Chrome Embedded Framework Viewer also reached RC status with the release of version 4.0.0.307894 on Wednesday, November 18th.

Group Ejections and Group Chat

SVC-32, “Person kicked remains in chat session” came up for discussion at the Simulator User Group Meeting on Tuesday, November 17th.

This is the ongoing issue where those who are ejected from a group whilst in a group chat session still being able to continue using the chat until they close the window (with various permutations on the issue vis muting and unmuting people).

The cause of the problem seems to be quite deeply rooted in how the group chat service works, with Simon Linden noting, “I hate that one … and I’ve been lurking in some large groups (like FS) and seeing it abused. It’s an ugly problem and I’ve looked into it … basically because the info about how to add the logic isn’t where the log needs to be run … I looked into revoking caps and other ideas but didn’t find a silver bullet, or a pile of them, that would work … I’ll try to get some time to see if there’s anything that can be done without major architecture changes or new servers.”

Object and Linkset  Contents and Limits

Changes were recently made to how many objects can be placed within an individual in-world / worn object  and linkset. The limit per individual objects is now 10,000, and the limit per linkset is 30,000. The limits were revised to help deal with people using objects which massive contents from eating up server resources and impacting region performance as a form of griefing. However, a side issue with the changes has been that tools such as texture organisers with huge contents spread through the linkset can stop working – permanently.

To help people check their organisers and make suitable adjustments in line with the new limits, the Unpackistan region has a 90K limit for linksets (although there was an issue with this region as well – BUG-10695 – which should now have been corrected).

Concerns were raised the simulator User Group over the lack of follow-through when limits like this are changed (there has been no Wiki update in this instance, for example).

While it is understood limits may be changed to help reduce problems with griefing, etc., and thus the Lab may no wish to announce forthcoming changes in case it leads to an increase in abuse before the door is shut, so to speak, it was pointed out that providing some indicators as to what has changed, and ensuring reference documentation is properly updated could go a long way to help people figure out what might have happened as a result of such changes having an unexpected repercussion (as in this case).

Acknowledging this point, Simon agreed to take the concerns raised back to the office for further discussion.

Second Life projects update 46/2: RC server deployment, viewer updates

Saint Pete City; Inara Pey, November 2015, on Flickr Saint Pete City (Flickr)blog post

Server Deploys Week 46 – Update

As always, please refer to the server deployment thread for the latest updates.

  • On Tuesday, November 10th, the Main (SLS) channel was updated with the same server maintenance package as deployed to the RC channels in week #45, comprising a fix for group invite throttle notifications, and an internal server code clean-up

Release Channel Deployments – Server-side Validation Enforcement

A new server maintenance package had been due for deployment on Wednesday, November 11th. This packed includes simulator-side enforcement checks against attachment point IDs (attachments using invalid attachment point ID will be moved to the chest attach point), and validation checks against animation and mesh to prevent the upload of animations and meshes weighted to invalid attachment points.

However, it was initially postponed after testing revealed the upload validation checks to be too stringent. (see BUG-10607 ‘Back end enforcement for animation upload is blocking upload of many of my animations – “references unknown joints”‘). As a result, the deployment was postponed while the Lab revisited the code to address issues, and a revised package was then rolled to all three RCs on Thursday, November 12th.

Firestorm 4.4.2 Bridge

In terms of the attachment ID and valid attachment points, one group of users who will be affected by the update are those on Firestorm 4.4.2, which uses an invalid attachment point ID for the Firestorm Bridge. Such users will only be affected if they log-in to an updated RC channel (keeping in mind this code will likely be deployed grid-wide in week #47), with the result that the bridge creation process will fail, leaving them with a boulder belly – a default rock shape embedded in their avatar.

Firestorm 4.4.2 users may find themselves with a boulder through their avatars as a result of the simulator code deployed to the RC channels this week. The issue will not affect Firestorm versions from 4.6.9 onwards (image courtesy of Whirly Fizzle)
Firestorm 4.4.2 users may find themselves with a boulder through their avatars as a result of the simulator code deployed to the RC channels this week. The issue will not affect Firestorm versions from 4.6.9 onwards (image courtesy of Whirly Fizzle)

As Whirly Fizzle explains in the deployment thread, to avoid this, those users on Firestorm 4.4.2, and who can, should update to 4.6.9 or later. Those who can’t, and who wish to avoid getting stoned, should disable the bridge creation process (Avatar -> Preferences -> Firestorm -> General -> Untick “Enable LSL-Client bridge” -> Apply -> OK  and detach the Bridge attachment). 

SL Viewer Updates

The Quick Graphics viewer containing the new Avatar Complexity capability and the ability to set multiple graphics presets was updated to version 3.8.7.306758 on Thursday, November 12th. This update sees the removal of a number of issues previously marked as resolved:

  • MAINT-5541 “[QuickGraphics] 0 complexity avatar renders as jelly”
  • MAINT-5567 “[QuickGraphics] A mesh attachment causes avatar to be jellybaby while Avatar complexity is set to No Limit”
  • MAINT-5620 “[QuickGraphics] clicking on Graphics Preset title triggers favorite”

The removal of these fixes would seem to suggest this viewer will remain at RC for a while longer.

HTTP RC Viewer Withdrawn

Also as noted in an addendum to the first part of this week’s report, the HTTP RC viewer (Project Azumarill) was withdrawn from the viewer release channel. This viewer had been suffering from a number of issues, including disconnecting from the simulator during teleports, and also issues such as BUG-10230 – “Voice often fails to connect on the Azumarill viewer”, and BUG-10391 – “Avatar often bakes fails on Azumarill”. These may all have contributed to the removal of the viewer from the channel while the issues are worked on.

Second Life projects update 46/1: server, viewer, issues

Endless Summer; Inara Pey, October 2015, on Flickr Endless Summer (Flickr) – blog post

Update November 11th: The HTTP RC viewer has been with withdrawn – see notes below.

Server Deploys Week 46

As always, please refer to the server deployment thread for the latest updates.

  • On Tuesday, November 10th, the Main (SLS) channel was updated with the same server maintenance package as deployed to the RC channels in week #45, comprising a fix for group invite throttle notifications, and an internal server code clean-up
  • There will be no RC deployment on Wednesday, November 11th.

Simulator Behaviour Changes

A RC deployment had been planned for week #46, which would have included new simulator side validation checks for mesh and animation uploads and attachment point validation.

However, when testing the simulator code on Aditi, Whirly Fizzle found that some of the checks are a little too stringent, and that even valid animations and mesh might be blocked – see BUG-10607 ‘Back end enforcement for animation upload” is blocking upload of many of my animations – “references unknown joints”‘, with even the default official bvh animations being blocked.

As a result of this, and following discussions at the Simulator User Group meeting on Tuesday, November 10th, the Lab postponed the deployment pending a re-examination of the new validation code.

SL Viewer Updates

A new Maintenance viewer was released on Tuesday, November 10th. Version 3.8.7.307376 brings with it some 38 fixes and improvements, including updates for some regressions introduced into the viewer with the current release viewer (version 3.8.6.305981).

The CEF project viewer (Project Valhalla) was also updated on Tuesday, November 10th. Version 4.0.0.307535 includes 17 fixes and updates compared to the initial release of the viewer.

Update November 11th: The HTTP (Project Azumarill) release candidate viewer was removed from the viewer release channel. As noted in my last TPV Development meeting report, the latest iteration of this viewer was generating assorted issues, including viewer disconnects on teleporting, and this may have led to the removal, although no official reason has been cited.

Other Items

HTTP Request Throttling

The server deployment to the main channel in week #45 introduced a throttle on HTTP requests. It had been introduced to prevent a griefing vector which could crash a region. However, the throttle was sufficient to cause llHTTPRequest to give NULL_KEY returns, which impacted a range of in-world systems include skill gaming systems, breedables, and tools such as Sculpt Studio (see BUG-10627 “llHTTPRequest appears to return NULL_KEY even if throttle is not reached”).

Commenting on this at the simulator user group meeting, Simon said:

I’m looking at relaxing that more with (hopefully) next week’s maintenance release; perhaps sooner (but no promises). I also need to update the llHTTPRequest() wiki page.  It really needs details on how scripters should handle a NULL_KEY return; basically they need to code for it … wait about a minute before retrying, and don’t kill your breedables if you can’t talk to your server. [The griefing vector] really caused some problems.   So we need that limit, but hopefully without killing off herds of SL animals.

Scripted normal & specular texture change Issues

Whirly Fizzle has reported a viewer-side issue within the current release viewer (version 3.8.6.305981) which can occur when manipulating materials on in-world objects or worn attachments via script.

The problems are outlined in BUG-10601, “Scripted normal & specular texture changes do not work correctly and screw up other materials enabled objects on the region”, and the problems affect normal and specular maps, and as Whirly notes in the report, the impact can be a little erratic. In sort, using scripts to toggle normal and / or specular maps on a surface on / off fails to result the expected behaviour.

Normal Maps expected Normal Maps 386
On on the Left, how things should work as Whirly uses a script to toggle a normal map on / off to reveal the “001”; on the right, how the same script fails to achieve the same result in the 3.8.6.305981 viewer – click either images for full size

Whirly provides a full description of the problems which can occur, and it should be noted that while annoying when seen, the circumstances under which they occur are most likely narrow enough so as not be to be a major issue for users. The bug has been accepted by the Lab, and Marine Kelley has offered a hotfix which, if needed, might help in formulating a “full” fix for the problem.

Aditi Log-in Sync Issues

A recent breakage in the update mechanism for syncing passwords and inventory between the main grid and Aditi, led to further issues for some users in trying to use both grids (see this forum thread and BUG-10634).

At the start of week #46, the Lab deployed a new method of syncing between Agni (the main grid) and Aditi, which it was hoped would resolve problems. Unfortunately, things didn’t work out as planned, and the new system had to be rolled back.

As a result, and as Simon notes on the forum thread, those wishing to sync their grid passwords and inventory between the two grids should file  a Support ticket asking that your account be copied from the main grid to Aditi.