A note card containing a message from Nyx Linden is doing the rounds in-world concerning Server-side Baking / Appearance. The note card reads in full:
Greetings all,
Nyx Linden (stock)
Wanted to give everyone a quick update on the status of Server Side Appearance. First of all, thanks to all who helped participate in last week’s pile-on test, and a special thanks to those viewers who are already integrating the RegionHandshakeReply flag posted recently. We’ll likely be doing one more pile-on next week, targeting a smaller set of users (to avoid inventory limits that have caused attachments not to load, etc) next week. Let me know if you’d like to participate.
We currently are not aware of any major release-blocking bugs and are starting to look at scheduling the roll-out process. We have a number of QA passes and tests to run through before we can get the final greenlight to do so, so we are currently targeting July 9th as the earliest date at which we will enable SSA for a significant portion of the grid (a server RC channel). Please note that if we find additional bugs in the meantime, or run into other scheduling difficulties this date could be pushed back. We will not be going to RC before this date however.
Please consider this an official warning that this is imminent – We’ve been saying for a while that we’re getting ready for release. We hope with a solid date in mind, all viewers can start messaging to their users that they will need to update or they will start to see issues. There are a few methods by which we will be messaging to the SL community as a whole about this, but we highly encourage you to use your judgement in the best way to reach out to your users and transition them to SSA-compatible viewers.
As always, if you have any questions, or see any issues that could be worrisome if they are not fixed before release, please do not hesitate to reach out to me and/or file a JIRA. Thanks for all your work in preparing for this release!
Nyx Linden
[link and all emphasis mine]
If you want to avoid seeing increasing numbers of grey avatars and / or avoid people telling you, “you’re a cloud” when you appear perfectly fine to yourself, update to a version of a viewer supporting SSB/A
As previously noted in this blog, the plan from the Lab has been to deploy SSB/A gradually. Speaking on this at the TPV Developer meeting on Friday June 14th, Nyx indicated the deployment would be a small group of regions prior to moving to a Release Candidate channel (either in whole or in part) and then progress from there.
However, with the preliminary date for this work to commence, and given that almost all maintained SL viewers are now SSB/A-ready (Dolphin and Exodus are the only exceptions at the time of writing), there really is no excuse for people not to update their viewer. The choices are arleady available:
Those who prefer the v1-style of UI have the choice of Cool VL Viewer and Singularity
Those who prefer the V3-style have the choise of the SL viewer, Catznip, Firestorm, Kokua, Niran’s, and RLV.
In addition, the Lumiya, Meltabolt and Radegast clients are also SSB/A ready.
As always, please refer to the week’s forum deployment thread for news, updates and feedback.
Second Life Server (Main) Channel
On Tuesday June 18th, the SLS main channel received the interest list improvement project which have been previously deployed to Magnum (week 22) and BlueSteel and LeTigre (week 24). This includes:
A fix for excessive AvatarAppearance packets being sent to the viewer [in which the simulator would send many unnecessary AvatarAppearance messages to the viewer]
A final fix for the “meeroo problem” whereby animations on Meeroos and other animals fail to update correctly when camming around.
Release Candidate (RC) Channels
On Wednesday 19th June, all three RC channels (Magnum, BlueSteel and LeTigre) should receive the server maintenance package briefly deployed to BlueSteel and LeTigre in week 24. to fix a number of crash modes, addresses an issue with neighbouring region visibility, and adds new LSL pathfinding capabilities and object return capabilities:
The new pathfinding property CHARACTER_STAY_WITHIN_PARCEL, which can be used with llCreateCharacter() and llUpdateCharacter(), and is intended to help with keeping characters within parcel boundaries – see my week 19 report for details
The new object return functions I reported on in week 23, namely llReturnObjectsByOwner and llReturnObjectsByID, are intended to provide an automated means of returning objects to their owners – see my full update on these functions for details.
This package also includes the following:
An update to llReturnObjectsByID() to prevent it from returning other objects which are owned by the parcel owner or estate owner/manager
A fix for an issue in which LSL HTTP-in scripts would sometimes see the incorrect URL (BUG-2833)
A fix for Bug 2850 (Cannot rez objects in Bluesteel and LeTigre parcels which disallow object entry) – which caused this deployment to be replaced by the Magnum RC package in week 24.
SSB/A Pile-on Test Update
Nyx Linden (stock)
On Friday June 14th, a Server-side Baking / Appearance pile-on test was conducted on the main grid (see my report on events). The Lab is still going over the results of the test and all JIRA filed and log files submitted. Giving a preliminary summary of the test at the Content Creation User Group meeting on Monday June 17th, Nyx Linden said:
We actually just recently got through looking at the bug reports that were filed. Things are looking good, if you know anyone who saw anything major during the pile-on test please encourage them to file a bug ASAP if they have not already done so.
The system seemed to work quite well for most people, and we’re looking closely at the people who were having trouble resolving to try to figure out exactly what happened. The baking service was doing fine, there were some other services that weren’t used to that many people changing their outfits that close together (hence some attachments had difficulty resolving, etc). If anyone knows of failure cases for SSA (aside from those reported if you log in with very old viewers), please let us know asap.
The test did not include a minor update intended for the viewer-end of things, or the code change to help avoid SUN-74. However, as mentioned in my last SSB/A update, the former isn’t required prior to SSB/A starting its deployment across the grid, while the safest way to avoid encountering problems with non-maintained viewers / viewer without the necessary SSB/A updates is to upgrade your viewer. Now.
Viewer News
Materials Processing
The final beta release (3.6.0.277409) performed well over the weekend, with a crash rate “comfortably under” 9%. The code has been merged with the release viewer and is in its final QA testing ready for deployment. Providing nothing unexpected happens, it should appear on the viewer download page as the release viewer very soon.
Viewer Release Process
Work is continuing on the new viewer release process, which may go live later in week 25 or early in week 26. In the meantime, and as reported in week 24, a viewer source repositories page has been produced on the wiki. There is also a further wiki page explaining the release process, although it is still under development. You can find it listed as the Viewer Integration and Release Processes.
Note that the new process does not mean there will be multiple versions of the release viewer available for download (although there will potentially be multiple project / beta / release candidate versions available for download).
Should two projects reach a point of being ready to go to a release status at the same time (such as with “project 1” and “project 2” in the diagram above), a decision will be made by the Lab as to which should go first. That viewer then changes status to release, with the code pulled back to the viewer-release repository. The second viewer awaiting release will then merge with the changes and put out a further release candidate, and will then move to a release status from there.
For ease of reference, the viewer download page and the Alternate Viewers wiki page remain the default places for most users to obtain versions of the SL viewer.
Other Bits
Object Contents Loading
We’re all familiar with using prims as storage for other items (e.g. “boxed” items sold through the Marketplace or using a prim to store items in inventory we don’t frequently use). when a prim has a large number of items in it, there can be a noticeable delay in seeing the contents listed in the Contents tab of the Build floater. In addition, adding objects can be prone to a slow response as well – and can cause problems such as the loss of No Copy items when dropping more objecting into the Contents tab while the system is already copying / adding items to a prim’s contents. I
n terms of the slow loading issue, some have reported times of 30-40 seconds when trying to list the contents of a prim with 100 or so items, and a question was asked at the Simulator User Group meeting on Tuesday June 18th on whether there was any particular reason for this.
Replying to the question, Andrew Linden started with a cautionary note, “Sure, people can create that many items in contents, but I wouldn’t rank it as a good idea.” He then went on, “I don’t know the exact nature of the bottleneck there, but 30 seconds sounds too long. I’m pretty sure it could be sped up, but I’d have to dig around to see why it is slow.”
Kelly Linden then added: “Object inventory transfer from server to viewer uses one of the oldest legacy methods in Second Life. Updating that is probably a good idea but would require a joint viewer and server version change, or some acrobatics on managing compatibility.”
While agreeing this might be the case, Andrew went on, “The UDP protocol should be able to transmit 300 items in much less than 30 seconds. I’ll try to look around to see what is limiting that. However, I should note… I won’t be attending next week. I’ll be on vacation.”
So, there may be further updates on this in the future.
SUN-74 – Asset Corruptions With Non-SSB/A-enabled Viewers
I’ve recently reported on the issue of BUG-74 in relation to server-side baking / appearance. This affects some non-maintained viewers which do not have SSB/A support and which might result in some worn modify assets (skin, hairbase and eyes) being corrupted – see here for details. The issue was finally repro’d successfully by the Lab in week 23. Since then, investigations have been ongoing.
Commenting on the situation during the TPV Developer meeting on Friday 14th June, Nyx Linden said, “we have a technical solution for SUN-74. I have tested it against 1.23[.5], and the old behaviour is no longer reproducing. So hopefully that will mean that once we get it in a place where we can test against Phoenix there should be no more asset corruption.”
Nyx Linden (stock)
It’s not clear when this will happen, but it seems likely the updates will be deployed to the “closed beta” regions where TPV testing of the SSB/A code has been ongoing for the last couple of weeks, and tests will be taken there to ensure non-SSB/A viewers will not be negatively impacted when moving between SSB/A-enabled and non-SSB/A regions during the initial deployment of Server-side Baking / Appearance.
However, this does notmean that people on older, non-maintained viewers no longer need to update to an SSA/B-compatible viewer.
Regardless of the fix for SUN-74. people on non-maintained viewers will start to see increasing numbers of grey avatars around them as SSB/A is rolled out, and willfind that others see them as a permanent cloud. So the only way to be sure of being ready for the deployment of SSB/A is – update or upgrade your viewer if you have not already done so.
Additional Viewer Patch
A side effect of the work carried out on this issue is that the Lab will also be producing a small viewer-side patch which is not any kind of “bug fix” for SUN-74, but which will help viewers get their own appearance messages “just a little bit faster” than is currently the case.
While TPVs are encouraged to incorporate the patch once it becomes available, it is not seen as a mandatory requirement ahead of SSB/A being enabled on the main grid. As such, TPVs have been encouraged to integrate the patch once it becomes available and as it best fits their own release schedules.
Grid-wide Deployment
When asked about how the Lab plans to deploy SSB/A server-side at the TPV Developer meeting on Friday June 14th, Nyx replied:
Carefully. Definitely carefully. We are doing a lot of testing, and as most of you know, we’re doing an Agni pile-on [see later in this report] … and we have been doing a lot of load testing and we’re pretty confident we have enough hardware on the back-end to handle the load.
[So] We’re going to start with a small group [of regions] and go to an RC channel, and then more, and then take over the entire grid.
Whether “go to an RC channel” means enabling SSB/A across an entire RC channel (Magnum, BlueSteel or LeTigre) or enabling across a portion of regions on the selected RC channel is currently unclear. That decision doesn’t rest with Nyx, but will be dependent upon on number of factors including how well the initial steps in the deployment go.
In light of things like the pile-on test (see the next section) and readying the SUN-74 patch, the Lab remains unwilling to commit to specifying a date by which SSB/A deployment might be expected to start. This is understandable as there is still no guarantee that further issues such as SUN-74 won’t be uncovered as a result of either the pile-on test or as a result of further closed beta testing, which the Lab continues to monitor.
Main Grid Pile-on Test
On Friday 14th February pile-on test was conducted across a number of regions which had been specifically set-up to stress test Server-side Baking with some “real world” avatar numbers. Some fifty or so people turned up for the tests using various viewers with Appearance debugging enabled, including a version of the official viewer which had been pre-set with debugging enabled. The test was in three parts:
Baseline testing on regions using the current avatar baking mechanism
Testing on regions in the Snack RC channel running a version of the SSB/A code
Testing in the “closed beta” region specifically set-up for TPV testing running the SSB/A code
The precise differences between the code on the Snack regions and the code on the TPV test region (the Testylvania Sandbox). Questions were asked in open chat, but the nearest answer which seemed to be given was that the Testylvania region was the one the Lab “cared about the most”.
Testing on the current baking mechanism saw familiar issues of slow clothing / skin rendering and the need to use manual rebakes to try to encourage non-blurred appearances. Things appeared to be a lot better on the SSB/A-enabled regions (I personally experienced no issues in changing skins / clothing layers and found rendering of both fast and error-free, for example). However, some did report issues with rendering, and filed JIRAs / provided log files as a result.
There were multiple reports of attachment rezzing failures as of the asset service coming under pressure as a result of so many outfit / look changes going on simultaneously and in rapid succession. Whether these will see further work undertkaen on the inventory system (some work has already been carried out in a wider context by the Sunshine team), remains to be seen.
Expect more on the tests once LL have had time to chew on the data gathered and review the logs of those who did encounter issues.
Update: In further tests of the FIRE-9097 “fix” at lower resolutions (e.g. 2650 pixels across), I found it can re-introduce the tiling artefacts in snapshots.
General Viewer News
Materials Processing
The release of an update to the materials beta viewer on Wednesday June 5th (3.6.0.276961) was followed at the weekend by the arrival of a further beta version – 3.6.0.277049 – with accompanying release notes. Commenting on the rapid-fire releases, Oz linden said at the Content Creation User Group meeting on Monday June 10th, “We’re getting close to the end of its beta cycle (or put another way… report your bugs now).”
Snapshot Issues
We’re all aware of the snapshot tiling issue which plagued SL photographers for a good while, which would leave “tiling” artefacts on images taken at higher resolutions than the user’s monitor resolution when running in deferred mode (now known as Advanced Lighting Model). A fix for this issue (MAINT-628) finally reached the public in late 2012, but brought with it some additional issues. On of the most notable of these was the appearance of black rectangles in very high resolution images.
Very high-resolution “black rectangle” issue common to snapshots taken with viewers utilsing the MAIN-628 “tiling” fix (image courtesy of Dil Spitz)
This latter problem most recently caused an additional outcry when the LL “tiling” fix was finally incorporated in Firestorm viewer earlier in 2013, with many users incorrectly blaming the Firestorm team for the problem.
Well, for all and sundry, Firestorm users or otherwise, there is some potentially good news on the horizon.
Firestorm image artefact fix: image at 6000 pixels across, saved as JPG (click to enlarge)
Commenting on the broader issues reported with snapshots duing the Open-source Dev meeting on Monday June 10th, Oz Linden said:
It looks like there are fixes in the MAINT pipeline for those. I don’t know how soon those will be out… I can try to find out if they have a project build ready.
Additionally, there is further news specifically for Firestosm users. While it is somewhat outside the scope of “SL project news”, it is neverthless reporting here.
Nicky Dasmijn has been working on the problem, and has implemented a fix for the issue (see Firestorm JIRA FIRE-9097), which should correct matters for snaps of up to 4096×4096 pixels without any “rectangle” artefacts appearing or with any regression to issues of tiling, and which may work at high resolutions than that for some.
Firestorm image artefact fix: image at 6000 pixels across, saved as PNG – note rectangle artefact (click to enlarge)
I tried a very rough-and-ready test of the fix. I found that capturing images up to 5,000-5,600 pixels across with the aspect ratio maintained worked OK for me. Anything around 6,000 pixels across saw JPG images save OK, but rectangle artefacts begin to appear when saving in PNG (see both images on the right)
However, as I’ve recently been experiencing other GPU issues, I’ve been unable to ascertain if the rectangles are down to an issue with the code or simply a matter of my GPU running out of resources when processing PNG images above 5600 pixels across.
The fix is currently in a recent Firestorm pre-release, and will hopefully make the cut for the next formal release. It is currently unclear whether the code has been / will be contributed to LL, and if so, whether they will adopt it or opt to go with their own forthcoming updates (as indicated by Oz in his statement above) or opt to combine it with their own fixes (depending on the nature & scope of the latter).
Future “STORM Project Viewer” Release
There are a number of code contributions which have come via the Snowstorm route which have been queued awaiting a suitable release. These cover a range of additions to the viewer, and example of which is STORM-68 (As a Builder, I want that ability to set default permissions on creation of objects, clothing, scripts, notecards, etc.).
Commenting on STORM contributions in general, and in light of the forthcoming changes to the viewer release process, Oz said, “I’m trying to get all the storm issues merged up so that I can be ready to put out a project viewer as soon as the new viewer version manager is deployed.” Whether STORM-68 (which is apparently seen as “largely good to go”, although it may also require a server-side change), or the fixes for snapshot issues mentioned above will be among them remains to be seen. However, a “STORM” project viewer could well be adding even more features to the SL viewer in the near future.
As usual, the latest updates, feedback and comments can be found on the deployment discussion thread. Anyone encountering a specific bug is asked to file a JIRA.
There was no SLS Main channel roll out this week
On Wednesday June 5th, the Release Candidate channel received the following packages:
BlueSteel and LeTigre were updated with a new server maintenance project. This project addresses a disconnection issue and also fixes a crash mode – see my notes from week 22, and a fix for a crash mode
Magnum remained on the same interest list improvement project as originally deployed to LeTigre in week 21, and to Magnum and BlueSteel in week 22, with some updates. Two of these fix what Andrew Linden describes as “two rare crash modes”. The package should also include the same disconnection issue fix.
Interestingly, the “large” scripts issue I reported on in part 1 of this update was given as the reason why there was not Main channel deployment this week. As previously reported, the fix for this issue, which prevents the text of previously saved scripts from being displayed in the script editor for users on slow connections, failed to make the cut for the week 23 deployments.
There are continuing reports of “invisible avatars” on Magnum regions. This issues was first reported following the week 22 deployments, and described as “on an in-region teleport when landing all surrounding avatars de-rezz and cannot be seen until the person re-logs. Everything else appears normally.” The problem appears to be random in nature, and was also noticed at one club which enjoys a high level of attendance. and which still appears to be encountering the same issue. During the Beta Server meeting on Thursday June 6th, this issue was discussed, and appeared to be most strongly linked to v1-based viewers.
SL Viewer News
The next release of the materials beta viewer arrived on Wednesday June 5th (3.6.0.276961 with the release notes here), and as I indicated in part one of this report, sees the beta once more installed into the “correct” folder (SecondLifeBetaViewer). This means the initial beta release needs to be uninstalled separately, as the updated version obviously won’t over-write it.
The materials project beta viewer had its first update on June 5th, with the release of 3.6.0.276961
In terms of beta releases in the future, it’s worthwhile again pointing-out that once the new viewer release process comes into effect, beta viewers will be installed into folders identified by their project name (e.g. something akin to “SecondLifeBetaMaterials”). Viewers should be fairly well self-contained (although they may still share the same default settings location, as that remains to be seen once things start rolling), so the uninstalling of individual beta versions (or RC versions) shouldn’t be a problem once they reach release status.
Work continues in preparing the new viewer release process for … release (or implementation). It’s still unclear whether it will arrive before or after materials moves to the release viewer. As reported in week 22, the current pipeline of releases we should be seeing as the new release process rolls forward includes:
A collection of open-source contributions to the viewer which is hoped will appear as a release candidate viewer pretty quickly
A “pretty substantial batch” of maintenance fixes for the viewer
Vivox updates, which Oz described as, “Finally getting attention again, and will probably be in a release candidate version ‘real soon’ now”
An Experience Tools viewer which is also expected to appear “real soon”
An interest list update viewer, which is believed to be getting closer to being stable
Server-side Baking / Appearance
Investigations have been continuing into the SUN-74 issue which affects non-SSB/A updated viewers (notably Phoenix), with Nyx Linden commenting at the Server Beta meeting that, “Thanks to the support from the phoenix/firestorm team, we’ve been able to identify the cause of that issue. We’re looking into what options are available to us. [It] took some backflips to get it in a debugging environment, but managed to hunt it down – a combination of factors from not having the last 4 years of appearance fixes :)”
Apologies for the slight delay in getting this update out, real life is proving a little time-consuming at the moment.
Server Deployments, Week 23
As usual, the latest updates, feedback and comments can be found on the deployment discussion thread. Anyone encountering a specific bug is asked to file a JIRA.
Second Life Server (Main) Channel
No rolls are planned for the week 23.
Release Candidate Channels (RC)
On Wednesday June 5th, the Release Candidate channel should receive the following:
BlueSteel and LeTigre should get a new server maintenance project. This project addresses a disconnection issue and also fixes a crash mode – see my notes from week 22, and a fix for a crash mode
Magnum will remain on the same interest list improvement project as originally deployed to LeTigre in week 21, and to Magnum and BlueSteel in week 22, with some updates. Two of these fix what Andrew Linden describes as “two rare crash modes”. The package should also include the same disconnection issue fix.
A further fix had been planned for the RC channels. This relates to people’s inability to download “large” scripts. This relates to the code path used for script uploads / downloads having a bug, such that you can write a lengthy LSL script and save (upload) it, but on trying to edit it once more, the text of the script will not display. The issue is thought to be related to bandwidth use, and while a fix has been developed by Andrew Linden, but it failed to make the QA cut for this week’s RC releases.
Viewer News
There have been reports of issues with the materials beta viewer, including:
Crashing when logging-in to SL on systems using Intel graphics
Issues with transparent alphas showing as white and semi-transparent alphas showing as black, which also appears linked to systems with Intel-based graphics
The Lab is currently working on a fix for the Intel issue, but the alpha issue is apparently providing difficult to consistently reproduce
That the materials beta viewer (3.6.0.275764) installs into a different folder to previous SL beta viewers (SecondLifeBeta rather than SecondLifeBetaViewer), as reported in my overview of the materials beta release, appears to have been an error on the Lab’s part, and it appears likely the additional releases will revert to the SecondLifeBetaViewer folder until such time as the new viewer release process comes on-stream.
Server-side Baking / Appearance
The Lab continues to investigate SUN-74, although there has been no major progress since my last update. The JIRA itself has been updated as a result of further TPV testing.
In terms of any deployment time frames, the Lab still will not be drawn on dates at the moment (again, understandably, even the likes of SUN-74 and the need to try to push more users into updating to viewers which do support SSB/A). Replying to a question on possible deployment beyond the current close beta regions at the Content Creation UG meeting on monday June 3rd, Nyx Linden would only say, “SSA will be deployed slowly and carefully when its ready, we’re working with third-party devs to make sure the last of the bugs are found and hunted down.”
Interest List Update
Andrew Linden – who marked his 11th rezday on June 4th, 2013!
As Andrew has been involved in trying to resolve the “large” script bug described above, he’s not had time to make further progress on the “Meeroo issue”, which can affect other scripted animals, etc., as well, and which he describes as:
If you turn your camera away from a crowd of Meeroos, wait several seconds, then turn back around… the Meeroos will be updated, but not quite in the right order. So sometimes you’ll see a head move to the new position, then a fraction of a second later the rest of the body. So I have a theoretical fix that doesn’t crash the simulator (anymore).
As noted recently, he has developed a partial fix for this problem was deployed as a part of the current interest list updates, and he now hopes he’ll be able to focus on developing a more complete fix, which will mark the final aspect of server-side interest list work for the moment.
Inara Pey: Living in a Modemworld 