As noted in part 1 of this report, the Second Life Server (SLS / Main channel) received the server maintenance project which was on all three RC channels in week 8, comprising miscellaneous bug fixes, and the improved region restart notification. Deployed on Tuesday 26th February, the roll-out was followed by a number of issues being reported via the deployment forum thread.
Issues have included:
Parcels / regions vanishing from search for a period of time following the restart (an issue WolfBaginski Bearsfoot gave some thought to in commenting on part 1 of this report)
Further region crossing issues, with loss of control followed by “broken” camera positioning after recovery. This issue has been reported at some of the Simulator and Server Beta meetings previously, and Eric Gregan has produced a video demonstrating the problem as it occurs on some aircraft, although he’s also come up with a means of avoiding the issue which may or may not work for people.
Problems with Dwarfins attachments.
Release Candidate Channels
On Wednesday February 27th, all three Release Candidate channels received a new server maintenance package which includes a fix for a crash mode (see the release notes (BlueSteel)). No significant issues have been reported for this deployment.
SL Viewer
The Communications Hub User Interface finally made its debut in both the SL development and SL beta servers, arriving as release 3.5.1.270826 for the development viewer on Tuesday 26th February, and 3.5.0.270825 for the beta viewer on Wednesday 27th. I’ve provided a brief introduction to CHUI, and there is also a video from Torley Linden.
CHUI is liable to remain in beta for a “Good long run”, to quote Oz Linden. Hopefully, this may mean that materials processing will be the next significant viewer update to arrive as a project viewer.
Interest List
The recent updates to Andrew Linden’s interest list work are apparently won’t reach a RC channel in week 10, but should see deployment in week 11 (week commencing Monday 11th March). This work includes fixing issues with avatar appearance teleporters which use llSetRegionPos(), as well as correcting problems where objects which change appearance behind the camera ‘snapping’ into place when you rotate the camera back.
Server-side Baking
On Friday March 1st, Linden Lab unexpectedly released an updated version of the SSB project viewer (release 3.4.5.271118), which includes their approach to overcoming the problems of the Z-offset capability common to many TPVs being broken as result of the SSB code. The new approach, which I was able to outline when it “launched”, introduces an additional shape slider into the Edit Shape floater; as such, the ease-of-use of the approach, particularly for those who may swap between different avatar shapes (e.g. “normal” and petite) is very questionable.
The new “Hover” option in the Edit Shape panel for adjusting avatar height offsets in the Sunshine project viewer. Not the most elegant solution
It’s a “lightweight” week for deployments, with all channels receiving one of two maintenance packages.
Server Life Server (Main) Channel
Region restart notifications: the “old” style (top), and the “new” style (bottom)
On Tuesday February 26th, the Second Life Server (SLS / Main channel) received the server maintenance project which was on all three RC channels last week. This project has miscellaneous bug fixes, along with an impoved restart notification (which will only be apparent when this new version restarts for the next update) – see the image on the right.
While this is much better than the previous notification, it does suffer from the fact it will still fade from the viewer’s window and become a chiclet whether or not the OK button is clicked – and thus can still be missed if attention is not on the screen when it appears. To help avoid this, it is possible to set a sound for the alert using the Debug Setting UISndAlert, which requires the UUID of the sound, which coule be one you’ve uploaded to inventory yourself (Firestorm additionally has UISndRegionRestart, which can be set through Debug Settings or Preferences > Sound & Media > UI Sounds 2).
This release means the until the RC channel deployments take place (see below), almost the entire grid is running the same server code – the only exceptions being a few “special” regions (such as the Linden Realms regions) – release notes.
Release Candidate Channels
On Wednesday February 27th, all three Release Candidate channels should receive a new server maintenance package which includes a fix for a crash mode – release notes (BlueSteel).
As always, there is a forum thread for discussing the deployments.
Server-side Baking
Following-on from his initial feedback given at the TPV Developer meeting on Friday 22nd February, Nyx Linden gave a further update on the recent Sever-side Baking (SSB) pile-on / load test when chairing the Content Creation User Group meeting on Monday 25th February
Thanks all for everyone who helped out with the pile-on test of server appearance last Thursday. The results tended to be that when the system worked, it was nice and speedy, however there were a number of issues that prevented it from working for everyone all the time. We got a good bit of data on the bugs we knew were there as well as a couple new ones. we’ve been crunching on the results and are starting to attack the issues, so it was definitely successful in achieving what we wanted: more data :). There will be future rounds, so thanks in advance for anyone who comes back to help us test again.
Z-offset Issue and SUN-38
As noted in the comments following my quick SSB test using currently available TPVs with SSB enabled, there is an issue with the “Z-offset” capability common to many third-party viewers, which allows the vertical height of an avatar above the ground to be adjusted, such that sits and kneels don’t leave the avatar apparently floating in the air, and which allow those with very tall / giant avatars or very small / petite avatars and those wearing full body mesh to similarly adjust their vertical placement relative to the ground / floor.
The problem was reported to Nyx in week 8, and SUN-38 was subsequently raised by Henri Beauchamp on February 24th to officially log the matter. Commenting on the problem during Content Creation meeting, Nyx Linden stated. “It hasn’t escaped our notice. We’re considering a couple different approaches … we’re considering a few different options. Suggestions appreciated, but we haven’t officially settled on an approach we’re going to commit to publicly just yet.” He also suggested that any questions on the subject be asked again next week, presumably by which time LL will have had time to consider the problem in detail, and hopefully consult with TPV developers.
Interest List Updates
Andrew Linden is continuing to work on the interest list code, with more refinements to come, as well as looking at a number of bug fixes. The updates about to go to LL’s QA include changes which should make scenery load faster / more correctly when first logging-in to SL or teleporting to a region.
Future updates to the interest list code should see scenery in your world view rezzing a lot faster
Update February 24th:Metabolt 0.9.64.0 (Beta) was released on February 23rd to address the issue of nearby objects not being recorded following theserver-side interest list updates.
Update February 22nd: Radegast 2.8 has been released, which both provides support for server-side baking and resolves the interest list releated issue of failing to correctly report nearby objects noted in the first poart of this report.
Week 8 Main (Second Life Server / SLS) Channel Deployment
The expected Main channel (SLS) deployment took place on Wednesday 20th February, as anticipated. The SLS channel deployment came a day late as a result of the Presidents Day holiday in the US – Main channel deployments will be back to their normal Tuesday slot from week 9 (week commencing Monday 25th February.
Issues
There is a report that both the Radegast and Metabolt text clients are either incorrectly seeing local objects, or failing to see them at all, possibly as a result of the interest list code deployment.
There is a bug in the interest list code which means that child prims within a linkset do not always render from a distance – you need to cam in towards them for them to appear, and they can then vanish again on camming away. A fix for this is expected in the “next round” of interest list updates.
Week 8 Release Channel Deployments
There have been signficant changes to the RC channel deployments occurring on Thursday 21st February (again a day late due to Presidents Day).
Due to a last-minute bug being found in Baker Linden’s large object rezzing code updates, the package intended for both BlueSteel and LeTigre has been pulled. Instead, these channels will receive the code targeted for Magnum, together with the updates made to the Main channel.
Diagonal Region Rendering Issues
I’ve reported on this on a number of occasions recently, and JIRA SVC-8130 is still open on the matter. The incidence of occurrences seems to be on the increase, with more people having anecdotal evidence of problems. Following his comments in the second part of my week 7 report, Simon believes a fix is in the works, with Simon reporting at the Simulator User Group meeting on Tuesday 19th February that it is, “With QA now.”
Missing regions: bug fix on the way …
Region Crossing Issues
Numerous problems are being reported with region crossings, some of which appear to be on the increase of late.
Content “Warping”
I first reported on this in week 48 of 2012, as one of a number of region crossing issues being reported. When crossing between regions, the contents of the region you’ve just left suddenly “warp” into the region you’ve just entered, appearing for a time and perhaps somewhat deformed.
When the matter was first raised, Andrew Linden thought it might be more a server-side issue than a viewer-side problem – although all the issues raised at that time seemed to be split between possible issues within the server code and the viewer code.
Sometimes on crossing between regions, objects from the region you’ve just left seem to appear “warp” into the region you’ve just entered, and may appeared deformed…
In looking at the problem as a server-side issue, Andrew had hoped that the matter would be resolved via the interest list code roll-out. However, it has persisted despite the interest list code having been deployed to an RC channel, and is now largely regarded as a viewer-side issue. The status of any fix for the problem is unclear.
…only to disappear after a few seconds / as you approach them
Camera Position Lost on Region Crossings
There has ben a return / increase in issues of the camera position getting “messed up” when crossing between some regions in a vehicle. This can take a variety of forms, most notably with the camera suddenly shifting to show a side view of the vehicle while driving, rather than the usual forward-looking view SVC-7684 provides the details, complete with a video demonstrating the issue. It appears that fixes for this problem have thus far been on a per-region basis, as there is no clear indicator as to the underlying issue – although it is recognised enough such that Motor Loon now incorporates a scripted workaround for the problem in lieu of a fix, which forces the correct camera positioning to be re-applied on a script crossing. The Lab have been trying to investigate the issue, but are having a hard time reproducing it in a consistent manner. Those routinely experiencing the problem on given region crossings should consider filing a JIRA specifying the issue and region crossings where it occurs and reference SVC-7684.
Increase in Crossing Disconnects and Other Issues
The Simulator User Group meeting on Tuesday 19th February saw an increase in the number of other region crossing issues, with people reporting several issues, ranging from a disconnect on attempting a region crossing through to an increase in teleport failures when moving between issues and getting stuck in the corner of a region and spinning during a region crossing. Again, it is unclear what may be causing these issues, some of which have been reported as increasing somewhat sharply over the last 2-4 weeks.
Sudden Region Lag
It has been a while since I last updated on this issue, as reported in both a forum thread by Toysoldier Thor and others and in a JIRA BUG (BUG-355). In sort, the problem remains, with live event venues reporting issues, and the Lab uncertain as to the cause. When the issue was raised at the Simulator User Group meeting, Simon Linden replied, “Hmm, that sounds like it needs a fresh look. It seems like there’s a spike in networking traffic, performance goes bad and thus all sorts of things don’t work well.”
Interestingly, the four regions of the recent OBR in SL event did not suffer any such issues, despite a constant and high volume of traffic, although these had apparently received additional attention from the Lab to help ensure problems might be minimised.
Server-side Baking Load Test
A final reminder that there will be a Server-side Baking load test on Aditi on Thursday February 21st, following the Server Beta meeting. For details, please see my original announcement.
The deployments scheduled for the week commencing Monday 18th February are detailed below. Please note that due to Monday 18th being a holiday in the United States (Presidents Day), the deployments will be taking place one day later than usual.
Main (Second Life Server / SLS) Channel
The Main channel should receive the following two projects:
The Interest List Improvement project, which has been on the Magnum RC channel for the past few weeks
Server-side support for materials processing.
Note that there is still no publicly available project viewer to take advantage of the materials project code, although there may be news on this later in the week.
This deployment should take place on Wednesday 20th February – release notes.
Bluesteel and LeTigre Release Candidate (RC) Channels
Baker Linden in a change from his usual rooster avatar
Bluesteel should receive Baker Linden’s object rezzing code changes, which were reported here in week 1. These updates have nothing to do with the interest list code changes. Baker describes the aim of the work as, “Hopefully significantly decreasing lag spikes when rezzing large, complex objects. Large does not necessarily imply size, but size of the files being read. When an object is rezzing, we have to parse the object / mesh files and create our in-world objects with that data.”
Until now, reading and parsing of any files related to objects which require rezzing has been on the main thread. When several such objects requiring rezzing at the same time, the simulator stalls. Baker has been moving the reading / parsing operation to a background thread in the expectation that rezzing multiple “large” (again, in terms of file size, not the size of the object itself) objects will not choke the simulator, causing performance issues.
These deployments should take place on Thursday 21st February – release notes (Bluesteel).
Magnum Release Candidate (RC) Channel
Magnum should receive the same maintenance server update deployed in week 7 to LeTigre, intended to fix miscellaneous crash modes.This deployment also includes the following:
An improvement to the rolling restart notifications so that they appear in an alert format (as with manual region restarts) rather than an easily missed notification. This change will only be apparent in restarts following the code deployment restart (as per JIRA SVC-7759)
A fix to an encroachment / return problem: if you’re banned from the neighbour’s parcel, you couldn’t select / return items that encroached on your parcel (see JIRA SVC-496)
Instant messages are now truncated to 1024 bytes to prevent certain types of delivery failure. Currently, the IM database supports larger messages than the delivery system can handle. This change will enforce a limit of 1024 bytes when processing messages coming into the database as well as those being sent out.
This deployment should take place on Thursday 21st February – release notes.
SL Viewer News
The release version of the SL viewer moved to the 3.4.5 code base on the 14th February, with the release of viewer 3.4.5.270263 (release notes). At the same time, the Server-side Baking project viewer received its second update with the release of version 3.4.5.270409, od which more below.
The CHUI viewer received a further update to the development version, reaching 3.4.6.270520 on February 18th. This project is currently the next in line to merged into the viewer-dev code base (development viewer) and then into the beta code base.
Server-side Baking Load Test
Serer-side baking – load test February 21st
A reminder that if all goes according to plan, there should be a special load test for Server-side Baking on Thursday February 21st, and volunteers are being sought to assist.
This will take place on the SSB test regions on the beta grid (Aditi), immediately following the Server Beta User Group meeting which take place at 15:00 SLT on Thursdays in Morris, also on Aditi with the aim of placing the SSB code under a stress test representative of the loads it will face when deployed to the main grid, with people routinely changing outfits, updating their appearance, enter / leave regions running the SSB code (given that the grid will, for a time, be running both the current avatar baking service and SSB as the latter is initially deployed), and so on.
Test Requirements
While final details of the test have yet to be confirmed, key requirements for those wishing to participate in the test are as follows:
Participants must be able to log-in to Aditi and attend the Sunshine test regions from 16:00 SLT onwards (participants can attend the Server Beta UG meeting ahead of the test if they wish)
Participants must be running the latest version Server-side Baking project viewer (version 3.4.5.270409 or later) – this viewer has been specifically configured to report statistics required by LL for the test
Participants should have a number of outfits of system clothing, preferably with multiple layers, which they can swap between during the course of the test. Library outfits are acceptable, but LL are keen for people to use their own outfits to add greater weight to the tests
Clearing the viewer cache prior to the test is suggested, but not an absolute requirement.
“if you have specific failures we’ll ask for your viewer logs, otherwise just running through the test will help us gather data,” Nyx added when explaining what is required by way of feedback from those opting to take part.
the server deployments planned for week 7 all went ahead as scheduled.These comprised:
Main channel: (Second Life Server / SLS): received the maint-server package focused on crashes fixes which was deployed in week 6 to LeTigre – release notes
Bluesteel: retained the materials processing project code and received the same fixes and updates being deployed to the SLS channel (above) – release notes
Magnum: retained the interest list project code and received the same fixes and updates being deployed to the SLS channel (above) – release notes
LeTigre received a new maintenance server update to fix miscellaneous crash modes – release notes. This deployment also included the following:
An improvement to the rolling restart notifications so that they appear in an alert format (as with manual region restarts) rather than an easily missed notification. This change will only be apparent in restarts following the code deployment restart (as per JIRA SVC-7759)
A fix to an encroachment / return problem: if you’re banned from the neighbour’s parcel, you couldn’t select / return items that encroached on your parcel (see JIRA SVC-496)
Instant messages are now truncated to 1024 bytes to prevent certain types of delivery failure. Currently, the IM database supports larger messages than the delivery system can handle. This change will enforce a limit of 1024 bytes when processing messages coming into the database as well as those being sent out.
Feedback on all the deployments has so far been muted, with only a couple of issues having been reported via the forum thread,
There is no news on deployments for week 8 (commencing Monday, 18th February), as the meeting to determine upcoming deployments does not take place at the Lab until Friday of the current week.
Server-side Baking (SSB) Load Test
Serer-side baking – load test February 21st
On Thursday February 21st there will be a special load test for Server-side baking, and LL are looking for volunteers to help.
This will take place on the SSB test regions on the beta grid (Aditi), immediately following the Server Beta User Group meeting which take place at 15:00 SLT on Thursdays in Morris, also on Aditi.
The aim is to place the SSB code under a stress test which is representative of how SSB will be used once it is deployed to the main grid – with people routinely changing outfits, updating their appearance (as SSB handles appearance updates differently to the current service), enter / leave regions running the SSB code (given that the grid will, for a time, be running both the current avatar baking service and SSB as the latter is initially deployed), and so on.
“We have a few other internal stress tests, but wanted to do one with real-world conditions on real connections,” Nyx linden explained when announcing the test opportunity. “The test will run through switching from the old system to the new system, which is a transition where issues may pop up. if enough people are changing outfits simultaneously it should get us some valid data.”
Test Requirements
While final details of the test have yet to be confirmed, key requirements for those wishing to participate in the test are as follows:
Participants must be able to log-in to Aditi and attend the Sunshine test regions from 16:00 SLT onwards (participants can attend the Server Beta UG meeting ahead of the test if they wish)
Participants must be running the latest version Server-side Baking project viewer (version 3.4.5.270409 or later) – this viewer has been specifically configured to report statistics required by LL for the test
Participants should have a number of outfits of system clothing, preferably with multiple layers, which they can swap between during the course of the test. Library outfits are acceptable, but LL are keen for people to use their own outfits to add greater weight to the tests
Clearing the viewer cache prior to the test is suggested, but not an absolute requirement.
“if you have specific failures we’ll ask for your viewer logs, otherwise just running through the test will help us gather data,” Nyx added when explaining what is required by way of feedback from those opting to take part.
Aditi Log-in
As has been reported in this blog on a number of occasions, the Aditi grid is itself subject to a number of issues, both in terms of access and inventory support.
If you have not logged-in to the beta grid at all, or in the last several months and wish to participate in the SSB tests, it is recommended that you test your Aditi log-in (you use the same user name and password as you use to access the main (Agni) grid) sooner rather than later. If you find you are unable to log-in, then you should try changing your password. to refresh your Aditi access (this should also update your beta grid inventory).
HOWEVER, be warned that this process isn’t always successful, due to the issues mentioned above, and that it can take up to 48 hours before you can access Aditi, and even then, your inventory may not be successfully updated.
If you have recently updated your password and have reasonable inventory access on Aditi, the recommendation is that you don’t force any inventory update by running a further password change, as there is a risk you may either lose access to Aditi or that you may become subject to inventory change failures.
Those who wish to participate in the load test, and who encounter either issues with their Aditi inventory or accessing the beta grid can, as a last resort, contact Nyx Linden at least 24 hours ahead of the test. Nyx will then endeavour to see if LL can fix matters.
Other Items
Vanishing Regions
Last updated in SL project news: week 6 (2), wherein diagonally adjacent regions fail to render until such time as the observer moves to a region immediately bordering the “missing” region. This has been an ongoing problem for some time, as reported in SVC-8130, and commenting on it at the Server Beta meeting, Simon Linden indicated that the maint-server code deployed to LeTigre may help with some of the issues being encountered, but also admitted, “There’s been some improvement but it looks like there are still bugs to chase there.”
Missing regions: new Maint-server code may help…?
There is a forum post on the matter – if you are encountering this issue in a reproducible manner, and in lieu of SVC-8130 being open to comment, you might want to note your experiences on the thread.
On Tuesday February 12th, the Main channel (Second Life Server / SLS) received the maint-server package focused on crashes fixes which was deployed in week 6 to LeTigre – release notes.
On Wednesday February 13th the three RC channels are scheduled to receive the following packages:
Bluesteel: retains the materials processing project code – please see the SL Viewer section below for information regarding the current status of viewer development work for thi project – and should receive the same fixes and updates being deployed to the SLS channel (above) – release notes
Magnum: retains the interest list project code and should receive the same fixes and updates being deployed to the SLS channel (above) – release notes
LeTigre should receive a new maintenance server update to fix miscellaneous crash modes – release notes. This deployment also includes the following:
An improvement to the rolling restart notifications so that they appear in an alert format (as with manual region restarts) rather than an easily missed notification. This change will only be apparent in restarts following the code deployment restart (as per JIRA SVC-7759)
An avatar banned from a parcel can now select an object located in that parcel if they have ‘Return’ powers over that object (see JIRA SVC-496)
Instant messages are now truncated to 1024 bytes to prevent certain types of delivery failure. Currently, the IM database supports larger messages than the delivery system can handle. This change will enforce a limit of 1024 bytes when processing messages coming into the database as well as those being sent out.
As always, a forum thread has been created for discussions / feedback on the deployments.
SL viewer Updates
Beta Viewer
While the current beta release of the 3.4.5 code remains stable, problems are being reported in merging various project code updates with the 3.4.5 code base, some TPV developers are reporting issues attempting to merge the 3.4.5 code into their viewer builds, notably with Linux 64-bit and Windows, although the same issue does not appear to be occurring with Mac OSx.
In the meantime, one more 3.4.5 beta release of the SL viewer is anticipated during week 7.
CHUI
As noted in part 3 of the week 6 report, the CHUI project viewer is now largely on a par with the development viewer, although it has yet to be actually merged with the main viewer-dev code trunk. This is due to happen very soon. In week 6, Oz Linden commented that it is anticipated that CHUI will be the first of the upcoming projects to merge with the beta viewer code, and would then be in for a (quot) “Good long run” in beta. This week, it has been hinted that CHUI could actually have a lengthy run in both the development viewer and the beta viewer as final priorities are sorted out.
Server-side Baking Project Viewer
Problems remain in merging the SSB code with viewers using RLVa, although good progress is being made in addressing them, although they have yet to be merged-up to a TPV which supports RLVa and have a version under development using the SSB code.
In the meantime, the latest updates to the viewer code from the Lab were pushed to the project repositories on Monday 11th February, and a new version of the Sunshine project viewer for Windows emerged on Tuesday February 12th – Sunshine 3.4.5.270231, with Mac and Linux builds to follow.
Materials Viewer
Problems remain with the materials processing viewer-side code. Precise details are not clear, but the latest private (LL and project members only) Windows builds are being reported as being, “Conspicuously more unstable than the Mac builds”, in that they tend to crash when connecting to a materials-enabled server. Investigations into the problems (notably by TPV developers engaged in the project) are ongoing to try to locate the underpinning cause.
Materials processing: viewer issues still being resolved, so no public project viewer as yet
There are a couple more features to go into the materials viewer as well prior to it reaching a public project viewer status, but these are apparently being held-off until the current server interaction problems are resolved.
Experience Permissions
July saw the launch of the first phase of the Advanced Creation Tools, also referred to as experience tools. Following problems with an initial deployment of the tools in June, which resulted them being exploited as a means of griefing, the “first phase” of the release saw the tools implemented with existing permissions system in place, with the intention of updating the permissions system to allow the tools to be more fully used “in the future”. Since then, it has been something of a waiting game to see if / when the promised new experience permissions would be deployed, and my own last report on the general status of the permissions system being in December 2012.
Questions are still being asked at various LL meetings, he most recent being tabled at the Content Creation User Group on Monday 11th February. In reply, Nyx Linden indicated he would attempt to get an update from Falcon Linden, who is leading this project.
Object Inventory Issues
Some people may have encountered problems when trying to transfer a number of inventory items into a box (prim rezzed in-world) or similar. There are two known issues with this:
The first is (non-publicly viewable) JIRA BUG-383: “When HTTP Inventory is disabled, mass selecting a bunch of inventory items and attempting to add to an objects contents will fail”. This will apparently happen if the HTTP inventory setting in a viewer is changed without cache also being cleared
The second is (non-publicly viewable) JIRA SEC-1041, which is apparently (according to Kelly Linden) related to “An internal TCP error between sim and dataserver.” The problem manifests with the in-world error-message “Inventory creation on in-world object failed” on trying to transfer a large number of items (e.g. 50+) from inventory into the contents of a prim in one go.
As the JIRA on both of these issues are non-viewable, their status isn’t clear, other than the problems described in them can be reproduced. SEC-1041 is the more serious of the two problems and might result in a crash, although it will not result in physical damage to a person’s inventory.
Inara Pey: Living in a Modemworld 