Tag: SL Issues

SL project news 4 (2): server and materials processing

Posted on by Inara Pey

Server Deployments Week 4

The RC channel deployments went ahead as planned on Wednesday 23rd January. BlueSteel and LeTigre Received the Threaded Region Crossing Code, and Magnum the Interest List code.

The week 3 maint-server release had been the only contender for the Main channel on Tuesday 22nd January, but was not deployed due to the issue of places not indexing correctly in search. A fix was available for this issue, but was rolled to Magnum as a part of that deployment.

Magnum Issues

There have been widespread issues in regions on the Magnum channel since the deployment, some of which have been reported in the deployment discussion thread on the forum, others of which have been reported in groups such as the Firestorm Support group. These issues may be related to a high bandwidth issue leading to packet loss for people on slower connections. Symptoms include:

  • High packet loss / ping to the simulator
  • Avatars initially unable to move on arriving in a region
  • Avatars apparently unable to move on arrival, but then suddenly “warping” to new position if movement keys have been pressed
  • Avatars already in the region being visible to a newly arrived avatar, then vanishing the first time the avatar moves or rotates camera
  • Objects failing to rez or to rez correctly (including attachments on avatars)
  • Other issues as reported in the forum discussion thread, including notable issues being encountered with JVA bots.

At this time, the issues remain under investigation by the Lab.

Deployments for Week 5

There is currently no clear news on deployment plans for the week commencing Monday 28th January. However, if there is a Main Channel release, Maestro Linden is of the opinion that it will likely be the threaded region crossing code.

The maint-server release originally on Magnum in week 3 will be re-deployed on an RC (which one is TBA at this point), which includes a number of additional fixes, including the fix for the search indexing issue.

Materials Processing

There has been little direct news on the materials processing project from those working on it. However, at the Server Beta User Group meeting on the 24th January, Maestro Linden reiterated that the server-side code is very close to a Release Channel deployment and, “May even get a slot next week [week 5].”

Maestro also gave some information on new options for what the alpha channel means during rendering, with four settings available:

  1. None –  the alpha channel is ignored, rendering the face opaque, or
  2. Alpha blending – essentially the same as we currently have for any alpha texture, or
  3. A 1-bit alpha mask with each pixel either 100% transparent or 100% opaque, with a cutoff setting to determine where the threshold is (alpha masks should render faster than alpha blending, and eliminate issues with alpha layer sorting), or
  4. Emissive mask – so the alpha layer is interpreted as a per-pixel glow setting.

(Note that these options appear to suggest a surface cannot have both an alpha mask (or alpha blending) and an emissive mask; however, as Maestro Linden pointed out, there will still be options in the build floater to set glow or face transparency.)

Maestro additionally indicated that the remaining “mesh sandbox” regions on Aditi (DRTSIM-169 channel: Mesh Sandbox 3, Mesh Sandbox 6, Mesh Sandbox 22) have been updated with the server-side materials processing code, so that builders can test the new capabilities once a materials processing viewer is available.

Testing materials processing in SL. Note the texture of some surfaces and reflections on others (image courtesy of Geenz Spad) - click to enlarge
A “sneak peek” at materials processing in SL, provided by Geenz Spad. Note the texture of some surfaces and reflections on others – click to enlarge. (Note this image should not be taken as indicating materials processing is available for public use as yet – see below.)

Viewer Status

Even though the server-side code is available on selected Aditi regions and may be deployed to a Release Channel in week 5, it will not be usable until such time as a materials processing project viewer is made available. Currently, this is unlikely to be for another few weeks, as there is further work to be completed on the viewer itself, including UI updates (such as the new build floater), code for fetching the additional materials data and for rendering it.

Server Object Rezzing Code

As I outlined back in week 1,  Baker Linden is working on improving how objects with large file sizes (as opposed to them being physically large when rezzed in-world) are handled by the simulator software when being rezzed.

In the first part of this update, I reported that Baker hoped the code would be progressing through LL’s QA for deployment to an RC channel in the next few weeks. In the meantime, the code is now available on the DRTSIM-194 channel (e.g. CCMTEST26).

The key point about this work is that it is specifically aimed at preventing the simulator processes from choking and a region stalling when there are a number of large object files being read / parsed, not at actually “speeding up” the physical rezzing process. As such, it is unlikely that objects will appear any faster in people’s in-world view as a result of this work. However, what it does mean is that the simulator code will be better able to handle rezzing multiple “large file” objects without the attendant region lagging which can occur as a result of the simulator being unable to process messages from viewers and other simulators, etc.

Aditi Issues

Issues continue with Aditi, both in terms of people being able to log-in, and with inventory syncing problems. It is currently unclear as to what the status is regarding potential solutions which have been mentioned in the past – culling “stale” accounts from Aditi and obtaining additional hardware to handle the load. In the meantime, problems continue for some, and the advice remains to keep trying a password change and waiting 24-48 hours for it to fully propagate before attempting to log-in to Aditi once more.

Related Links

Note that all SLurls given in this report are for the Aditi (Beta) grid.

SL project news: week 2 (2): Server deployments and network issues

Posted on by Inara Pey

Update:  The Interest list release to Magnum has been postponed due to a last-minute bug being found. Magnum has instead received the same code  as deployed to Bluesteel and LeTigre, containing the threaded region crossing update.

Server Deployments for Week 2

Server deployment recommenced this week, with a main channel roll-out on Tuesday 8th January, which saw that channel get the same code as released to the RC channels just before the Christmas / New Year break and as reported here.

Wednesday January 9th will see two major deployments to the RC channels. LeTigre and BlueSteel should be receiving a package which includes threaded region crossing code while Magnum gets Andrew Linden’s work on Interest Lists.

The threaded region crossing code should help improve simulator performance when avatars are region crossing to and from a region. However, and as previously noted, while the code did give clear improvements when crossing between regions on foot, the results were less positive when using vehicles during a recent test. Whether this has been improved as a result of those tests remains to be seen.

Release notes are available for both for LeTigre and Bluesteel, and the forum discussion thread is available for feedback / questions.

Interest List Deployment

The Interest List work should reduce the bandwidth usage of viewers due to object updates, and should improve simulator performance, especially in sims with many connected avatars and which is busy in terms of object updates.

This is the first phase of work planned around interest lists and object caching as a part of the Shining project. Andrew’s work has primarily been focused on improving the manner in which object updates are handled, etc., in order to provide improved performance. However, even with this initial phase of the work, there should be some improvements in the actual order in which objects appear in world (those closer to your camera position appearing prior to items further away – although there is far more work to be done in this regards before the project is finished. 

As it stands, there are already some updates to the work in the offing, as Andrew indicated at the Simulator User Group meeting on the 8th January, saying, “I’ve got some interest list work that didn’t quite make it into the Magnum RC … this work is almost ready for testing, so I’ll be trying to get it into a maintenance branch or something for a follow-up release.”

Interest list updates should be deployed to the Magnum RC on Weds 9th Jan.
Interest list updates should be deployed to the Magnum RC on Weds 9th Jan.

The new interest list code will see the final removal of the “legacy cloud” layer (at around the 170-200m mark). While this has been disabled viewer-side since around the introduction of Viewer 2, the data relating to the legacy cloud layer has still been sent out by the servers, allowing viewers which still incorporate the necessary code to render the clouds in-world. However, the server-side code relating to legacy clouds has been removed from the interest list code, so viewers will no longer be able to render the clouds.

The Magnum release notes are available here.

Continue reading “SL project news: week 2 (2): Server deployments and network issues” →

SL project news: week 51

Posted on by Inara Pey

Server Deployments week 51

Tuesday 18th December saw the Main Release channel get the maint-server release as deployed to Magnum in week 50, as Maestro Linden indicated might be the case during the Server Beta meeting on the 13th December. There have been some reports of teleport issues following the deployment, but these are thought to be a network-side issue, rather than a problem with the server code itself.

Release notes.

All three RC channels should be receiving the same package on Wednesday 19th December, 2012 which is also the Magnum deployment from week 50, with a single additional bug fix.

Release notes (for Magnum, but applicable to all three RC channels)

Forum thread for the deployments.

This is the last set of server-side deployments for 2012.

A low-key Simulator User Group meeting on 18th December, the last for 2012
A low-key Simulator User Group meeting on 18th December, the last for 2012

SL Viewer Updates

The release version of the official SL viewer rolled to the 3.4.3 code on Tuesday 18th December, with the release of 3.4.3.268262. Among a range of updates, this release include the major GPU table updates seen in recent beta viewer releases, the additional graphics settings and the tiling fix for high-resolution snapshot (MAINT-638). This release also sees the return of JIRA numbers to the release notes, matching the recent beta viewer release notes update.

The new "intermediary" graphics settings intended to better represent the capabilities of different GPU classes
The new “intermediary” graphics settings intended to better represent the default SL capabilities of different GPU classes

The beta viewer should be moving to the 3.4.4 code with the next release, although it is not clear whether this will be before or after the holiday period.

Interest List Update

Andrew Linden has been tweaking and optimising his Interest list code. The first phase of the new interest list code is aimed at reducing the amount of information being sent to the viewer by the server. Speaking at the Simulator User Group meeting on the 18th December, Andrew commented, “Last week I managed to find an optimisation opportunity in my Interest List project, which shaves a few hundred usec off of some cases where there are a lot of Full Object Updates. So this would translate to better server FPS when in large groups of avatars or when in the presence of stuff that is generating lots of full updates.”

As I reported last time, object updates take three forms, Full, Terse and Delete. Full updates are generated when an object is seen for the first time, or which is undergoing a constant change in appearance / position. Terse updates are sent only when an object changes its appearance / position. Under the current Interest list system, both types of update are sent to the viewer, regardless as to where the viewer actually needs to process them. A primary part of Andrew’s work has been reworking the code so that updates – particularly terse updates – are only sent to the viewer when the object generating them is in the camera’s field-of view. This new optimisation work should further help reduce the number of updates being sent to the viewer regardless as to whether or not it has to process them.

Continue reading “SL project news: week 51” →

SL project news: week 50/1: Server, JIRA, mesh and Shining

Posted on by Inara Pey

Server Deployments

Due to the offline e-mail issue involving scripted objects, as reported in my last news update, there has been no Main Channel deployment this week. Two RC deployments are currently planned for Wednesday 12th December, however. These are:

  • BlueSteel and LeTigre: should receive the same maint-server project that rolled to Magnum in week 49, with bug fixes arising from that deployment. The release notes are available for review
  • Magnum should receive a superset of the changes scheduled for BlueSteel and LeTigre, which includes extra bug fixes, including stability improvements and a memory leak fix.  
    • The only new feature new to Magnum is an increase in the allowed animation asset size – the 60KB size limit on animation assets has been raised to 120KB. This change is to allow for longer and more complex animations to be made in the future, once an viewer-side update to allow 60-second animation loops has been implemented. Magnum’s release notes can be read here.
So be sure to read them :-) (with thanks to Whirly Fizzle for the link)
So be sure to read them 🙂 (with thanks to Whirly Fizzle for the link)

Update on Key Region Issues

Physics Memory / Region Performance

As reported last time, the physics memory issues affecting some regions, which I reported in week 47, had been tracked down by Simon Linden to a Havok issue related to navmesh rebakes. His fix for this problem cleared QA and forms a part of the RC deployments for the 12th December, together with a fix for a low-level threading problem within the simulator code which has also been causing region crashes.

Offline IMs from In-world Objects Failing to Forward to E-mail

This issue, linked to llInstantMessage(llGetOwner(), caused the RC deployments in week 49 to be rolled back on Thursday 6th December. A fix has been developed and tested and is included in all three RC deployments planned for Wednesday 12th December.

Code Freeze / No Change Windows

Again, to re-iterate from my last report, there will be no server-side code changes over the holiday period as follows:

  • Week 52  – commencing Monday December 24th
  • Week 1, 2013 – commencing Monday December 31st

Simon Linden still hoped that one of the code being deployed to the RC channels this week can be rolled to the Main Channel in week 51. There will likely be a further update on this following the Thursday Server Beta UG.

JIRA / Bug Tracker Update

Linden Lab are still mulling the September closure of the old public JIRA system. Since the initial shut-down, things have opened up a little. Additional JIRAs have been left open as read only beyond the initial triage, while others have been opened and have had their comments enabled in order to allow feedback – such as the CHUI JIRA, which is being very constructively used for comments and feedback and shows how, in an ideal world, the system might work.

Currently, it appears that “nothing definitive” has been decided on the change, although it has been under internal discussion.

Feedback from those in the two JIRA support groups (developers who have significantly contributed code and those who have in the past supplied significant support in handling JIRAs) has been interesting. It appears that the number of feared duplicates on issues has been a lot smaller than had been feared. The overall quality of input given using the new form also appears to have been significantly improved since it was introduced.

Continue reading “SL project news: week 50/1: Server, JIRA, mesh and Shining” →

SL project news week 49

Posted on by Inara Pey

SL Viewer Updates

Things have been relatively quiet viewer-wise with only two updates to the official viewer branches. The development viewer rolled to 3.4.4.267614 December 4th, while on December 5th, the beta viewer rolled to 3.4.3.267755. The latter included a good crop of updates, including a number of graphics and GPU support related changes, and the long-await snapshot tiling fix.

Rough the same image shot using the new beta viewer at the same resolution - no tiling line (click to enlarge)
The MAINT-628 fix in action: an image taken with the latest beta viewer running in deferred mode and at a resolution of 3500×2154 pixels, well above the 1400×900 of my monitor – and no tiling! There are limits to how well the fix works at ultra-high resolutions, as noted in the JIRA comments included in my report on the release.

Server Deployments

Following-on from last week’s RC deployment issues, there was no main channel deployment on Tuesday December 4th, although a number of regions were restarted during the course of the day.

Wednesday December 5th saw the same maintenance release rolled to all three RC channels. This comprised the release originally aimed at Magum in week 48 and which included all bugs fixes for the problems which required the roll-back on Thursday 29th November. Initial statistics for this update during the brief time it was available last week showed a clear improvement in stability, and this seems to have continued with this week’s release, although there has been one major issue come to light and is under investigation.

This relates to IM messages sent by scripted objects failing to trigger e-mails to the object’s owner if they are off-line. The problem appears to be related to the use of llInstantMessage(llGetOwner(), and appears to affect regions on all three RC channels, but not every case where llInstantMessage(llGetOwner() is used appears to be affected.

Currently, it is thought that a fix will be available for deployment during week 50, and should reach the RC channels om Wednesday December 11th.

Details of the week’s RC release can be found in the release notes and in the forum discussion thread (including some discussion on the current scripted object / e-mail issue).

Continue reading “SL project news week 49” →

SL project news: week 48/2: RC issues, region performance and Aditi issues

Posted on by Inara Pey

Server Deployments Week 48

After a smooth deployment to the main channel on Tuesday 27th November, things got a little unsettled on Wednesday 28th November with the deployments to the RC channels. As noted in part 1 of this report, these were supposed to comprise a maint-server release to BlueSteel and LeTigre, with the same package and a few extras going to Magnum.

The problems started during the actual deployment on Wednesday, wherein after successfully updating Magnum and BlueSteel, the deployment team started noticing issues unrelated to the deployment which caused Coyot Linden to call off the LeTigre roll-out until things were sorted. However, as Maestro Linden takes up the story:

Then there were reports in the forums about offline IM emails from objects being broken; if an object sent you an IM and you were offline, the offline email would contain all the usual details *except* for the message. This bug affected both BlueSteel and Magnum since they both shared the responsible change. Then, after digging into offline emails a bit more, we noticed that the ‘To’ field of offline emails would show the object owner’s name instead of the recipient’s name … which was a little confusing … Anyway, these bugs were kind of bad, but we weren’t sure that they were worth the trauma and downtime of an emergency rollback …But then this morning, we became aware of a 3rd bug, from support. It turned out that deeding parcels to groups was failing.

It was this third bug which was deemed sufficiently serious enough to warrant a roll-back of the RC deployments, which took place on Thursday 29th November, with the result that all four channels are now running on the same release – 12.11.09.266804. Kelly Linden has fixes for all three issues, but they are currently in testing, and the plan is to try again next week with the RC deployments.

Region Performance / Memory Issues

The physics memory issues which I reported in week 47, and then provided an update on in Part 1 of this report have received further attention from Linden Lab. The problem has been with some regions experiencing severe physics memory bloat within a short time of being restarted, with the result being that they rapidly reach a threshold of memory use (~230MB for homesteads, ~920MB for full regions) which prevents rezzing of any objects, in-world or attached.

In investigating the issue, Simon Linden located a source of memory leak related to the Havok system which may address the issue, and is hopeful he has found a fix. Commenting on the matter at the Server Beta meeting on 29th November, “I’m in the midst of hacking a special test mode for a region … I’m going to make it continuously re-bake the navmesh and terrain data. I’ll let that run overnight and see what happens … I’m not going to claim victory quite yet … this is like that point where you hit the zombie hard, but you have to see if it comes back again.”

Aditi Grid Log-in Issues

It had been hoped that the Aditi situation, wherein problems with inventory-related data is preventing people from being able to log-in to the beta grid – would be discussed at the Server Beta meeting, but these were somewhat side-lined by discussions on other projects, most notably Interest Lists (work on which prevented Andrew Linden from digging into the matter following the Simulator User Group meeting on Tuesday 27th November).

While Maestro was able to confirm that a member of Linden Labs is, “Working on a script to help the Aditi inventory situation”, little more information was provided at the meeting due to other ongoing conversations. Hopefully, this matter will be picked up next week.

PATHBUG-183

In working on Interest Lists, Andrew Linden had hoped he’d sorted a fix for PATHBUG-183, which relates to offscreen physical objects flying across your in-world view. The code for the fix has been available on Ahern, on the Aditi grid, but there are reports that while the problem may have been somewhat reduced in scale, it is still very apparent. Andrew hopes to look into this during week 49.

Region “Flicker” and  Content “Warping”

There has been some issues related to viewing neighbouring regions and crossing region boundaries. These issues take a number of forms, and are related to both server and viewer issues:

  • The entire neighbouring region “flickers” if you are near its edge
  • After crossing between regions, the content of the region you have just left seems to appear in the region you’ve just entered, usually somewhat warped / deformed
  • “Cachable” objects (e.g. unscripted and non-physical) objects vanish from your view of a region on leaving it
  • A region appears to “reset” (re-renders) itself shortly after leaving it
warp-4
Sometimes on crossing between regions, objects from the region you’ve just left seem to appear warped and deformed in the region you’ve just entered…
...only to disappear after a few seconds / as you approach them
…only to disappear after a few seconds / as you approach them

The first two of these issues are considered to be viewer-related bugs which are thought to have been resolved in code currently in the viewer development. The final item in the list is related to a server issue which Andrew Linden believes to have been fixed as a part of his work on Interest Lists. The “cacheable” objects problem was also considered to be a viewer-side bug which has also been addressed in viewer development, but in considering the problem at the Server Beta meeting, Andrew Linden thought there might actually be more than one bug responsible and indicated he would be looking into this some more.