Tag: Server Updates

SL project news week 51/2

Posted on by Inara Pey

Server Deployments

The RC channel deployment took place on Wednesday 19th December as planned, with no major hiccups or issues apparently being felt / reported. This release saw all three RC channels receive the same package, which is the Magnum deployment from week 50, with a single additional bug fix.

The release notes (for Magnum, but applicable to all three RC channels) can be found here, and the forum discussion thread is still open should anyone have comments (allowing to the start of the holiday period & delays in any replies from LL).

SL Viewer Update

The last beta release of 2012 arrived on Thursday December 20th with the release of 3.4.4.268479 – which as the name suggests, uses the 3.4.4 code from viewer-development. Chief among the updates in this release include:

  • Moving viewer crash logger and SLPlugin from the Carbon API to Mac OSX Cocoa for the Mac viewer
  • Fixes for fontconfig crash-on-start issues experienced with Linux
  • Revisions for pathfinding options, including additional of tear-off option for Pathfinding menu when opened from Build menu; introduction of Region Rebake option to Pathfinding menu
  • Fixes for 3.4.2.266708 release issue of flexiprims stopping flexing after teleports
  • Further fixes for llTargetOmega() issues

The full set of release notes can be found here.

Forthcoming Viewer Graphics Enhancements

Alongside of the upcoming materials processing capabilities of normal and specular maps, the SL viewer will be receiving further rendering improvements in 2013, comprising:

  • Gamma correction capabilities
  • Improvements to the “Shiny” capabilities in the viewer (Build floater > Texture tab >Shiny button) when running in deferred mode, to make shine appear more as it does in the real world, with the level of shine based on the amount of reflected light / angle from which a shiny object is viewed from relative to the light source
  • In-world scene reflections on shiny surfaces.
Gamma correction: from Exodus to the SL viewer in 2013
Gamma correction: from Exodus to the SL viewer in 2013

Gamma correction is a process by which lighting on a screen can be adjusted to make it look more natural to the human eye by adjusting the RGB channels to give a more natural-light contrast. It can also be used to produce interesting photographic effects. Exodus viewer already has gamma correction, which can be adjusted via the Visual Settings floater (above).

A simple, if extreme, example of gamma correction: both images were taken under the same lighting conditions
A simple, if extreme, example of gamma correction used for a photographic effect: both images were taken under the same deferred lighting conditions, the image of the left with “normal” (default) gamma correction in the viewer, the one on the right taken with all three channels increased in value to produce an image with a much deeper contrast

As with materials processing, all three of the new capabilities are being introduced to the SL viewer via the Exodus team, lead by Geenz Spad, who indicated that in the case of gamma correction, the new capabilities will be somewhat more capable as a process within the SL viewer than is currently the case with the Exodus viewer.

As with material processing itself, which is still progressing, as I reported last time, there are no time frames as to when each of these new capabilities will be available within the viewer.

Threaded Region Crossing Code Test

Wednesday 19th December saw a “pile-on” test take place on Aditi for the new threaded region crossing code. The test has been called by Caleb Linden with a view to trying-out region crossing under a range of tests, including avatars with heavy scripted loads, repeated crossings by vehicles, via teleporting, etc.

In all, six regions were / are available for tests: GC Test 9, 10, 15 and 16 forming a block of four running in “threaded” mode, and GC Test 2 and 8 running in “unthreaded” mode, but otherwise the same server code.

Turn-out for the tests wasn’t particularly great (around 15 people together in the test region at any one time), with a range of ground and air vehicles being used to try-out the crossings and compare them directly with crossing between the two unthreaded regions (and testing between threaded / unthreaded).

The overall consensus from those at the tests were that crossings between threaded regions were somewhat improved – but the overall improvements were not that spectacular. At times recovery from loss of control of a vehicle appeared to be faster – a matter of one or two seconds – but at other times, things seemed to be the same, with vehicles ploughing through the ground / flying into the air for 6-7 seconds at a time before recovering, giving pretty similar results to those encountered crossing between threaded / unthreaded and between unthreaded regions.

Crossing between threaded regions in a vehicle: some improvements, but not a lot
Crossing between threaded regions in a vehicle: some improvements, but not a lot

Crossing between regions on foot, even under an excessive HUD / attachment script load (373 scripts accounting for 2.87800ms cpu time) and with vehicles zinging back and forth, did appear to me to be somewhat improved, with little or no “rubber banding” or walking off into the sunset, but a recovery of control after just a few paces.

Post-test, Caleb has requested those participating in the test file JIRA on any specific issues they encountered, and include “Region Crossing” in the description / subject heading for the report.

With thanks to Darien Cauldwell for information on the viewer enhancements discussed at the Opensource Dev meeting.

SL project news: week 51

Posted on by Inara Pey

Server Deployments week 51

Tuesday 18th December saw the Main Release channel get the maint-server release as deployed to Magnum in week 50, as Maestro Linden indicated might be the case during the Server Beta meeting on the 13th December. There have been some reports of teleport issues following the deployment, but these are thought to be a network-side issue, rather than a problem with the server code itself.

Release notes.

All three RC channels should be receiving the same package on Wednesday 19th December, 2012 which is also the Magnum deployment from week 50, with a single additional bug fix.

Release notes (for Magnum, but applicable to all three RC channels)

Forum thread for the deployments.

This is the last set of server-side deployments for 2012.

A low-key Simulator User Group meeting on 18th December, the last for 2012
A low-key Simulator User Group meeting on 18th December, the last for 2012

SL Viewer Updates

The release version of the official SL viewer rolled to the 3.4.3 code on Tuesday 18th December, with the release of 3.4.3.268262. Among a range of updates, this release include the major GPU table updates seen in recent beta viewer releases, the additional graphics settings and the tiling fix for high-resolution snapshot (MAINT-638). This release also sees the return of JIRA numbers to the release notes, matching the recent beta viewer release notes update.

The new "intermediary" graphics settings intended to better represent the capabilities of different GPU classes
The new “intermediary” graphics settings intended to better represent the default SL capabilities of different GPU classes

The beta viewer should be moving to the 3.4.4 code with the next release, although it is not clear whether this will be before or after the holiday period.

Interest List Update

Andrew Linden has been tweaking and optimising his Interest list code. The first phase of the new interest list code is aimed at reducing the amount of information being sent to the viewer by the server. Speaking at the Simulator User Group meeting on the 18th December, Andrew commented, “Last week I managed to find an optimisation opportunity in my Interest List project, which shaves a few hundred usec off of some cases where there are a lot of Full Object Updates. So this would translate to better server FPS when in large groups of avatars or when in the presence of stuff that is generating lots of full updates.”

As I reported last time, object updates take three forms, Full, Terse and Delete. Full updates are generated when an object is seen for the first time, or which is undergoing a constant change in appearance / position. Terse updates are sent only when an object changes its appearance / position. Under the current Interest list system, both types of update are sent to the viewer, regardless as to where the viewer actually needs to process them. A primary part of Andrew’s work has been reworking the code so that updates – particularly terse updates – are only sent to the viewer when the object generating them is in the camera’s field-of view. This new optimisation work should further help reduce the number of updates being sent to the viewer regardless as to whether or not it has to process them.

Continue reading “SL project news: week 51” →

SL project news week 50/2: server and viewer

Posted on by Inara Pey

Server Deployments Update

The RC channel deployments went head as scheduled, and included the promised fix for offline IMs from scripted objects failing to reach people’s e-mail (BUG-1002). A further issue (BUG-1027) with group owners receiving garbled messages on ejecting group members was reported over the weekend of the 9/10th December, and this also received a fix which formed a part of the deployments.

As reported in part 1, Magnum received code to double the server-side memory allocation from 60KB to 120KB. Animations within SL have two core limits: loop time (30 seconds) and memory allocation (60KB). Apparently, some complex animations which ran within the 30-second time frame have been hitting the memory allocation limit. This initial change should help to ease that issue when encountered. A future change on the viewer-side of things should eventually increase the animation run-time as well, allowing for animations longer than 30 seconds to be uploaded and used.

Server Deployments Week 51

Given Magnum has all the same changes as both BlueSteel and Le Tigre, plus the additional stability improvements and memory leak fixes, it now looks likely that this code will be promoted to the Main Channel on week 51 (week commencing Monday 17th December).  No news was provided during the Server Beta meeting on any proposed RC releases during week 51.

Again, as a reminder, there will be no server code releases during weeks 52 (commencing Monday 24th December) and week 1, 2013 (commencing Monday 31st December). There will also be no further Server Beta meetings until Thursday January 3rd, 2013.

Viewer Updates

Linden Lab continue to work on the beta and development viewers in order to clear the backlog of releases resulting from the memory leak / crash issues. Currently, they are “almost” at the end of catching-up on the release schedule. Some of the focus at present is on the Mac side of things, with Oz Linden reporting that there should be “A bunch of changes for the Mac build and the Mac implementation coming into viewer development” over the next few days.

The beta viewer has seen a further 3.4.3 code release (3.4.3.268139 on December 14th). This should mark the last of the 3.4.3 code releases for the beta viewer prior to that code moving to the release viewer, possibly in week 51. After this, the beta viewer will move to the 3.4.4 code base, which will include the changes for the Mac side of things as well. This should then see the viewer branches all more-or-less back to a normal pace of development and update, with fresh releases on the order of every three weeks or so, including more HTTP service updates and improvements.

Tcmalloc has been set off to one side in order to clear the backlog, but “has not been forgotten”. Currently it is still enabled in the beta release, but appears to be disabled in upcoming viewer development versions.

Avatar Baking

The biggest news of the week came with the announcement that for Avatar Baking, the countdown has commenced. This is going to take the next few months to implement, and requires both changes on the server-side of things and significant changes to the viewer. An update had actually been promised at the Content Creation User Group on Monday 17th December, but given the large impact the changes have on viewers, Nyx Linden rightly announced the news relating to the project at the TPV Developer meeting on Friday 14th December.

Nyx linden discusses server-side baking at the TPV Developer meeting, Friday 14th December
Nyx linden discusses server-side baking at the TPV Developer meeting, Friday 14th December

Threaded Region Crossings

The work on multi-threaded region crossings is still with the LL QA team. In the meantime, further regions have been added to the simulator version (server code  DRTSIM-184) running the new code. Four of the latter are GC Test 9, GC Test 10, GC Test 15 and GC Test 16, which form a block of four regions which may assist with testing the capability (remember these SLurls are all to Aditi!). Caleb Linden has been testing the capability and reports that he has encountered some issues himself, with crashes during “automated horde testing” and with repeated crossing with heavy scripts. He’s interested in hearing constructive feedback from anyone willing to carry out informal tests on the code.

Related Links

SL project news: week 50/1: Server, JIRA, mesh and Shining

Posted on by Inara Pey

Server Deployments

Due to the offline e-mail issue involving scripted objects, as reported in my last news update, there has been no Main Channel deployment this week. Two RC deployments are currently planned for Wednesday 12th December, however. These are:

  • BlueSteel and LeTigre: should receive the same maint-server project that rolled to Magnum in week 49, with bug fixes arising from that deployment. The release notes are available for review
  • Magnum should receive a superset of the changes scheduled for BlueSteel and LeTigre, which includes extra bug fixes, including stability improvements and a memory leak fix.  
    • The only new feature new to Magnum is an increase in the allowed animation asset size – the 60KB size limit on animation assets has been raised to 120KB. This change is to allow for longer and more complex animations to be made in the future, once an viewer-side update to allow 60-second animation loops has been implemented. Magnum’s release notes can be read here.
So be sure to read them :-) (with thanks to Whirly Fizzle for the link)
So be sure to read them 🙂 (with thanks to Whirly Fizzle for the link)

Update on Key Region Issues

Physics Memory / Region Performance

As reported last time, the physics memory issues affecting some regions, which I reported in week 47, had been tracked down by Simon Linden to a Havok issue related to navmesh rebakes. His fix for this problem cleared QA and forms a part of the RC deployments for the 12th December, together with a fix for a low-level threading problem within the simulator code which has also been causing region crashes.

Offline IMs from In-world Objects Failing to Forward to E-mail

This issue, linked to llInstantMessage(llGetOwner(), caused the RC deployments in week 49 to be rolled back on Thursday 6th December. A fix has been developed and tested and is included in all three RC deployments planned for Wednesday 12th December.

Code Freeze / No Change Windows

Again, to re-iterate from my last report, there will be no server-side code changes over the holiday period as follows:

  • Week 52  – commencing Monday December 24th
  • Week 1, 2013 – commencing Monday December 31st

Simon Linden still hoped that one of the code being deployed to the RC channels this week can be rolled to the Main Channel in week 51. There will likely be a further update on this following the Thursday Server Beta UG.

JIRA / Bug Tracker Update

Linden Lab are still mulling the September closure of the old public JIRA system. Since the initial shut-down, things have opened up a little. Additional JIRAs have been left open as read only beyond the initial triage, while others have been opened and have had their comments enabled in order to allow feedback – such as the CHUI JIRA, which is being very constructively used for comments and feedback and shows how, in an ideal world, the system might work.

Currently, it appears that “nothing definitive” has been decided on the change, although it has been under internal discussion.

Feedback from those in the two JIRA support groups (developers who have significantly contributed code and those who have in the past supplied significant support in handling JIRAs) has been interesting. It appears that the number of feared duplicates on issues has been a lot smaller than had been feared. The overall quality of input given using the new form also appears to have been significantly improved since it was introduced.

Continue reading “SL project news: week 50/1: Server, JIRA, mesh and Shining” →

SL project news week 49

Posted on by Inara Pey

SL Viewer Updates

Things have been relatively quiet viewer-wise with only two updates to the official viewer branches. The development viewer rolled to 3.4.4.267614 December 4th, while on December 5th, the beta viewer rolled to 3.4.3.267755. The latter included a good crop of updates, including a number of graphics and GPU support related changes, and the long-await snapshot tiling fix.

Rough the same image shot using the new beta viewer at the same resolution - no tiling line (click to enlarge)
The MAINT-628 fix in action: an image taken with the latest beta viewer running in deferred mode and at a resolution of 3500×2154 pixels, well above the 1400×900 of my monitor – and no tiling! There are limits to how well the fix works at ultra-high resolutions, as noted in the JIRA comments included in my report on the release.

Server Deployments

Following-on from last week’s RC deployment issues, there was no main channel deployment on Tuesday December 4th, although a number of regions were restarted during the course of the day.

Wednesday December 5th saw the same maintenance release rolled to all three RC channels. This comprised the release originally aimed at Magum in week 48 and which included all bugs fixes for the problems which required the roll-back on Thursday 29th November. Initial statistics for this update during the brief time it was available last week showed a clear improvement in stability, and this seems to have continued with this week’s release, although there has been one major issue come to light and is under investigation.

This relates to IM messages sent by scripted objects failing to trigger e-mails to the object’s owner if they are off-line. The problem appears to be related to the use of llInstantMessage(llGetOwner(), and appears to affect regions on all three RC channels, but not every case where llInstantMessage(llGetOwner() is used appears to be affected.

Currently, it is thought that a fix will be available for deployment during week 50, and should reach the RC channels om Wednesday December 11th.

Details of the week’s RC release can be found in the release notes and in the forum discussion thread (including some discussion on the current scripted object / e-mail issue).

Continue reading “SL project news week 49” →

SL project news: week 48/2: RC issues, region performance and Aditi issues

Posted on by Inara Pey

Server Deployments Week 48

After a smooth deployment to the main channel on Tuesday 27th November, things got a little unsettled on Wednesday 28th November with the deployments to the RC channels. As noted in part 1 of this report, these were supposed to comprise a maint-server release to BlueSteel and LeTigre, with the same package and a few extras going to Magnum.

The problems started during the actual deployment on Wednesday, wherein after successfully updating Magnum and BlueSteel, the deployment team started noticing issues unrelated to the deployment which caused Coyot Linden to call off the LeTigre roll-out until things were sorted. However, as Maestro Linden takes up the story:

Then there were reports in the forums about offline IM emails from objects being broken; if an object sent you an IM and you were offline, the offline email would contain all the usual details *except* for the message. This bug affected both BlueSteel and Magnum since they both shared the responsible change. Then, after digging into offline emails a bit more, we noticed that the ‘To’ field of offline emails would show the object owner’s name instead of the recipient’s name … which was a little confusing … Anyway, these bugs were kind of bad, but we weren’t sure that they were worth the trauma and downtime of an emergency rollback …But then this morning, we became aware of a 3rd bug, from support. It turned out that deeding parcels to groups was failing.

It was this third bug which was deemed sufficiently serious enough to warrant a roll-back of the RC deployments, which took place on Thursday 29th November, with the result that all four channels are now running on the same release – 12.11.09.266804. Kelly Linden has fixes for all three issues, but they are currently in testing, and the plan is to try again next week with the RC deployments.

Region Performance / Memory Issues

The physics memory issues which I reported in week 47, and then provided an update on in Part 1 of this report have received further attention from Linden Lab. The problem has been with some regions experiencing severe physics memory bloat within a short time of being restarted, with the result being that they rapidly reach a threshold of memory use (~230MB for homesteads, ~920MB for full regions) which prevents rezzing of any objects, in-world or attached.

In investigating the issue, Simon Linden located a source of memory leak related to the Havok system which may address the issue, and is hopeful he has found a fix. Commenting on the matter at the Server Beta meeting on 29th November, “I’m in the midst of hacking a special test mode for a region … I’m going to make it continuously re-bake the navmesh and terrain data. I’ll let that run overnight and see what happens … I’m not going to claim victory quite yet … this is like that point where you hit the zombie hard, but you have to see if it comes back again.”

Aditi Grid Log-in Issues

It had been hoped that the Aditi situation, wherein problems with inventory-related data is preventing people from being able to log-in to the beta grid – would be discussed at the Server Beta meeting, but these were somewhat side-lined by discussions on other projects, most notably Interest Lists (work on which prevented Andrew Linden from digging into the matter following the Simulator User Group meeting on Tuesday 27th November).

While Maestro was able to confirm that a member of Linden Labs is, “Working on a script to help the Aditi inventory situation”, little more information was provided at the meeting due to other ongoing conversations. Hopefully, this matter will be picked up next week.

PATHBUG-183

In working on Interest Lists, Andrew Linden had hoped he’d sorted a fix for PATHBUG-183, which relates to offscreen physical objects flying across your in-world view. The code for the fix has been available on Ahern, on the Aditi grid, but there are reports that while the problem may have been somewhat reduced in scale, it is still very apparent. Andrew hopes to look into this during week 49.

Region “Flicker” and  Content “Warping”

There has been some issues related to viewing neighbouring regions and crossing region boundaries. These issues take a number of forms, and are related to both server and viewer issues:

  • The entire neighbouring region “flickers” if you are near its edge
  • After crossing between regions, the content of the region you have just left seems to appear in the region you’ve just entered, usually somewhat warped / deformed
  • “Cachable” objects (e.g. unscripted and non-physical) objects vanish from your view of a region on leaving it
  • A region appears to “reset” (re-renders) itself shortly after leaving it
warp-4
Sometimes on crossing between regions, objects from the region you’ve just left seem to appear warped and deformed in the region you’ve just entered…
...only to disappear after a few seconds / as you approach them
…only to disappear after a few seconds / as you approach them

The first two of these issues are considered to be viewer-related bugs which are thought to have been resolved in code currently in the viewer development. The final item in the list is related to a server issue which Andrew Linden believes to have been fixed as a part of his work on Interest Lists. The “cacheable” objects problem was also considered to be a viewer-side bug which has also been addressed in viewer development, but in considering the problem at the Server Beta meeting, Andrew Linden thought there might actually be more than one bug responsible and indicated he would be looking into this some more.