Tag: Aditi

SL project news: week 7 (2): Server-side Baking load test

Posted on by Inara Pey

Server Deployments

the server deployments planned for week 7 all went ahead as scheduled.These comprised:

  • Main channel: (Second Life Server / SLS): received the maint-server package focused on crashes fixes which was  deployed in week 6 to LeTigre – release notes
  • Bluesteel: retained the materials processing project code and received the same fixes and updates being deployed to the SLS channel (above) – release notes
  • Magnum: retained the interest list project code and received the same fixes and updates being deployed to the SLS channel (above) – release notes
  • LeTigre received a new maintenance server update to fix miscellaneous crash modes – release notes. This deployment also included the following:
    • An improvement to the rolling restart notifications so that they appear in an alert format (as with manual region restarts) rather than an easily missed notification. This change will only be apparent in restarts following the code deployment restart (as per JIRA SVC-7759)
    • A fix to an encroachment / return problem:  if you’re banned from the neighbour’s parcel, you couldn’t select  / return items that encroached on your parcel (see JIRA SVC-496)
    • Instant messages are now truncated to 1024 bytes to prevent certain types of delivery failure. Currently, the IM database supports larger messages than the delivery system can handle. This change will enforce a limit of 1024 bytes when processing messages coming into the database as well as those being sent out.

Feedback on all the deployments has so far been muted, with only a couple of issues having been reported via the forum thread,

There is no news on deployments for week 8 (commencing Monday, 18th February), as the meeting to determine upcoming deployments does not take place at the Lab until Friday of the current week.

Server-side Baking (SSB) Load Test

Serer-side baking - load test February
Serer-side baking – load test February 21st

On Thursday February 21st there will be a special load test for Server-side baking, and LL are looking for volunteers to help.

This will take place on the SSB test regions on the beta grid (Aditi), immediately following the Server Beta User Group meeting which take place at 15:00 SLT on Thursdays in  Morris, also on Aditi.

The aim is to place the SSB code under a stress test which is representative of how SSB will be used once it is deployed to the main grid – with people routinely changing outfits, updating their appearance (as SSB handles appearance updates differently to the current service), enter / leave regions running the SSB code (given that the grid will, for a time, be running both the current avatar baking service and SSB as the latter is initially deployed), and so on.

“We have a few other internal stress tests, but wanted to do one with real-world conditions on real connections,” Nyx linden explained when announcing the test opportunity. “The test will run through switching from the old system to the new system, which is a transition where issues may pop up. if enough people are changing outfits simultaneously it should get us some valid data.”

Test Requirements

While final details of the test have yet to be confirmed, key requirements for those wishing to participate in the test are as follows:

  • Participants must be able to log-in to Aditi and attend the Sunshine test regions from 16:00 SLT onwards (participants can attend the Server Beta UG meeting ahead of the test if they wish)
  • Participants must be running the latest version Server-side Baking project viewer (version 3.4.5.270409 or later) – this viewer has been specifically configured to report statistics required by LL for the test
  • Participants should have a number of outfits of system clothing, preferably with multiple layers, which they can swap between during the course of the test. Library outfits are acceptable, but LL are keen for people to use their own outfits to add greater weight to the tests
  • Clearing the viewer cache prior to the test is suggested, but not an absolute requirement.

“if you have specific failures we’ll ask for your viewer logs, otherwise just running through the test will help us gather data,” Nyx added when explaining what is required by way of feedback from those opting to take part.

Aditi Log-in

As has been reported in this blog on a number of occasions, the Aditi grid is itself subject to a number of issues, both in terms of access and inventory support.

If you have not logged-in to the beta grid at all, or in the last several months and wish to participate in the SSB tests, it is recommended that you test your Aditi log-in (you use the same user name and password as you use to access the main (Agni) grid) sooner rather than later. If you find you are unable to log-in, then you should try changing your password. to refresh your Aditi access (this should also update your beta grid inventory).

HOWEVER, be warned that this process isn’t always successful, due to the issues mentioned above, and that it can take up to 48 hours before you can access Aditi, and even then, your inventory may not be successfully updated.

If you have recently updated your password and have reasonable inventory access on Aditi, the recommendation is that you don’t force any inventory update by running a further password change, as there is a risk you may either lose access to Aditi or that you may become subject to inventory change failures.

Those who wish to participate in the load test, and who encounter either issues with their Aditi inventory or accessing the beta grid can, as a last resort, contact Nyx Linden at least 24 hours ahead of the test. Nyx will then endeavour to see if LL can fix matters.

Other Items

Vanishing Regions

Last updated in SL project news: week 6 (2), wherein diagonally adjacent regions fail to render until such time as the observer moves to a region immediately bordering the “missing” region. This has been an ongoing problem for some time, as reported in SVC-8130, and commenting on it at the Server Beta meeting, Simon Linden indicated that the maint-server code deployed to LeTigre may help with some of the issues being encountered, but also admitted, “There’s been some improvement but it looks like there are still bugs to chase there.”

Missing regions: new Maint-server code may help...?
Missing regions: new Maint-server code may help…?

There is a forum post on the matter – if you are encountering this issue in a reproducible manner, and in lieu of SVC-8130 being open to comment, you might want to note your experiences on the thread.

Related Links

SL project news 4 (2): server and materials processing

Posted on by Inara Pey

Server Deployments Week 4

The RC channel deployments went ahead as planned on Wednesday 23rd January. BlueSteel and LeTigre Received the Threaded Region Crossing Code, and Magnum the Interest List code.

The week 3 maint-server release had been the only contender for the Main channel on Tuesday 22nd January, but was not deployed due to the issue of places not indexing correctly in search. A fix was available for this issue, but was rolled to Magnum as a part of that deployment.

Magnum Issues

There have been widespread issues in regions on the Magnum channel since the deployment, some of which have been reported in the deployment discussion thread on the forum, others of which have been reported in groups such as the Firestorm Support group. These issues may be related to a high bandwidth issue leading to packet loss for people on slower connections. Symptoms include:

  • High packet loss / ping to the simulator
  • Avatars initially unable to move on arriving in a region
  • Avatars apparently unable to move on arrival, but then suddenly “warping” to new position if movement keys have been pressed
  • Avatars already in the region being visible to a newly arrived avatar, then vanishing the first time the avatar moves or rotates camera
  • Objects failing to rez or to rez correctly (including attachments on avatars)
  • Other issues as reported in the forum discussion thread, including notable issues being encountered with JVA bots.

At this time, the issues remain under investigation by the Lab.

Deployments for Week 5

There is currently no clear news on deployment plans for the week commencing Monday 28th January. However, if there is a Main Channel release, Maestro Linden is of the opinion that it will likely be the threaded region crossing code.

The maint-server release originally on Magnum in week 3 will be re-deployed on an RC (which one is TBA at this point), which includes a number of additional fixes, including the fix for the search indexing issue.

Materials Processing

There has been little direct news on the materials processing project from those working on it. However, at the Server Beta User Group meeting on the 24th January, Maestro Linden reiterated that the server-side code is very close to a Release Channel deployment and, “May even get a slot next week [week 5].”

Maestro also gave some information on new options for what the alpha channel means during rendering, with four settings available:

  1. None –  the alpha channel is ignored, rendering the face opaque, or
  2. Alpha blending – essentially the same as we currently have for any alpha texture, or
  3. A 1-bit alpha mask with each pixel either 100% transparent or 100% opaque, with a cutoff setting to determine where the threshold is (alpha masks should render faster than alpha blending, and eliminate issues with alpha layer sorting), or
  4. Emissive mask – so the alpha layer is interpreted as a per-pixel glow setting.

(Note that these options appear to suggest a surface cannot have both an alpha mask (or alpha blending) and an emissive mask; however, as Maestro Linden pointed out, there will still be options in the build floater to set glow or face transparency.)

Maestro additionally indicated that the remaining “mesh sandbox” regions on Aditi (DRTSIM-169 channel: Mesh Sandbox 3, Mesh Sandbox 6, Mesh Sandbox 22) have been updated with the server-side materials processing code, so that builders can test the new capabilities once a materials processing viewer is available.

Testing materials processing in SL. Note the texture of some surfaces and reflections on others (image courtesy of Geenz Spad) - click to enlarge
A “sneak peek” at materials processing in SL, provided by Geenz Spad. Note the texture of some surfaces and reflections on others – click to enlarge. (Note this image should not be taken as indicating materials processing is available for public use as yet – see below.)

Viewer Status

Even though the server-side code is available on selected Aditi regions and may be deployed to a Release Channel in week 5, it will not be usable until such time as a materials processing project viewer is made available. Currently, this is unlikely to be for another few weeks, as there is further work to be completed on the viewer itself, including UI updates (such as the new build floater), code for fetching the additional materials data and for rendering it.

Server Object Rezzing Code

As I outlined back in week 1,  Baker Linden is working on improving how objects with large file sizes (as opposed to them being physically large when rezzed in-world) are handled by the simulator software when being rezzed.

In the first part of this update, I reported that Baker hoped the code would be progressing through LL’s QA for deployment to an RC channel in the next few weeks. In the meantime, the code is now available on the DRTSIM-194 channel (e.g. CCMTEST26).

The key point about this work is that it is specifically aimed at preventing the simulator processes from choking and a region stalling when there are a number of large object files being read / parsed, not at actually “speeding up” the physical rezzing process. As such, it is unlikely that objects will appear any faster in people’s in-world view as a result of this work. However, what it does mean is that the simulator code will be better able to handle rezzing multiple “large file” objects without the attendant region lagging which can occur as a result of the simulator being unable to process messages from viewers and other simulators, etc.

Aditi Issues

Issues continue with Aditi, both in terms of people being able to log-in, and with inventory syncing problems. It is currently unclear as to what the status is regarding potential solutions which have been mentioned in the past – culling “stale” accounts from Aditi and obtaining additional hardware to handle the load. In the meantime, problems continue for some, and the advice remains to keep trying a password change and waiting 24-48 hours for it to fully propagate before attempting to log-in to Aditi once more.

Related Links

Note that all SLurls given in this report are for the Aditi (Beta) grid.

SL project news: week 51

Posted on by Inara Pey

Server Deployments week 51

Tuesday 18th December saw the Main Release channel get the maint-server release as deployed to Magnum in week 50, as Maestro Linden indicated might be the case during the Server Beta meeting on the 13th December. There have been some reports of teleport issues following the deployment, but these are thought to be a network-side issue, rather than a problem with the server code itself.

Release notes.

All three RC channels should be receiving the same package on Wednesday 19th December, 2012 which is also the Magnum deployment from week 50, with a single additional bug fix.

Release notes (for Magnum, but applicable to all three RC channels)

Forum thread for the deployments.

This is the last set of server-side deployments for 2012.

A low-key Simulator User Group meeting on 18th December, the last for 2012
A low-key Simulator User Group meeting on 18th December, the last for 2012

SL Viewer Updates

The release version of the official SL viewer rolled to the 3.4.3 code on Tuesday 18th December, with the release of 3.4.3.268262. Among a range of updates, this release include the major GPU table updates seen in recent beta viewer releases, the additional graphics settings and the tiling fix for high-resolution snapshot (MAINT-638). This release also sees the return of JIRA numbers to the release notes, matching the recent beta viewer release notes update.

The new "intermediary" graphics settings intended to better represent the capabilities of different GPU classes
The new “intermediary” graphics settings intended to better represent the default SL capabilities of different GPU classes

The beta viewer should be moving to the 3.4.4 code with the next release, although it is not clear whether this will be before or after the holiday period.

Interest List Update

Andrew Linden has been tweaking and optimising his Interest list code. The first phase of the new interest list code is aimed at reducing the amount of information being sent to the viewer by the server. Speaking at the Simulator User Group meeting on the 18th December, Andrew commented, “Last week I managed to find an optimisation opportunity in my Interest List project, which shaves a few hundred usec off of some cases where there are a lot of Full Object Updates. So this would translate to better server FPS when in large groups of avatars or when in the presence of stuff that is generating lots of full updates.”

As I reported last time, object updates take three forms, Full, Terse and Delete. Full updates are generated when an object is seen for the first time, or which is undergoing a constant change in appearance / position. Terse updates are sent only when an object changes its appearance / position. Under the current Interest list system, both types of update are sent to the viewer, regardless as to where the viewer actually needs to process them. A primary part of Andrew’s work has been reworking the code so that updates – particularly terse updates – are only sent to the viewer when the object generating them is in the camera’s field-of view. This new optimisation work should further help reduce the number of updates being sent to the viewer regardless as to whether or not it has to process them.

Continue reading “SL project news: week 51” →

SL project news week 49

Posted on by Inara Pey

SL Viewer Updates

Things have been relatively quiet viewer-wise with only two updates to the official viewer branches. The development viewer rolled to 3.4.4.267614 December 4th, while on December 5th, the beta viewer rolled to 3.4.3.267755. The latter included a good crop of updates, including a number of graphics and GPU support related changes, and the long-await snapshot tiling fix.

Rough the same image shot using the new beta viewer at the same resolution - no tiling line (click to enlarge)
The MAINT-628 fix in action: an image taken with the latest beta viewer running in deferred mode and at a resolution of 3500×2154 pixels, well above the 1400×900 of my monitor – and no tiling! There are limits to how well the fix works at ultra-high resolutions, as noted in the JIRA comments included in my report on the release.

Server Deployments

Following-on from last week’s RC deployment issues, there was no main channel deployment on Tuesday December 4th, although a number of regions were restarted during the course of the day.

Wednesday December 5th saw the same maintenance release rolled to all three RC channels. This comprised the release originally aimed at Magum in week 48 and which included all bugs fixes for the problems which required the roll-back on Thursday 29th November. Initial statistics for this update during the brief time it was available last week showed a clear improvement in stability, and this seems to have continued with this week’s release, although there has been one major issue come to light and is under investigation.

This relates to IM messages sent by scripted objects failing to trigger e-mails to the object’s owner if they are off-line. The problem appears to be related to the use of llInstantMessage(llGetOwner(), and appears to affect regions on all three RC channels, but not every case where llInstantMessage(llGetOwner() is used appears to be affected.

Currently, it is thought that a fix will be available for deployment during week 50, and should reach the RC channels om Wednesday December 11th.

Details of the week’s RC release can be found in the release notes and in the forum discussion thread (including some discussion on the current scripted object / e-mail issue).

Continue reading “SL project news week 49” →

SL project news: week 48/2: RC issues, region performance and Aditi issues

Posted on by Inara Pey

Server Deployments Week 48

After a smooth deployment to the main channel on Tuesday 27th November, things got a little unsettled on Wednesday 28th November with the deployments to the RC channels. As noted in part 1 of this report, these were supposed to comprise a maint-server release to BlueSteel and LeTigre, with the same package and a few extras going to Magnum.

The problems started during the actual deployment on Wednesday, wherein after successfully updating Magnum and BlueSteel, the deployment team started noticing issues unrelated to the deployment which caused Coyot Linden to call off the LeTigre roll-out until things were sorted. However, as Maestro Linden takes up the story:

Then there were reports in the forums about offline IM emails from objects being broken; if an object sent you an IM and you were offline, the offline email would contain all the usual details *except* for the message. This bug affected both BlueSteel and Magnum since they both shared the responsible change. Then, after digging into offline emails a bit more, we noticed that the ‘To’ field of offline emails would show the object owner’s name instead of the recipient’s name … which was a little confusing … Anyway, these bugs were kind of bad, but we weren’t sure that they were worth the trauma and downtime of an emergency rollback …But then this morning, we became aware of a 3rd bug, from support. It turned out that deeding parcels to groups was failing.

It was this third bug which was deemed sufficiently serious enough to warrant a roll-back of the RC deployments, which took place on Thursday 29th November, with the result that all four channels are now running on the same release – 12.11.09.266804. Kelly Linden has fixes for all three issues, but they are currently in testing, and the plan is to try again next week with the RC deployments.

Region Performance / Memory Issues

The physics memory issues which I reported in week 47, and then provided an update on in Part 1 of this report have received further attention from Linden Lab. The problem has been with some regions experiencing severe physics memory bloat within a short time of being restarted, with the result being that they rapidly reach a threshold of memory use (~230MB for homesteads, ~920MB for full regions) which prevents rezzing of any objects, in-world or attached.

In investigating the issue, Simon Linden located a source of memory leak related to the Havok system which may address the issue, and is hopeful he has found a fix. Commenting on the matter at the Server Beta meeting on 29th November, “I’m in the midst of hacking a special test mode for a region … I’m going to make it continuously re-bake the navmesh and terrain data. I’ll let that run overnight and see what happens … I’m not going to claim victory quite yet … this is like that point where you hit the zombie hard, but you have to see if it comes back again.”

Aditi Grid Log-in Issues

It had been hoped that the Aditi situation, wherein problems with inventory-related data is preventing people from being able to log-in to the beta grid – would be discussed at the Server Beta meeting, but these were somewhat side-lined by discussions on other projects, most notably Interest Lists (work on which prevented Andrew Linden from digging into the matter following the Simulator User Group meeting on Tuesday 27th November).

While Maestro was able to confirm that a member of Linden Labs is, “Working on a script to help the Aditi inventory situation”, little more information was provided at the meeting due to other ongoing conversations. Hopefully, this matter will be picked up next week.

PATHBUG-183

In working on Interest Lists, Andrew Linden had hoped he’d sorted a fix for PATHBUG-183, which relates to offscreen physical objects flying across your in-world view. The code for the fix has been available on Ahern, on the Aditi grid, but there are reports that while the problem may have been somewhat reduced in scale, it is still very apparent. Andrew hopes to look into this during week 49.

Region “Flicker” and  Content “Warping”

There has been some issues related to viewing neighbouring regions and crossing region boundaries. These issues take a number of forms, and are related to both server and viewer issues:

  • The entire neighbouring region “flickers” if you are near its edge
  • After crossing between regions, the content of the region you have just left seems to appear in the region you’ve just entered, usually somewhat warped / deformed
  • “Cachable” objects (e.g. unscripted and non-physical) objects vanish from your view of a region on leaving it
  • A region appears to “reset” (re-renders) itself shortly after leaving it
warp-4
Sometimes on crossing between regions, objects from the region you’ve just left seem to appear warped and deformed in the region you’ve just entered…
...only to disappear after a few seconds / as you approach them
…only to disappear after a few seconds / as you approach them

The first two of these issues are considered to be viewer-related bugs which are thought to have been resolved in code currently in the viewer development. The final item in the list is related to a server issue which Andrew Linden believes to have been fixed as a part of his work on Interest Lists. The “cacheable” objects problem was also considered to be a viewer-side bug which has also been addressed in viewer development, but in considering the problem at the Server Beta meeting, Andrew Linden thought there might actually be more than one bug responsible and indicated he would be looking into this some more.

SL project news week 48/1: server and beta, viewer, maps and memory

Posted on by Inara Pey

Server Deployments

After indications from LL that there may not be a Main channel deployment on Tuesday 27th November, restart commenced as the deployment made to the RC channels last week went ahead as per the usual schedule.

Wednesday 28th November should see the three main RC channels updated as follows:

  • BlueSteel and LeTigre: should receive a maint-server project.  There are a few new flags for the LSL function llGetObjectDetails(), but the most important changes are some fixes for physics and mesh-based crash modes – see the server release notes
  • Magnum should receive the say package, with additional stability improvement changes – see the Magnum server release notes.

As usual there is a forum thread for the week’s server deployments.

Viewer News

Release Viewer

The 3.4.2 viewer code finally reached the release (production) version of the LL viewer with the release of 3.4.2.267137 on Monday 26th November, which I briefly reviewed here.

Beta Viewer

The beta viewer, now cleared of the crash issue bottleneck, moved rapidly through the 3.4.2 code base prior to Thanksgiving in the US, as previously reported in the news updates, and then reached 3.4.3 with the surprise release of 3.4.3.267135 during Thanksgiving week, after it had been indicated there would be no viewer releases during the week due to decreased support staff availability during the long weekend period. As reported last week, this release includes the first phase of Monty Linden’s HTTP texture fetch project, which should see people experiencing significantly faster texture rezzing when in-world.

CHUI Viewer

The CHUI – the Communications Hub User Interface – project viewer is due to go through another couple of iterations before moving towards a development / beta viewer code merge. There has already been one update since the project viewer, which is aimed at improving the capabilities and reliability of in-world text and Voice conversations, first appeared.

CHUI: potentially a couple more iterations to come

While he has not followed the project first-hand, Oz Linden believes CHUI to be nearing a “feature complete” status. The advice is that if you haven’t tried it out and wish to give feedback, now is the time to do so.

Mesh Deformer

Nalates Urriah provides an update on some of the ongoing work around the mesh deformer. In the meantime, speaking at the Open Development User Group meeting on Monday the 26th November, Oz linden responded to a question from White Rabbit as to what garments are still required for testing by saying, “That’s a great question. I’m setting up a meeting with the people responsible for avatars to try to get a proper acceptance test defined for both that and STORM-1800.” STORM-1800 relates to the vertex weights of the default avatar character mesh.

While Oz didn’t specify a date for the meeting, those with a direct interest in either supplying mesh clothing for testing or in the JIRA should be hearing from him in the near future on the meeting details.

Continue reading “SL project news week 48/1: server and beta, viewer, maps and memory” →