Tag: Server Updates

SL project news week 6 (2): server deployment updates

Posted on by Inara Pey

Server Deployments – Week 6

The planned server deployments for week 6 occurred as anticipated:

  • On Tuesday 5th February, the Main channel received the server maintenance project deployed to LeTigre in week 5. This has miscellaneous minor bug fixes and new features – release notes
  • On Wednesday 6th February, the RC channels received the following:
    • BlueSteel: code for materials processing (project viewer still pending) – release notes
    • LeTigre: a new maint-server project to fix miscellaneous crash modes, and with minor performance improvements – release notes
    • Magnum: interest list code update to specifically address the bot / bandwidth problem reported on in last week’s update and also support for materials processing – release notes

Server Deployments – week 7

There is no advanced news on potential deployment for the week commencing Monday 11th February, 2013.

SL Viewer Updates

The beta viewer was updated on February 6th with the release of 3.4.5.270034. Please refer to the release notes for details of all changes and updates. The CHUI project development viewer also updated to 3.4.6.270114 on February 6th.

Updates – Issues and Other Bits

Bot / Bandwidth Issues

Speaking at the Server Beta user group meeting on Thursday February 7th, Maestro Linden indicated the ongoing bot / bandwidth issue related to the interest list code and as pointed to by Latif Khalifa and confirmed by Andrew Linden (reported in more detail here), appears to have been resolved. Commenting on the bug fix in the server deployment thread, Triple Peccable, who was one of those being badly impacted by the problem, comments:

Maestro and Andrew,

I wanted to report on the bot’s usage. Fixed!

Before this incident the bot’s “normal” usage was 5 MB / hr. That is so normal no one would suspect anything.

But now it is 1 MB / hr! It has never been that low before, ever.

The improvement might be from the interest list changes, but since the bot is parked 3300m up with a very limited draw distance, I think it is from this UDP bug fix, and will help with more than just bots. :smileyhappy:.

Estate Ban Issues

Two issues have been reported in relation to estate bans recently.

One is the use of LSL commands for estate moderation, as mentioned in the second part of my report for week 5. While it is not clear how widespread the issue is (the reports received so far appear to relate to four regions), it had been hoped that the code deployment to LeTigre might have fixed the problem, but tests with an affected region move to LeTigre showed this was not the case. However, Maestro Linden believes LL may have a match between the issue and a bug that was filed internally after  crash report fingerprints were browsed, so investigations are liable to continue.

In the second, Whirly Fizzle has reported an issue with the “GTFO” ban feature in Phoenix. While this adds the banned individual’s name to the banlist for an estate, the individual isn’t actually barred from accessing the estate. As such, it is thought that this issue might contribute to recent problems in people apparently circumventing estate bans, and is something which will not be rectified by the estate ban improvements currently being deployed by LL, as it is an issue within the Phoenix viewer code itself.

Region Crashes on Restarts

In addition to the restart performance issues related to physics memory use previously reported and updated in part 1 of this report, some regions are experiencing issues with the physics engine during a restart, with all scripting capabilities being disabled as the physics engine is overloaded. Scripting must then be re-enabled by the region owner / estate managers. A fix for this is being worked on, and should be available soon.

Vanishing Regions

Following the week 5 deployments, Alvid Majestic contacted me concerning issues with regions diagonally opposite Brocade, on the Mainland, failing to render in the viewer’s world view, and would not render until such time as a person moved into one of the regions immediately adjacent to it / moved into it.

Missing regions: Mullein and (beyond it) Ear fail to render from Brocade, which sits diagonally opposite them
Missing regions: Mullein and (beyond it) Ear fail to render from Brocade, which sits diagonally opposite them

This is not a new issue, having previously been reported in SVC-8130, although there was some confusion as to whether or not it had been resolved. Commenting on it in general at the Server Beta User Group meeting, Maestro Linden informed me, “It’s somewhat rare, but it was never officially fixed.”  As the JIRA is closed to comment, Shug Maitland has raised a forum thread on matter, so if you are witnessing the same issue on an ongoing basis, consider adding your comments there as well as raising reports.

Region Crossings

There has been mixed feedback to the results of the deployment of the new region crossing code across Agni.

Regular commentator on this blog, Wolf Baginski Bearsfoot has put together a report on his findings in the SL Server sub-forum, which builds on his initial impressions posted in this blog.

Some feedback given through the User Groups suggest that in some instances region crossings – such as with sailing – are improved, and at the Simulator User Group meeting on Tuesday 5th February, Simon Linden indicated LL were seeing fewer instances of stuck teleports.However, there have also been reports passed through the Server Beta group of automated cars on the Mainland encountering problems at region crossings while following Linden Roads and piling-up at the boundaries of regions such as Furness to Ravenglass, although instances appear to have calmed down. More updates on this as they come.

SL project news week 6 (1): servers, viewer, materials

Posted on by Inara Pey

Deployments for Week 6

A full set of deployments on the channels this week.

On Tuesday 5th February, the Main channel received the server maintenance project deployed to LeTigre in week 5. This has miscellaneous minor bug fixes and new features – release notes.

On Wednesday 6th February, the Release candidate channels should receive the following:

  • BlueSteel should receive code for materials processing. There is still no project viewer publicly available for this project. When one becomes available, notices will be posted on the Project Viewer page, the Tools and Technology Blog, and STORM-1905 – release notes
  • LeTigre should receive a new maint-server project to fix miscellaneous crash modes, and which  offers minor performance improvements – release notes
  • Magnum should receive an update to the interest list code and support for materials processing. The interest list update should specifically address the bot / bandwidth problem reported on in last week’s update – release notes

As ever, a forum thread has been created for the discussion of the deployment packages, or any issues arising therefrom.

In late 2012 some regions (noticeably Homesteads) starting experiencing issues related to their physics memory. Investigations by Simon Linden revealed that part of the problem lay with these regions experiencing repeated navmesh rebakes, with each rebake consuming server memory with the result that multiple rebakes were leaving regions in need of a restart. Simon also confirmed that not all of the triggers generating a request appear to be linked to the actual need for a rebake (altering some estate / parcel settings can trigger a request, for example), and developed a fix for the issue. Simon believes this fix will be promoted to a Release Channel this week, although it is absent by name and description from the release notes at present.

Viewer News

Viewer releases are again unblocked, with further development viewer (incl. CHUI) made at the end of the week 5. Currently, CHUI looks to be the next project in line to be merged to the 3.4.5 code (with the project version already merged to the viewer-dev 3.4.6 code). This could make CHUI the first of the current projects affecting the viewer to reach a beta viewer release, but the timescales and order are far from definitive at this point – so it is possible CHUI may still be delayed in reaching the 3.4.5 beta code.

Materials Processing

Further to the week 4 update, it now appears scripting support may become available with  materials processing, although a) it will not be in the initial release; b) there appear to be considerable concerns on the Lab’s side of things as to the potential impact. Speaking at the Server Beta meeting on Thursday 31st January, Maestro Linden said the option for scripted control of normal and specular maps had been removed from the original proposal out of concern for it being exploited and used to thrash the server and rendering pipeline.

Speaking at the Content Creation User Group meeting on Monday 5th February, Geenz Spad, who co-authored the original proposal and who is working on the viewer side of the project, struck a more conciliatory tone. While confirming script support will not be available for normal and specular maps, he commented that this is in part because normal and specular maps don’t “plug into” existing means to manipulate diffuse (texture) maps using scripts. He went on, “I’m not saying no one would add scripting for the *new* parameters. Just that it won’t make it as part of the first release; think of it as a ‘we didn’t have time’ sort of thing.”  Whether or not support for scripted control of normal and specular maps remains to be seen, commenting on the matter, Nyx Linden said, “That would be a feature request to submit after the first release :),” – so it is likely the Lab will see what the demand is like prior to committing to anything, one way or the other, again allowing for the network / rendering concerns which have been voiced on their side.

In terms of animating normal and specular maps, Geenz confirmed that all current methods of animating textures will work with the additional maps, which I had more-or-less confirmed through my own rough tests, as reported in my sneak peek at the (then) latest version of the pre-release materials viewer.

Back in week 3, I discussed the fact that normal and specular maps require a viewer to be running in deferred mode (“Lighting and Shadows” in the Advanced options of the Graphic tab in v3-based viewers) in order for their effects to be seen, and gave a short overview how deferred can be used without actually having to have shadows enabled. This post was followed by a short discussion on possibly renaming the option to something more obvious to users.

Well, it appears that someone is a few steps ahead of things on this. In the most recent versions of the pre-release materials viewer, Lighting and Shadows is renamed to “Advance Lighting Model”.

Materials processing: the option formerly known as "Lighting and Shadows" - soon to appear in a project viewer Materials processing: the option formerly known as “Lighting and Shadows” – soon to appear in a project viewer

It’s still a little bit of a mouthful, but it may help when it comes to explaining how materials processing works. As it stands a project viewer for materials might be available by the end of week 6.

Other Items

What’s in a Name?

Those who make  full permission items intended for use by other creators as a part of their products can often face a frustrating problem: finding themselves in receipt of a call for assistance about the items in which their products have been used – as it is their name recorded in the Creator field, rather than the name of the person who made the item itself.

While this can be negated in some degree, results aren’t always perfect, and requires no small amount of fiddling around when it comes to full perm mesh items. This being the case, there was some discussion at the Content Creation user Group Meeting on Monday 5th February as to how the situation might be improved through the introduction of an additional field which could sit alongside the Creator and Owner fields and  which would identify the person who utilised a full permission mesh in their own work as well as the maker of the mesh itself – so that support questions could then be addressed to that person. One suggestion has even been to call the new field “Support”.

However, such a change could have wide-ranging impact, both in the viewer and server, making it a potentially complex matter to implement. During the Content Creation meeting on Tuesday February 4th, it was clear that there were several views on the subject of how to handle things, as well as some discussion on the complexities of actually implementing it.

Commenting on the matter, Nyx Linden requested that if a consensus view can be reached on the matter – or if people do feel it is a pressing matter which needs more consideration / discussion – that it should be raised as a feature request on the JIRA (i.e. file a bug report, but put “Feature Request” in the title / subject), so that it is at least on the Linden radar.

SL project news week 5 (2): servers, issues, viewer, SSB, deformer

Posted on by Inara Pey

Server Deployments week 5

On Tuesday 29th January, the Main channel received the threaded region crossing code previously deployed to BlueSteel and LeTigre in week 4 – release notes.

On Wednesday 29th January, the following updates were deployed to the Release Candidate channels:

  • BlueSteel received the server-side materials processing code – release notes
  • LeTigre received the maint-server project originally deployed to Magnum in week 3. This deployment includes the bug fix for the places / search indexing issue which occurred following the original deployment of the code to Magnum – release notes
  • Magnum received a further update to the interest list code deployed in week 4, with fixes intended to deal with issues of high packet loss – release notes.

Reported Issues

Main Channel

A number of initial problems were reported  in the forum discussion thread, Several of which  appeared to be related to initial region caching / capabilities problems which lead to some strange initial experiences for users, ranging from teleport failures / potential viewer crashes on region crossings (physical or teleport) and  out-of-range controlled flight (i.e. avatars could continue to fly under full control when apparently at coordinates well outside the region (e.g. 320, 128, 75). These issues seem to cease once regions were correctly caching following a (protracted) restart period.

There were also some reports in the forum discussion thread that use of the estate tools to eject / teleport home / ban an avatar can lead to a region cross. However, as the reports appear to be the same / similar to previously filed SEC JIRA (SEC-1215 and SEC-1204, neither of which are open to public viewing), and may be an isolated series of incidents (others are not reporting the same problems, so they have not been repro’d outside of the affected region), there is a reluctance within LL to attribute them directly to the region crossing code.

Magnum RC (Interest List Code)

The Interest List Code problem relates to the use of bots. Originally, it had been hoped that one of the fixes deployed this week would correct the issue; however this has not entirely been the case.

Following further investigation by Latif Khalifa and Andrew Linden, an explanation of the problem is provided in the deployment forum thread. The issue has its routes in the fact that the Second Life UDP protocol requires an acknowledgement (or “ACK”) for all messages sent to the viewer. However, rather than sending a distinct “message received” back to the server for each packet received, certain types of bot append the acknowledgement to other outgoing messages. A bug in the interest list code means these appended ACKs aren’t read. This results in original data been regarded as a packet loss, so it is repeatedly re-sent to the viewer controlling the bot, leading to the huge increases of bandwidth which have been witnessed. Andrew is currently working on a fix for this issue.

Deployments for Week 6

There is currently no clear news on deployment plans for the week commencing Monday 4th February. Currently, it appears that the main-server release re-deployed to LeTigre will most likely be promoted to the Main channel.

SL Viewer Updates

Issues relating to the beta viewer 3.4.5 code have been resolved, and an updated beta viewer (3.4.5.269698), was released on Thursday January 31st – see the release notes for updates. Also released on the 31st was a new version of the CHUI development viewer, 2.4.6.269797. The SL development viewer rolled to 3.4.6.703 on Wednesday January 30th.

An interesting issue has been noted recently by those using multiple versions of the SL viewer. When swapping between more recent versions of the SL development viewer and older versions of the code base, toolbar buttons can vanish from the viewer’s UI. The exact cause is unclear, and the problem appears intermittent (I’ve personally only encountered it once jumping between project and release versions of the viewer).

Mesh Deformer Project Viewer

Sovereign Engineer assisted with code merges for the parametric deformer project viewer with the result that the code is now merged-up to the 3.4.4 codebase with the latest release of the mesh project viewer (3.4.4.269558) on January 29th. There are apparently no functional changes to the deformer itself.

The updated shape selection options for mesh clothing and human shapes in the mesh upload floater
The mesh deformer project viewer has now been merged-up to the 3.4.4 viewer codebase. There are apparently no updates to the deformer itself with this release

This removes one of the three bottlenecks to the project as noted in the first part of this week’s reports. However, the issue of internal resources at the Lab remains a problem, although Oz Linden is attempting to address this and there may be further news in week 6.

Continue reading “SL project news week 5 (2): servers, issues, viewer, SSB, deformer” →

SL project news: week 5 (1): servers, viewer, deformer

Posted on by Inara Pey

Server Deployments Week 5

There are a full set of deployments this week, Main channel and RCs.

On Tuesday 29th January, the Main channel received the threaded region crossing code deployed to BlueSteel and LeTigre in week 4. This project makes sim performance smoother when objects and avatars cross between regions. Please refer to the release notes for further information.

On Wednesday 29th January, the Release Candidate channels should receive the following deployments:

  • BlueSteel should receive the server-side materials processing code. This code will require a a project viewer in order to be used, As reported in week 4, there is no definitive timeframe as to when a viewer is liable to be available, but it would be reasonable to expect something in the next few weeks (possibly sooner) – release notes
  • LeTigre should receive the maint-server project originally deployed to Magnum in week 3. This deployment includes the bug fix for the places / search indexing issue which occurred following the original deployment of the code to Magnum – release notes
  • Magnum will receive a further update to the interest list code deployed in week 4. This update addresses high packet loss problems, leading to issues such as not seeing avatars or objects in-world, specifically:
    • A fix for bots which do not specify a valid draw distance in order to prevent them downloading huge amounts of unwanted data
    • A fix to keep ObjectUpdate packets within an ethernet MTU of 1500
    • Release notes.

Please check with the forum discussion thread for these deployments to keep abreast of updates / changes / issues.

Materials processing: server-side code to go to BluseSteel on Wednesday 30th January - but no project viewer yet
Materials processing: server-side code to go to BlueSteel on Wednesday 30th January – but no project viewer yet

SL Viewer News

Problems remain with the beta viewer code, but it is hoped a new beta will be appearing shortly. In te meantime, Kelly Linden has committed the viewer-side code for extending the maximum avatar animation length from 30 seconds to 60 seconds (MAINT-1492). This should be appearing in the 3.4.5 beta viewer, and should also be appearing in TPVs in the near future.

Mesh Deformer

Delays continue with the mesh deformer, which can be summarised as:

  • The Lab has limited internal resources for testing it – although attempts are being made to address this
  • There is concern that there is no clear way for a designer and a user to be sure that they are using the same avatar base model – and if they are not, then the deformer will very likely make things worse. While there are some provisions within the current deformer for the base model to be taken into consideration, there is apparently concern that it may not be sufficient
  • The project veiewer code is now significantly behind the current viewer release, although work is underway to merge it up (and I received a report that this has been done while preparing this update).

A further potential issue in progressing the deformer  – as Karl Stiefvater (Qarl Fizz), who wrote the original code points out – is there are a lot of expectations that the deformer will do a lot of things which were not in the original specification (or indeed, the code as originally written). Some of this scope creep has resulted in further updates to the code from other contributors. However, there appear to be concerns that if the Lab prematurely release if, there will be a risk that a) it breaks existing content, b) it is viewed as not addressing problems correctly.

This latter aspect was also commented upon by Oz Linden, when speaking Open-source Development meeting on Monday 28th January. He said, “We are often (and frequently with justification) accused of throwing things out that are not really finished enough… indeed, the lack of some solution for the problem the deformer is trying to solve is one of the more prominent recent examples, but I don’t want to make the situation even worse by putting out a ‘solution’ that turns out not to solve enough of the problem and/or creates bigger ones down the road.”

Avatar Baking (SSB)

A set of code updates for the viewer are in progress. However, an issue with the Mac build means they have not been pushed as an update to the project viewer as yet. It is hoped that the updates will reach all three versions of the viewer (Windows, Linux and Mac) once the Mac build problem has been resolved.

SL project news 4 (2): server and materials processing

Posted on by Inara Pey

Server Deployments Week 4

The RC channel deployments went ahead as planned on Wednesday 23rd January. BlueSteel and LeTigre Received the Threaded Region Crossing Code, and Magnum the Interest List code.

The week 3 maint-server release had been the only contender for the Main channel on Tuesday 22nd January, but was not deployed due to the issue of places not indexing correctly in search. A fix was available for this issue, but was rolled to Magnum as a part of that deployment.

Magnum Issues

There have been widespread issues in regions on the Magnum channel since the deployment, some of which have been reported in the deployment discussion thread on the forum, others of which have been reported in groups such as the Firestorm Support group. These issues may be related to a high bandwidth issue leading to packet loss for people on slower connections. Symptoms include:

  • High packet loss / ping to the simulator
  • Avatars initially unable to move on arriving in a region
  • Avatars apparently unable to move on arrival, but then suddenly “warping” to new position if movement keys have been pressed
  • Avatars already in the region being visible to a newly arrived avatar, then vanishing the first time the avatar moves or rotates camera
  • Objects failing to rez or to rez correctly (including attachments on avatars)
  • Other issues as reported in the forum discussion thread, including notable issues being encountered with JVA bots.

At this time, the issues remain under investigation by the Lab.

Deployments for Week 5

There is currently no clear news on deployment plans for the week commencing Monday 28th January. However, if there is a Main Channel release, Maestro Linden is of the opinion that it will likely be the threaded region crossing code.

The maint-server release originally on Magnum in week 3 will be re-deployed on an RC (which one is TBA at this point), which includes a number of additional fixes, including the fix for the search indexing issue.

Materials Processing

There has been little direct news on the materials processing project from those working on it. However, at the Server Beta User Group meeting on the 24th January, Maestro Linden reiterated that the server-side code is very close to a Release Channel deployment and, “May even get a slot next week [week 5].”

Maestro also gave some information on new options for what the alpha channel means during rendering, with four settings available:

  1. None –  the alpha channel is ignored, rendering the face opaque, or
  2. Alpha blending – essentially the same as we currently have for any alpha texture, or
  3. A 1-bit alpha mask with each pixel either 100% transparent or 100% opaque, with a cutoff setting to determine where the threshold is (alpha masks should render faster than alpha blending, and eliminate issues with alpha layer sorting), or
  4. Emissive mask – so the alpha layer is interpreted as a per-pixel glow setting.

(Note that these options appear to suggest a surface cannot have both an alpha mask (or alpha blending) and an emissive mask; however, as Maestro Linden pointed out, there will still be options in the build floater to set glow or face transparency.)

Maestro additionally indicated that the remaining “mesh sandbox” regions on Aditi (DRTSIM-169 channel: Mesh Sandbox 3, Mesh Sandbox 6, Mesh Sandbox 22) have been updated with the server-side materials processing code, so that builders can test the new capabilities once a materials processing viewer is available.

Testing materials processing in SL. Note the texture of some surfaces and reflections on others (image courtesy of Geenz Spad) - click to enlarge
A “sneak peek” at materials processing in SL, provided by Geenz Spad. Note the texture of some surfaces and reflections on others – click to enlarge. (Note this image should not be taken as indicating materials processing is available for public use as yet – see below.)

Viewer Status

Even though the server-side code is available on selected Aditi regions and may be deployed to a Release Channel in week 5, it will not be usable until such time as a materials processing project viewer is made available. Currently, this is unlikely to be for another few weeks, as there is further work to be completed on the viewer itself, including UI updates (such as the new build floater), code for fetching the additional materials data and for rendering it.

Server Object Rezzing Code

As I outlined back in week 1,  Baker Linden is working on improving how objects with large file sizes (as opposed to them being physically large when rezzed in-world) are handled by the simulator software when being rezzed.

In the first part of this update, I reported that Baker hoped the code would be progressing through LL’s QA for deployment to an RC channel in the next few weeks. In the meantime, the code is now available on the DRTSIM-194 channel (e.g. CCMTEST26).

The key point about this work is that it is specifically aimed at preventing the simulator processes from choking and a region stalling when there are a number of large object files being read / parsed, not at actually “speeding up” the physical rezzing process. As such, it is unlikely that objects will appear any faster in people’s in-world view as a result of this work. However, what it does mean is that the simulator code will be better able to handle rezzing multiple “large file” objects without the attendant region lagging which can occur as a result of the simulator being unable to process messages from viewers and other simulators, etc.

Aditi Issues

Issues continue with Aditi, both in terms of people being able to log-in, and with inventory syncing problems. It is currently unclear as to what the status is regarding potential solutions which have been mentioned in the past – culling “stale” accounts from Aditi and obtaining additional hardware to handle the load. In the meantime, problems continue for some, and the advice remains to keep trying a password change and waiting 24-48 hours for it to fully propagate before attempting to log-in to Aditi once more.

Related Links

Note that all SLurls given in this report are for the Aditi (Beta) grid.

SL project news: week 4 (1): server, permissions, terrain and teleporting

Posted on by Inara Pey

Server Deployments Week 4

There was no Main Channel deployment on Tuesday 22nd January due to the issues encountered in week 3, which saw the interest list code on BlueSteel and LeTigre rolled back due to stability issues, and the maint-server release on Magnum having problems of its own (including regions not showing up in search, etc.).

There will be RC deployments on Wednesday 23rd January, as follows:

  • BlueSteel and LeTigre should receive the region crossing improvement project.  This project makes sim performance smoother when objects and avatars cross between regions.  This is the same project which was on BlueSteel and LeTigre in week 2; the only change is a fix for a crasher – release notes (BlueSteel)
  • Magnum should receive the interest list improvement project.  This update should reduce the bandwidth usage of viewers due to object updates, and should improve simulator performance, especially in sims with many connected avatars.  This is the same project which was on BlueSteel and LeTigre briefly in week 3; the only change is a fix for two crash modes – release notes
3rd time lucky? Interest List updates due for deployment to Magnum on Wednesday 23rd January
3rd time lucky? Interest List updates due for deployment to Magnum on Wednesday 23rd January

According to Simon Linden, speaking at the Simulator User Group on Tuesday 22nd January, the issue(s) relating to Magnum regions / parcels failing to show up in search also has a fix, which is apparently included in code due to be deployed to those regions, although there is no comment on this in the release notes themselves at the time of writing.

SL Viewer

Work continues on the viewer development side of things with further updates an merges and new versions appearing almost every other day, but little is currently filtering through to the beta viewer code branch at present.

Similarly, the CHUI developement viewer is going through a rapid series of updates, reaching 3.4.4.269264 on Tuesday 22nd January, although updates from the last series of development releases have yet to reach the CHUI project viewer.

Work has been promised to update the Sunshine (avatar baking) viewer code, whether this has been done or not is unclear as there was no Content Creation meeting on Monday 21st January due to Martin Luther King’s birthday; however, the current release of the project viewer remains 3.4.4.268071 at the time of writing.

Merges are also still awaiting on the Mesh project viewer in order to bring that back into line with more recent viewer-dev code updates.

Continue reading “SL project news: week 4 (1): server, permissions, terrain and teleporting” →