Tag: Server Updates

SL project updates week 49/1: server, viewer, issues

Posted on by Inara Pey

Pinoy Hideout, Lions Hill; Inara Pey, May 2014, on FlickrPinoy Hideout, Lions Hill (Flickr) – blog post

Server Deploys for Week 49

  • There has been no deployment to the Main channel, following-on from the lack of any deployments to the RCs in week #48.
  • On Wednesday, December 3rd, all three RC channels should receive the same server maintenance package, comprising the following updates:
    • A fix for BUG-7515 “Restarting region turns off ‘block parcel fly over’”
    • A fix for BUG-4949 (non-public) “Cannot manage block list with certain object names”
    • A fix for BUG-7850 “Experience tools: ‘Script trying to teleport other avatars!’ script error” appears incorrectly
    • A fix for BUG-6789 “Spelling mistake in llGodLikeRezObject”
    • Minor server change to help configure the texture and mesh CDN.

A question was asked at the simulator User Group meeting on Tuesday December 2nd, whether the fix for BUG-4949 would also address SVC-7550, relating to previously blocked avatars reverting to being blocked after someone has cleared them from their block list. Simon indicated that while BUG-4949 would not fix this issue, it is being addressed, but requires further back-end work.

SL Viewer

On Monday, December 1st, the de facto release viewer was updated to version 3.7.20.296724, formerly the Snowstorm RC viewer, which includes update for Japanese input; improved rendering of projectors; fixes for object editing bug when rotating and for crash on exit on OS X Yosemite in full screen mode, and more – release notes.

This viewer edged-out the current Maintenance RC viewer (version 3.7.21.296734) in terms of promotion. However, the latter has had a growing list of  issues, most of which are related to fitted mesh, and which may be addressed by the attachments update viewer (RC version 3.7.21.296904).

Experience Keys / Tools

There have been a lot of requests and ideas around expanding / improving the upcoming Experience Keys / Tools (for example: BUG-6912). Commenting on the general status of the project, Simon Linden indicated he’s doing some work related to attachments – although he wasn’t able to go into specifics. However, commenting on the ideas put forward a JIRAs submitted in general, he said:

To put things in perspective, this release is the first one for this feature.   I’d really like to see it succeed and have us move on to an update that adds more features like these. And while they aren’t going into this first release, the JIRA’s aren’t being ignored … we’ve gathered them and had to pick an choose what we could release with and without.

Oz also added, in response to, and alongside of, Simon’s comments:

Exactly. There will doubtless be many ways to improve and expand the experience concept. We want to get more applications built with it and see which are the best ways to invest in it further.

Other Items

3D Connexions Support

3D Connexion is the brand name for the space Navigator range by Logitech. As has been reported through this and other blogs, and also on the forums. More recent versions of the 3D Connexions drivers have been causing some problems on installation. The API support for these devices within the viewer is also quite old (although Linux has been recently updated), and Cinder Roxley is working to try to improve things in this area. However, updating to the newer API support may force things to be platform-dependent, and so take longer to implement, but it might result in the support of more 3D connexions devices.

Hair Base Cloud Issue

Some users are encountering issues of seeming themselves rendered as a cloud in their own world view. This may be connected to them where a hair base with a missing texture (essentially a bad asset), which causes the server to reject the hair base when worn. The result leaves the avatar as a cloud in the user’s own view, although to everyone else they appear to be rendered correctly. An error message is actually generated on the matter:

newview/llvoavatarself.cpp(2273) : 2014-12-01T15:32:45Z INFO: LLVOAvatarSelf::getIsCloud: Self is clouded because of no hair texture

However, this only goes to the viewer log, and is not made visible through the UI. As the viewer thinks you’re wearing the hair base, the problem is not obvious to many users. So, should you find this happening, try replacing it with a different hair base.

Current Outfit Folder Issues

When Server-side Appearance (SSA) was being deployed in 2013, some users on certain TPVs encountered issued with duplicate current outfit folders being generated – see SUN-99. A fix was made to prevent this from happening, however, there have been a couple of recent reports that it is occurring one again (see BUG-7880 and BUG-7920 – interestingly, the latter case appears to have had all system folders duplicated). The cause of these problems isn’t currently known, but requests to support for tun the SUN-99 inventory fix does appear to resolve problems, once the user has cleared cache locally and re-logged.

Teleport Queueing

A feature request (BUG-7945) for a teleport queueing feature to be implemented to ease access into very busy regions. As noted during the Simulator User Group meeting, such an automated queuing system as proposed might be labour-intensive to implement and also be open to exploitation. However, commenting on the idea of improving how teleport queues and notifications might be better handled, Simon Linden said:

I was talking with someone about the incoming TP queue yesterday. It seems like it needs some attention. I don’t think we’d build a queue and later TP anyone, but it’s doing too much work before it decides you can’t get in.

 Whether this will result in some clean-up to the current process of handling teleports into busy regions remains to be seen.

SL project updates week 48/1: server, viewer, Experience Keys, Cocoa bugs

Posted on by Inara Pey

It All Starts with a Smile, March 2014It All Starts with a Smile, March 2014 (Flickr) – Blog post

Server Deployments – Week 48

On Tuesday, November 25th, the Main channel received the server maintenance package previously deployed to the release candidate channels in week 46. The update comprises “minor improvements” to help configure the texture and mesh CDN, by allowing the Lab to reconfigure the CDN URL if they need to, with the intention of the of making it a more dynamic host name in the future.

There are no deployments to the release candidate channels this week, due to the Thanksgiving no change window opening, which runs from Wednesday, November 26th through until start of business on Monday December 1st.

Server Beta Meeting – Thursday, November 27th

Just a reminder, there is no Server Beta group meeting this week, due to Thanksgiving in the USA. Happy Thanksgiving, all of you in the States!

Viewer Updates

The Attachment RC viewer was updated on Tuesday, November 25th to version 3.7.21.296904. This release adds fixes for two additional problems compared to the November 12th release of the RC:

  • MAINT-4537 “Change in Maintenance Viewer breaks my joint rigged mesh avatar”
  • MAINT-4687  “Petite” avatars render deformed for the wearer but not for observer.

Saving and Loading Graphics Settings

The option to save certain graphics settings in the viewer (STORM-2082) is moving forward; there is currently an initial test viewer undergoing trials, but things are in a state of flux.

The idea behind this change is to provide a means by which users can quickly switch between two sets of graphics pre-sets they have created and saved locally, allowing them to quickly adjust the graphics setting to assist with performance as they move around the grid (so a user would have a set of “low” graphic settings they could switch-on in order to maintain performance in busy regions, and a set of “high” graphics settings, with as many bells and whistles turned on as they like, for use in quieter regions).

As noted in my week 46 report, the initial work saw a “Quick Preferences” floater added to the viewer, which allows users to set various settings and would likely include options to save said settings.

Initially, this was accessed via the Setup tab in the official viewer’s Preferences, but a suggestion has been made to keep everything accessible under the Graphics tab to prevent unnecessary fragmentation of options. A suggestion has also been made to change the name of the floater, as “Quick Preferences” is a term used by several TPVs, where it has a different context.

Experience Keys / Tools

Progress continues with the Experience Keys (Tools) project. The project viewer hasn’t been updated in a while, but work has apparently been going on elsewhere  – including a decision on what the charge will be for an Experience Key, although this has yet to be made public.

As a quick recap on this, and from my original overview on Experience Keys / Tools:

Every experience using the Experience Tools capabilities must be governed by an Experience Key supplied by the Lab – think of it as a licence applied to the experience and to all control scripts used within that experience, and which directly links the experience / experience scripts directly back to the experience owner, providing an audit trail of accountability.

Thus, the Experience Key allows the Lab to instantly revoke all permissions used by a given experience, stopping al the scripts associated with it, in necessary. This is intended to reduce the risk of people using Experiences as a means of griefing. As a further deterrent, the Lab will be charging some form of fee for the “licensing” of an Experience Key.

The cornfield, revamped in July, still provides a taster for SL Experiences - access it via the Portal Park
The cornfield, revamped in July, still provides a taster for SL Experiences – access it via the Portal Park

Apparently, Experience Keys / Tools were supposed to be released this quarter. However, with the Christmas season fast approaching, it seems questionable as to whether this will be achieved. As Oz Linden said in the Simulator User Group meeting on Tuesday, November 25th, “wish me luck”!

As previously noted, the initial release of Experience Keys / Tools will not support grid-wide experiences – although this is still on the Lab’s list for future enhancements to the platform.

Other Items

Cocoa Bugs

Many Mac users are still experiencing Cocoa-related issues, finding them something of a bone of contention as the Lab doesn’t regard some of the issues being experienced (such as excessive typing lag) as specific to the viewer, but rather endemic to the OS X operating system. All told, here is quite a wide range of issues, and TPV developer Cinder Roxley is attempting to resolve a number of them.

Part of the problem lies in the way the cursor position on the screen is translated to the cursor position in-world, which is in turn very screen resolution specific. This makes bug testing / fixing particularly hard: as fixes need to be tested against multiple monitor types. Cinder has been in touch with Apple engineers and has found one of their suggested solutions –  removal of deprecated calls – hasn’t helped in resolving problems, so fixes may yet be a while off.

Lab blogs about the Nov 17th-21st region restarts

Posted on by Inara Pey

secondlifeUpdate: At the time this article went to press, it appeared the daily restarts were still in progress (hence the reference to the restarts being Nov 17th-21st). Subsequent to this article appearing, the Lab updated the Grid Status report to indicate the work has actually bee completed, therefore the Lab’s blog post did in fact mark the end of the work.

The week of November 17th – 21st 2014 has been marked with daily periods of region restarts. Notice that these would be going on was first posted via a Grid Status update on Friday, November 14th.

As I noted in the first of my SL project updates for the week, Simon Linden indicated that restarts and the attendant maintenance was hardware-related, requiring servers to be taken down and physically opened-up, although precise details on what was being done was still scant.

In a blog post published on Thursday, November 20th, the Lab provided a detailed explanation on the reasons for the restarts, which reads in full:

Keeping the systems running the Second Life infrastructure operating smoothly is no mean feat. Our monitoring infrastructure keeps an eye on our machines every second, and a team of people work around the clock to ensure that Second Life runs smoothly. We do our best to replace failing systems pro actively and invisibly to Residents. Unfortunately, sometimes unexpected problems arise.

In late July, a hardware failure took down four of our latest-generation of simulator hosts. Initially, this was attributed to be a random failure, and the machine was sent off to our vendor for repair. In early October, a second failure took down another four machines. Two weeks later, another failure on another four hosts.

Each host lives inside a chassis along with three other hosts. These four hosts all share a common backplane that provides the hosts with power, networking and storage. The failures were traced to an overheating and subsequent failure of a component on these backplanes.

After exhaustive investigation with our vendor, the root cause of the failures turned out to be a hardware defect in a backplane component. We arranged an on-site visit by our vendor to locate, identify, and replace the affected backplanes. Members of our operations team have been working this week with our vendor in our data centre to inspect every potentially affected system and replace the defective component to prevent any more failures.

The region restarts that some of you have experienced this week were an unfortunate side-effect of this critical maintenance work. We have done our best to keep these restarts to a minimum as we understand just how disruptive a region restart can be. The affected machines have been repaired, and returned to service and we are confident that no more failures of this type will occur in the future. Thank you all for your patience and understanding as we have proceeded through the extended maintenance window this week.

Once again, it’s good to see that Landon Linden and his team are keeping the channels of communication open, and working to keep users appraised of what’s happening whenever and wherever is necessary / they can.

SL project updates week 47/1: server, viewer, RenderAutoMute functions

Posted on by Inara Pey

Nordan om Jorden; Inara Pey, November 2014, on FlickrNordan om Jorden (Flickr) – blog post

My apologies for the lateness of this update; have been busy with a variety of things, both SL and non-SL.

Server News – Week 47

There are no server deployments during the week, due to the hardware inspections taking place, which involved restarts taking place from Monday, November 17th through Friday November 21st, as detailed in the Grid Status updates (as a reminder, there is an additional period of maintenance due on Wednesday, November 18th, commencing at 13:00 SLT).

According to Simon Linden, speaking at the Simulator User Group on Tuesday, November 18th, the work on the servers requires each box being taken down, opened-up and physically inspected, and parts (unspecified) possibly being swapped-out.

Exactly what amount of work (if anything) is required on each server may vary, making the process something of a piece of string when it comes to how long it will take per box.

What prompted the work isn’t clear, but there was muted speculation that some servers may need a physical update of some description to avoid the potential of failing due to a defect. Whether this means one of the Lab’s suppliers altered them to a problem or not or something else has come up, isn’t clear.

However, none of this work should, outside of the rolling restarts, affect the performance of any regions.

SL Viewer

HTTP Pipelining Viewer

A new HTTP Pipelining RC viewer appeared on Monday, November 17th. Version 3.7.21.296736, see a “reduced pipelined texture and mesh fetching time-out so that stalled connections fail quickly allowing earlier retry. Time-out value changed from 150 seconds to 60 seconds.”

It is hoped that this viewer fixes the following issues:

  • BUG-7686 – “Avatars are not coming on viewer”
  • BUG-7687 – “Nothing is rezzing in SL,, av’s are all gray and textures will not rez”
  • BUG-7688 – “Since the last restarts I cant seen to see things I rez from my inventory or wear mesh in my inventory. I have done numerous clean installs of the latest LL viewer. I have also made sure I am not running the beta version of the AMD CCC”
  • BUG-7690 – “Textures and Meshes abruptly stopped rendering”
  • BUG-7691 – “Won’t rez properly”
  • BUG-7694 – “Textures and meshes loading slow or refusing to load”
  • BUG-7698 – “Textures much slower to load on a CDN region then on a clone of the same region not running CDN”

See the release notes for further details.

Maintenance RC Viewer

There are reports that the Maintenance RC viewer currently in the viewer release pipeline (version 3.7.21.296734) contains a number of regressions for joint position bugs. These issues are apparently known by the Lab, which hopes to have them corrected before the code merges with anything else.

Group Chat

There have been various reports of further group chat issues doing the rounds. On Monday, November 17th, for example, it was noted through Firestorm Support chat that at least two group chat server were down. Asked about this during the Simulator User Group meeting, Simon Linden replied:

Yes some of the chat servers have been having troubles in the last few days.   I’ve been looking into that … the code running there isn’t super new, and the outages might be timed with some of those restarts. In any case, there is an update soon for the chat servers, and already another in the pipeline.

Experience Keys

No major news here, other than those trying the Experience Keys in the current beta are being urged to file any additional issues they may have found as BUG reports as soon as they can. Simon Linden added to the request, “we’re trying to finish off the last few issues and have that Real Soon Now (sometime in the future, no promises).”

As previously indicated in my coverage of Experience Keys, the first release of the capabilities will not allow for grid-wide experiences, although this is something that is on the Lab’s list. Commenting on the plans, Oz Linden said:

The first general release of experiences won’t include being able to get a grid-wide key. It’s not so much that as there are more issues for us to deal with for grid-wide experiences, and we don’t want to make people wait for the ones …. Being able to do grid-wide experiences isn’t going to fall of the to-do list or anything.

As Oz was speaking, Simon Linden added:

We’ve talked about it before, and having a widely available data system like the key-value store would be really great, but there are a ton of issues with that being available, scaling it to cover the full SL population and all that. So we’re going ahead with a more feasible sized feature set.

RenderAutoMute Functions (Avatar Complexity)

One of the heaviest impacts on viewer performance comes not from issues with the SL servers or in rendering the contents of the the region you’re in, but from avatars themselves, particularly in crowded or busy regions. The Avatar Imposters option within the view can help with this, however, the Lab is looking to bring a debug setting to the fore within the viewer’s UI to further help users control their viewer’s performance.

The setting in question is RENDERAUTOMUTERENDERWEIGHTLIMIT, which is somewhat tied to the Avatar Render Weight (once aka Avatar Render Cost), the colour-coded render value assigned to avatars which can be displayed over their heads via the Advanced menu (CTRL-ALT-D to enable if not visible): Advanced > Performance Tools > Show Draw Weight for Avatars.

Essentially, the idea is that by entering a value against this setting, you can define a limit above which the viewer will cease rendering avatars fully, and instead will render them as a sold colour imposter, regardless as to how near / far they are from your point-of-view, reducing the rendering load on the viewer / your computer.

Currently, you can use the RENDERAUUTOMUTERENDERWEIGHTLIMIT option within the viewer to set a limit on rendering high-ARW avatars as solid colours in your viewer. You'll need to have RENDERAUTOMUTEFUNCTIONS set to 7 for it to work smoothly, and should note
Currently, you can use the RENDERAUUTOMUTERENDERWEIGHTLIMIT option within the viewer to set a limit on rendering high-ARW avatars as solid colours in your viewer. Note how the debug setting doesn’t correlate with the ARW for the avatar – something that will be fixed when the setting becomes a UI option (which will also see the dependency on setting RENDERAUTOMUTEFUNCTIONS removed – see the notes below)

I used the term “somewhat tied” above, because there is currently no obvious correlation between a number set within the debug setting and Avatar Render Weight, which is a figure that is in turn impacted. A further problem with the setting as it currently stands is that it is actually calculated by multiplying the number you enter against RENDERAUTOMUTERENDERWEIGHTLIMIT by a certain LOD (level of detail) factor (so if you set RENDERAUTOMUTERENDERWEIGHTLIMIT to 60,000, the actual figure being used might be 92,000 – 60K x the LOD factor).

Both of these points of confusion will be addressed by the Lab in making the option directly available through the viewer UI, so that there is a much clearer and obvious correlation between the setting and ARW.  Oz Linden is also working on colour-coding the resultant solid avatars so that it is possible to determine those avatars which are just over any limit set in the viewer and those which are well over the limit, allowing users to further fine-tune their settings according to needs / circumstance.

The two debug settings: you'll need to set RENDERAUTOMUTEFUNCTIONS to 7, and then experiment with RENDERAUTOMUTERENDERWEIGHTLIMIT
The two debug settings: you’ll need to set RENDERAUTOMUTEFUNCTIONS to 7 in order to experiment with RENDERAUTOMUTERENDERWEIGHTLIMIT

The option can actually be experimented with at the moment, although it currently has a dependency on another debug setting – RENDERAUTOMUTEFUNCTIONS - which must be set to 7 in order for any of the RenderAutoMute functions (5 in all) to work. Again, the Lab indicate that this dependency will be removed when RENDERAUTOMUTERENDERWEIGHTLIMIT becomes a UI option.

Again, the emphasis is on “experiment”, simply because of the lack of a direct correlation between values entered into the debug setting and the ARW values of surrounding avatars.  However, if you do want to have a play with the setting as it is at the moment, Oz Linden suggests starting with a value of around 60,000 and working up or down from there, depending on your needs / circumstances.

There’s no time frame as to when this work may find its way into a viewer, but Oz is actively working on it, following a prompt from third-party contributor, Jonathan Yap.

More server maintenance, Weds November 19th

Posted on by Inara Pey

In addition to the server maintenance taking place throughout this week, the Lab has  announced an additional period of maintenance which will be taking place on Wednesday, November 19th. The Grid Status post announcing the work reads in full:

Scheduled Server Maintenance

[Posted 12:10 PM PDT, 18 November 2014] We will be performing additional scheduled maintenance on 19 November at approximately 1:00 AM PST. During this time, some residents may experience login issues and residents in-world may be logged off or experience degraded performance. Additionally, during this time Group Chat may fail. Please refrain from rezzing no-copy objects and making L$ transactions during this maintenance. Please check back here for updates.

 

Don’t forget: scheduled maintenance means daily restarts Nov 17th-21st, 2014

Posted on by Inara Pey

Don’t forget that the coming week (commencing Monday, November 17th, 2014) will see all regions on the main SL grid undergoing daily restarts – which may occur more than once for each region.

The restarts are a part of grid-wide maintenance which requires all of the sim host servers to be inspected, with work commencing at 07:00 SLT every morning from Monday through Friday.

The Grid Status announcement on the work reads in full:

Scheduled Maintenance

[Posted 10:00 AM PST, 14 November 2014] Beginning Monday, November 17, we will be undertaking a full week of scheduled maintenance.  This maintenance will take place at approximately 7:00 AM each day next week. Each region will be subject to one or more restarts, as its host takes its turn for inspection.  A five-minute warning will be broadcast for each restarting region. Additional warnings will be broadcast to regions where multiple restarts are necessary. Please be cautious about going AFK for extended periods in the middle of build sessions or handling no-copy objects, as you may miss the shutdown warnings. Please keep an eye on this blog for further updates.