SL projects updates 20/2: server, group bans

Server Deployments Week 20 – Recap

There was only one server channel deployment in week 20, which went to the Magnum RC.

This was a new sever maintenance project deployed on Wednesday May 14th, which included a bug fix for a networking-related issue that sometimes affects busy sims, which Maestro Linden described as, “busy sim hosts would suddenly run into a bunch of networking issues, where you’d see failures creating inventory, accessing capabilities, etc.” The problem can also apparently affect LSL email registration for receiving email from outside the region, causing it to break without automatic recovery.

Maestro indicated that the Lab has a hotfix for regions reporting the problems (although that number appears to be low) which involves a configuration change for the sim host, but the update deployed to Magnum (and which will obviously progress to the other sever channels) has this config change set by default.

SL Viewer

There have been no further updates to any of the SL viewers currently in the release channel or available as project viewers.  Releases are as per my Current Viewer Releases page.

Group Bans

The group ban work is inching closer to the main grid. However, it’ll still be another couple of weeks (ish) before anything visible is seen as far as this capability is concerned. As noted in week 19, the Lab will be deploying things cautiously, with an initial back-end host code update being undertaken first, prior to anything being seen on the simulator channels.

“We’re just running some final tests at this point; the server which runs the group ban service also provides some other services, and we want to check that those didn’t break,” Maestro Linden informed the Server Beta User Group on Thursday May 15th.

He went on, “After the back-end is out, we’ll want to give it a few days to verify that nothing broke because we don’t want to roll back the backend service after group ban is on server RC.”

This probably means it’ll be another couple of weeks before the server-side code appears on a server RC channel. The plan is that when that happens, a formal project viewer with the viewer-side group ban code will appear for public use.

When formal deployment to one more simulator RCs does commence, it is important to remember that until the server-code has been fully deployed across the main grid, the group ban functionality will only work as advertised on those servers / simulators / regions which have the necessary server-side code. It therefore may appear to give unpredictable results.

For example, you will only be able to effectively ban people from you group when the viewer is connected to a server supporting group bans (although they do not need to also be using the group ban viewer in order to be banned). Also, even when someone is banned from your group, they could successfully rejoin it from any simulator / region which does not have the server-side code deployed to it (leading to further confusion as they’ll appear in both your group members list and your group ban list).

Obviously, these issues will go away once the server-side code is fully deployed across the main grid. However, until such time as that has been achieved, people should be aware they may encounter what appear to be “issues” with the functionality simply because it isn’t available right across the grid.

More information will be posted on this when the project viewer surfaces and the code has been made available of a server-side RC.

SL projects updates 20/1: server, viewer, LSL and materials

Server Deployments, Week 20

There was no Main channel deployment or rolling restart on Tuesday May 13th, and neither the BlueSteel or the LeTigre RC channels will receive an update or should undergo a restart on Wednesday May 14th.

The Magnum RC should receive a new sever maintenance project on Wednesday May 14th, which includes a bug fix for a networking-related issue that sometimes affects busy sims.

SL Viewer Updates

The SL Maintenance viewer was updated on Monday May 12th to version 3.7.8.289922. This viewer includes multiple fixes to Mac viewer; fixes in Recent tab, Chat, LSL editor, land management, etc; GPU table updates; crash fixes & performance improvements.

LSL Functions for Materials

The subject of scripted control for materials was once again raised at the Simulator User Group meeting on Tuesday May 13th. Commenting on the matter, Simon Linden said:

I am looking at it but not promising anything. We’re trying to be really careful to understand how the server and viewers will react when stressed with a lot of material churn. From what I can tell, fast-moving material-based animation will not work well … that’s likely to be throttled or blocked somehow. But supporting something like a hud or other control that could adjust the look of an object … where it’s done rarely … is definitely possible.

As noted the last time this subject was raised, there are concerns over how LSL control of materials might impact system performance, either deliberately (via rapid and multiple flipping of maps, hence Simon’s comment on throttling the speed at which changes could be made), or unintentionally, such as using them with objects which may already have a large performance impact (such as animated mesh tails).

During the meeting, there was discussion on options for animating normal and diffuse maps, remembering that they can already be animated in lockstep with their attendant texture (diffuse) map. During this discussion, Simon commented on some of the difficulties in animating  materials independently of the texture map:

The materials LSL support would include changing the offset, repeat and rotation values for the two maps, just like for regular textures. The update problem hits if you look at the way materials have been optimised between the server and viewer and how updates are sent. Materials are referred to by a number ID … so you get updates that say “this face has material 1234” on it, the viewer, if it doesn’t know what 1234 is, has to ask the server.

Now, if you change the offset … you have a new material 34356, the viewer has to again find out what that is, but this time it already has the actual specular and normal maps, so no download there.  And when you switch back to 1234, it has all the info and can draw it faster.

Summing-up the situation in general, Simon concluded, “I hope there will be something to play with eventually on the beta grid … we’ll probably want to experiment there and figure out what kind of limits are effective.”

SL projects updates 19/2: group bans, miscellaneous items

Server Deployments Week 19 – Recap

There were no server deployments!

Group Chat

As noted in part one of this report, the group chat updates were deployed to the back-end chat servers on Monday May 5th. The changes to group chat should be subtle, and may not be observable to many. Additional analytics are included in the code, which should provide further pointers on what else may need addressing going forward.

Group Ban Lists

Obligatory Baker Linden shot :)
Obligatory Baker Linden shot 🙂

Baker Linden’s work on adding the ability to ban troublemakers / spammers, etc., from groups with open enrollment is now getting relatively close to becoming available.

Baker has recently closed what is believed to be the last of the server-side issues, BUG-5929. This meant that if the name of the group owner was accidentally added to a list of people to be banned from a group, the ban process would fail, with no-one in the list either being added to the ban list or banned from the group (although other than the group owner, anyone selected for banning would be ejected from the group).

The expected behaviour would be for all those named (other than the group owner) to be added to the ban list, with those who were already members of the group also being ejected and banned. Baker’s fix is to ensure this is now the case, and it should be available shortly on Aditi for testing (channel DRTSIM-234 14.05.05.289712 – which includes the Morris region where the Server Beta meeting is held).

Viewer-wise, a project viewer with the new code is expected to appear very shortly (it was running through the build process during the Server Beta meeting on Thursday May 8th). This should be added to the Alternative Viewers wiki page when available. The repository for the code has now been made public, so TPVs can start looking at it – but again, given the status of the viewer as a project release, don’t expect the code to immediately start popping-up in TPVs.

HOWEVER, it may be a while before the new group ban functionality can be used on the main grid, as there is an initial back-end host code update required prior to anything being deployed to any simulator channel. According to Maestro Linden, the Lab will likely want to run those updates for a week to check for any unexpected regressions prior to putting any simulators on a group ban RC.

In the meantime, the group ban capabilities can be tested on Aditi either using the project viewer (when available) or the existing test viewer.

Other Items

“Welcome to the Hotel California” – BUG-5961

Trying to leave a group with a large membership list can prove problematic if the memebrship list takes time to load
Trying to leave a group with a large membership list can prove problematic if the membership list takes time to load

An old issue recently came to light once more with BUG-5961 (originally entitled “I cannot leave a group that I joined”, but with the description subsequently updated by Maestro to “Viewer attempts full fetch of member list before allowing user to leave group” in order to better reflect his findings following investigation).

It’s not actually clear if this is a one-off situation, or possibly more widespread, as the bug report is specific to the group “Akeyo”.

However, Maestro’s thinking is that the problem is linked to the download of the membership list, which even with the Group Services fixes introduced in late 2012, can still take time to complete with some larger groups.

Essentially, you cannot leave a group until the membership list has been loaded, as the viewer must check to ensure that when leaving, you’re not the last owner of the group. Should the membership list take time to download, this can lead to a temptation to click the Leave button again, causing the download to start-over, resulting in the list not loading, thus preventing you  from leaving it (hence the Hotel California quip, which I admit I stole from Maestro!).

The Lab is looking into this issue further, although it may be a while before any resolution is found. One workaround in the meantime is to run a client such as Radegast, which handles groups slightly differently to the viewer, and use that to leave the offending group.

Restore to Last Position

Restore to Last position was a popular feature which allowed anyone to take content to inventory and then re-rez it later at the same position. While there were issues with the capability (such as using it to rez an object in a different region, with a different topology to the one where it was originally taken back to inventory resutling in an object to “vanish”, as it rezzed underground or something), it was broadly seen as beneficial.

However, it was also subject to exploitation, which is why the server-side behaviour for it was changed by the Lab some time ago such that the function will only work if you have rezzing rights at 0,0,0 in a region. If you do not, any attempt to use Restore to Last Position will fail with a notification that you don’t have the required rezzing permissions. The viewer-side code for the capability was also removed from the SL viewer, although TPVs have retained it.

A further issue with the capability has been with No Copy objects. If Restore to Last Position is used on these when the user doesn’t have rezzing rights at 0.0.0 in a region, they not only fail to rez – they also vanish from inventory, requiring a relog in order to get them listed again.

However, BUG-5955 “Restore to Last Position (used only by TPVs) causes content loss” highlights a problem where at least one type of No Copy object can be permanently lost from inventory if Restore to Last Position is used even in a region where the user has rezzing permissions at 0,0,0. Not even a subsequent re-log sees the item reappear in inventory.

Given the unpredictable nature of Restore to Last Position, the Lab is considering removing or blocking all support for it viewer-side until such time as a fix for issues can be found / it can be made to work more predictably in all cases.

As an alternative, and given the function’s popularity, it has been suggested a restriction preventing its use on No Copy objects should be implemented. The Lab may be taking this under consideration. This is the option Firestorm have indicated that they intend to implement with their upcoming release (which may as a result be delayed until the code is implemented and tested).

SL projects updates 19/1: SL viewer, group chat and miscellaneous things

Server Deployments

There are no scheduled simulator deployments this week to either the Main or RC channels, and so no associated rolling restarted expected.

SL Viewer

The Interest List RC finally made it to the de facto release viewer with its promotion on Tuesday May 5th (version 3.7.7.289461). This leaves just three RC viewer in the release channel at present: SL Share 2 project viewer version 3.7.7.289497; Sunshine / AIS v3 RC  version 3.7.7.289441; and the Maintenance RC viewer version 3.7.7.289405. Please refer to my Current Viewer Release page for up-to-date information on all viewer releases.

 Group Chat

Simon Linden’s optimisation work for group chat was deployed across all of the back-end chat servers on Monday May 5th. while these should see some improvements in group chat (particularly in sending / receiving chat and moving between regions), Simon does warn that these optimisations are not expected to “fix” all of group chat. However, he will continue to work on further improvements as well.

Other Items

New Starter Avatars

Ebbe Altberg used one of the upcoming new starter avatars at the VWBPE conference in April (image: Strawberry Singh)
Ebbe Altberg used one of the upcoming new starter avatars at the VWBPE conference in April (image: Strawberry Singh)

During his appearance at the VWBPE conference in mid-April, Ebbe Altberg appeared using one of the new starter avatars. At the time he did, it was hinted that the new avatars would be appearing relatively imminently. However, almost a month on and they have yet to officially appear, although there is some speculation they’ll do so in May.

these new avatars are said to take advantage of some of the latest features in SL, which is being taken to mean that some / all are full or partial mesh. This has in turn raised questions as to whether it is wise giving new starters full mesh avatars, given they may not work with freebie items often offered to or picked-up by new starters.

LSL Functions for Materials

While there is no confirmation any work is being carried out on this (except, as Simon quipped, “perhaps in a parallel universe or something”), the Lab is still sounding out how and where such calls would likely be used, and the frequency with which such calls would be made.

The option of having scripted control of materials has been debated often, and still remains a desired item among builders and scripters. However, some of the concerns still remain – notably, have such capabilities might end up causing performance issues, deliberately or otherwise. Much has already been written on how rapid map flipping on multiple objects could deliberately impact performance and potentially result in viewer crashes, plus there are already animated mesh elements available which can also have a significant impact on viewer performance (some types of animated mesh tail can reportedly overload a viewer on a 32-bit system with out-of-memory errors in a matter of seconds), so there are also concerns that were this to be combined with the ability to change textures via script, they could (even unintentionally) have further dramatic impacts on performance.

One way around this would be to throttle the rate at which material maps can be changed via scripted command. What is interesting for the moment is that the Lab appears to have not completely closed the door on scripted control of materials, but is considering options and informally seeking feedback on potential use cases.

 

SL projects updates 18/2: group chat; group bans

Server Deployments Week 18- Recap

  • On Tuesday April 29th, the Main channel received the server maintenance project that was on the Magnum RC in week 17, comprising a fix for BUG-5533 and a crash mode fix.
  • On Wednesday April 30th, the Magnum RC had the server-side Sunshine  / AIS v3 code re-enabled (this code requires the use of the Sunshine RC viewer), and all three RCs were updated with the bug fixes deployed to the Main channel.

SL Viewer

There have been no updates to the RC viewers in the release channel during week 18  and no further releases, either RC or project viewers, so the SL viewer releases remain as per the last update to my Current Viewer Releases page.

Group Chat Optimisation

Simon Linden dancing at a Server Beta User Group meeting
Simon Linden dancing at a Server Beta User Group meeting

The code Simon Linden has been working on to improve group chat was deployed to a single group chat server, where it has been running for all groups starting with group_id “b”. Commenting on the work at the Server Beta meeting on Thursday May 1st, Maestro Linden said:

Simon’s been looking at the performance of that group chat server, and it seems to be running fine. So there are plans to update the rest of the group chat servers to the new version early next week. We won’t go so far as to say that group chat has been totally fixed, though – Simon has identified some other changes which could improve performance further.

A recent fix was made to IM sessions to correct the issue where it is possible to see “typing…” in an IM window when the other person isn’t actually typing (see STORM-1975), and questions were raised on whether this fix might be adding a load to group chat sessions, as the viewer-side code appeared to send the message during group chat sessions as well as person-to-person IMs. However, both Simon and Maestro Linden indicated that the notifications are simply ignored by the chat servers during a group chat session, so no additional load is created, although Maestro admitted it would be nice if the viewer didn’t send meaningless messages.

Aside from the back-end load, the biggest issue which occurs in group chat is when someone using it changes regions. When this happens, the chat service has to figure out where you are.

“The region you are on, your viewer and the back-end database all know where you are, and keep updated very fast. The chat servers, however, aren’t kept in perfect sync because that would be very hard to do with 50000+ people moving around who are all in 42 groups,” Simon said, in explaining the problem. He added, “It [the chat service] doesn’t track missing messages … but if it can’t send one to your agent, it then has to ask where you really are and then it sends there,” all of which takes time, delaying the receipt of group chat messages.

In describing the changes made, Simon concluded:

The new code we have out now is a bit more efficient, but more importantly it has more metrics and it showed me that the performance problems are in a few other areas … it turns out the updates to keep the list of people in the group chat updated are really significant. It gets worse, of course, in large groups – more people coming and going, and more people who need the updates.

Group Ban Lists

It appears a server-side deployment of the code require to manage the new group ban list functionality is drawing close. There is still work to be done viewer-side, but recent testing on Aditi resulted in a number of JIRAs being filed, and the associated server bugs have been stomped on by Baker Linden, with help from the likes of Caleb Linden.

SL projects updates 18/1: miscellaneous items

Server Deployments Week 18

As always, please refer to the server deployment thread for the latest updates and news.

Main (SLS) Channel

On Tuesday April 29th, the Main channel received the server maintenance project that was on the Magnum RC in week 17, comprising:

Release Candidate Channels

On Wednesday April 30th, all three RCs should receive the same maintenance package. This comprises the same fixes as deployed to the Main channel and sees the Magnum RC rejoin BlueSteel and LeTigre with AIS v3 support, which requires the use of the Sunshine RC viewer.

SL Viewer

As per my recent post, the Lab have updated their SL system requirements page for Windows and Mac OSX to better reflect the public status of both operating systems.  There is an official blog post on the subject as well, which essentially confirms that Windows XP and OSX 10.6 are considered as no longer supported. While accessing SL from computers using these OS versions will not be blocked, users will no longer be able to obtain SL-related support should they encounter issues, and the Lab is advising people update wherever possible.

Additionally, the Lab is updating the Windows installer so that it will verify whether the latest service packs for Windows XP versions have been installed, otherwise viewer installation on XP will be blocked until such time as the relevant service packs have been installed.

The aim here is to help people enjoy improved stability in the SL experience. However, once these new requirements come into effect, they may cause some upset among those affected. How widespread this is liable to be is debatable; the Firestorm team have been running a similar process since the release of Firestorm 4.6.1, and they are reporting minimal complaints from among users.

Group Chat Optimisation

Following the last group chat tests on Aditi, Simon Linden reports that the optimised code has been deployed to a production server this week, but it is liable to be a little while longer before it is deployed to all of the chat servers, and time will be taken to see how it behaves with traffic on the server on which it is currently running.

Group Bans

Baker Linden is back on code merges for the viewer. It is thought he may have hit one or two problems, so assuming he makes the Server Beta meeting on Thursday May 1st, we may find out if this is the case and what is happening.

Experience Keys (/Permissions)

Expectations on what these are liable to be, how they will work, what limitations may be placed on them, and so on, is still running high, particularly after Danger Linden (Don Laabs, Linden Lab’s Senior Director of Product) mentioned them recently. As I’ve covered previously, the experience keys are essentially the culmination of a project which was initially prototyped with the Linden Realms game, and which have since been going through continued development, re-definition and enhancement and which should – hopefully – be appearing in the near future.

Other Items

URL Errors When Connecting to a Region

There is a known issue which can cause issues when the viewer is trying to connect to a region. In these circumstances, the viewer receives URLs containing the host name, but not the required “.agni.lindenlab.com” (e.g. to something like “https://sim10586:12043/cap/…” is received, rather than “https://sim10586.agni.lindenlab.com:12043/cap/…”).

This causes the viewer to fail to connect to the various capabilities using the URL calls, which in turn results in things like mesh load failures, inventory load failures,  L$ failures, and so on, as well as having bad URLs shown for http-in. See also BUG-4704.

Should this happen, the advice to region owners is to contact support, indicating the region where the issues are occurring. The problem can be identified in the viewer log file, which will contain entries similar to the following:

2014-04-29T16:44:05Z INFO: LLCurl::completedRaw: Failed to deserialize LLSD. https://sim10586:12043/cap/01a64236-9a2b-4008-bef0-46a0f7afecae [499]: STATUS_ERROR
2014-04-29T16:44:05Z WARNING: BaseCapabilitiesComplete::errorWithContent: [status:499:] {'reason':'STATUS_ERROR'}
2014-04-29T16:44:05Z INFO: failedSeedCapability: posting to seed https://sim10586:12043/cap/01a64236-9a2b-4008-bef0-46a0f7afecae (retry 23)
2014-04-29T16:44:05Z WARNING: LLURLRequest::smileytongue:rocess_impl: URLRequest Error: 6, Couldn't resolve host name, https://sim10586:12043/cap/01a64236-9a2b-4008-bef0-46a0f7afecae