A TPV developer meeting took place on Friday August 1st, 2014. The core items discussed in the meeting are reported below, with timestamps in the relevant paragraphs indicating the point at they are discussed in the video embedded here. My thanks, as always, to North for the video.
SL Viewers
[00:15] There have been no changes to the RC viewer currently in the release channel or to the project viewers. The limited release log-in viewer which is being used for some unspecified A/B testing is still in the wild as well, although this is not available for manual download via the Alternate Viewers wiki page.
The Zipper viewer (for faster installation), which reappeared as a RC viewer on Wednesday July 30th (version 3.7.13.292263), is reported as doing “pretty terribly” crash-wise on older operating systems which may not support the archive format used to zip the viewer skins for faster installation. As a result, the Lab is currently unsure as to whether the project will be continued or not. Until a decision has been made, TPV developers have been advised not to port the code.
The Oculus Rift project viewer was updated to the current viewer code base in week 30. However, it is not currently clear when any updates related to the Oculus DK2 kit will start appearing in the viewer.
As indicated in part 1 of this week’s update, the Experience Keys beta project is going well. There are some issues to be addressed, although these are describing as being “nothing major”, and it’s not clear if they are viewer-related or not.
Unsupported Operating System Versions and Windows 8.0
[38:03] A reminder was issued at the TPV Developer meeting that the Lab will no longer be providing assistance with unsupported operating systems. This includes Windows XP and, once the Library Refresh RC viewer reaches a release status, Max OS X 10.6.
[39:05] It was also reiterated that Windows 8.0 users who are experiencing crash issues with the viewer should consider the free upgrade to Windows 8.1, which has a much lower crash rate, with overall improvement in viewer stability being described as a “really big difference”. A blog post on this subject (and viewer crashes in general) was posted by the Lab towards the end of July 2014.
Group Chat
[03:00] Following-on from the discussion about group chat and the initial testing of updates which took place at the Server Beta meeting on Thursday July 31st, Oz said of the work, “we’re continuing with making back-end changes, there’ll be another roll-out of some experiments shortly and we’ll see how they go. Like I’ve said before, we’re not going to generally announce when those happen, because we don’t want to change the experiment by changing people’s perceptions.”
[10:27] Although at the moment the focus is very much on what can be achieved on the back-end services without the need for changes within the viewer itself, Oz gave notice that the Lab may want to talk to TPVs about possible changes to group functionality and viewer behaviour at some point in the future. One of the questions being asked within the Lab is whether or not the members list needs to be displayed for all groups. Some groups, for example have a “no chat” policy and / or are only for the outward flow of information (e.g. product update groups); so are these a category of group for which the updates of people coming on-line or joining / leaving the session are no actually relevant, and could be eliminated or suppressed, thus reducing the volume of update messages?
On Tuesday July 29th, the Main channel was updated with remaining recent feature changes and bug fixes previously deployed to the RC channels – release notes
There were no RC deployments.
SL Viewer
The Zipper viewer, offering a faster install, reappeared on Wednesday July 30th, after vanishing from the Alternate Viewers wiki page in May. There are apparently an issue with the XUI Preview Tool being broken, which has now been resolved.
The new version of the viewer – 3.7.13.292263 – appeared as a release candidate in the release viewer channel, rather than a project viewer, where it resides with the group ban viewer and the library refresh viewer, both of which were updated in week 30, are likely the strongest candidates for promotion as the next de facto release viewer.
Group Chat
Testing of on-going group chat updates took place during the Server Beta meeting on Thursday July 31st.
Simon Linden is once more digging into the group chat code
As noted in a previous report, one of the major causes of issues with group chat lies not with the actual messages being sent back and forth, but rather as the chat server tracks who in online or not. The server maintains a list of who is online and in the group chat at a given moment, and is constantly updating the list as people join / leave the session; these updates are then sent to everyone else still active in the group, which interferes with the sending / receiving of actual messages.
“Imagine a popular group with, say 120 people online,” Simon said during the meeting. “Let’s guess the average online time is an hour … and that number varies widely, as there are a LOT of people who are connected for only a minute or two, maybe just checking IMs, see who’s online, or trying to fix something. But with 120 people … that’s very roughly an update every 30 seconds [14,400 updates an hour], sent to the whole group.”
Not only does this impact the sending / receiving of chat messages within the group, it can also impact other group chat sessions which are running on the same back-end server, as they are being starved of resources.
The code being tested on July 31st had been set to delay the sending of these updates from the server in order to see if it improves the throughput of actual messages. The downside of this is that the member list updates are somewhat delayed; however, this would seem to be a small price to pay in order for an increase in the reliability of messages actually getting through the system. As it is, the delay is configurable, so Simon was gathering data to see how the updated code works in terms of people joining / leaving chat sessions and sending messages. The results are liable to be known next week.
One possible future option for group chat is for people to be offered the ability to opt-in or out of receiving group chats until such time as they join a group chat (some TPVs already have an option to disable group chats until such time as you opt to join them).While this may help with the “I’m here!” messages sent to all groups on log-in, and which exacerbate the problem somewhat (again as described in the update linked-to above), such an approach is not seen as optimal, as it is possible users won’t change their behaviour, but will simply opt-in to all group chat sessions anyway.
Simon has also been tracking down an odd bug with joining a group and being able to open a chat session with it. “It’s really an odd one where opening the group is very slow or times out,” he said at the July 31st meeting, “and then can be immediate the next time you try. From what I can tell the chat server isn’t getting the messages … so somewhere between the viewer, simulator and chat server it gets lost.”
Other Items
BUG-6736 is a feature request for the updating or removal of the current limit on the distance at which objects can be linked (see linkablility rules). The advantage in increasing the limit is that it could allow for bigger builds (in terms of footprint) without having to rely on scripted rezzing systems. A problem here is the if increased, there is a risk that the ability to link objects over greater distances might cause issues were said distances are close to or exceed the draw distance / interest list distance.
“You will get some really funky update issues if the link size is larger,” Simon said at the Server Beta meeting. “As soon as it gets close to your draw distance, things go bad, as in … you stumble against something you can see.”
Commenting on this, Lucia Nightfire added, “I noticed a selection bug/gripe with multi selection and one prim being out of interest range on rez, you deselect everything by clicking on something else then if you pull your cam back you magically select stuff that was out of your interest range.”
Responding to this, Simon said, “Yeah, that’s the kind of thing that can get confusing, you won’t see what you expect because the root might be farther away than your draw distance … That said, I understand the builder desire to make larger parts, but those limits are there because it can conflict with the interest list logic about your updates.” As such, it would seem unlikely that there will be much in the way of change to the linkability limit.
There will be no Release Candidate channel deployments in week 31. This means there is unlikely to be any main channel deployment in week 32 (week commencing Monday August 4th), although RC channel deployments are anticipated for that week.
Experience Keys
There is a little more news on the Experience Keys beta, which Oz Linden, attending the Simulator User Group meeting on Tuesday July 29th, indicated was going well. A possible extension to the beta had been sort-of hinted at in previous meetings, but Oz indicated that whether this will or won’t be the case has yet to be decided.
It was also indicated at the meeting that the idea fees which may be associated with the use of Experience Keys by creators may also have had further discussion within the Lab.
The idea of charging creators a nominal fee for the use of Experience Keys was first discussed at the Experience Keys preview during a TPV Developer meeting. A nominal fee or some restriction on accessing the tools (e.g. such as having to be a Premium member) was seen as a means of perhaps discouraging those who might otherwise use the capabilities with intent to grief harass others. While at the time it was admittedly an exploit of the original Advanced Content creation permissions system, the June 2012 incident did demonstrate how someone with access to the tools might use them to cause upset.
Commenting on the likely charges (if any) which may be levied and whether they have been agreed internally, Oz Linden said at the Simulator User Group meeting on Tuesday July 29th, “We don’t discuss pricing here… not our area. I believe they have, but I don’t know details.” Simon Linden then added, “We can’t talk about any details like that until they’re officially announced.”
llSetLinkAlpha Updates Issue
BUG-1786 has been around for over a year, and takes the form of multiple prims in a linkset randomly failing to update correct when using llSetLinkAlpha. For example, you have a cube made up of a number of smaller cubes so that when clicked, the entire linkset should switch from solid to transparent or back again, but only some of the prims in the linkset appear to change on being clicked. with some remaining solid when they should be transparent. The problem has been reported as being more noticeable on linksets of 34 or more prims, but has also been reported on smaller linksets.
The problem seemed to first occur with the 26th February 2013 server deployment 13.02.15.270481, and the issue has been thought to be server-side. However, the problem seems to be more prevalent with lower bandwidth settings (e.g. 500 or lower – 500 being the SL viewer default), and can vanish with higher bandwidth settings (e.g. around 1200-1500), suggesting the issue might be related to packet loss, particularly if there are a lot of simultaneous updates going on, or a possible race condition within the viewer. A sample object has been passed to the Lab to let them poke at things further.
Group Chat
The group chat test which suffered cancellation at the July 24th Server Beta meeting may get to be run at the upcoming meeting on Thursday, July 31st.
The Next Simulator UG Meetings
Simon Linden is on vacation next week, and may not be able to attend the week 33 Simulator UG meeting either, due to other commitments. He’s hoping to get someone at the Lab to cover for him at both meetings.
On Tuesday July 22nd, the Main channel was updated with the infrastructure project deployed to the Magnum RC in week 29, and which adds support for the upcoming changes to the Skill Gaming policy, and includes the updates previously on LeTigre and BlueSteel- release notes
On Thursday July 24th, all three RC channel were updated with the infrastructure support for the upcoming changes to the Skill Gaming policy, and the updates previously deployed to LeTigre and BlueSteel – release notes.
SL Viewer
The Library Refresh viewer was updated to release candidate status with the release of version 3.7.13.292194 on July 23rd. This viewer contains an update to a large set of the libraries used by the viewer to provide security, stability and consistency improvements to this and future viewers.
Group Chat
Simon Linden: continuing to work on group chat issues
The anticipated group chat test didn’t materialise at the Server Beta meeting on Thursday July 24th as a result of Simon Linden coming across a last-minute issue which needed to be resolved ahead of further tests. He and Oz did, however, explain some recent discoveries within the chat system.
“While the earlier update to group chat didn’t give us any significant performance boost, we got a lot more information out of the servers,” Simon said. “And what we found was a big part of the group chat system load is not the chat messages you care about, but the updates to who is in the session or not.”
Oz added, “Those updates happen whether or not you’re displaying who’s in the session, in every group you’re in.”
Simon continued, “You can actually see this in the viewer if you add a line of code to log something whenever an update comes in to tell you who’s in the group chat … you’ll be surprised how many you get. The load goes up as the group size goes up … with a larger group, people are joining and leaving more often, and there are more people to update.”
People joining / leaving a session are recorded by the chat server. “It has a list of who’s online and in the group chat at that moment,” Simon explained, “it’s adding and removing from that list, and [generating] the resulting updates, that are the problem, [causing them] to be sent to everyone else still active in the group as they do so.”
The growth curve of these updates is described as exponential, and there is a knock-on effect with them as well; as group chat sessions share server resources, it is possible that a large group chat session, with multiple users joining / leaving it and thus causing it to generate lots of updates can affect other group chat sessions hosted on the same group chat server, slowing them down as well.
While the chat servers are due for a hardware change, which is hoped will improve performance to a degree, simply adding more hardware to the chat service back-end isn’t seen as a solution, as it’s the exponential manner in which the updates grow which needs to be reduced and controlled. The testing Simon had hoped to run during the Server Beta meeting was to test some improvements he had been making to the queuing of the updates and in combining messages to hopefully reduced the load. However, in running over the code, he encountered a glitch that he needs to resolve before the testing can proceed.
Another issue with the group chat system is that when users log-in to a Second Life session, they automatically join all 42 of their groups, sending a “I’m here!” message to all 42 groups so that they can start receiving messages from active groups. This has obviously been exacerbated each time the limit on how many groups a person can join has been raised, so as Oz pointed-out during the meeting, “upping it again would make it even worse, so until this is dealt with, don’t even ask… 🙂 .”
HTTP Updates
As indicated by Monty Linden at the last TPV meeting, there are further server-side updates which should further assist with improvements to texture and mesh asset downloads. These are now nearing the point where they are likely to be surfacing (although quite when isn’t clear), prompting Oz to comment, “We’re setting up some experiments with server side changes that will complement the pipeline viewer, but are not strictly speaking dependent on it. When we’re confident that our test setup is ready, including how to measure the results, we’ll invite you folks to help us test.”
As always, please refer to the server deployment thread in the forums for the latest information and updates.
Main (SLS) Channel
On Tuesday July 22nd, the Main channel was updated with the infrastructure project deployed to the Magnum RC in week 29, and which adds support for the upcoming changes to the Skill Gaming policy, and includes the update to improve L$ transaction logging for payments made by scripted objects previously on LeTigre and BlueSteel- release notes.
Release Candidate Channels
On Thursday July 24th, all three RC channel should be updated with the infrastructure support for the upcoming changes to the Skill Gaming policy, and includes the L$ transaction logging update and Temp attachment fix previously deployed to LeTigre and BlueSteel.
SL Viewer
On Tuesday July 22nd, the de facto release viewer was updated to version 3.7.12.291824, previously the Maintenance RC release candidate. This viewer includes over 40 MAINT fixes, including:
MAINT-3135 Cocoa Viewer: Mac: Maximizing the viewer leaves garbage on the screen
MAINT-3171 Alt-clicking while moving mouse can move the camera significantly
Tuesday July 22nd also saw two other viwer updates:
Tthe Group Ban RC viewer updated to version 3.7.13.292225. This allows group owners and those assigned to a specific role to ban (and eject) others from joining the group. Useful for dealing with spammers and other trouble-makers in open enrollment groups. For an overview of the group ban viewer, please refer to my preview notes
The Oculus Rift project viewer also updated to version 3.7.12.292141, to bring it up to parity with the release viewer 3.7.12 code base. Note at the time of writing, there were no supporting release notes for this update – please refer to the Alternate Viewers wiki page for the downloads.
Group Chat
Simon Linden is planning on a further group chat test at the Server Beta User Group meeting on Thursday July 24th. Commenting on this at the Simulator UG meeting on Tuesday July 22nd, he said, “those have been really helpful with our development. The more people who can come, the better. The test is usually at the end, a little before 4:00 if we don’t run out of other topics.”
My apologies for this appearing a little on the late side; things have been a bit manic in the physical world of late (not helped by the Tour de France and the German GP this weekend!), and I’ve been slipping behind on blog posts (I’ve also got to talk to my minions about vacation scheduling….).
Note that the following notes are taken from both the Server Beta User Group meeting of Thursday July 17th and the TPV developer meeting on Friday July 18th, the video of which is supplied below (my thanks to North, as always). Items taken from the later are time stamped within the text, so you can locate and listen to the discussion in full via the video.
Server Deployments Week 29 – Recap
On Tuesday July 15th, the Main channel was updated with the Experience Keys project, which had previously been running on Magnum. This roll-out coincides with the release of the Experience Keys project viewer (see below) and the release of the Lab’s first Experience Keys demonstrator game, The Cornfield. Please refer to the release notes for further information
On Wednesday July 16th, the Magnum RC was updated a new infrastructure project that adds support for the upcoming changes to the Skill Gaming policy. Release notes
On Thursday July 17th, BlueSteel and LeTigre were both be updated with the Experience Keys project, but otherwise remained on the same server maintenance project as week 28, which addresses a JSON-related bug, an interest list related race condition, and to improve L$ transaction logging for payments made by scripted objects. See the release notes (BlueSteel) for details.
SL Viewer
Group Ban Viewer
The Group Ban viewer reached release candidate status on Wednesday July 16th, with the release of version 3.7.12.292031. This viewer allows certain group members to ban avatar from a group or from joining a group When an existing group member is banned, they are also automatically ejected from the group. Please refer to my Group Bans overview for further information, if required.
Maintenance Release RC
[04:20] This viewer, version 3.7.12.291824, has been tracking with the same crash rate as the current release viewer (3.7.11.291465), and as such is expected to be promoted to the de facto release during week 30 (week commencing Monday July 21st). However, it has been reported that the Mac Alt-Cam bug (BUG-6760) fix doesn’t work and has been referred back to the Lab for further investigation.
Oculus Rift Project Viewer
[04:56] It is anticipated that an updated version of the Oculus Rift project viewer will be appearing soon, potentially in week 30. The update will bring the viewer up to par with the current 3.7.12 release code base.
Log-in Test Viewer
[04:44] There is a special log-in test viewer currently on closed use (there is no publicly available version), which is being used for some kind of A/B testing related to logging-in to Second Life. Precisely what this testing is geared towards is unclear.
Viewer Autobuild Process
[05:50] Oz Linden has been working on improving the viewer autobuild process, and there is a new version of autobuild, together with a wiki page on the changes and improvements. The new version brings with it a number of improvements, such as stricter library version checking, full transitive dependency checks, additional error checks, etc. This is considered to be one of the steps required in order for the viewer to be compiled using Visual Studio 2013. Full details in the video for those into self-compiling viewers.
Third-party Viewer Directory Updates
[00:20] The Third-party Viewer Directory, which lists all Second Life viewers and clients which have gone through the self-certification process, has been revised.
Until recently, the directory was listed by viewer crash rate – with the most stable at the top. However, this was something of a hit-and-miss approach due to a number of factors, including significant changes made to the code within the viewer which is used to detect and report crashes. So instead, viewer and clients are now split into three categories:
Those which are actively maintained “full” viewers which are updated regularly to track new developments in the Linden Lab viewer, and implement a full graphical environment
“Lightweight”, text and mobile clients, such as Lumiya, Group Tools, Radegast and so on
Those viewers which have not been updated recently enough to be considered fully compatible with current Second Life services (e.g. they lack things like server-side appearance, etc.).
Group Chat
[11:56] Work is continuing on group chat. At the TPV Developer meeting, Oz Linden summarised this work as:
We are working on group chat; I don’t really have much to report on that this week. We’re doing a set of experiments and collecting a lot of data, and then we’re going to come up with the next round of changes to make. One of the things we’ll try to do, once we think we’re done with this project – and I have no predictions for when that will be – is tell people how it went and what we’ve done.
It may well be that before we’re done, we’ll come back to this group and say we’re making changes to interfaces to viewers for group chat in order to improve the situation. I don’t know of any of those yet, but I’m not ruling them out. We’re going to try to make group chat a lot better, and if that means not being 100% backwards compatible, then that’s what it means. At this point we’re not looking at changing the protocol with anything else. Not ruling it out, but that’s not the correct direction.
Experience Keys
The Cornfield, the Experience Keys demonstrator game
The current creator beta programme for Experience Keys has now been filled. Commenting on it at the Server Beta meeting, Coyot Linden referred to it as proving “wildly popular” and that the Lab have “heard some really cool ideas for new experiences”. He also referred to this being “round on” of the beta programme – so there may be more opportunities for creators to be involved in the future.
During the Server Beta meeting, a request was made for the Linden to consider allowing the popularity of an Experience (e.g. the number of people engaged on it) to be made available, with the suggestion it could be done in a number of ways:
As information made available only to the Experience owner (so they can see how popular a given experience they’ve created is proving to be
As information which can be (perhaps optionally) published by the Experience owner (e.g. via the Experience Profile)
As information which can be displayed in the Search tab of the Experience floater, allowing users to search for the most popular experiences at any given time.
Commenting on this, Simon Linden said, “The numbers will definitely be interesting, but we’ll have to think carefully about what and how to expose it. As an owner, it makes sense for you to have an idea what’s going on with your experience. I’m not so sure about others.”
Part of the concern here is about the popularity figure potentially being used by griefers as a means of targeting popular regions / activities and causing disruption. Following Simon’s observation, Dolphin Linden added, “yeah, technically the number can be made available. But how and to whom needs to be thought about, but numbers about your own xp can be tracked if you want with a little bit of work. We might also be able to just get an ordered list of the top 10 experiences or something, without disclosing actual numbers.”
Inara Pey: Living in a Modemworld 