SL projects updates week 13/3: TPV developer meeting March 28th

A TPV developer meeting took place on Friday March 28th. The core items discussed in the meeting are reported below, with timestamps in the relevant paragraphs indicating the point at they are discussed in the video embedded here. My thanks as always to North for the latter.

Release Candidate Viewers Status

Stat Test RC  and Google Breakpad RC

[00:10] The Stat Test release candidate viewer (version 3.7.5.288371 at the time of writing) is not intended to ever move to a formal release status. Its function is to help with identifying how big an impact a bug within the Google Breakpad RC was having on viewers by comparing crash rates.

The fix contained in the Stat Test viewer is now also in the Google Breakpad RC, which was updated on Thursday March 27th to version 3.7.5.288464 (download and release notes). It’s thought that this RC is getting “pretty close to final form”, something that will likely be confirmed once the new update has been in the release channel sufficient long enough for decent statistics to be gathered in week 14 (week commencing Monday March 31st).

Interest List RC and Merchant Outbox RC

[01:00] As noted in part 2 of this report, the Interest List RC returned to the release channel wither version 3.7.5.288404 (download and release notes). This RC had been removed from the release channel on March 21st due to the number of RC viewers in the channel at that time.

The Mechant Outbox RC, which contains fixes for accurately detecting Merchant status and improves recovery for Merchant Outbox errors, was updated on Friday March 28th to version 3.7.5.288408 (download and release notes). This update means that as of Friday March 28th, all RC viewer in the release channel were up-to-date with the current release version of the viewer.

Voice RC and Sunshine / AIS v3 RC

[01:35] These two release candidates were withdrawn from the release channel on March 21st to reduce the number of RC viewers vying for promotion to release status. These have yet to be updated with the current release viewer code, however, it is anticipated that the Voice RC will be returning to the release channel early in week 14, and that the Sunshine AIS will hopefully reappear later in the week.

The binaries for the Vivox updates are currently available to TPVs for testing an incorporation, so the removal of the Voice RC from the release channel should not have unduly impacted any work being undertaken by TPVs in updating or offering Vivox 4.6.x to Windows and Mac users. Unfortunately, Vivox have yet to supply updates for Linux.

Project Viewers

Project Zipper

[01:52] The Project Zipper “fast installer” viewer was updated to version 3.7.5.288507 on March 28th, also bringing it up to parity with the current release viewer, although it remains at project status (download and release notes).

Oculus Rift

The Oculus Rift project viewer is currently on a closed beta, and so the code is not publicly viewable. The beta is reported to be going “really well”, although other sources indicate that there is a lot of dissatisfaction in how the UI works, up to an including demands that the entire UI needs to be re-thought.

Given any change in the viewer UI tends to attract a high degree of negativity right out of the gate (and also given the fact I’ve not actually seen the “Riftlook” UI), it’s hard to say how many of the concerns are objective as opposed to subjective in nature (e.g. the issue is genuinely one of ease-of-use as opposed to it being one that is considered “unusable” because it’s either not how it’s worked in the past, or requires a re-training of muscle memory) .  However, if the issues are objective and indicate significant issues of usability, it will be interesting to see how the Lab responds.

Upcoming Viewers

Group Bans

[02:24] The Group ban viewer is progressing, as noted in part 2 of this report, a new version is available through the Server Beta wiki page, and can be tested on the Morris region on Aditi. There is yet more work to be done on this, so it’s unlikely to move to an official project viewer or RC status in the short-term.

Snowstorm

A new Snowstorm RC viewer is in development, and Oz Linden is hoping to include a few more things in it. I believe this viewer may include the LSL syntax updates of STORM-1831, although the hope is apparently to get some more items into it alongside what is already there. As there are already so many viewers either in, or queuing-up for RC status, there is currently no rush to move this viewer forward.

Release Viewer

[04:02] Commenting on the current release viewer version: 3.7.4.288138, formerly the FmodEx Hotfix RC, Monty Linden indicated that there are almost a year of fixes the stream threading between the current viewer and previous versions. These include many fixes for issues where the viewer can drop a stream entirely. However, whether these include a fix for the issues where a stream containing externals ads (for cars, insurance, etc. – see FIRE-12798) can be completely dropped by the viewer and require a re-log in order to reconnection, is unclear.  However, Monty has offered to take a poke at things.

SLS Share / Facebook Photos Upload

[07:26] As reported on March 20th, the ability for users to upload photos from the viewer to their Facebook accounts was disabled by Facebook, apparently because of the inclusion of SLurl with the photo uploads.

The Lab has made changes in the viewer and in the intermediate server layer which should address Facebook’s concerns, however, Facebook has yet to unblock the upload capability at their end, and at the time of writing there has been no word on when this might be done.

The viewer-side updates have yet to reach the release viewer, but are currently in all of the release candidates in the viewer release channel.

The SL Share photo upload panel as it is in the current release viewer (), left, and as it appears in the RC viewers awaiting promotion, right. The SL option has been removed to meet Facebook's requirements
The SL Share photo upload panel as it is in the current release viewer (3.7.4.288138) on the  left,complete with the option to include a SLurl; and as it appears in the RC viewers on the right, with the SLurl option removed to meet Facebook’s requirements

Mac Issues

[08:30] A request has been made for the Lab to provide a list of Mac / Cocoa issues to help TPVs keep track of things. As reported last time this issue was raised, the Lab are hampered in trying to deal with Cocoa-related issues, as their Mac developers are working on a number of projects, and so specific issues are dealt with individually, rather than as a part of an overall project.

HTTP Issues

Monty Linden:
Monty Linden:

[19:53] The Firestorm team asked if the Lab had any data on how well the HTTP code changes were going and if they were seeing anything unexpected with server-side loads. The prompt from this question came by the fact that as downloads of the Firestorm 4.6.1 release have increased, so the number of user complaints which appear as though they might be related to HTTP have increased. Monty Linden replied:

Not that I can tell. Nothing is getting to me if there is any such story. I’ve heard the anecdotes, and whenever I’ve looked into them, the stories behind them are generally pretty random. It’s people associating recent changes with behaviour they’re seeing and assuming some correlation where there isn’t any.

Now, that said, there are some patterns that continue to persist that have existed for the past few years that I’ve been dealing with transport issues. The simulators always have had problems in the lib services with connection quotas being exceeded, and that’s still going on today. A busy region on a busy sim hots is still a problem, but I can’t say it’s worse than before. It should be better, but the data I’m getting – which is almost none – doesn’t point to any problem. I’m sorry that’s not definitive, but it’s all I’ve got at this point.

It has been reported that HTTP downloads on sims which have a lot of textures and objects, but few avatars are significantly better than HTTP downloads where there is a large number (20+) avatars. In the latter circumstance, it has also been reported that switching back to UDP appears to improve downloads. However, Monty warned against this, as UDP puts significantly more load on the simulator, which can have a noticeable impact for every user connected to that simulator.

In discussing how to confirm whether or not users really are seeing HTTP issues, Monty continued:

The first thing would be to look at the log file. If you’re getting permanent failures in texture HTTP, it’s going to show up in the log file with a fail on a final retry with an error code and a bunch of related information.So you can pretty much confirm pretty quickly whether or not they’re getting serious permanent failures for texture operations … and these will be distinct from 404s. 404s are nots founds, and you’ll see those every now and then simply because the texture reference is not valid. But other reasons for a permanent failure would be interesting.

When I updated the texture console, I put mesh errors in, but I didn’t do anything about HTTP errors, but there would be another useful area to put things in the future for monitoring error counts. but for now the log file will have the truth. Let me know what you find, because that will point us in one direction or the other.

On our services, I absolutely definitely see busy sim hosts having their input channels saturated. And when that happens, you are going to have retries, and I’ve seen some pretty hard cases of that. Some of the big social things that get put on Second Life [such as RFL] often get put on concentrated sim hosts, and these really get victimised by this problem. But again, permanent failures, they’ll be in the logs and you can look at that. Beyond that, I’ll se what i can find out.

 A problem with obtaining logs is that people will often encounter problems like this a few times and the revert back to an earlier release of the viewer. So, if you are using a viewer (SL or Firestorm) with the HTTP updates and encounters persistent rendering issues or failures when moving around SL, please consider taking a couple of minutes to fill-out a JIRA, attach your log file and submit it. Monty is very sensitive to HTTP issues, but in order to respond to them, he needs data, not anecdotes.

Other Items

AIS v3

[02:56] Oz Linden reported that ther server-side AIS v3 code will be enabled across all three RC channels (LeTigre BlueSteel and Magnum) in week 14. As BlueSteel and LeTigre had AIS v3 enabled with server update 14.03.07.287758 and Magnum with  update 14.02.21.286985, I can only assume that it was disabled again with the Magnum update 14.03.12.288004 in week 13.

Male Avatar Chest Rendering Broken / Mesh Clothes Breaking on Male Avatars

This issue related to BUG-5537, where there is a discrepancy on how the AV mesh shape is rendered/viewed by oneself and by others in the male AV chest portion, so that while an item of rigged mesh clothing looks fine to the user when viewing his avatar, other people see parts of the avatar’s chest still poking through the outfit. There has also been discussion of the issue on the forums.

The Lab is currently investigating the issue, but have no definitive answer on the problem as yet.

 

 

 

SL projects updates 11/3: TPV developer meeting, March 14th

A TPV developer meeting took place on Friday March 14th. The core items discussed in the meeting are reported below, with timestamps in the relevant paragraphs indicating the point at they are discussed in the video embedded here. My thanks as always to North for the latter.

SL Viewer Updates

[0:01:37] The list of release candidates in the release channel remains unchanged from part two of this week’s projects updates, and as per my Current Viewer Releases page.

FmodEx RC

[0:01:44] The FmodEx Hotfix viewer RC (version 3.7.4.287875), is a fix Monty Linden has been working on, and is described by Oz Linden as:

A threading problem that at least manifests when there are various FmodEx things going on, but is not strictly speaking an FmodEx problem. We think that was a good and important fix, but it doesn’t seem to have done all we hoped it would do yet.

Whether or not this is a fix TPVs would need to implement quickly or not is down to how they have implemented FmodEx.

Voice RC

[0:02:38] The Voice RC is essentially the release viewer with the Vivox 4.6.x SLvoice plugin packaged with it for Windows and Mac. Commenting on in from a Mac perspective, Oz Linden indicated that it does appear to solve a number of issues, such as working with an iPhone headset adaptor, which was an issue with earlier versions, as well as addressing some Mavericks related issues.

[0:11:11] There has been some confusion over the latest SDK supplied by Vivox, in that only the Windows and Mac versions of 4.6.x have so far been supplied; the Linux version is still an older version. It’s unclear as to when the Linux Vivox SDK will be supplied, as this is apparently seen as a “lower priority” compared to Windows and Mac, although the Lab is working on Vivox to try to improve matters. The Lab is also working to try to get 64-bit versions of the Vivox SDK, which could then be made available to those TPVs building 64-bit versions of their viewers.

Interest List RC

[0:41:54] Concern is raised as the number of updates which form a part of the interest list RC viewer, and whether these may leave TPVs with another “CHUI situation” when trying to merge things.  The repository for the viewer has been available since the viewer reached RC status, however, Oz went on to comment:

There’s a bunch of refactoring of things that people decided needed refactoring as a part of the process [and] which may or may not have been strictly needed as [a] part of interest lists; that is, part of the functional change that that branch is doing. Some of it was a new trace capability that’s used in a bunch of places where they wanted to take the measurements they wanted to take about it.

The interest list RC is working its way towards release status ... slowly ...
The interest list RC is working its way towards release status … slowly …

There have been various stability issues with the interest list RC, hence why it has remained an RC rather than being promoted to the de facto release viewer. However, it is now reaching the point where its stability is comparable to that of the other RCs in the release channel – and is actually better than some.

In terms of merges, there is the potential for the interest list viewer to cause TPVs some problems, as there appear to be changes to llCommons and other libraries which are causing issues for those TPVs which have attempted a merge.

Google Breakpad

[0:04:53] The Google Breakpad RC is due to make another appearance, as a “bunch of issues have been wrestled to the ground”, and the hope is that when it does appear in the release channel, it will mark the last round of updates for that particular project, and those TPVs using Google Breakpad are advised to take a look at what the Lab has done.

Overall Status for RCs

[0:04:10] Overall, it appears as if none of the RCs are performing as well as the Lab would like them to be in terms of crash rates. It had been hoped that the FmodEx Hotfix RC would get the Lab back below what Oz referred to as “an acceptable, if not admirable, crash rate”, but it has not done so as yet.However, the other RCs in the channel should see updates released in week 12 (week commencing Monday March 17th), one or more of which may improve the crash rates.

[0:43:56] In terms of what does get promoted next, the most likely candidate will be the RC which shows clear evidence that it is reducing the crash rate compared to current levels across the release and RC viewers.

[0:05:27] In the meantime, because of the volume of RCs sitting in the release channel, the Lab are holding back a number of further RCs,. These include the Project Zipper (faster installer) viewer being updated to RC status, and the group ban viewer (although there are bugs in this which are still being worked upon). There is also likely to be a further Snowstorm RC appearing with a mix of code contributions, again once the number of viewers currently in the release channel is thinned-down a little.

Continue reading “SL projects updates 11/3: TPV developer meeting, March 14th”

SL projects update week 9/3: viewer, Vivox and Fitted Mesh

Server Deployments: week 9 – recap

As always, please refer to the server deployment thread in the forums for the latest updates / changes.

On Tuesday February 25th, the Main channel was updated with the server maintenance package previously on all three RCs, which comprises the following:

  • Fix for BUG-5034 “If an EM restarts a region and then teleports out immediately, the EM will disconnect just after teleport”
  • Fixed a rare case in which e-mails read by LSL scripts immediately after rez or region change would sometimes be missing the message body
  • Fixed some crash modes.

On Wednesday February 26th, the BlueSteel and LeTigre Release Candidate channels were updated with an infrastructure update, with no functional changes. The Magnum RC received the same infrastructure update and had the server-side AIS v3 code is enabled. As noted in part 1 of this report, the new AIS code is designed to improve performance and reliability when changing outfits, but is reliant on new viewer code, which is currently available from the Lab in the Sunshine Project viewer (of which mor below), and will doubtless start appearing in TPVs soon.

While no details were given, the infrastructure updates deployed to the RC channels are apparently system changes to make simulators easier to manage.

TP Developer Meeting Video

A TPV developer meeting took place on February 28th. The core items discussed in the meeting are reported below, with timestamps in the relevant paragraphs indicating the point at they are discussed in the video embedded here. My thanks as always to North for the latter.

SL Viewer Updates

Release Candidates

  • The “Project Interesting” interest list RC  viewer updated to version 3.7.3.287127 on February 26th
  • The Maintenance RC viewer updates to version 3.7.3.287263 on February 27th.

[0:29] These updates put both of these viewers on a par with the current release viewer, and both are doing “well” in terms of low crash rates, but as Oz Linden said at the TPV Developer meeting on Friday February 28th, “it’s still early days”.

Upcoming Release Candidates

[0:43] The Google Breakpad viewer is expected to reappear as a release candidate “pretty soon”. The last version of this viewer had significant issues related to Windows XP, and so a “different tack” will be taken to handling XP and older versions of Windows with the next update, XP itself having a “truly horrific” crash rate for the viewer. However, and to ease any concerns,  the “new tack” doesn’t involve blocking Windows XP systems from accessing Second Life.

[2:33] The Breakpad update is described as being “important” for TPVs to pick-up and include in their own code, in order to ensure viewers are all reporting stats in the same manner.

[6:48] Both the Sunshine (AIS v3) project viewer and the Merchant Outbox project viewer are expected to move to release candidate status very shortly. These are described as being in the “last stages of the QA pipeline”, and so will be entering the viewer release channel,  once again increasing the number of RC viewers vying for promotion.

Project Zipper Viewer

[7:20] On Wednesday February 26th, the Lab issued the “Zipper” project viewer (version 3.7.2.286810), designed to speed-up viewer installation time (cutting it by roughly 50% in a clean install on my own PC) – from 35.6 seconds to  16.4 seconds. Most of this appears to be the result of compressing / combining files, and there have been questions raised on whether doing so might result in an actual performance hit.

It is anticipated that this viewer will also be shortly rebuilt as a release candidate viewer in the release channel.

Other Viewer Projects

  • [9:12] Group Bans – as I’ve noted in part 2 of this week’s report, the viewer-side code for Group Ban (lists) should be appearing as a project viewer in week 10, ready for the Group Ban functionally to be tested on Aditi
  •  [10:07] Oculus Rift – the Rift viewer is still with LL’s QA team.

Continue reading “SL projects update week 9/3: viewer, Vivox and Fitted Mesh”

SL project updates 7/3: viewer updates, AIS and misc items

The following notes are taken from the TPV Developer meeting of Friday February 14th, a video of which is included below. My thanks, as ever, to North for the latter. Timestamps relative to the recording are given in braces both at the start of each section and within the text where appropriate.

SL Viewer

Release Viewer Update

[01:20] The Facebook Hotfix RC (version 3.7.1.286557) released on February 12th was promoted to the de facto release viewer on Friday February 14th. Based on the Fitted Mesh viewer code, this viewer has a single fix for a  problem in the in-viewer web browser that made it impossible to login correctly into Facebook. The rapid promotion was made because the issue was seen as obnoxious by those people who have connected their SL and Facebook accounts, and it was felt those who do may want to post Valentines-related messages and images to their Facebook accounts.

HTTP RC

Robbie Monty Linden sports new look
Robbie Monty Linden sports new look

[02:32 and 36:56] As noted in part 2 of this report, the HTTP viewer has been rebuilt using the Facebook Hotfix / Fitted Mesh viewer code to version  3.7.1.286567.

However, it has suffered from the number of RCs currently in the queue, or as poor Monty put it, getting “stuck behind everyone”.  This  viewer has one of the lowest crash rates on record as an RC, and given this, the expectation is that it will be promoted to release status “pretty soon”.

One of the major issues Monty faced with the viewer-side updates was directly related to mesh, and thread race conditions, and he admits that not all of these have been resolved. This is partially due to some of them being  infrastructure-related Heisenbugs, which are time and labour-intensive to resolve. However, they shouldn’t impact the stability of the updates made to date.

Remaining RC viewers

[03:54] The three remaining RC viewers  – the Maintenance RC (3.6.14.285499), Interest List RC (3.6.14.285213) and Google Breakpad RC (3.6.14.285686) are in the process of being rebuilt to the 3.7.1 release code, so updated versions should be appearing in the release channel in the next week (ish).

Project Viewers

[04:23]

  • It is unclear whether the Merchant Outbox project viewer (3.6.13.284731) will move forward or pulled back to have some more work done on it, and it is unlikely to move towards viewer release “any time soon”
  • The Sunshine / AIS v3 project viewer was rebuilt to the 3.7.1 code (Fitted Mesh and the Facebook Hotfix), with a new version (3.7.1.286565) appearing on February 14th. It is anticipated that this viewer will move forward to a release candidate status fairly quickly now that the Facebook Hotfix has been promoted, reducing the number of viewer RC cohorts currently in the release channel.

AIS v3

[05:16 and 14:50] A surprising piece of news passed-on in the meeting was that the AIS v3 server-side code has been deployed across all channels on the Main grid. This initially caused some confusion during the meeting , as there has been no mention of this in any server-side release notes. Nyx Linden queried the situation with the ops tem and received a confirmation that the new AIS capabilities had been deployed to Agni [21:40, via text], but are currently disabled [22:29 via text]. It would seem likely that the capabilities will be enabled once the Sunshine / AIS v3 project viewer moves to release candidate status.

Voice

vivox[07:35] There have been a number of issues with regards to voice in SL, particularly of late. As noted in my week 6 report, there was some recent back-end work carried out which should improve things for those using viewers running with the most recent versions of the voice SDK (SLvoice.exe).

Discussing the matter at the TPV Developer meeting, Oz Linden revealed that Vivox had reached out to the Lab to assist with issues being experienced, and as a result of this underlined the issues with viewers using older versions of the SDK (and which will not see any real improvements to their voice performance as a result). Vivox have requested that TPVs provide details on any older versions of the SDK they are running, and details of specific issues they are encountering, as well as offering encouragement to update.

As a new version of the SDK is due to be released in the near future (hopefully within a couple of weeks), it may prove to be an opportunity for TPVs to update, given it has a number of audio quality fixes and Vivox have offered to assist in dealing with issues being experienced with voice in SL.

In addition to this, Oz is looking to work with Vivox to try to get any new versions of the SDK used by the viewer made available to TPVs at the same time it is made available to the Lab, thus eliminating the need for TPVs having to wait for LL to QA and integrate the package into the LL code prior to being able to merge it into their own code, allowing them to test new SDK releases in parallel with the Lab. These will hopefully include 64-bit binaries of the SDK as well a 32-bit versions.

Group Ban Lists

[48:23] It had been hoped that Baker Linden might be providing an update on the overall status of his group ban list work. However, this was unfortunately not the case, although Oz provided a small update on things, stating that he has been able to sit down with Baker to review the updated viewer code, which is now with QA. Hopefully this means it will be appearing in at least a project viewer in the near future.

Continue reading “SL project updates 7/3: viewer updates, AIS and misc items”