The following notes are drawn from the TPV Developer meeting held on Friday October 10th, and shown in the video above. Time stamps, where relevant, have been included for ease of reference to the video. Note that items are listed according to subject matter, rather than chronologically, so time stamps may appear out-of-sequence in places. My thanks as always to North for the recording.
SL Viewer
HTTP Pipeline Viewer
[0:26:35] As noted in part 2 of this week’s report, the HTTP pipelining viewer was looking close to being ready for deployment as an RC viewer, possibly in the next week. However, it hit a final QA snag, and in Oz’s words, “Monty is busily correcting the problem and getting it back in the queue.” Or as Monty put it in chat “:(“. Apparently a workaround for a CURL bug “tends to disable pipelining”.
Benchmark and Experience Tools Viewers
[0:27:20] It is hoped that these two project viewers will be updating and moving into the viewer release channel as release candidates “pretty soon”.
Viewer Build Tool Chain
[0:41:51] The tools upgrade project for building the viewer has been subject to a few delays. Currently, the Lab has just about finished putting together all the build prerequisites for building the Mac version of the viewer directly on OS X 10.9, and are about to commence test builds of the viewer using the new tool chain. This has also led to some progress being made on updating the Linux build process as well. The windows environment will require further work, so it will be a little longer before things are fully in place.
Group Chat
[0:28:58] Again, as noted in part 2 of this week’s report, the latest updates for group chat are being deployed to various back-end chat servers by the Lab, and may be deployed to all of the chat servers in the next week.
While investigating group chat, the Lab has noticed that in general it is “unbelievably spikey”, with chat sessions having peaks of really good performance followed by troughs of really bad performance regardless as to whether the groups are regarded as being “good” or “bad” in handling group chat. There is no real pattern to these peaks and troughs, other than the larger the group, the more it seems to swing between the two extremes, and nothing to correlate them with anything in particular beyond the bad times occurring when a lot of people in a group are online.
Z-offet height Adjustment
[0:31:52] Work is continuing on the z-offset height proposal, which Vir Linden has been engaged upon. However, it appears the Lab has encountered some issues which have made it “a little more complicated” than had initially be thought. hopefully, these will be overcome, and they won’t bring the work to a halt.
CDN
[0:32:28] As noted in part 2 of this week’s report the number of regions on Snack and utilising the Highwinds CDN for mesh and texture data servicing had reached around the 270 mark, but has since come down a little as a result of the Lab overloading the Snack sim hosts with regions running high volumes of users.
Metrics gathered by the Lab have been positive, and even though the Snack sim hosts were initially overloaded, the Lab feel they performed significantly better thanks to texture and mesh fetching being off-loaded to the CDN than would have been the case had the “old” method of texture / mesh fetching still been in use.
One aspect of the move to using the CDN is that until now, the sim host Apache service was being used for texture and mesh data handling and “lots” of other things which are timing critical to operations such as region crossings. With the move to the CDN, much (if not all) of the texture and mesh data handling is removed from the Apache service, making it easier for it to better handle time-critical activities.
If all goes according to plan, the CDN support will be expanded to the BlueSteel release candidate channel in week 42 (commencing Monday October 13th). This will allow the Lab to gain data on performance using the CDN support which can be directly compared with historical data available for region / sim host performance in BlueSteel. Those regions already running on Snack will continue to do so alongside of BlueSteel, so that around 5% of the main grid will be using the CDN service.
Goodnight Photography, Berlengas (Flickr) – blog post
Viewer Managed Marketplace
[0:00:28] The Lab has announced further changes to the Marketplace and Direct Delivery. These changes will commence a long lead-in period starting around the end of October / beginning of November 2014, before a full switch-over occurring in around March 2015. Rather than cover them here, I have a separate blog post providing an overview of what is being planned, including audio comments from Brooke Linden.
Other Items
AIS v3 Issues
[0:49:49] The Firestorm team are working on the next release of their viewer, which may be available at the end of November or early December as they try to keep to their 3-month release cycle. However, concerns have been raised around a series of issues which appear to be related to AIS v3, and which are serious enough for the team to feel that they will result in user upset were Firestorm to release a version of their viewer with AIS v3 included. The bugs specifically mentioned comprise:
- BUG-6197 “Avatar distorted when changing outfits”, which has in turn apparently spawned:
- BUG-7433 “Change in Maintenance Viewer breaks my joint rigged mesh avatar” (version 3.7.17.294943)
- BUG-7436 “[Maintenance-RC] Attachments appear in wrong position for observers on the Maint-RC but appear in the correct position to yourself on the Maint-RC. Observers on default release see attachments correctly” (version 3.7.17.294943)
- BUG-6439 “Shape distortion when wearing / taking off rigged mesh with joint offset (BUG introduced in version 3.7.9.290582)”
It is thought that a fix for BUG-6197 (internally referenced as MAINT-4158 by the Lab) is available (and is listed in the latest Maintenance release viewer, version 3.7.17.294943), although it has apparently spawned two additional bugs, as noted above. There may in addition be further work forthcoming on this issue (currently within the Lab’s internal viewer-lion branch), together with a fix for BUG-6439 (referred to internally by the Lab as MAINT-4189). It is also thought that fixes for joint offset issues need to be propagated back into the appearance service before they’ll be completely fixed, and this has yet to occur.
In addition, the following bugs were also mentioned:
- BUG-6487 “‘Wear’ Behaves Erratically For Attachments on Same Point (this has been imported by the Lab as MAINT-4196, but is still awaiting a fix)
- BUG-6908 “Attachments scripted to llDetachFromAvatar on region change show as “Worn on invalid attachment point” in inventory after region change on AIS3 enabled viewers only”
- BUG-6925 “HUDs and attachments intermittently and randomly detach after teleports, sometimes reattaching on their own shortly after, sometimes staying detached completely, or showing as “worn on Invalid Attachment Point” while still detached”
- BUG-7131 “Unexpected behaviour of on_rez event and llDetachFromAvatar()”
The last three bugs may be related to changes the Lab has been making in support of Experience Keys, and may require adjustments on the server-side. Further updates to follow.
Changing Group Tags / Scene Load Issues
As noted in my week 39 update, some of the recent interest list changes have resulted in various additional scene rendering issues, including BUG-6299, wherein changing your group tag while the scene around you is still rendering causes the region update top stop, and the scene failing to load. The only way to correct this when it happens is to relog.
The Lab is looking into this, although beyond confirming that there is an interaction between changing your tag and receiving a full update from the region which causes the latter to fail, there is as yet no sign of a fix. So the workaround remains either a) changing your group tag before you teleport to a region (if you need to wear a specific tag there), or b) wait until the scene around you has loaded prior to accepting any invitation to join a group.
Inara Pey: Living in a Modemworld 
I have a suspicion that the CDN will need us to adjust viewer caching of textures. At the moment we can have a large texture cache on disk—up to 10GB for Firestorm—which can be fairly quickly checked to trigger a data fetch over the internet. But CDN can be modelled as a huge cache which takes a long time to check for a hit, and this might lead to a very slow response if a texture isn’t in the CDN system.
I’ve also been seeing BUG-6925 and wondering if it was a sign of a bad connection (teleports into and out of the Snack RC have been bad)
The mention of CDN taking a load off the sim-host Apache service, giving room for time-critical data traffic is a good sign, but if the timing is getting a bit odd, it still has to get into the Viewer. It might almost be as if we are running an independent data download. And there is some old stuff about the internet which suggests there are bad interactions between TCP/IP and UDP. Could the HTTP, arriving delayed by a few seconds because of a cache-miss, be trampling over time-critical UDP being sent into a datalink that the server thinks is running near-empty?
It needs somebody who knows more about both the proprietary Linden code than any of us, and is familiar with how the internet works.. And there is only one Monty Linden
LikeLike
You apparently understand this stuff better than I do; but we were told from the beginning that textures which had never been called to the CDN server you are accessing would take longer to load since the CDN server had to get it from LL then pass it on to you.
LikeLike