Tag: Group Chat

SL project updates week 41/3: TPV Developer meeting

Posted on by Inara Pey

The following notes are drawn from the TPV Developer meeting held on Friday October 10th, and shown in the video above. Time stamps, where relevant, have been included for ease of reference to the video. Note that items are listed according to subject matter, rather than chronologically, so time stamps may appear out-of-sequence in places. My thanks as always to North for the recording.

SL Viewer

HTTP Pipeline Viewer

[0:26:35] As noted in part 2 of this week’s report, the HTTP pipelining viewer was looking close to being ready for deployment as an RC viewer, possibly in the next week. However, it hit a final QA snag, and in Oz’s words, “Monty is busily correcting the problem and getting it back in the queue.” Or as Monty put it in chat “:(“. Apparently a workaround for a CURL bug “tends to disable pipelining”.

Benchmark and Experience Tools Viewers

[0:27:20] It is hoped that these two project viewers will be updating and moving into the viewer release channel as release candidates “pretty soon”.

Viewer Build Tool Chain

[0:41:51] The tools upgrade project for building the viewer has been subject to a few delays. Currently, the Lab has just about finished putting together all the build prerequisites for building the Mac version of the viewer directly on OS X 10.9, and are about to commence test builds of the viewer using the new tool chain. This has also led to some progress being made on updating the Linux build process as well. The windows environment will require further work, so it will be a little longer before things are fully in place.

Group Chat

[0:28:58] Again, as noted in part 2 of this week’s report, the latest updates for group chat are being deployed to various back-end chat servers by the Lab, and may be deployed to all of the chat servers in the next week.

While investigating group chat, the Lab has noticed that in general it is “unbelievably spikey”, with chat sessions having peaks of really good performance followed by troughs of really bad performance regardless as to whether the groups are regarded as being “good” or “bad” in handling group chat. There is no real pattern to these peaks and troughs, other than the larger the group, the more it seems to swing between the two extremes, and nothing to correlate them with anything in particular beyond the bad times occurring when a lot of people in a group are online.

Z-offet height Adjustment

[0:31:52] Work is continuing on the z-offset height proposal, which Vir Linden has been engaged upon. However, it appears the Lab has encountered some issues which have made it “a little more complicated” than had initially be thought. hopefully, these will be overcome, and they won’t bring the work to a halt.

CDN

[0:32:28] As noted in part 2 of this week’s report the number of regions on Snack and utilising  the Highwinds CDN for mesh and texture data servicing had reached around the 270 mark, but has since come down a little as a result of the Lab overloading the Snack sim hosts with regions running high volumes of users.

Metrics gathered by the Lab have been positive, and even though the Snack sim hosts were initially overloaded, the Lab feel they performed significantly better thanks to texture and mesh fetching being off-loaded to the CDN than would have been the case had the “old” method of texture / mesh fetching still been in use.

One aspect of the move to using the CDN is that until now, the sim host Apache service was being used for texture and mesh data handling and “lots” of other things which are timing critical to operations such as region crossings. With the move to the CDN, much (if not all) of the texture and mesh data handling is removed from the Apache service, making it easier for it to better handle time-critical activities.

If all goes according to plan, the CDN support will be expanded to the BlueSteel release candidate channel in week 42 (commencing Monday October 13th). This will allow the Lab to gain data on performance using the CDN support which can be directly compared with historical data available for region / sim host performance in BlueSteel. Those regions already running on Snack will continue to do so alongside of BlueSteel, so that around 5% of the main grid will be using the CDN service.

Continue reading “SL project updates week 41/3: TPV Developer meeting” →

SL project updates 41/2: server, HTTP, Group Chat

Posted on by Inara Pey

Pigeon Island, Neverending; Inara Pey, June 2014, on FlickrPigeon Island, Neverending (Flickr) – blog post

The following notes are taken from the Server Beta meeting held on Thursday October 9th, 2014. The transcript is available here, and the support agenda notes here.

Server Deployments Week 41 – Recap

  • Server deployment thread
  • The Main (SLS) channel and the Snack RC received the same server maintenance package as had been deployed to the three primary RCs in week 40,which fixes a bug related to viewing parcel details in gaming regions
  • There was no deployment to any of the primary RC channels (LeTigre, BlueSteel and Magnum).

CDN and the Snack RC

There are now some 270 regions on the Snack RC and which use the CDN for texture and mesh data fetching. The majority of these are mainland regions, although there is still no public list of all of them. Those who would like their region added to Snack are asked to contact the Lab at cdn-test@lindenlab.com.

If you’re using a TPV that can display region details in a dialogue box following a region crossing, you can use it to identify regions using the CDN, or you can check the About Land floater for the region you’re in. In both cases look for “Second Life RC Snack” in the region’s descriptive text. If you see it, the region is using the CDN.

Group Chat

Simon Linden’s work to both reduce the volume of additional group chat information messages (people joining leaving group chat sessions, updates to the group members list as people log-in or out of SL), and the frequency with which they are sent appears to have satisfied the Lab that they will produce some measure of improvement. There has already been a deployment of the updates to some of the back-end chat servers, and according to Maestro Linden, it is anticipated they should be on all of the chat servers some time in the next week. There may be more to add to this following the TPV Developer meeting.

HTTP Pipelining

Monty Linden: HTTP guru
Monty Linden: HTTP guru

The new HTTP pipelining viewer (referred to by the Lab’s QA team as the “weaponized” viewer, it is apparently so slick), is due to hit RC status. “Project Drano”, as Monty Linden, who has been spearheading the HTTP work over the last 2+ years, jokingly calls it, offers both improvements of its own and should further assist in data downloads via the CDN. Commenting on the viewer, Monty said:

The next viewer adds HTTP pipelining support to mesh and texture fetches. This allows multiple asset requests to be issued without waiting for intervening responses from the server, and it goes very far towards making client-to-server ping time irrelevant in throughput metrics. It’s a huge step on its own making our services perform as well as they can. Combined with CDN it’s even better. I won’t mention specific numbers but a region can show up in a very few seconds … Right now, we plan on skipping the project viewer stage. We’re going out the door [as an RC] once it passes QA (maestro heading that up).

Mesh fetching via the CDN should see an improvement because the viewer-side throttle of 100 meshes/second has been removed (this won’t affect non-CDN regions, as the throttle is also applied server-side on those).

As well as the core HTTP work, Monty has also done some work on inventory fetching within this viewer, particularly the code which is used to initially populate the inventory floater. This has been converted to use HTTP, and reduces the number of connections it uses. Again, no precise figures were given, but it should lead to improvements in inventory loading in situations where inventory data has been removed from cache (see the Firestorm wiki for information on removing inventory data files from cache). In one test with no inventory data, Monty saw a 100K inventory load around 10 times faster with the new viewer.

Whirly Fizzle (based in the UK) carried out inventory load tests for an 105K inventory between what was at the time the SL release viewer and a pre-release version of the HTTP pipelining viewer, both starting from a clear cache, and achieved some impressive results:

  • Second Life 3.7.16 (294015) Sep 10 2014 11:08:26 (Second Life Release)
    • Session 1: 16 mins 28 secs
    • Session 2: 17 mins 53 secs
    • Session 3: 17 mins 18 secs
    • Session 4: 17 mins 51 secs
  • Second Life 3.7.17 (294571) Sep 26 2014 12:32:36 (HTTP pipelining viewer)
    • Session 1: 2 mins 29 secs
    • Session 2: 2 mins 17 secs
    • Session 3: 2 mins 11 secs
    • Session 4: 2 mins 27 secs

I tend to have a very organised inventory, old / no longer used items are boxed, and anything not used in 6 months is similarly boxed (see: zdrop). Thus I only have an “active” inventory of around 10K (9,140 items at the moment), out of a total of around 70-100K. However, even with a “small” inventory like that, and working with a cleared cache, my results were impressive. The latest project version of the HTTP pipelining viewer (3.7.18.295146) averaged 9-10 seconds for inventory to download compared with an average of 2 minutes 50 seconds to 3 minutes for the current release viewer (3.7.17.294959).

Again, overall mileage on inventory downloads will vary, hence why Monty is hesitant to mention figures. However, it is thought that those with excessively “flat” inventories should see particular benefit (although they’d also be better off nesting their inventory into groups of folders).

SL project updates week 40/2: server, viewer, CDN news, group chat update

Posted on by Inara Pey

Square Pegs, Round Holes, Kashmir Dreams; Inara Pey, September 2014, on FlickrSquare Pegs, Round Holes, Kashmir Dreams (Flickr) – blog post

Server Deployments Week 40 – recap

As always, please refer to the server deployment forum thread for the latest news and updates.

  • On Tuesday September 30th, the Main (SLS) channel received the server maintenance package previously deployed to the three primary RC channels (BlueSteel, Magnum and LeTigre), and which focuses on the Experience Tools project
  • On Wednesday October 1st, the three primary RC channels all received the same new server maintenance package which fixes a bug related to viewing parcel details in gaming regions.

The RC update was to fix BUG-7329 “RemoteParcelRequest cap returned by a skill gaming region does not exist (returns HTTP error code 404)”.  In the official viewer, this capability is mainly used to show parcel script info, but some TPVs use it to show the parcel_id of parcels as well.

SL Viewer

The new log-in screen RC viewer was updated to version 3.7.17.294762 on September 29th, although it didn’t appear on the Alternate Viewers wiki page until either September 30th or October 1st.

A new Maintenance RC entered the viewer release channel on Thursday October 2nd. Version 3.7.17.294943 contains around 40 updates focused on voice, privacy, rendering, texture animation, avatar distortion, inventory management, sounds, and mouselook in Mac, together with a scripting crash fix and multiple UI fixes in script editor, Pay flow, chat, stats floater, edit menu etc. See the release notes (linked-to above) for details.

CDN News

Speaking at the Server Beta Meeting on Thursday October 2nd, Maestro Linden confirmed that there are now 28 regions using the CDN service for texture and mesh asset data. A full list of regions is still not available, but among the new additions are Morris, Dore, Ahern, and Bonifacio.

These regions are all running on the Snack RC. There is unlike to be any wider deployment to a larger RC in week 41 (commencing Monday October 6th), as the Lab is still fine-tuning their gathering data. However, there are still a few slots left on Snack for those wishing to have their regions added to it. Requests should be sent to cdn-test@lindenlab.com. Note that regions should be those with a relatively high texture / mesh count,

The CDN service being used by the Lab is operated by Highwinds Network Group, which the Lab has also been using in support of server-side appearance.

Highwinds supply their CDN service to a broad range of businesses, including a number of games companies such as Valve (Steam), Funcom, Meteor Entertainment, GameFly and Virgin Gaming. They operate 25 centres around the world, with 11 in North America, seven in Europe, 4 in Asia, one in Australia and two in South America, over their own network infrastructure, which they call “RollingThunder”, which  peers with more than 1,600 provider networks worldwide and over 14,000 ASNs, and uses the anycast network addressing and routing methodology.

Highwinds data centres - click for full size, or see additional information on the centres and Highwinds PoPs here (image courtesy of Highwinds)
Highwinds data centres – click for full size, or see additional information via Highwinds here (image courtesy of Highwinds)

Group Chat

As I’ve previously reported, one of the biggest issues of chat delays in group chat sessions is to do with the numbers of updates the chat server has to send as people join / leave session and log-in / out of SL (which causes an update to their status in the group member’s list).

Some work has already been carried out in ways to reduce the volume of update messages being sent with the aim of lessening the impact they have on the flow of actually text messages, and Simon Linden is attempting to further refine this work, again with the agin of reducing the volume of update messages and their impact on group chat test messages.

The Server Beta User Group meeting on October 2nd saw a further test of his work in order for him to gather data on the effectiveness of these changes, and feedback on his findings will likely be given at one of the user group meetings next week.

SL projects updates 38/1: server, group chat, CDN

Posted on by Inara Pey

The Colder Water, Daydreams; Inara Pey, August 2014, on FlickrThe Colder Water, Daydreams, (Flickr) – blog post

Server Deployments

As always, please refer to the server deployment thread for the latest updates.

  • On Tuesday September 16th, the Main (SLS) channel received the server maintenance package previously deployed to the three RC channels, which includes crash fixes and a fix for a bug that prevented some people from leaving Skill Gaming regions – notably Linden staff who entered a region using their admin powers …
  • On Wednesday September 17th, the three RC channels should each receive a new server maintenance package, which includes some bug fixes, and tweaks some behaviours related to experience tools.

CDN Micro Channel Deployment

The CDN micro channel deployment is due this week. As noted in mt last TPV meeting report, this will see a small number of regions updated to use the CDN test configuration for texture and mesh fetching, bypassing the simulator entirely. No viewer-side update is required for this (that will come in the future), so the benefits should be felt by anyone entering those regions (assuming the region is already cached by their local CDN node).

There currently isn’t a list of regions which will be included in the micro channel, although it has been suggested that the Lab consider including the Home and Garden Expo and the Kustom 9 event regions are included, as these all have considerable mesh and texture use. Two regions that will be included in the micro channel are Hippotropolis and Hippo Hollow.  The latter can be a reasonable text of texture downloads, as it has a wide variety of items sitting on it.

Group Chat

While Izzy Linden had reported something of a drop in the number of reports of requests for the back-end chat servers to be restarted at the end of week 37, reports continued through the weekend and the start of this week. The changes aimed at fixing the problem (essentially a network issue which causes the chat server to run out of ports) has been delayed, but commenting at the Simulator User Group meeting on Tuesday September 16th, Simon Linden said he hopes to see them deployed soon.

There is also a further update aimed at improving group chat performance that is also in the pipeline, the result of recent testing on Aditi; Simon Linden has “high hopes” for these changes once deployed.

Other Items

The Simulator User Group meeting on Tuesday September 16th saw renewed discussion on a number of subjects which have been previously raised, such as improvements to scripted sounds / sound functionality (see my January 2014 and November 2013 updates) and the environment improvements project. While no conclusions were drawn on specific improvements to the sound functionality (it doesn’t appear to be on the current roadmap), Oz Linden did confirm the environment improvements are still on the roadmap, but are not currently being worked on.  It’s not clear whether the SL roadmap will be made public, although it was indicated this might be the case when it was first raised a couple of months ago.

SL project updates week 37/2: miscellaneous items

Posted on by Inara Pey

Server Deployments, Week 37 Re-cap

  • There was no deployment to the Main (SLS) channel on Tuesday September 9th, largely as a result of there having been no RC deployments in week 36.
  • On Wednesday September 10th, the RC channels all received the same server maintenance project, which includes crash fixes and a fix for a bug that prevented some people from leaving Skill Gaming regions – notably Linden staff who entered a region using their admin powers …

SL Viewer

The Experience Keys projects viewer got its long-awaited update on Wednesday September 10th, with the release of version 3.7.16.293901. The viewer remains at a project viewer status, but now has parity with the current release viewer.

Group Chat

There is no major news here; as noted in part one of the week’s report, the Lab are still working on adjustments to the group chat code, but according the Maestro Linden, there isn’t anything as yet ready for further testing on Aditi – although the implication seemed to be that there may well be in the near future, as the comment on the work was made in terms of “upcoming stuff”.

In the meantime, people have been reporting further increases in group chat issues, prompting questions on whether increasing the number of servers used in support of the chat service might help. However, when this idea was last discussed, it was indicated that increasing the amount of available hardware isn’t considered to be a suitable solution at the moment, as it is the exponential manner in which ancillary messages (status updates, etc.) flood the service is seen to be the issue; something which just the provision of new hardware may not resolve / ease. As such, bringing the volume of the messages under better control is seen as key step in improving the service.

Other Items

Gesture and Object Sounds Parcel Restrictions

During the Server Beta meeting the subject of restricting sounds between parcels came up, with a request for a parcel setting to restrict sounds from other parcels being heard within a parcel. Currently, there is an option to restrict sounds from objects and gestures played within a parcel to that parcel, this can be found under About Land > Sound.

The option to restrict sounds from gestures and objects to a parcel will also block the same sounds originating from other parcels
The option to restrict sounds from gestures and objects to a parcel will also block the same sounds originating from other parcels from being heard within the parcel where set

The question led to an experiment with this option, which revealed, to the surprise of several at the meeting (myself included), that the setting is actually bi-directional: enabling it not only restricts sounds from objects and gestures being heard outside of a parcel, it blocks such sounds played from other parcels from being heard within the parcel as well. In some respects, that the options is bi-directional shouldn’t really have been a surprise; the other parcel options of a similar nature – restricting voice, restricting avatar visibility and chat, for example, are bi-directional, so why not gestures and sounds?

The option may not be perfect; looped sounds, for example, may not be restricted, and during a short test at the meeting,  some indicated they could still hear a sound generated on one parcel when moving from it to a parcel with the gesture and objects sounds restriction set, prompting Maestro Linden to muse if the restriction was enforced server-side or in the viewer. However, if you are troubled by sounds from a neighbouring or nearby parcel, it might be worth trying the option if you haven’t already.

SL project udates week 35/2: server; CDN; TPV meeting

Posted on by Inara Pey

NorderNey, Heaven Scent; Inara Pey, July 2014, on FlickrNorderNey, Heaven Scent (Flickr) – blog post

Server Deployments, Week 35 Recap

As always, please refer to the server deployment thread on the forums for the latest updates and information.

  • On Tuesday August 26th, the Main channel was updated with the server maintenance release previously deployed to all three RC channels in week 34, which contains a single crash fix.
  • On Thursday August 28th, after a delay from the planned deployment, the three RC channels were all updated with the same server maintenance package, which contains further crash mode fixes and, fixes for SVC-2262 – “Incorrect height value in postcard which sent from above 256m” (a postcard being a snapshot sent to e-mail) and BUG-6466 – “Numbers expressed in scientific notation and include a plus sign in the exponent are not parsed as JSON numbers by LSL”, which was thought to have been fixed a while ago, but which in fact resulted in BUG-6657 – “Valid JSON numbers like 0e0 no longer valid after 14.06.26.291532″, prompting the original fix to be rolled back.

The crash mode fix deployed on the RC channels in week 34 and the Main channel in week 35 is apparently related to Skill Gaming regions (SEC-1458).  Essentially, if you sent somebody a teleport offer from a Skill Gaming region, you could, depending on the circumstances, crash the region. Which sort-of sounds like a skill game in its own right …

The RC rolls were postponed 24 hours as a result of problems with the simulator deployment tool, rather than with the RC code itself. The postponement allowed the deployment team locate and fix the problem.

CDN and Map Tile Images

During the Server Beta meeting on Thursday August 28th, Maestro revealed that as well as looking to move mesh and texture fetching to a CDN, the Lab is considering using the same service to deliver map tile images for the world map. To test the idea, and while acknowledging it doesn’t have much of a load test, as the beta grid is much smaller than the main grid, map tiles are now being delivered to any viewers connected using Aditi via the CDN.

Hopefully, this should result in faster and more reliable map tile loading than when relying on the Amazon S3 servers.

Moving the delivery of world map tiles to the CDN should hopefully improve the loading time for the world map
Moving the delivery of world map tiles to the CDN should hopefully improve the loading time for the world map

Commenting on the current state of play with Aditi, Maestro said, “Aditi doesn’t have too many regions up, so there’s no exciting load test to do with a regular viewer, but for what it’s worth, I do see much lower latency when fetching map tiles from the CDN than from Amazon S3 (from my home, about 15ms vs 30-42ms, round trip).”

It’s not clear when this might be available on Agni, “I think it’s a matter of the ops team setting up an Agni-pointing CDN,” Maestro said.

The following notes are drawn from the TPV Developer meeting held on Friday August 28th, and shown in the video above. Time stamps, where relevant, have been included for ease of reference to the video. Note that items are listed according to subject matter, rather than chronologically, so time stamps may appear out-of-sequence. My thanks as always to North for the recording.

 SL Viewer

There are no major changes to the various LL viewers; the Experience Keys project viewer has yet to be updated, and the Oculus Rift viewer will be getting updated as the Lab continue to work with their Oculus DK2 sets.

The Snowstorm RC is  liable to remain in RC for a while yet, although it appears to be doing well in terms of crash rates. The Experimental log-in viewer is still available and being tested.

Autobuild work

[01:48] As reported following the last TPV Developer’s meeting, the Lab is updating the tool chain used to compile the Windows and Mac versions of the viewer, together with implementing a new viewer autobuild process. Referring to the latter at the TPV Developer meeting on Friday August 28th, Oz described the work as going “really well” and pointed to the fact that there are now two versions of the autobuild process that re “in theory” available for testing, “one that builds 64-bit, and one that does not.” These are liable to be merged in the near future.

Some TPV Devs have reported a few problems reported with the process, which may be library or environment related, but these still need to be looked into further to determine what the problems being experienced are and where they actually reside.

[03:50] There is some further work to be done on the new process, notably in making the handling of licenses and copyrights a lot stricter, but otherwise it is viewed as being very close to being ready to go. Oz hopes to have the latest bug fixes and updates merged into the process in week 36, and once that is done, the new autobuild process will be used in compiling the Lab’s viewers going forward as a part of the overall tool chain update.

Continue reading “SL project udates week 35/2: server; CDN; TPV meeting” →