SL project updates week 37/1: server, viewer

GFrisland  - Laluna Island; Inara Pey, March 2014, on FlickrFrisland – Laluna Island (Flickr) – blog post

Server Deployments

As always, please refer to the server deployment thread for the latest information and updates.

There was no deployment to the Main (SLS) channel on Tuesday September 9th, largely as a result of there having been no RC deployments in week 36.

On Wednesday September 10th, the RC channels should all receive the same server maintenance project, which includes crash fixes and an “obscure bug”.

Describing the crash mode fixes at the Simulator User Group meeting, Simon Linden said, “The crash fixes aren’t normal usage but some edge cases that needed attention. The “obscure bug” was one that Maestro Linden had mentioned at the last Server Beta meeting, describing it as preventing Lindens from being able to leave Skill Gaming regions. Simon expanded on that saying, “It wasn’t just Lindens, but it was really anyone that didn’t have proper access.” Quiet what the proper access might be, isn’t clear.

A separate issue with Skill Gaming regions is that some who have had payment information files with the Lab for some time, but who have not actually used it for a period of time, may have to refresh their information prior to being able to access such regions.

SL Viewer

Release Viewer

Monday September 8th saw the Snowstorm contributions RC promoted to the de facto release viewer, version 3.7.15. 293376. This viewer includes assorted contributed fixes and updates, including the unified snapshot floater and support for ‘older than’ when inventory filtering).

The new snapshot floater by Niran V Dean is now available within the official viewer
The new snapshot floater by Niran V Dean is now available within the official viewer

There has been a report of this installation of this viewer pausing with the message “unable to find Second Life Crash Logger”, but that clearing the message will see the viewer complete installation OK. As there are reports that this has happened with other releases of the viewer and where allow automatic updates has been enabled, it might be connected to the automatic update process within the viewer itself, possibly as a result of the viewer not shutting down correctly as a part of the automatic update process, although this is by no means certain.

RC Updates

A new maintenance release viewer arrived in the release channel as an RC on Friday September 5th. Version 3.7.15.293253, which includes fixes for inventory and outfit management; appearance editing; group & group ban management; camera controls; multi-grid support for favourites; notifications management; stability, plus bug and crash fixes (download and release notes).

New Viewer Autobuild Process

The Lab’s new viewer autobuild process is now in the final phases of preparation for deployment, with Oz Linden reporting at the Open-source Dev meeting that there are only “minor fixes here and there” that seem to be required. There are expected to be some further updates to the new autobuild wiki page to assist self-compilers.

Other Items

Viewer Fixes in the Works

In week 34, I referred to a Mac Cocoa issue with theapplicationShouldTerminate function,   reported by Cinder Roxley, and which she described as:

Say you are logged into sl, and you open appstore and there is an update that needs a restart to complete. Normally you click restart and it shuts down all apps completes the update and when you login your apps are reopened. Because the viewer is sending a bad value back to OSX, OSX will not close it, and it stops the restart from happening.

The fix for this, submitted by Cinder under STORM-2053, is now under review, and so should be progressing forward soon.

STORM-2053, “Viewer no longer asks for confirmation before ejecting a member from a group like in previous releases, also no confirmation for banning”, also submitted by Cinder is also in the same state.

OPEN-268, “FFLOAD_XML missing on linux and darwin, FFSAVE_XML missing on darwin”, also from Cinder Roxley, may be a little more obscure, given it is apparently a long-standing issue which hasn’t previously been reported, but it to is progressing the Lab’s internal review phase.

Opting Out of Group Chat

As noted in my SL project updates, the Lab is working on trying to improve group chat (see the Group Chat tag list for articles). While this doesn’t form a part of the Lab’s work, Cinder Roxley has submitted STORM-2079, “As a group member, I would like to opt out of joining chat”.

The JIRA itself is still pending, but it highlights one of the ways in which some of the message load face by the group chat service might additionally be reduced. As previously noted, the major issue with group chat is not so much the volume of actual chat messages being sent, but with the number of updates the service must send as people log-in and out of SL, and as they join / leave group chat sessions. However, cumulatively, this still adds up to a lot of message traffic. So having a means by which users can opt-out of group chat altogether (and just receive notices, for example), could help reduce the load in terms of actual messages being sent and with the volume of updates for people joining / leaving group chat sessions.

Firestorm already provides a toggle for opting-out of group chat either on a group-by group basis from within a Group Profile floater, or en masse via options in Preferences > Chat > Firestorm. However, these options only cause the viewer to discard incoming messages, they don’t actually reduce the volume of messages being sent by the chat servers.

Firestorm includes options to discard group chat messages  - but this doesn't reduce the volume of messages being sent
Firestorm includes options to discard group chat messages – but this doesn’t reduce the volume of messages being sent

As a result of the JIRA and a brief discussion at the Open-source Dev meeting, there is likely to be some more poking around to identify where values relating to received group notices and chat are stored in the viewer with a view of seeing what might be leveraged towards helping reducing the volume of chat related traffic.

SL project updates week 36/2: server releases, CDN news, projectors and materials

Kats, Love Kates; Inara Pey, February 2014, on FlickrKats, Love Kats (Flickr / Blog post)

Note that the majority of information in this article was gathered at the Server Beta User Group meeting on Thursday September 4th, the transcript for which is available here.

Server Deployments, Week 36 – Recap

On Wednesday September 3rd, the Main channel received the server maintenance project previously deployed to the three RC channels. This comprises crash mode fixes and fixes for the following:

  • SVC-2262 – “Incorrect height value in postcard which sent from above 256m” (a postcard being a snapshot sent to e-mail)
  • BUG-6466 – “Numbers expressed in scientific notation and include a plus sign in the exponent are not parsed as JSON numbers by LSL”, which was thought to have been fixed a while ago, but which in fact resulted in BUG-6657 – “Valid JSON numbers like 0e0 no longer valid after 14.06.26.291532″, prompting the original fix to be rolled back.

There were no planned deployments to the RC channels for week 36.

Week 37 Releases

There will be a Release Candidate channels deployment in week 37 (week commencing Monday September 8th), which will comprise crash fixes and bug fixes. Interestingly, one of the fixes will be to prevent Linden personnel from getting stuck in the new Skill Gaming regions – which Maestro jokingly describes as, “once that fix is out, I’ll no longer have an excuse to play cards all day 🙂 .”

CDN Work

Map Tiles

April Linden has been doing further refinements to the use of the CDN environment for map texture fetching on Aditi (see my notes from week 35). This work is liable to be moving to a main grid RC in the near future, and is being viewed as a “good dry run  for using the CDN for texture and mesh fetching in the future”, according to Maestro.

Viewer Mesh Request Throttling

“During mesh fetch testing with the CDN, we realized that we were throttled by a viewer’s internal throttle,” Maestro said of the ongoing texture and mesh CDN testing currently underway on Aditi (again, please refer to my week 35 and  week 33 reports). He went on:

The viewer ‘only’ requests 100 meshes/second because the simulator has a similar throttle for answering those requests, but with the CDN, the simulator’s throttle is irrelevant. Monty did a special build of his viewer (I’m not sure if the change was permanent) which removed the throttle, and I benchmarked an average of ~365 meshes/second on the CDN region with it. Which, in my mind, is approaching “fast enough” 🙂 .

This probably means that when the CDN work is completed (which will include viewer-side changes at some point), the viewer’s throttle will likely remain, but will be set higher and perhaps with a debug setting. “Because,” as Maestro said, “if mesh loading got way too fast, eventually you might have viewer performance issues from the insane download speed.”

Yuzuru Jewell (of Kane projects fame), has been carrying out tests from Japan, which saw his mesh load speed double via the CDN when using an unmodified viewer.

 Other Items

HUD Detaching / Reattaching Following Teleport

This was first commented on in my week 32 meeting update, and referred to again in week 33 (both alongside BUG-6908). Commenting on the state-of-play with investigations into the issue, Maestro indicated that there is some thinking that it may have the same cause as BUG-7131 (Unexpected behaviour of on_rez event and llDetachFromAvatar() ), which is being looked into, although there is no news on a fix as yet.

Materials Rendering via Basic Shaders and Improving Projectors

In week 34, I carried news about Geenz Spad’s proposal for introducing materials rendering into basic shaders (i.e. so users would not necessarily need to have Advanced Lighting Model enabled in order to see materials), and to improve the functionality of projectors in Second Life.

The JIRA submitted for this work were respectively:

  • STORM-2077 – Add support for materials in basic shaders
  • STORM-2056 – Projector reflections do not respect the environment intensity parameter
  • STORM-2067 – Glossy Projectors

The news on adding materials to basic shaders isn’t currently encouraging. The most recent comment from Marissa Linden (August 20th) states, “Moved to STORM. However our internal engineers do not believe that this can safely be implemented.”

Gennz's work on projectors an glossiness: top: as test items currently appear in Firestorm. Bottom: as they appear in the test viewer
Geenz’s work on projectors an glossiness: top: as test items currently appear in Firestorm. Bottom: as they appear in the test viewer

News is a little more positive with the projector work, with a test viewer having been built, offering those who are interested with an opportunity to play with the updated projector capabilities. There is also a test area on Hippo Hollow.

SL project updates week 36/1: server releases

Server Deployments, Week 36

As always, please refer to the server deployment thread for the latest information and updates.

On Wednesday September 3rd, the Main channel should receive the server maintenance project previously deployed to the three RC channels. This comprises crash mode fixes and fixes for the following:

  • SVC-2262 – “Incorrect height value in postcard which sent from above 256m” (a postcard being a snapshot sent to e-mail)
  • BUG-6466 – “Numbers expressed in scientific notation and include a plus sign in the exponent are not parsed as JSON numbers by LSL”, which was thought to have been fixed a while ago, but which in fact resulted in BUG-6657 – “Valid JSON numbers like 0e0 no longer valid after 14.06.26.291532″, prompting the original fix to be rolled back.

Note that due to the Labor Day holiday in the US, this deployment is taking place a day later than usual.

There are no planned deployments to the RC channels for week 36.

SL Viewer

There have been no changes to any of the SL viewers in the release and project channels since my last SL projects update. They remain as per the SL viewer section of my Current Viewer Releases page.

SL project udates week 35/2: server; CDN; TPV meeting

NorderNey, Heaven Scent; Inara Pey, July 2014, on FlickrNorderNey, Heaven Scent (Flickr) – blog post

Server Deployments, Week 35 Recap

As always, please refer to the server deployment thread on the forums for the latest updates and information.

  • On Tuesday August 26th, the Main channel was updated with the server maintenance release previously deployed to all three RC channels in week 34, which contains a single crash fix.
  • On Thursday August 28th, after a delay from the planned deployment, the three RC channels were all updated with the same server maintenance package, which contains further crash mode fixes and, fixes for SVC-2262 – “Incorrect height value in postcard which sent from above 256m” (a postcard being a snapshot sent to e-mail) and BUG-6466 – “Numbers expressed in scientific notation and include a plus sign in the exponent are not parsed as JSON numbers by LSL”, which was thought to have been fixed a while ago, but which in fact resulted in BUG-6657 – “Valid JSON numbers like 0e0 no longer valid after 14.06.26.291532″, prompting the original fix to be rolled back.

The crash mode fix deployed on the RC channels in week 34 and the Main channel in week 35 is apparently related to Skill Gaming regions (SEC-1458).  Essentially, if you sent somebody a teleport offer from a Skill Gaming region, you could, depending on the circumstances, crash the region. Which sort-of sounds like a skill game in its own right …

The RC rolls were postponed 24 hours as a result of problems with the simulator deployment tool, rather than with the RC code itself. The postponement allowed the deployment team locate and fix the problem.

CDN and Map Tile Images

During the Server Beta meeting on Thursday August 28th, Maestro revealed that as well as looking to move mesh and texture fetching to a CDN, the Lab is considering using the same service to deliver map tile images for the world map. To test the idea, and while acknowledging it doesn’t have much of a load test, as the beta grid is much smaller than the main grid, map tiles are now being delivered to any viewers connected using Aditi via the CDN.

Hopefully, this should result in faster and more reliable map tile loading than when relying on the Amazon S3 servers.

Moving the delivery of world map tiles to the CDN should hopefully improve the loading time for the world map
Moving the delivery of world map tiles to the CDN should hopefully improve the loading time for the world map

Commenting on the current state of play with Aditi, Maestro said, “Aditi doesn’t have too many regions up, so there’s no exciting load test to do with a regular viewer, but for what it’s worth, I do see much lower latency when fetching map tiles from the CDN than from Amazon S3 (from my home, about 15ms vs 30-42ms, round trip).”

It’s not clear when this might be available on Agni, “I think it’s a matter of the ops team setting up an Agni-pointing CDN,” Maestro said.

The following notes are drawn from the TPV Developer meeting held on Friday August 28th, and shown in the video above. Time stamps, where relevant, have been included for ease of reference to the video. Note that items are listed according to subject matter, rather than chronologically, so time stamps may appear out-of-sequence. My thanks as always to North for the recording.

 SL Viewer

There are no major changes to the various LL viewers; the Experience Keys project viewer has yet to be updated, and the Oculus Rift viewer will be getting updated as the Lab continue to work with their Oculus DK2 sets.

The Snowstorm RC is  liable to remain in RC for a while yet, although it appears to be doing well in terms of crash rates. The Experimental log-in viewer is still available and being tested.

Autobuild work

[01:48] As reported following the last TPV Developer’s meeting, the Lab is updating the tool chain used to compile the Windows and Mac versions of the viewer, together with implementing a new viewer autobuild process. Referring to the latter at the TPV Developer meeting on Friday August 28th, Oz described the work as going “really well” and pointed to the fact that there are now two versions of the autobuild process that re “in theory” available for testing, “one that builds 64-bit, and one that does not.” These are liable to be merged in the near future.

Some TPV Devs have reported a few problems reported with the process, which may be library or environment related, but these still need to be looked into further to determine what the problems being experienced are and where they actually reside.

[03:50] There is some further work to be done on the new process, notably in making the handling of licenses and copyrights a lot stricter, but otherwise it is viewed as being very close to being ready to go. Oz hopes to have the latest bug fixes and updates merged into the process in week 36, and once that is done, the new autobuild process will be used in compiling the Lab’s viewers going forward as a part of the overall tool chain update.

Continue reading “SL project udates week 35/2: server; CDN; TPV meeting”

SL project updates week 35/1: server, viewer, unified snapshot floater and CDN

Matoluta Sanctuary, Sartre; Inara Pey, July 2014, on FlickrMatoluta Sanctuary, Sartre (Flickr) – Blog post

Server Deployments, Week 35

As always, please refer to the server deployment thread on the forums for the latest updates and information.

Main (SLS) Channel

On Tuesday August 26th, the Main channel was updated with the server maintenance release previously deployed to all three RC channels in week 34, which contains a single crash fix.

RC Channels

On Wednesday August 27th, all three RC channels should be updated with a new server maintenance package, which contains further crash mode fixes and, as indicated in the second part of my week 34 report, fixes for the following:

  • SVC-2262 – “Incorrect height value in postcard which sent from above 256m” (a postcard being a snapshot sent to e-mail)
  • A “re-fix” for BUG-6466 – “Numbers expressed in scientific notation and include a plus sign in the exponent are not parsed as JSON numbers by LSL”, which was thought to have been fixed a while ago, but which in fact resulted in BUG-6657 – “Valid JSON numbers like 0e0 no longer valid after 14.06.26.291532”, prompting the original fix to be rolled back.

SL Viewer

A new Snowstorm contributions RC arrived in the viewer release channel on Tuesday August 26th. The viewer, version 3.7.15.293295, includes a number of fixes for viewer issues and the new unified snapshot floater. As such, the list of contributions comprises:

  • OPEN-213 Broken header guard in llaudiodecodemgr.h
  • OPEN-217 Change to media_plugin_quicktime.cpp causes Windows compile to fail
  • STORM-2030 Rapidly clicking the refresh button in selected floaters may result in duplicate entries
  • STORM-2031 Display issues with Top colliders/Top scripts floater
  • STORM-2034 Support ‘older than’ when inventory filtering
  • STORM-2037 LSL syntax fetching for new keywords appears to be broken
  • STORM-2038 On the Edit tab of the build menu, clicking the area in between the Full Bright check box and the materials drop down box opens the color picker.
  • STORM-2040 Unified Snapshot floater
  • STORM-2076 Width spinner is broken in the Snapshot->Save to disk floater.
The new snapshot floater by Niran V Dean: note the button options for Flickr, Twitter and Facebook uploads
The new snapshot floater by Niran V Dean is now available in a new Snowstorm contributions RC viewer. Note the button options for Flickr, Twitter and Facebook uploads, the position of the filter option and the enlarged preview panel. Please refer to my preview of the floater for further details

The new snapshot floater provides an enlarged preview panel, plus buttons to access the Facebook, Twitter and Flickr upload floaters. However, what’s possibly going to be very popular among SL users is the ability to use the filter options with snapshots saved to inventory, disk or to e-mail – and the ability to use them on snapshots to be uploaded to the SL profile feed. Further information on the floater can be found in my preview article about it.

While there have been no changes to either the Experience Keys project viewer or the Oculus Rift project viewer, an update to the Experience Keys viewer is also anticipated soon.

Texture and Mesh CDN

Testing is continuing on the Aditi set-up to test the initial configuration of the proposed texture and mesh fetching CDN, and results are seemingly positive.As noted in my last reports, if successful, this approach will see texture and mesh fetching bypass the simulator entirely, being routed instead directly between the viewer and asset servers via the CDN, which should see improvements in the speed and reliability of such transfers.

Given the CDN nodes are located around the globe, they offer significantly faster response times than trying to reach the SL servers. For example: ping times to the SL servers from the East coast US or the UK can be measured at an average of around 110ms or 180 ms respectively; ping times to a local CDN however, have been reported at 10ms and 30 ms respectively.

The way the system works means that when a new asset (with a new ID) is created, it is initially held by the Lab’s servers. However, the first time the asset is called, it is delivered to the CDN nearest to the person calling it, and cached there, and once cached, is then served locally whenever called by anyone connecting to that CDN, thus making for faster delivery to the viewer.

Whirly Fizzle has been driving a lot of the tests, carrying out direct comparisons in texture fetching via the CDN and via the current method of running things through the simulator, and her results are promising:

The results from Whirly Fizzle's testing with the CDN and classi download mechanism for textures, using the test regions on Aditi - click for full size, or go here for the original document
The results from Whirly Fizzle’s testing with the CDN and classic download mechanism for textures, using the test regions on Aditi – click for full size, or go here for the original document

However, a couple of points need to be remembered when looking at these figures. Firstly, the work is still in its infancy, and there is a way to go before the Lab is in a position to announce anything is ready for prime-time use. The test environment, while producing valid figures, isn’t something that will scale particularly well, a fact which caused Oz Linden to comment at the Simulator User Group meeting that while the Lab does have a plan for building a scalable solution, hasn’t as yet actually been built.

An additional point worth noting is that while the test environment doesn’t require a special viewer (any viewer will work in the test regions on Aditi), any solution which will eventually be deployed will require viewer-side updates. To ensure that all viewers can leverage the new system as quickly as possible when both it and the viewer changes are ready, the Lab plans to release the viewer updates separately from everything else so that TPVs can quikcly merge them into their own viewers.

Even so, the Lab welcomes people testing the new set-up, as it helps them to gather more data. Details on the testing and the Aditi regions can be found in the Server Beta meeting agenda notes.

Oz is particularly excited about this work, commenting, “It’s something I’ve wanted to see done since I got here.”

Other Items

Not strictly an SL project update, but something related to SL project work.

Loki Eliot has posted a couple of blog posts related to Experience Keys / Tools. In the first, he provides notice that his Escapades Island has now been updated to use Experience Keys – which should make for some interesting fun! In the second, he provides some feedback on his dabbling with Experience Keys, highlighting one or two shortfalls which, while not devastating to the system, do point at least one area where the Lab might want to consider extending the permissions system (seating avatars, which might need a little careful thought), and on how things might be improved on the Oculus Rift side of things for those wanting a really immersive experience with Exp Keys.

SL projects Updates week 34/2 server, texture / Mesh CDN, group chat

Santaurio, Cala del Barronal; Inara Pey, April 2014, on FlickrSantaurio, Cala del Barronal (Flickr) – blog post

Server Deployments week 34 – Recap

There was no deployment on the Main (SLS) channel on Tuesday August 19th. All three RC channels received the same server maintenance project  on Wednesday August 20th, aimed at fixing a crash mode.

There may be news on the crash mode fixed in the RCs once it has deployed to the Main (SLS) channel in week 35.

Upcoming Server Deployments

There will be a new server maintenance package deployed to the RCs in week 35, which includes a couple of visible changes:

  • SVC-2262 – “Incorrect height value in postcard which sent from above 256m” (a postcard being a snapshot sent to e-mail)
  • A “re-fix” for BUG-6466 – “Numbers expressed in scientific notation and include a plus sign in the exponent are not parsed as JSON numbers by LSL”, which was thought to have been fixed a while ago, but which in fact resulted in BUG-6657 – Valid JSON numbers like 0e0 no longer valid after 14.06.26.291532. prompting the original fix to be rolled back.

SL Viewer News

TPV developer Niran V Dean has been working on a new unified snapshot floater which encompasses the “standard” floater, plus the Flickr, Twitter and Facebook upload options, under STORM-2040. Those who use the most recently releases of Niran’s Black Dragon viewer will be familiar with the approach, as he initially worked on the idea in that viewer.

NiranV Dean has been working on a more unified snapshot floater, which is currently undergoing testing. It will hopefully appear soon
NiranV Dean has been working on a more unified snapshot floater, shown here in the viewer window, which is currently undergoing testing. It will hopefully appear soon (click for full size)

The work is now progressing to a point where we should be seeing the fruits of his labour in the near future. In the meantime, I’ve previewed the work as it stands at the moment.

Texture and Mesh Fetching CDN

As reported in my last TPV developer meeting update, the Lab is looking to move texture and mesh fetching to a content delivery network (CDN). If successful, this approach will see texture and mesh fetching bypass the simulator entirely, being routed instead directly between the viewer and asset servers via the CDN, which should see improvements in the speed and reliability of such transfers.  Explaining the new approach further at the Server Beta meeting on Thursday August 21st, Maestro Linden said,

The viewer currently fetches all meshes and textures through the sim, which gets them from the asset server, and the sim gives the viewer a ‘capability’ URL, which the viewer uses for fetching. However, with this change to use the CDN, the sim instead gives the viewer the URL of the CDN, and the CDN has hosts all over the world. 

It’s good for two reasons: 1) the sim isn’t burdened with texture/mesh transfers; 2) you’ll often have lower latency to the CDN than the sim, which means more speed.

Allowing for the fact the Lab is accessing the service through their own network (although the CDN is a commercial service), both Maestro and April Linden report it as being a noticeable improvement on things, with texture and mesh fetching having double the performance compared to the current means of fetching via the simulator. Initial results of testing from Europe show similar improvements.

The Server Beta meeting agenda has further information on new method for mesh and texture fetching, including details on the Aditi stress test regions (one for textures, one for meshes), and those wishing to try them out are invited to do so. No special viewer is required in order to carry out testing at present, and the agenda includes notes on what to do. Note that the test regions are set to no-build so that people don’t rez extra things that would skew results; they are also likely to be limited in terms of the maximum number of avatars able to access them at one time.

If / when the new approach is more broadly rolled-out, people will be able to see which service (CDN or via the simulator) they are using is to set Develop > Consoles > Capabilities Info To Debug Console. Those using the CDN will see the GetMesh, GetMesh2, and GetTexture URLs will all be  http://asset-cdn.aditi.lindenlab.com/, while those using the current method will see it give some URL pointing at the sim host. However, we’re still some way from seeing the new service deployed further than the Aditi test regions.

Should this work prove successful, and once it and other HTTP work such as pipelining, as being developed by Monty Linden, is completed, the Lab hopes that they’ll have a fast, robust series of HTTP services such that they can look to retire UDP texture fetching – although this will be some way down the road, and in the interim, UDP will offer people something of a fallback for texture fetching should they have issues with HTTP as the various new services are deployed.

 Skill Gaming Regions

Simon Linden reported that the first of the Skill Gaming regions has arrived on Agni (the main grid). Called Crunchy, it doesn’t have any gaming parlours or anything on it, but appears to be set-up for testing (such as accessibility). There are a few things going on there, most under the control of Gecko Linden. Also, and as pointed-out by Simon, the first skill games and operators have started to appear on the Lab’s Skill Gaming Participants wiki page.

Group Chat

Work has resumed on group chat after a brief pause, and the Server Beta meeting saw a very brief test take place. The aim of this was to test delays that have been introduced into the members list updates sent by the chat server.

As I’ve previously reported, one of the biggest issues of chat delays in group chat sessions is to do with the numbers of updates the chat server has to send as people join / leave session and log-in / out of SL, changing their online status within the groups they’re a member off. Recent changes to the code are intended to queue these updates and reduce the load they are placing on the servers, interrupting the flow of text messages.

The test was brief, but appeared to give Simon Linden enough information to be able to go back and poke at things some more.

Group Chat Server Issues

There have been further reports of group chat servers at times becoming non-responsive. This issue was initially raised in week 33, after the server supporting all group chats with a key starting with “b”. A further issue was identified at the start of week 34 affecting the server supporting all group chats with a key starting with “d”. While the Lab is aware of ongoing problems, there is also a request for JIRAs to be submitted on specific issues.