2021 TPV Developer meeting week #15 summary

Centre for Mindfulness, February 2021 – blog post

The following notes are taken from the TPV Developer meeting held on Friday, April 16th, 2021.

These meetings are generally held every other week.  They are recorded by Pantera Północy, and her video of the meeting is embedded at the end of this report – my thanks to her for allowing me to do so – and it is used with a transcript of the chat log from the meeting and my own audio recording to produce these notes.

SL Viewer

[0:00-3:35]

The current run of official viewers is as follows:

  • Release viewer: Custom Key Mappings RC viewer, version 6.4.17.557391, dated March 24, promoted March 27 – No Change.
  • Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
    • Maintenance RC viewer – Eau de Vie, version 6.4.18.557782, dated April 12.
    • Love Me Render (LMR) 5 project viewer, version 6.4.18.557797, dated April 7.
  • Project viewers:
    • Legacy Profiles viewer, version 6.4.11.550519, dated October 26.
    • Copy / Paste viewer, version 6.3.5.533365, dated December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, dated November 22, 2019.
    • 360 Snapshot project viewer, version 6.2.4.529111, dated July 16, 2019.

General Viewer Notes

  • LMR 5 could now be set for promotion in the next week or so. No further active work is being carried out, and the current RC version has an “acceptable” crash rate.
  • Love Me Render 6 (LMR 6) is already in preparation with a further mix of fixes, including some that are EEP related (e.g. rendering the Moon with haze).
  • The current Maintenance RC viewer is also in good shape, and also have a further Maintenance RC rolling along behind it that is getting close to being ready for issuing.
  • The Legacy Profiles project viewer is now on hold pending the outcome of internal discussions related to it. These are related to whether or not there needs to be further UI changes within the viewer or changes made on the web side of things.
  • The Simplified Cache viewer has been overhauled, and should be re-appearing as an RC viewer very soon.
  • A viewer focused on correcting the problems of Apple notifications popping-up when trying to run the viewer on a Mac system is also getting close to appearing as a project viewer.
  • The “518” viewer (the Lab’s internal reference for an in-development viewer project) that has the UI updates specifically geared towards the new user experience changes LL have been working on is liable to be appearing “fairly Soon™”.

Fee Changes, 2FA and Map Tile Generation

[6:05-8:00 for map tile and 2FA / TPV comments]

The recently announced changes to Linden dollar “buy” fees, together with the upcoming (at some point in the future) shift to Two-Factor Authentication (2FA) has apparently given ground to worries it could spell the end of third-party viewers, and an update on map tile generation were all mentioned during the meeting.

All all three have broader interest than represented by those who follow viewer news, I’ve provided a separate summary of all three – please see: Linden Lab: L$ buy fee changes + 2FA and Map tiles news.

In Brief

  • [8:56-most of the rest of the meeting] Object spamming: the issue of widespread IM-to-e-mail spamming by a range of in-world objects (e.g. scanners that record your avatar UUID and then send out IMs on “offers” and “sales”, regardless of whether or not you’ve joined a related group) is a topic that does get periodically raised within the Lab, but tends to get pushed to one side while other issues are dealt with. It is possible something might be done to mitigate the problem, but further time is needed to determine what and how.

2021 TPVD meetings week #13: summary

The Handmaiden Estate – blog post

The following notes are taken from the TPV Developer meeting held on Friday, April 2nd, 2021.

These meetings are generally held every other week.  They are recorded by Pantera Północy, and her video of the meeting is embedded at the end of this report – my thanks to her for allowing me to do so – and it is used with a transcript of the chat log from the meeting and my own audio recording to produce these notes.

This as a very short meeting.

SL Viewer News

[1:13-4:55]

As noted in my CCUG summary, the Custom Key Mappings viewer, version 6.4.17.557391 was promoted to de facto viewer release status on Thursday April 1st.

The Key Mapping viewer adds the ability to assign your own keyboard short-cuts to a list of defined commands

The rest of the current pipeline of upcoming RC and project viewers remains as follows:

  • Release channel cohorts:
    • Maintenance RC viewer – Eau de Vie, version 6.4.17.557412, dated March 25.
    • Love Me Render (LMR) 5 project viewer, version 6.4.14.556118, dated February 23.
  • Project viewers:
    • Legacy Profiles viewer, version 6.4.11.550519, dated October 26,   2020.
    • Copy / Paste viewer, version 6.3.5.533365, dated December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, dated November 22, 2019.
    • 360 Snapshot project viewer, version 6.2.4.529111, dated July 16, 2019.

General Viewer Notes

  • LMR 5 is being used to address crashes relating to older hardware using Intel Graphics drivers. These have proven difficult to track down as almost nothing is reported on where in the viewer the issue causing the crash occurred. In the meantime, those on systems using older Intel HD graphics drivers are encouraged to update to more recent versions.
  • Both LMR-5 and the Maintenance RC are awaiting merges with the new release viewer code.
  • The Simple Cache Viewer has been updated following the forced roll-back, and should be appearing as a new RC viewer some time in the next week.
  • The recent hold-up in the viewer update and release cycle means that there are now a number of Maintenance RC viewers, each with its own focus, awaiting initial release.
  • A further Lover Me Render viewer (LMR-6) is also in development,  with around 8 known issues awaiting resolution before it can be considered for initial release as an RC viewer.

In  Brief

  • [7:20-8:06] the re-working of the Map Tile code continues. It is believed “significant progress” has now been made, but no ETA on when the fully revised code will be in operation.

2021 CCUG and TPV Developer meetings week #11 summary

Osta Nimosa – blog post

The following notes were taken from my audio recording and chat log of the Content Creation User Group (CCUG) meeting held on Thursday, March 18th 2021 at 13:00 SLT, and Pantera’s video recording of the TPV Developer’s meeting of Friday, March 19th, a copy of which is embedded at the end of this article.

These meetings are chaired by Vir Linden, with dates available via the SL Public Calendar. The venue for the CCUG is the Hippotropolis camp fire, and the TPV Developer meeting is held at the Hippotropolis Theatre.

SL Viewer

There have been no changes to the current pipeline of SL viewers since the update to the Key Mappings viewer at the start of the week. This leaves the pipelines as follows:

  • Release viewer: version 6.4.13.555567 (Jelly Doll improvements) originally promoted February 17th.
  • Release channel cohorts:
    • Custom Key Mappings project viewer, version 6.4.17.556726, dated March 15.
    • Maintenance RC viewer – Eau de Vie, version 6.4.14.556149, dated March 4.
    • Love Me Render (LMR) 5 project viewer, version 6.4.14.556118, dated 23, 2021.
  • Project viewers:
    • Legacy Profiles viewer, version 6.4.11.550519, dated October 26.
    • Copy / Paste viewer, version 6.3.5.533365, dated December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, dated November 22, 2019.
    • 360 Snapshot project viewer, version 6.2.4.529111, dated July 16, 2019.

Viewer Status

  • The Simple Cache viewer is being updated and will re-enter circulation as a new RC viewer. Depending on the outcome of further testing this man or may not be the next viewer promoted to release status.
  • The other RC viewer with the potential to be promoted is the Key Mappings Viewer.
  • LMR 5 has encountered some additional crash issues  centred on Intel GPU drivers, and so is unlikely to be in line for promotion at present.

Graphics Work

The graphic team is addressing bugs relating to lighting underwater and to the Moon haze. This work, together with the LMR 5 issues mean Euclid Linden’s work to separate out UI rendering from scene rendering is currently on hold.

ARCTan

Summary: An attempt to re-evaluate object and avatar rendering costs to make them more reflective of the actual impact of rendering either in the viewer. The overall aim is to try to correct some inherent negative incentives for creating optimised content (e.g. with regards to generating LOD models with mesh), and to update the calculations to reflect current resource constraints, rather than basing them on outdated constraints (e.g. graphics systems, network capabilities, etc).

As of January 2020 ARCTan has effectively been split between viewer renderings focused on revising the Avatar Rendering Cost (ARC) calculations and providing additional viewer UI so that people can better visibility and control to seeing complexity. This will be followed in the future by work on providing in-world object rendering costs (LOD models, etc.) which might affect Land Impact will be handled as a later tranche of project work, after the avatar work.

This project has reached a point where consideration needs to be given to how performance controls that can leverage the avatar-related ARCTan data can be implemented. However, this work is waiting on Steeltoe Linden.

In Brief

  • There was more general discussion on improved avatar scaling  – uniform / proportional scaling, etc. However, as has been pointed out in the past, the general design of the avatar skeleton, coupled with the morphing capabilities (sliders) do not make uniform scaling easy to implement.
    • One of the calls for making such scaling possible is to allow users scale down their avatars so that regions feel much “bigger”  and thus can present larger settings. However, this view ignores the fact that there are other practical constraints on the region and the underpinning simulator that mean just because avatars are smaller, “more” can be packed into a given space.
  • The majority of the meeting was general spitballing on options for revising the avatar per above, requests to implement Marvelous Designer, providing morph targets, and so on. However, none of the chat related to projects the Lab are currently working on or plan to implement in the foreseeable future.
  • The TPV Developer meeting amounted to some 6 minutes of discussion, ergo no timestamps to the video.

Date of Next Meetings

  • Content Creation: Thursday, April 1st, 2021.
  • TPVD: Friday, April 2nd, 20221.

2021 CCUG and TPV Developer meetings week #9

Wildwood Gardens, January 2021 – blog post

The following notes were taken from my audio recording and chat log of the Content Creation User Group (CCUG) meeting held on Thursday, March 4th 2021 at 13:00 SLT, and Pantera’s video recording of the TPV Developer’s meeting of Friday, March 5th, a copy of which is embedded at the end of this article.

The majority of the TPVD meeting was given over to a discussion of the texture cache issue, together with a general discussion on the simplified cache structure and also on Jelly Doll avatars.

These meetings are chaired by Vir Linden, with dates available via the SL Public Calendar. The venue for the CCUG is the Hippotropolis camp fire, and the TPV Developer meeting is held at the Hippotropolis Theatre.

SL Viewer

[From the CCUG meeting & TPVD Meeting]

Release Roll-Back & Issues

  • The Simple Cache viewer, version 6.4.14.556088, had to be rolled back following its March 2nd promotion due to a number of bugs (e.g. cache location not being adhered to – BUG-230337; uploads of textures above a certain size were failing – BUG-230295).
  • The roll-back saw the Jelly Doll improvements viewer, version 6.4.13.555567 (originally promoted on February 17th) revert to being the de facto release viewer.
  • However, the roll-back resulted in some users who had updated to the Simple Cache viewer users experiencing a corrupted texture cache when using an older version of the viewer.
  • Unfortunately, LL are having problems consistently reproducing the texture caching issues, so it is unclear how matters will be resolved.

This means that for the meantime:

  • Those experiencing texture issues on the official viewer as a result of having used the Simple Cache viewer and then rolling back to, or installing, another version should try clearing cache and re-starting the viewer.
  • Until the Lab have more of a handle on the texture cache issues:
    • Those RC viewers that had been internally merged with the Simple Cache viewer will also be rolled-back to an earlier version.
    • Internal viewer testing procedures will be reviewed.
    • TPVs that have also merged the Simple Cache code will also likely need to roll back their code.
  • One suggestion for resolving this issue is for the official viewer to increment the texture cache version number, forcing a complete wipe / reset of the cache
  • Independent of the texture caching issues, and due to BUG-230337 and BUG-230295, the Simple Cache viewer will go back to a development status so the issues can be fixed.
    • Firestorm apparently has fixes for these particular issues, and these may be contributed to LL.

General Viewer Notes

  • A new Maintenance RC viewer – Eau de Vie, version 6.4.14.556149, was released on March 4th.
  • The Custom Key Mappings viewer updated to version 6.4.14.556098, also on Thursday, March 4th.

The rest of the official viewers in the pipeline remains as follows:

  • Release channel cohorts:
    • Love Me Render (LMR) 5 project viewer, version 6.4.14.556118, February, 23, 2021.
  • Project viewers:
    • Legacy Profiles viewer, version 6.4.11.550519, October 26.
    • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

There may be a viewer promotion during week #10 (commencing Monday, March 8th).  While no final decision has been made, it is possible the LMR 5 may be the viewer that gets promoted.

Viewer Rendering

[CCUG Meeting]

  • Euclid Linden continues to work on separating UI rendering from general scene rendering in order to improve overall viewer performance.
    • Currently the UI is completely redrawn every frame, along with the rest of the scene, whether or not it requires update.
    • This can take up to 40% of the rendering time, depending on the complexity of the scene being rendered, with around 10-20% being an average, so it is hoped the changes will particularly help those on less-capable systems.
    • There may be some subtle / slight reduction in UI responsiveness under certain circumstances, so a means to tweak the UI update rate may be provided via something like a Debug setting.
    • Note: this work is not physically altering the appearance of the UI, only the frequency with which it is drawn.
  • Ptolemy Linden continues to work on more general rendering performance improvements and rendering bug fixes.

In Brief

  • [CCUG] Map tiles are still being worked on. In the future the lab may look to implement a new means of generating the Map tiles in a more robust way.
    • The Lab has an internal proposal for updating terrain textures, but it has yet to be formally adopted.
  • [CCUG] Allow upload of ground  constrained animations – on the LL’s list,  but has not been looked at as yet.  Vir believes that as the capability is already supported (but undocumented) is should only require an alteration to the simulator animation code to ensure the constraints are correctly understood.
  • [TPVD] Firestorm has entered QA, with a release planned for mid-March.

Date of Next Meetings

  • CCUG: Thursday, March 18th, 2021.
  • TPV Developer: Friday, March 19th, 2021.

2021 TPVD meetings week #7: summary

Eulennest, January 2021 – blog post

The following notes are taken from the TPV Developer meeting held on Friday, February 19th, 2021.

These meetings are generally held every other week.  They are recorded by Pantera Północy, and her video of the meeting is embedded at the end of this report – my thanks to her for allowing me to do so – and it is used with a transcript of the chat log from the meeting and my own audio recording to produce these notes.

The latter two-thirds of the meeting included a large amount of local text chat related to VRAM and texture handling. Please refer to the video for details.

End of an Era

[0.00-1:06]

As I recently blogged, Oz Linden, who initiated the TPVD meetings and who has, with the exception of when he’s been on vacation, chaired them is departing the Lab for retirement on Friday, February 26th (see: Oz Linden announces his forthcoming departure from Linden Lab). As such, this marked the last TPVD meeting he would attend, with – I believe – Vir Linden now set to carry them forward.

Given this, Oz had a few words to say to the TPV community at the start of the meeting:

 I want to say that working with the third party viewer community has been  – I mean this is how it started for me, and it has been a theme throughout, and it has been a very great pleasure working with all of you. It’s really been terrific, and thank you. Thank you for making me look good, and thank you for all you contribute to this really impactful and fun product.

SL Viewer News

[1:21-2:52]

  • Current release viewer: Project Jelly viewer (Jellydoll updates), version 6.4.13.555567 and dated February 5, 2021, promoted February 17.
  • Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
    • Love Me Render (LMR) 5 project viewer, version 6.4.13.555871, February, 18, 2021.
    • Simple Cache project viewer, version 6.4.13.555641, February 16, 2021.
    • Custom Key Mappings project viewer, version 6.4.12.553437, January 7, 2021.
  • Project viewers:
    • Legacy Profiles viewer, version 6.4.11.550519, October 26.
    • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

Project Jelly Viewer

This viewer essentially improves the rendering of Jelly Doll avatars.

  • Originally introduced in 2015 (and with various improvements since) as a means to allows users reduce the avatar rendering load on their systems by having any avatars around them that exceed a certain complexity value (set via a slider) render as a solid colour and minimal detail.
  • There have always been a number of issues with the manner in which these avatars are rendered. For example: the colours used have been seen as intrusive so users often avoid the capability, while there have also been technical flaws such as the original code. attempting to render all of a Jelly Doll avatar’s attachments, defeating the intent of the code.
  • As a result, the Project Jelly viewer improves things by both rendering avatars as simplified grey humanoid shapes, and by not making any attempt to render attachments.
  • In addition it also improves to how avatar imposters are rendered and updated.
  • These improvements should result in demonstrable improvements in view performance in environments where there are a large number of avatars and the capability is sensibly used.

General Viewer Notes

  • Of the RC viewers, all three appear to be in good shape for promotion as the next de facto release viewer, although no decision has been taken on which will be promoted next.
  • At the CCUG meeting, some users expressed a preference to see the Love Me Render 5 viewer promoted next, but there is no commitment to this being the case, as promotions are governed by stability / crash rates.
  • A new Maintenance RC view is anticipated as appearing soon.

In  Brief

  • [12:36-14:09] Viewer rendering:
    • There are still no firm decisions as to how the viewer rendering API will be handled in the move away from OpenGL. for the most recent information I have on this, please see my February 5th CCUG meeting notes.
    • As per my February 18th CCUG meeting summary, the current focus is on bug fixes and UI performance improvements.
    • Other proposed UI work is related to the new user experience and making the viewer easier for new users to get to grips with the viewer.
  • [15:20-18:00] Chrome Embedded Framework (the media handler used by the viewer) will cease supporting Windows hardware that is pre SSE3 (2004). It is believed that few (if any) Sl users are running systems old enough to be affected by this – and if they are, they are liable to have more issues than simply losing their media playback capabilities.
  • Firestorm has entered a QA cycle in preparation for what will be something of a maintenance release with a focus on closing the gap between it and the more recent Lab viewer code releases. It is hoped this will be the first in a more regular cycle of 3-monthly releases.

2021 TPVD meetings week #5: summary

Grauland, December 2020 – blog post

The following notes are taken from the TPV Developer meeting held on Friday, February 5th, 2021.

These meetings are generally held every other week.  They are recorded by Pantera Północy, and her video of the meeting is embedded at the end of this report – my thanks to her for allowing me to do so – and it is used with a transcript of the chat log from the meeting and my own audio recording to produce these notes.

SL Viewer News

[0:00-4:10]

The Project Jelly viewer updated to version 6.4.13.555567 on Friday, February 5th. This presumably brings it to parity with the current release viewer code base, and moves it closer to potentially being the next viewer to gain promotion to de facto release status, although no decision has been made on this as yet.

  • Current release viewer Dawa Maintenance RC Viewer, version 6.4.12.555248, dated January 25, 2021, promoted February 1st, 2021 – NEW.
  • Release channel cohorts (please see my notes on manually installing RC viewer versions if you wish to install any release candidate(s) yourself):
    • Custom Key Mappings project viewer, version 6.4.12.553437, January 7, 2021.
  • Project viewers:
    • Love Me Render (LMR) 5 project viewer, version 6.4.12.553511, issued on January 7, 2021.
    • Simple Cache project viewer, version 6.4.11.551403, November 12.
    • Legacy Profiles viewer, version 6.4.11.550519, October 26.
    • Copy / Paste viewer, version 6.3.5.533365, December 9, 2019.
    • Project Muscadine (Animesh follow-on) project viewer, version 6.4.0.532999, November 22, 2019.
    • 360 Snapshot project viewer, version 6.2.4.529111, July 16, 2019.

General Viewer Notes

  • With Dawa now at release status, the current RC viewers are being bought up to parity with its code base.
  • As noted in my most recent CCUG summary, the Love Me Render (LMR) 5 graphics RC viewer is close to being ready for promotion from project to RC status.
  • The simple cache viewer (VFS replacement viewer) is in “decent shape” for promotion to RC status once updated to the Dawa code base.
  • There may be further UI work for the Legacy Profile project viewer (returns avatar profiles,etc., back into the viewer, a-la Firestorm) which may delay this viewer from progressing.
  • For OS X users, a viewer is in the works that will “get Apple’s notarisations working”. This has been something of a long standing issue, and the viewer should be appearing in the near future.

Project Jelly Viewer

[9:06-11:32]

  • This viewer essentially improves the rendering of Jelly Doll avatars.
  • The idea behind Jelly Dolls (first introduced in 2015, with various improvements since)  was to give users the means to reduce the load of having to render extremely complex avatars on computers with low hardware specifications by:
    • Explicitly selecting a nearby avatar and set its display value to Do Not Render, reducing them to a simplified, “flat” grey avatar form.
    • And / or setting a “complexity value” within the viewer that, if exceeded by any avatar in the field of view, will render it as a Jelly Doll.
    • Both the grey and the Jelly Doll forms are simplified avatar outlines, with the latter rendered as one of a number of solid colours selected by the viewer from a colour map. The term was coined by user Whirly Fizzle after the British Jelly Babies brand of sugared jelly sweets.
  • There have always been a number of issues with the manner in which these avatars are rendered. For example:
    • On the visual side, many users have avoided the viewer’s complexity setting (to the detriment of their viewer’s performance) as they do no like seeing solid, brightly coloured avatar forms in their field of view.
    • On the technical side, the code currently attempts to render all of  avatar’s attachments. As these tend to be the most costly to render, this can defeat he object of the code.
    • Also, the baseline formula for jelly doll calculations doesn’t allow for consistent results.
  • As a result, the Project Jelly viewer:
    • Does not attempt to render attachments, but instead renders affected avatars as a simplified, easy-to-render humanoid shape.
    • Render all such avatars in grey, no longer using the previous colour map, in the hope this will encourage more people on lower-end systems to use the capability, as the grey avatar forms tend to be less intrusive within a scene.
  • [13:38 via chat] In relation to avatar rendering / Jelly Dolls, it was asked if better global controls for rendering could be provided. In reply, Grumpity indicated that in addressing performance as a whole, such global controls might be considered in the future.

Viewer Log-in Changes

[4:41-8:05]

  • Oz Linden is working on the viewer log-in process that are designed to prevent people logging-in to Second Life when their inventory is “broken” (and potentially making their situation worse).
  • The updates to the log-in process means there will be additional checking on the status of an avatar inventory data on the back-end as a user logs into Second Life.
    • If errors within the data are found, the log-in process suspended, in order to prevent the errors being propagated to the viewer, and then viewer then exacerbating the situation by trying to manipulate the inventory database further on the basis of the invalid data.
    • Should this happen, the user will receive a massage that explicitly states that log-in has failed as a result of inventory issues with the request the user contacts Support. This message will also provide some specific information the user can pass to Support when they contact them.
    • Support will then use the information supplied to initiate the required corrective action to resolve the issue.
  • While this may impede users with inventory problems from logging-in, the hope is that these changes will actually make the resolution of inventory-related issues easier to correct at source and thus have less of an overall impact on affected users.
  • This is seen as a priority change, and the Lab hopes to be in a position to have the back-end changes tested and the viewer-side updates available by the end of the month.

In  Brief

  • [4:17-4:32] The Lab is making some changes to how deployments are managed within the AWS environment. If done correctly, this work should result in no user-visible changes.
  • [11:38-14:11] Post-Uplift issues:
    • There remain some issues still to be fully resolved as a result of the transition to AWS, including (but not necessarily limited to):
      • The problem with Map tile not being generated. This is being addressed.
      • The fact that the chat servers currently need to be restarted more frequently than pre-Uplift. This is still being diagnosed.
      • Teleport failures resulting in an “wrong” or “invalid” region message. This is also being diagnosed.
    • However, the Lab caution against assuming any issue that is encountered is a result of the AWS transition. The general rule remains, if you’re seeing a specific (and preferably reproducible) issue, raise a bug report.
  • [14:31-15:33] in response to a question on the avatar skeleton (why 159 bones, but capped at 110 on upload?),  vir pointed out the 110 bone cap is per sub-mesh in a character, rather than on a complete character (which can have several sub-mesh components. The reason for the cap is down to older GPUs that can be used in SL being unable to handle the transform matrices.
  • [19:54-20:28] Premium Plus: internal discussions have resumed on the deployment of Premium Plus, but there is nothing to share in terms of time frame, etc., at the moment.
  • [21:49-24:15] The recent Marketplace issues are seen as a combination of both the continuing work to improve the Marketplace experience and the work to transition it to AWS.
    • As the MP involves multiple back-end services, there are a lot of interdependencies that can be impacted particularly as a result of the AWS transition work, and not all of these can be accurately QA’d, as the sheer volume of transaction, etc., the MP sees hourly cannot be easily or accurately replicated.
    • The current focus is on general MP stability (including its various dependencies) in order to hopefully make future maintenance and update easier / smoother.
    • The most preferable way to deal with the MP would perhaps be to take the service down entirely for a a period of time and overhaul it, and then re-release it. However, given the impact this would have, it is simply not a viable option.
  • [24:42-25:45] Generally speaking, LL believe SL to be a lot more stable / robust now than previously, simply because it is running on much more recent hardware and within an infrastructure where they no longer have to worry about things like hardware that is well beyond its operational life failing, whilst any underpinning hardware / infrastructure issues are more-or-less immediately addressed by AWS. This in itself allows LL’s engineering and ops teams to be more focused on running the software side of things.