There are no server deployments planned for week #3 due to the next batch of updates being ready for deployment.
There are still reports of recent simulator updates causing issues for certain types of breedables. The Lab is aware of these problems, and while fixes are in the works, they may not be in the next set of simulator updates.
LL has continued a post-mortem into the region restart issues from week #2, with Simon Linden noting they have hopefully learned enough such that future restarts such run a lot better (and hopefully more smoothly!).
“A couple of hosts” that were not handling teleports as expected at the end of week #2, which should have been corrected.
Additional data that the Lab has been gathering on simulator performance does show that long-running simulators incur increased host resource utilisation, but for reasons yet to be identified.
Scripted avatar loads might in part be responsible, depending on what their scripts are doing, but LL would prefer to gather further data rather than speculate on possible causes.
However, there are regions where issues have been recorded, but which do not have large avatars loads.
LL have tried multiple approaches to making similar slow-downs occur on the beta grid (Aditi), but the problem doesn’t appear to occur “on demand”, making possible causes harder to identify.
The Xanté Maintenance RC viewer updated to version 22.214.171.1243748 on Monday, January 13th.
The remainder of the current official viewer pipelines remain unchanged for the end of week #2:
Current Release version 126.96.36.1993275, formerly the Wassail RC viewer, dated December 4, promoted December 12 – No change.
Release channel cohorts:
Love Me Render RC viewer, version 188.8.131.524234, January 9th.
Project Muscadine (Animesh follow-on) project viewer, version 184.108.40.2062999, November 22nd , 2019.
Legacy Profiles viewer, version 220.127.116.110836, September 17th, 2019. Covers the re-integration of Viewer Profiles.
360 Snapshot project viewer, version 18.104.22.1689111, July 16th, 2019.
Rider Linden is looking at region crossings, and hopes to have some code that should help improve things when it is ready for deployment. However, he notes that the improvements will be “nowhere near what I would ultimately like.
Both the Xanté Maintenance RC and the Legacy Profiles Viewer have updates that are “close” to appearing in the current viewers list, with Xanté possibly being close to promotion as the de facto viewer as well.
As per my week #2 CCUG summary, EEP is still pending the fixing of around a dozen graphics-related issues.
The Camera Presets Viewer, designed to allow users to set and save their own preferred camera presets in their viewer without having to use debug settings, is now closing on being “feature complete” and ready for release either as a Project or RC viewer.
This capability is being developed as a viewer code contribution by Jonathan Yap (who was responsible for developing the Graphics Presets options).
More information on the work can be found in STORM-2145.
Bitbucket, used to manage viewer repositories) will be sunsetting support for Mercurial in early 2020. Because of this, Linden Lab has been migrating their viewer repositories from Mercurial to Git on Bitbucket.
Most of the Lab’s viewer development is now processed via Git.
There is some work remaining on converting all of the third-party package repositories, and this is being done as part of the Visual Studio 2017 / Xcode upgrade, rather than being part of the Git migration per se, as these are relatively stable and not seen as a major problem in terms of migrating them.
A document outlining the steps in migration has been provided specifically for TPVs to allow them to keep in sync with how LL main the viewer code and its branches. See: Viewer Repository Migration (Google Docs).
This is not the only way for TPVs to manage their own repositories, but it is the one that LL recommend for those wishing to reflect how LL manages things.
The VS 2017 / Xcode upgrade has been caught with a couple of issues:
The Boost library used to implement coroutines in the build process would not build using VS 2017 & required conversion to Boost Fibre, which it turn had a number of impacts that also needed to be resolved.
Currently, the project is caught on a GIT-related issue, with validating the integration scripts used in the build process, and which is still being worked on.
It’s hoped that the additional work involved in moving the build process to support VS 2017 should make a future move to using VS 2019 easier.
[7:55-8:30] There has been no work on the Linux front for official viewer over that last few months, and while still on the road map, is seen as a fairly low priority item, sitting well behind the Git migration, build tools updates, general viewer development and – particularly – the work in transitioning Second life to the AWS cloud.
[9:32-10:55] Week #2 restarts. While not generally a part of TPVD meetings, the issues with the Tuesday, January 7th server restarts was mentioned, with Grumpity noting:
It was a rocky day on Tuesday. It came out of our desire to do things in a less disruptive manner and … once again, the road to downtime was once again paved with good intentions, because that way proved to be in no way less disruptive than if we had done things the regular way.
Essentially, the issue was with some restart code that hadn’t been used in a while, and which LL had thought had been fully tested, but when put to use proved to have issues.
[15:33-20:23] A reminder was given that – particularly after a viewer crash – it is better to wait for a short period, rather than immediately trying to re-log (particularly if the warning about logging back in is displayed). This is because each failed attempt refreshes your avatar presence, interfering with the post-crash clean-up (the “you are currently being logged out” part of the message).
Note: this part of the meeting included an extensive chat discussion on auto-relogging, as can be present in mobile and other clients, such as Lumiya and Radegast that continues through until the end of the meeting. Please refer to the video for more on this.
The following notes are taken from my audio recording of the Content Creation User Group (CCUG) meeting held on Thursday, January 9th 2020 at 13:00 SLT. These meetings are chaired by Vir Linden, and agenda notes, meeting SLurl, etc, are available on the Content Creation User Group wiki page.
Environment Enhancement Project
A set of environmental enhancements (e.g. the sky, sun, moon, clouds, and water settings) to be set region or parcel level, with support for up to 7 days per cycle and sky environments set by altitude. It uses a new set of inventory assets (Sky, Water, Day), and includes the ability to use custom Sun, Moon and cloud textures. The assets can be stored in inventory and traded through the Marketplace / exchanged with others, and can additionally be used in experiences.
Due to performance issues, the initial implementation of EEP will now likely not include certain atmospherics such as crepuscular rays (“God rays”).
The EEP RC viewer updated to version 22.214.171.1244193 on Thursday, January 9th.
Bug fixing continues, notably around alpha rendering issues.
It is believed there are about a dozen remaining issues to be dealt with before EEP may be ready for formal release.
First introduced in 2012 (and developed over the following year), Pathfinding was intended to provide a means for more interactive non-player characters (NPCs) in Second Life. Unfortunately, the implementation of the system proved to be so cumbersome (and leaving aside some of the incorrect perceptions about Pathfinding on the part of land holders), the it has never really seen that much use in Second Life.
With the arrival of Animesh, there has been renewed interest in using Pathfinding in conjunction with Animesh characters, but again, the current implementation is proving a bottleneck (e.g. highlighting / indicating “walkable” areas in the viewer; whether the navmesh is actually visible; the effort required to Pathfinding, etc.).
A forum thread highlights the issue, and it has been suggested that if a Jira can be raised highlighting the specific problems, it might be something the Lab could take a look at to try to improve some of the visualisation issues within the viewer (Navmesh visibility, etc.).
However, a broader pass at improving / overhauling Pathfinding is not on the Lab’s current road map for SL.
IP rights, UV Maps and “working copies”: there has been recent discussion on the forums, through various user groups (notably Governance, which I’ve been unable to attend for the last couple of months due to RL) concerning IP rights and things like mesh VW maps, compatibility, weight painting etc. The questions have arisen of late due to a mesh appearing on the Marketplace that achieves compatibility with all the other meshes of the same nature by providing amazingly close replicas of them.
Currently, the primary course of response to concerns over potential infringement – imperfect is it may appear to be where this issue may be more esoteric in nature, given that the meshes in question all tend to use things like UV maps derived from originals supplied by Linden Lab – is for a creator with concerns over infringement to file an DMCA complaint with LL.
BOM take-up: Bakes On Mesh take-up is seen as being a little slow. Some mesh body / head makers have yet to fully adopt BOM flagging on their products for example (so while Maitreya support BOM on their current body via a HUD, the body still has some 800 individual mesh elements that the viewer needs to handle, compared to the (roughly) less-than-fifty used by the Slink Redux (BOM) body). Also, there are continued concerns about BOM’s ease-of-use when compared with the use of HUD-based applier systems. While the latter can be more resource-intensive, the form is seen as requiring better scripted tools and / or better inventory visualisation mechanisms (even better base alpha support) in order to be more attractive to users.
The following notes are taken from my recording of the Web User Group (WUG) meeting, held on Wednesday, January 8th, 2020. These meetings are held monthly, with dates and details of the meetings available via the Web User Group wiki page.
When reading these notes, please keep in mind:
This is not intended as a chronological transcript of the meeting. Items are drawn together by topic, although they may have been discussed at different points in the meeting.
Similarly, and if included, any audio extracts appearing in these summaries are presented by topic heading, rather than any chronological order in which they may have been raised during the meeting (e.g. if “topic X” is mentioned early in a meeting and then again half-way through a meeting, any audio comments related to that topic that might be included in these reports will be concatenated into a single audio extract).
Recurring Events launched on Monday, January 8th, allowing Premium members to set a recurring events in the Upcoming Events lists available via users dashboards at secondlife.com.
Premium members will be charged L$10 for listing one-off events, or L$50 for filing an event that can recur up to 12 times (weekly or monthly).
Basic members are charged L$50 per event listing.
Recurring events are events that share the same location, time, duration, and description but occur on a weekly or monthly basis.
Recurring events are largely set-up exactly the same was as individual events:
Go to your dashboard at secondlife.com and select Events from left side menu to open the Events listing page.
Click on Create A New Event directly below the top left calendar on the page, then read and accept the Event Posting Rules to display the new Event Creation page.
Complete all the fields on the page.
For Premium members only, the page now includes two new drop-down options:
Repeating – allows you to select whether the event is a one-off (No Repeat) or should repeat on a weekly or monthly basis.
Number of Repeats (only available when either Weekly or Monthly is set from the Repeating options) – allows you to select the number of times the event is to be repeated, to a maximum of 12 weeks or months (so for a single L$50 fee you can schedule a monthly event to recur on the events listing page for up to an entire year, or a weekly event for up to 3 months at a time).
When selecting an event to repeat either weekly or monthly, the fee charged to your account (displayed at the bottom of the form) change from L$10 to L$50.
There are currently some minor adjustments required for monthly recurring events, as events recurring right at the end of a month may give some incorrect dates for months of different lengths. Such areas can be corrected manually, but updates to be deployed in the near future should correct the problem. In addition, LL will be monitoring the recurring events capability now it is live in order to identify and address any other issues that may arise now it is live.
Future Events Listings Work
Further improvements to events are planned (e.g. better searching for events), but these are dependent on other work being completed first.
This work may include things like being able to push events to your personal calendar (Google Calendar, Outlook, etc.).
No final confirmation on release date (although the hope has been for a mid-Q1 2020 release). The exact release time frame is dependent upon final checks on the capability being completed.
Even if delayed, Last Names will still be deployed before any release of the Premium Plus subscription package.
Over 2,000 entries have been received as a part of the (now closed) Last Names Contest, totalling some 6,000 last name suggestions in all, and LL have started going through them.
It is likely that when Name Changes is launched, users will be able to pick last names from a list of around 20-30.
One of the approaches being considered for changing the available last names is for:
Popular selections being removed from the list and replaced on reaching a certain number of people selecting them.
Having selections that appear “unpopular” (i.e. they are rarely / never selected) replaced if they fail to achieve a certain threshold of use over an (unspecified) period of time.
Marketplace search has been updated with options to exclude / include limited quantity items and / or demo items.
The check boxes for excluding (ticked) or including (unticked) such items can be found at the bottom of the Search Filters panel displayed on the left of Marketplace pages when either Search is used or See All Categories is clicked.
Both options can be found below the search by permissions options.
By default, both options are unchecked, so that searches will include either limited quantity or demo items in their results results.
Work is being put into performance improvements which, as Grumpity noted at the meeting, tend not to be noticed right up to the point where something goes wrong.
An example of this is the ability for merchants to search their invoices should be a lot faster following back-end updates.
There are still “quite a few” things the Lab have queued up to do in relation to Marketplace search, but no specific details are available as yet on what might surface when.
This issue with Destination Guide submissions is believed to be resolved.
The thread refers to the SLS Main channel being on simulator release 533895. However:
Following restarts, it appears that simulators on the Main channel (and some on RC channels) have remained on 2019-12-04T20:29:26.533447, which according to previous deployment threads, was the last formal deployment to the Main channel.
According to the release note page, 533895 was deployed on December 19th, but has channel names are now obfuscated, it is not clear which RC received the update.
The thread refer to restarts on the RC channels on Wednesday, January 8th, some RC servers should be restarted, but again without any deployment – although the 533895 version number is given for some when it should perhaps be 53447.
The Tuesday, January 7th restarts were repeated a number of times, for reasons Simon Linden explained:
So we haven’t had any server updates since mid-December. Today we restarted a bunch because that can help performance, and in the middle of that we uncovered a bug that’s been lurking there for months, so it got a bit exciting.
Mazidox Linden added that – as of the time of writing this update – further restarts could not be ruled out.
The bug itself was related to at least one directory required by the Mono compiler not being created correctly, preventing scripts from being saved. It is apparently a bug that has been around for some time, but only surfaced as a result of changes to the way simulator restarts are run. Ironically, the changes were intended to make simulator restarts faster and smoother
Numerous region holders reported significant performance issues over the holiday period. According to Grumpity Linden, part of the issue was a failure with an automated tool, itself something of a workaround for dealing with simulator performance issues:
Hi all! I was really hoping my first post of the new year would be more jolly, but here we are. Happy New Year, though!
For a couple of years now, we’ve had automated tools, aptly named “Grid Poking Bot” (GPB for short) responsible for doing region restarts, and this has been working quite well – most of the time. Very unfortunately, there was a problem with the GPB over the holidays, and due to a combination of events, it took us much too long to notice – and we finally caught it in part thanks to this very forum thread and a certain vigilant “Spray Can”. We’re now actively pursuing the least disruptive ways to address this problem as quickly as possible. We’ll have a more detailed post-mortem blog in a couple of days as well.
We’re very sorry about souring your holidays.
It’s possible there is more than one issue causing problems, and investigations are still in progress, including the distribution of poorly-performing regions (e.g. are they on their own on the grid, do they have neighbours) as well as looking at distribution by server.
There have been no viewer updates to mark the start of the week, leaving the current pipelines as follows:
Current Release version 126.96.36.1993275, formerly the Wassail RC viewer, dated December 4th, promoted December 12th – No change.
Project Muscadine (Animesh follow-on) project viewer, version 188.8.131.522999, November 22nd.
Legacy Profiles viewer, version 184.108.40.2060836, September 17th. Covers the re-integration of Viewer Profiles.
360 Snapshot project viewer, version 220.127.116.119111, July 16th.
Not strictly part of the engineering team’s remit, but it has been reported the Destination Guide is giving errors when trying to submit new destinations. This has been seen by the Web Team, and is apparently being looked at.
Each year through this blog I attempt to track news about, and changes to, Second Life, as driven by by Linden Lab. On the technical side, this is do through my weekly SL project summaries, whilst news and general updates are drawn from sources such as Lab comments on the official forums and official blog posts or as a result of attending public meetings and Q&A sessions, etc.
As a lot can happen during the course of the year, so in this article I’ve tried to summarise the more notable updates to occur during the course of 2019.
Unfortunately, these auctions had to be suspended in October / November 2019 due to unspecified “abuse” by users. The system is to be revised, but there is currently no indication of when the auctions will be re-enabled.
The first styles of the new Premium Homes were unveiled at the annual Home and Garden Expo in March 2019 in a single “preview” region that provided both the four types of Traditional homes and four types of Houseboat that would be the first of the Premium Homes themes that would be issued.
The Homes themselves launched on April 15th, 2019, with the opening of the new Bellisseria continent and a mass release of both Houseboats and Traditional homes. As with the original Linden Homes, they are available through Premium member’s Land Homes page, accessed through their secondlife.com dashboard.
The first mass release of houses and houseboats had been snatched up within 48 hours of the release, with the houseboats proving particularly popular – so much so, that the Lab immediately started planning a 709-parcel add-on to Bellisseria specifically to meet the demand. In the meantime, one of the points noted about the new region was its lack of airstrips and this was addressed with the opening of the continent’s first airstrip in late April 2019.
The new bolt-on for the houseboats opened on May 15th, 2019 – and were all gone within 27 minutes of the release. Then in June 2019, Patch Linden announced that releases would shift to a smaller-scale rolling basis with regions of new houses generally being made available on Mondays, Wednesdays and Fridays.
From June 2rd, 2019 Private region tiers changed as follows:
Full regions were reduced from L$249 a month to US $229.
Homestead private regions were reduced from US $195 to US $179.
These changes were exclusive of VAT, where applicable, and did not apply to Skilled Gaming region; however, Education / Non-profit (EDU/NP) discounted Full islands were be re-priced to maintain their 50% discount off the regularly priced Full island fees.
From June 24th, 2019, Premium fees were increased for the first time in their history:
The monthly fee increased from US $9.50 to US $11.99 (an annual increase of 26.21% from US $114 pa to US $143.88 pa)
The quarterly subscription increased from US $22.50 to US $32.97 (an annual increase of 46.53% a year from US $90 pa to US $131.88 pa). This fee was to be discontinued to users upgrading to Premium after July 24th, 2019, but a later decision saw it continued on a “temporary” basis that means it is still currently available.
The annual fee increased from US $72 to US $99 (an annual increase of 37.5%).
In addition, both existing quarterly and monthly subscriptions would again be subject to VAT.
Note: included with the announcement were proposed changes to Basic account users available off-line IMs and total group allowance. However, after receiving wide-ranging feedback (such as this letter from myself), the decision was made to not implement the group changes.
To help sweeten the Premium fees increase, between June 3rd through June 24th, Premium users were offered the chance to “lock-in” their Premium fee for an additional period commensurate to their subscription period from the end of their existing period. In addition, existing annual Premium subscription members were offered the chance to renew their subscription for an additional year from their next renewal date at the Winter Premium discounted rate (10% off).
There was also the 5% increase in Marketplace commission fees which caused some consternation. While the reason for the increase is understandable when put in the proper perspective, it could have been framed a little better.
The early part of 2019 was marked by users experiencing a significant number of teleport disconnects. These proved problematic for the Lab as well,with assorted causes from EEP deployments to server OS updates being suggested as a possible cause. A series of user-involved stress tests on the best (Aditi) grid to help with investigations, allowing adjustments to be made on the simulator side. These helped point towards a race condition, with LL implementing changes and updated monitoring to counter the issue.
Script Processing Changes
Over the course of the year, number of script-related issues have surfaced:
Issues following simulator deployments and restarts (see BUG-227688 and BUG-227897) which are still under investigation.
Script performance on Mainland regions sharply declining following a restart to a point where only around 20% or so of scripts are running per cycle, and the simulator has no spare time, forcing the region holder to submit a ticket requesting a move to a different server as a temporary fix.
As a result, LL has worked to improve script handling – such as adjusting how idle scripts are handled to reduce the overhead with place on CPU cycles – and these changes and adjustments have helped to eliminate some, but not all, of the problems encountered through the year.
Purchase notifications for merchants were introduced (providing details on item purchased, amount received and who made the purchase).
In May 2019, the Lab introduced new web-based release notes for the official viewer, together with a index page for said release notes. There were some initial teething problems with the system for those who track official viewer releases (sometimes a viewer update would appear on the index page, sometimes on the new Alternate Viewers page, sometimes on both that took a while to smooth out.
Server release notes made a similar move to web pages in September. After this, LL stopped breaking down simulator updates by release channel (e.g. BlueSteel, Magnum and LeTigre), listing all releases as “Second Life Server”, regardless of the actual channel used for a release.
In January, evidence surfaced that Linden Lab are working on a Second Life iOS. After enquiring with the Lab, I received confirmation the app was being actively worked on. As the year progressed, more details were revealed about the app, including: the app should work on both the iPhone and iPad, and will initially be more of a communicator / companion app than a fully-rounded client; it will provide a log-in option, and chat options (e.g. chat, group chat), but will not present users with an in-world location, or rez and avatar in-world. Over time it will be enhanced – but additional capabilities are still TBD. See my mid-year update for more.
Work – most of it transparent to users – has continued on the migration of Second Life to the cloud. Most of this work has been on the back-end services, notably the web services. Currently, no public-facing simulators have been transitioned to AWS provisioning. All of this work has been achieved without any significant disruption to services or – more particularly – without users actually being aware the services had been moved, and the Lab reports that the migrated services have been able to achieve almost 100% up time.