Grid Maintenance: all silent on the Linden front

On Friday, I drafted a piece on the lack of direct feedback from Linden Lab vis-a-vis the recent rounds of maintenance carried out on the 8th, 9th and 10th May. I held off publishing  because I opted to wait until Close of Business in California to see if anything would be forthcoming. Then I got involved in other things, and completely pushed the post out of my head.

In the meantime, Tateru has, with her usual incisiveness asked the question I was planning to ask, but far more succinctly and directly. It’s a good read, and I suggest that if you haven’t you go read it – if only to save yourselves from hearing the same old, same old from me.

Elsewhere – well, Nalates Urriah’s blog to be precise – it may appear that the poor state of communications isn’t restricted to the amount of information flowing out of the company to the users, but many extend to internal communications within the Lab. In her piece, and in reference to the maintenance periods, she reports that Andrew Linden commented:

I think part of that is an operating system upgrade on some hosts, not network level maintenance, but I’m not sure. We’re definitely working on migrating to later versions of Debian, but there will be a few upgrades along the way before we arrive at Debian/Squeeze.

Andrew, together with many of the other Linden staff involved in User Group meetings usually try to be as informative as possible. Following the outage of the 26th April, for example Oskar attempted to provide some explanation via the forums, as Jack Abraham pointed out on this blog, and Daniel Voyager later reported via his.

So when you do see normally forthcoming staff using terms like “I think,” and “I’m not sure,” when describing a situation that impacts the entire SL service, it’s a little disconcerting, and might be seen as suggesting internal communications at LL may be lacking. After all, Andrew is a part of a group of LL staff who do face users, and who are likely to get asked the hard questions when something major is going on, so you’d perhaps expect the company to ensure they are furnished with sufficient information to be able to field such questions with a measure of clarity.

There was a time when we would see blog reports related to upcoming service improvements, and posts on outage postmortems. But no more. Instead, we’re left hanging or having to resort to scrabbling around the forums in the hope that someone has had the foresight to say something. And while it is great that news is frequently passed out at User Group meetings, not everyone attends these, so the message tends to go more unheard than heard.

Simply put: there is no substitute for clear, open communications – and however you look at it, the SL blogs are the best place to communicate when it comes to major announcements and service issues. In these cases, all other channels should point back to the blog, not used in place of the blog. As Tateru points out, the Grid still isn’t the most stable of environments following the recent work; we have no idea as to what the general status of the grid is or what is going on – and that leaves us guessing.

Going on Andrew’s comments, we may not be the only ones. Either way, it would be nice if someone at Linden Lab stepped up to the keyboard and took a little time to let their users and customers know exactly what is / has been going on, and how things stand.

Microsoft Skydrive: Viewer crashes

Update: Latif Khalifa reports encountering the problem on Singularity, so the issue is likely to affect V1-style Viewers as well, and provides a potential workaround for those encountering the issue (see comments, below).

There have been a number of reports of mysterious Viewer crashes of late, specifically in relation to file operations (saving a texture or snapshot to disk, attempting to upload, setting your cache location, etc.).

The problem appears to be manifesting itself in V3-based Viewers, and the culprit appears to be Microsoft Skydrive.

A JIRA has been raised on the issue in general (VWR-28843), with reports that uninstalling Microsoft Skydrive eliminates the issue. This is obvious not the best solution for the situation, but those experiencing the issue may wish to check the JIRA and consider WATCHing it.

With thanks to Innula Zenovka. 

Marketplace update April 26th

With trying to cover / enjoy Fantasy Faire last week, I missed the following when it came out.

On Friday 26th April, Commerce Team Linden (CTL) provided an update on attempts to sort out the ongoing issues with the Marketplace. The update reads in full:

[UPDATE: April 26, 2012] We are continuing to work on outstanding Marketplace issues. 

The most significant issue that has been addressed over the past week was slowness on the Marketplace website, which occurred intermittently between April 18 and April 21. Page loads should be much faster now. 

Here is the status on additional JIRAs:

  • (WEB-4587/WEB-4601) Search results, best-selling lists and related items showing up on other merchants items or images. Current status: other merchants’ listings should not be appearing in store search results, best-selling lists, related items, etc. Other merchants’ images may still appear on your listings, but this is less likely to occur as we work on a solution.
  • (WEB-4676) ANS for SLM does not populate the Location field with internal ID. Current status: this is in progress and will be deployed soon. 
  • (WEB-4696) Deleted listings appearing in search results for consumers and in Merchant Admin. Current status: we are working on resolving this issue.
  • (WEB-4441) Orders stuck in Being Delivered. Current status:  the orders stuck here due to unicode problems was addressed, and we continue to investigate remaining orders stuck in this state.
  • (WEB-4567) Bulk delete failing. Current status: the work-around on this is to delete each item. We continue to work on resolving this. 
  • (WEB-4574) Direct Delivery products are re-delivered to purchaser instead of recipient. Current status: we are working on resolving this.

We continue to work on the other Marketplace JIRAs and will provide additional updates as soon as possible.

My apologies for not catching this at the time of publication.

Thursday’s downtime: network bug; more downtime to follow

The rumour mills (serious and highly humorous – the latter particularly on Twitter) churned mightily yesterday, Thursday April 26th, after SL log-ins were blocked. Users attempting to log-in to SL were treated to something of a dire message, rarely seen in recent years, prompting me to tweet at the time:

The outage certainly brought back memories for many, particularly of the old Black Wednesdays, when the Main grid would be down for anything up to eight hours (the rule of thumb being six) for weekly maintenance.

Yesterday’s outage brought with it memories of the “good old days”

A Grid Status update posted on April 27th later revealed that the cause of the fault to be a core network upgrade going awry, impacting both the Main grid and the SL website at secondlife.com. The Grid Status update also warns of further maintenance work that will take place on Monday April 30th as a result of the issue. The update reads in full:

[UPDATED] Scheduled Maintenance Monday, April 30, 2012

[UPDATED 2:30pm PDT, 27 April 2012] On April 26, 2012, the Second Life engineering team was performing an upgrade to the core network. The upgrade triggered a bug that led to downtime and instability throughout the day. The maintenance described in this post is scheduled for Monday, April 30 at 6am PT in order to correct the bug and may result in additional service interruption. We apologize for the service disruption and appreciate everyone’s patience.

[POSTED 1:00pm PDT, 27 April 2012] We will be performing scheduled maintenance Monday April 30, 2012 beginning at 6:00am until approximately 9:00am. Please save all builds and refrain from rezzing no copy objects or making inworld L$ transactions. Some residents may experience delays logging inworld. Please follow this blog for any updated information.

Marketplace: LL updates further, but communications hardly “regular”

Linden Lab has issued a further brief update about the on-going Marketplace issues, to whit:

Today we updated Marketplace to address two of the top three outstanding issues:

  • WEB-4580: purchases are now delivered to recipients with the inventory name (which does not allow unicode characters). This will prevent future orders from getting stuck in the Being Delivered state due to this issue. In addition, all orders affected by this problem have been pushed through.
  • WEB-4587:  updates have been made to support updating store search results, which we will process over the next week; we continue to work on the issue related to mismatched data on listings. We do know that this issue has existed since September 2010 (during the migration from Xstreet to the Second Life Marketplace).

We continue to work on the other Marketplace JIRAs and will provide additional updates as soon as possible.

Menwhile, Rodvik has stepped in to defend how matters have been handled in terms of communications, stating on Twitter:

While it is true that the Commerce Team clearly engaged with individuals experiencing problems through the medium of e-mail exchanged, it is nevertheless also true that feedback on this matter in the broader sense has been severely lacking from the Lab, with little or nothing being posted to either the main forum thread on the WEB-4587 issues or the JIRA itself. This left many merchants both frustrated and feeling as if they’d been abandoned, while those who had received some feedback from the Lab via e-mail tried to pass on the information to a wider audience in lieu of LL doing so.

It is good that progress is being made – but equally, it would be nice if LL would do more to keep users openly informed. As those responding to Rodvik’s tweets note:

Related Links

Marketplace listing issues: LL provide feedback

With listing errors still apparent on the SL Marketplace, Linden Lab have provided some feedback relating to their investigations. An update to their post relating to Marketplace JIRA currently under investigation reads:

[UPDATE: April 5, 2012]
We’ve identified two issues contributing to WEB-4587.

The first issue concerns mismatched data appearing on Marketplace listings. This impacts a very limited number of Merchants and has been occurring since the original migration from Xstreet to the Second Life Marketplace in September 2010. We continue to actively work on a resolution. 

The second issue concerns incorrect listings appearing in search results. We have identified the issue and are actively working on a fix. 

The information provided by Merchants on the forum and directly to the Marketplace development team helped greatly in identifying the root cause of these issues. Thanks to those who provided information.

We will continue to provide updates as we work through the issues.

There is no indication as to haw far along any resolution / fix for either issue might be – and probably wisely so. Event tentative dates might be taken literally and cause additional strain in LL / merchant relationships if missed.

That one of the errors dates back to the original migration from XStreet to the current  SL Marketplace back in 2010 will come as no real surprise to merchants who have been impacted by the issue; it is a view that was aired as a part of ongoing discussions into the matter. It might event be fair to say – as the update hints – that it was such speculation and  investigations by merchants themselves that may have helped Linden Lab confirm this particular root cause for the problems.

Even without potential dates for a fix or fixes to be rolled-out, the feedback from the Lab is to be welcomed. Hopefully, now issues have been identified and the line for updates has been re-established, the Commerce Team will take further steps to ensure the merchant community as a whole is kept informed of the situation through the Commerce forum.

Related Links