SL Marketplace Issues: July update

On July 31st, The Commerce Team issued the most recent update in the ongoing saga of SL Marketplace issues. The update reads in full:

UPDATE: July 31, 2012

We continue to work on testing the next Marketplace update, which includes a required upgrade (for the Marketplace, not for Residents). One benefit of this work is that we are seeing performance increases with page load and purchase completion during our testing. We are working to get this update completed as soon as possible.

Last week, fixes to help with WEB-4600 were deployed with viewer 3.3.4. We have also been working with Third Party Viewers to make sure they are handling the Merchant Outbox correctly going forward. In addition, some Third Party Viewers now support the Merchant Outbox on Linux. Please see the following Third Party Viewers if you would like to use the Merchant Outbox on Linux:

If your Third Party Viewer is not on this list, and it supports the Merchant Outbox on Linux, please send a notecard to CommerceTeam Linden. Please include a link to the download location, and it will be added to the above list.

Below is the updated set of outstanding issues with Direct Delivery and the Marketplace.

Direct Delivery

Here are the outstanding Direct Delivery issues:

  • WEB-4600 (Merchant Outbox failures): There are still outstanding issues with the Merchant Outbox, in addition to the issues addressed above. We continue to investigate and address these issues as they come up.
  • WEB-4554 (Test delivery permissions incorrect): This is on hold while we work on other issues.
  • Limited Quantity Support (Merchant does not have rights to copy the items for sale): This is currently being worked on. Magic Box migration will not be required until this is supported. (Note that Merchants can sell items that have next owner rights set to “No Copy”.)

Overall Marketplace

There are also several issues that occurred around the time of the Direct Delivery launch that we are still working to address, but are not issues with Direct Delivery.

  • WEB-4587 (listings with the wrong images): This will be addressed after the next Marketplace update.
  • WEB-4441 (Orders stuck in “Being Delivered” state): We have been able decrease the number of orders getting stuck and continue to work on preventing all orders from getting stuck.
  • WEB-4592 (Orders marked as “Delivery Partially Failed” on success): This issue is currently being worked on.
  • WEB-4138 (Confirmation emails failing to deliver): We are currently working on a solution to this issue.
  • WEB-2974 (Listing enhancement stuck in “Charging, cannot edit right now” state): This issue is on hold while we work on the other items on this list.
  • WEB-4696 (Deleted listings appearing in search results): This issue is on hold while we work on the other items on this list.
  • WEB-4567 (Bulk delete fails for some merchants): We will evaluate the priority of this once we have completed the above Direct Delivery fixes and features.

In the meantime, the due date for Magic Box migration has again been extended (as of July 26th) to October 1st, 2012.

SL Web profiles: my.secondlife.com errors

Update July 11th: Those impacted by the errors are reporting that their web profiles and feeds all appear to be displaying the correct pictures / information.

Update 16:35 UTC, July 9th: Some users appear to still be experiencing issues; LL have confirmed that the problem is not related to any compromise to their systems (which was unlikely anyway), but rather to a database issue. Soft Linden has requested that if anyone contacted SL Support about this issue, could they e-mail him with the details

Update 12:15 UTC, July 9th: The Grid Status page reports the “unscheduled maintenance” completed at 02:29 UTC. Again, it is unclear as to whether this work was in any way connected to the web profiles / my.secondlife issues, but my own feed history now appears to be displaying the “correct” information. Others are similarly reporting their own feeds seem to now be OK.

Update 01:47 UTC, July 9th: At 11:46 UTC (16:45 SLT) LL commenced a period of “unscheduled maintenance” on SL. Shortly thereafter people again reported that the my.secondlife.com feed were inaccessible (which had been the case on an intermittent basis alongside of the problems reported below). Whether the maintenance work – which references in-world services – also includes web profiles and my.secondlife.com is unclear. At the time of writing the maintenance work is still ongoing and feeds remain inaccessible. 

Update: People from the US are reporting they are experiencing the issue as well, confirming the problem is more widespread than initial Twitter reports suggested. 

Second Life users are reporting issues with my.secondlife.com. The problems are currently taking a number of forms:

  • The wrong pictures are showing up in people’s my.secondlife.com profiles
  • People’s posts are appearing in the wrong feeds
  • People are seeing posts from those they are not following appearing on their feeds.
Pallina60 Loon reports her Profile image appearing in Tifa Kobichenko’s my.secondlife.com feed

It is not clear how deep the problem runs, but many of those initially affected appear to be users based in Europe and there are concerns about accounts being compromised as a result of the issues. Issues are apparent whether viewing web profiles either via a browser or within the viewer.

There are some security concerns being voiced as well. While it is more likely that the issues are the result of a database error within LL’s servers rather than a deliberate hack or the result of the servers having been compromised, if you are affected and of a nervous disposition, you may want to consider changing your password.

Part of the issue on my feed: I’m receiving information on users I’ve not actually friended or am following

A JIRA has been raised on the issue – WEB-4844 – and you may want to check on your own feed and see if you are affected, particularly if you are in Europe. If you are, please ensure you WATCH the JIRA (votes do not count) and supply any information you can as to how the issue is affecting your web profile / feed in order to help bring it to LL’s attention.

Direct Delivery issues: June update

On June 7th, and missed in the build-up for SL9B, the Commerce Team issued a further update on the stains of ongoing work to fix various issues relating to both Direct Delivery and listings problems. This appears to be the latest in what seems to be monthly updates. The latest post reads:

06-07-2012 02:53 PM

Below is the updated set of outstanding issues with Direct Delivery and the Marketplace.

Direct Delivery

The following Direct Delivery issues have been verified, but have not yet been addressed:

  • WEB-4600 (Merchant Outbox failures): We have been working on this issue and will not shut down Magic Boxes until this is addressed. In some cases, logging out and in from the Marketplace and then the viewer may resolve this problem.
  • WEB-4554 (Test delivery permissions incorrect): This is currently under investigation.
  • Limited Quantity Support (Merchant does not have rights to copy the items for sale): This is currently being worked on. Magic Box migration will not be required until this is supported. (Note that Merchants can sell items that have next owner rights set to “No Copy”. Please see the Knowledge Base article on Object permissions for more details on how permissions work.)

Overall Marketplace

There are also several issues that occurred around the time of the Direct Delivery launch that we are still working to address, but are not issues with Direct Delivery.

  • WEB-4587 (Listings with the wrong images): This is currently under test.   
  • WEB-4441 (Orders stuck in “Being Delivered” state): We have been able decrease the number of orders getting stuck and continue to work on preventing all orders from getting stuck.
  • WEB-4567 (Bulk delete fails for some merchants): We will evaluate the priority of this once we have completed the above Direct Delivery fixes and features.
  • WEB-4592 (Orders marked as “Delivery Partially Failed” on success): This is currently under investigation.
  • WEB-4696 (Deleted listings appearing in search results): We continue to investigate this issue.
  • WEB-2974 (Listing enhancement stuck in “Charging, cannot edit right now” state): We are investigating this issue.
  • WEB-4138 (Confirmation emails failing to deliver): We are currently investigating this issue.

In addition to the above issues, there have been reports of Direct Delivery purchases silently being delivered and the Merchant not getting paid (the order is marked as “Failed”). We have not been able to confirm this report and would like to investigate further, so please file a support ticket with details if you see this.

While progress is welcome, for some merchants the wait is beginning to to tell – and quite understandably so.

Advanced user experience tools griefing update

Oskar Linden has provided an update / post-mortem on the recent bout of griefing that took place across the grid as a result of person or persons unknown abusing the advanced user experience code that was released onto the Magnum Release Channel two weeks ago.

The problem hit on Monday 4th June when the advanced teleport functions released to Magnum were used to teleport individuals or groups around the grid, with some people reporting they were teleported to the likes of The Cornfield, while others found themselves unexpectedly picked up and dropped into stores or meetings.

Linden Lab reacted rapidly to the issue, determining a fix for the exploit on the afternoon of the 4th (SLT) and deployed across the entire grid in a rolling restart that affected the main channel and all release channels.

The key points relating to the issue remain:

  • The exploit came about due to a permissions restriction within the advanced tools not working as anticipated
  • To prevent further misuse of the code, the advanced tools were also removed from the Magnum RC channel
  • Both the code and the associated test plans have been revised and are being run through LL’s QA process to better ensure the situation of the 4th June is unlikely to be repeated when the code is rolled-out once more
  • Coyot Linden estimates that the advanced experience tools project has been delayed by around 2-3 weeks as a result of these events
  • LL are at this point in time unclear as to when the tools are likely to be rolled back out onto a Release Channel; the slot assigned to the tools on the Magnum RC has now been taken by other security issues in preparation for their roll-out to the grid.

One immediate outcome of the griefing situation is that the teleport capability has been revised so that when someone is teleported, the function will tell them the name of the owner of the object that teleported them (thus allowing any potential abuser of the system to be reported to LL via an Abuse Report).

With thanks to Nalates Urriah.

Genie out of the bottle: advanced tools capability used for griefing

Update 11th June: Oskar Linden has provided further feedback on this situation. 

Update June 6th: Oskar Linden has confirmed that the Advanced Creation Tools capabilities that were rolled-out to the Magnum RC channel will remain disabled until at least next week, although no firm decision has been on re-enabling them.

Update June 5th 11:30 UTC: The rolling restarts completed at 03:39 UTC. At present, no further updates have been given on the forum post relating to the restarts, but this may change later during the course of today. Oskar Linden has reiterated that due to this rolling restart, there will be no main channel deploy today and that details on Wednesday’s RC deploys are still TBA. My apologies for late timing of this update; a small matter of real life prevented me putting my nose in front of the keyboard any sooner!

Last week saw elements of the new Second Life Advanced Creator Tools rolled-out to the Magnum channel. As I reported at the time, the tools were issued without the new permissions system, but with safeguards that (it was hoped) would prevent misuse.

Today, however, a party or parties unknown started to use teleport functions of the new tools outside of the Magnum Release Channel as a means of griefing. People first became aware of the issue as individuals and groups started finding themselves randomly teleported around the grid, which sparked speculation on Twitter. Later, messages started circulating in-world among groups, outlining issues, such as this one, sent out to the NCI Citizens Helper Group (with thanks to Raylene Gothly)

Ok to everyone, there is something seems grid wide, we do not know if its a bad code in certain sims or if someone has found a way to teleport grief. But its happening all over, so I’m not sure. however I’d like to let everyone know we are aware of this. Suddenly you are just teleported away, best to log off and relog to get out of it, as it seems to continue to teleport you, I was teleported 3 times when I relogged… Dont be frightened it just seems to be a mess up.

The only regions unaffected by the griefing tool appear to have been those on the Magnum RC (where the new teleport functionality has safeguards) and those sims that had script capabilities disabled. Messages were thus circulated to land holders to disable scripting in their regions to avoid the issue, at least until Linden Lab responded to the situation.

Linden Lab themselves commenced efforts to stop the problem with emergency rolling restarts across the grid, announced via a Grid status update and a forum post:

To solve a security issue with the Experience Tools that were deployed to Magnum last week we are doing an emergency simulator rolling restart deploy. This has already begun.

Regions on the following channels will be restarted with the fixed code:

Main Channel
BlueSteel
LeTigre

Magnum will not be restarted because the issue is not possible in Magnum regions. We will have no rolling restart Tuesday morning. The Wednesday morning RC channels will roll at the usual time.

We apologize for any inconvenience and appreciate your understanding.

I’ll update here when further information is available.

Marketplace Update: DD migration pushed to August

Updated 19th May: The Commerce team have responded to payment issues. See the comments after this article.

The Commerce Team issued a further update on the status of ongoing work to fix various issues relating to both Direct Delivery and listings problems. Indeed, the announcement itself may have been prompted by a cascade of Tweets directed at Linden Lab’s CEO by frustrated merchants late on the 16th May These both requested help with the non-payment issue (outlined later in this piece) and appealed for news on Direct Delivery issues given the revised migration deadline of June 1st is fast approaching without any clarification on matters from LL having been received since April 26th.

The update is reproduced here in full. For those concerned about Direct Delivery migration, please note the final paragraph of the announcement.

[UPDATE: May 17, 2012]

We continue to work outstanding Marketplace issues.

Performance continues to improve on the Marketplace web site. With our deploy last week, we fixed a couple of JIRAs and order processing sped up, which led to some orders getting stuck in the Queued state (which means no money has been taken from the customer nor have any items been delivered). Orders got stuck because other portions of the system struggled to keep up. Stuck orders were Aborted, and a fix was made to decrease the number of orders getting stuck in the Queued state going forward while we work on a longer term fix. 

Here is the status of the remaining issues:

  • (WEB-4587) Listings show up with images from other Merchants listings
    Current status: we have identified the problem and are working on testing the fix.
  • (WEB-4696) Deleted listings appearing in search results for consumers and in Merchant Admin.
    Current status: we continue to work on this issue, and have decreased the incidence of these occurrences.
  • (WEB-4441) Orders stuck in Being Delivered
    Current status:  we are actively working on this issue and will focus our fixes on Direct Delivery purchases.
  • (WEB-4567) Bulk delete failing.
    Current status: the work-around on this is to delete each item. We continue to work on resolving this.

One other question that has come up frequently as we get close to June 1, is around Direct Delivery migration dates. We will not be requiring Merchants to migrate to Direct Delivery before August 1, 2012, and will give at least a 4 week notice for any shutdown dates. [my emphasis]

The Commerce Team

In the meantime, Merchants are becoming increasingly concerned over the non-receipt of payments from good sold. The crux of the issue has been raised in a forum post (which itself links to an earlier and more involved post on the subject). A JIRA has also been raised on the issue as well. So far, there has been no response from the Commerce Team on this latter issue.