The SL Marketplace has appeared in these pages a few times these past couple of weeks, and not in a good way. First was my coverage of the latest errors relating to listing enhancements, to be followed by the news that LL’s Commerce Team had apparently seen the error and taken steps to correct it and refund people. Only as it turned out, they’d only fixed half the problem, which resulted in people again being incorrectly billed.
Marketplace: a dispiriting place?
Some Good News
On the 8th October the Commerce Team issued the following update:
Merchants,
There has been quite a bit of discussion on Product Listing Enhancements and stuck orders over the past few days on this Forum. Here is an update on these issues.
After we refunded merchants, billing for Product Listing Enhancements started again. Not all of the billed enhancements correctly updated their renewal date, so we stopped billing. We have updated the PLEs that had an incorrect date and will be releasing a fix this week before we start billing again.
We are aware that there are some Merchants who have Product Listing Enhancements stuck in the “Charging, cannot edit” state. We are continuing to look into this and investigate what we can do to get those PLEs unstuck.
Stuck Orders: Late Friday night through early Saturday morning, many orders got stuck in the being_delivered state. This morning we were able to force those orders to a completed state (allowing payments to Merchants to complete) and are working to prevent this from happening again.
If you are seeing anything different than [sic] the behavior described above (or have an additional problem), please contact support or file a JIRA. Please include order numbers or listing information as needed.
The Commerce Team
On the whole, this is welcome news, regardless as to how the problems originated.
Please use the page numbers below left to continue reading this article
As reported last week, Wednesday October 3rd saw a massive problem hit the LeTigre Release Candidate channel, which impacted over 1200 regions. This most visibly manifested itself as a large number of items (including partial builds) being returned to people’s inventories as a result of regions being seen as “full” by the software as a result of an error in the prim accounting code. This saw disruption across the grid throughout Wednesday and into Thursday, partially because those regions impacted by the error not only required a corrective deployment of RC code (from BlueSteel), but also had to be manually restored to a state prior to the LeTigre deployment occurring.
Since the problem occurred, Linden Lab has not only been looking into the bug within the prim accounting software, but also at their internal processes in terms of why the error wasn’t picked-up prior to the LeTigre deployment going ahead, and also in terms of what steps can be taken to curtail such a massive disruption in the future, should ever a similar problem occur, and how regions can be restored in a less manually intensive manner. Even so, sorting out a solution which fits every possible scenario by which a deployment may go wrong isn’t easy.
Speaking at the Sim / Server User Group meeting, Simon Linden commented on the matter thus, “The tough thing with SL testing is running it on all those combinations – it’s just never possible to have complete test coverage. The RC channels are actually designed to be representative of the whole grid, so we try to keep a mix of the different types of regions like full, mainland, Linden Homes, etc. On one hand, the system worked … we found a problem before it got to the whole grid, which might have been how this would have happened before we started the Release Channels. But it really was so bad we definitely want to catch something like that even earlier. So … we’ve had a bunch of meetings discussing what and why it happened, and have some better tests added to the regular test pass so this specific problem won’t happen again.”
Options to prevent a similar issue occurring again in the future which have apparently been discussed at these meetings include:
Improving the testing carried on Aditi. Part of the problem here is that as a representative testing environment, Aditi is very much smaller and much less diverse than the main grid, and as such it is harder to test for all possible failure conditions which may occur when deploying code to the main grid
Adding alarms to the deployment process so that when things do go wrong, such as a large number of object returns occurring, the process will automatically stop itself before the damage becomes widespread
Altering the deployment process so that code is initially rolled out to a subset of each Release Channel, prior to it being paused for a few hours to see if there are any reports from users of unexpected or undesirable results , and only resuming the deployments if it appears nothing untoward has happened
While it is the first time this particular problem has occurred in terms of selective region object returns, it has prompted the Lab into looking at ways and means to initiated an automated restore process in order to make the rollback of affected regions more time-efficient and less intensive.
It remains to be seen which of these – and any other ideas – which have been discussed at the Lab are implemented. However, it should be remembered that even with the best will in the world, and given the dynamic nature of Second Life with all the user-created content and scripting, it is impossible for Linden Lab to take into account every single possible error which may occur with a server deployment, and provide a means of avoiding it. Even so, as a result of the LeTigre event, LL are looking to further improve how server code is both tested and deployed in the future, and provide the means to better flag any negative impact occurring during a deployment in order to allow remedial action to be determined and actioned in a more timely manner.
Well, it seems news over the correction in one aspect of the ongoing SL Marketplace listing enhancements debacle (itself merely one part of the overall Marketplace debacle) was premature.
No sooner had the Commerce Team announced they were refunding people for the mess-up over payments, that automatic debiting for enhancements resumed, with the same level of confusion as to what is actually going on, and people unable to determine exactly what they have or have not been charged for. How this came to pass is unclear, although I do tend to agree with Darrius Gothly’s assessment of the situation, vis:
When your staff went through and refunded everyone, you should have AT THAT TIME tested to be sure your code modifications would not immediately undo everything just done. But did you? Nope. As a result it went through and lickety-split re-billed everyone .. not only for what they’d just been refunded but additional charges too. Pardon me but .. WTH?!? By dint of your lack of attention you have just completely undone everything your staff did .. by hand .. at great expense to your employer. You have WASTED a very large amount of money. Wasted because you could not or did not want to bother testing your changes.
It is perhaps bad enough that people have seen refunds enter their accounts only to evaporate once more. But it would also appear that people are again getting charged for enhancements they cannot cancel due to WEB-2974 (an issue now some two years old, and resolution of which was “on hold” as of July 2012).
This state of play is, frankly, ridiculous. While mistakes can and do happen, what has been going on within the Marketplace and on the part of the Commerce Team long ago reached a point of farce. Even the simplest of tasks appears to be beyond their capabilities (or the capabilities of the software they manage). Remember the change to the sales notification e-mail address I mentioned as being rolled-out on September 26th as a part of my last general SLMP update? Guess what was rolled back just 48 hours later, only to be rolled-out once more on October 4th?
One has to question a) the level of competence within those responsible for managing and coding SLMP; and b) the overall condition of the Marketplace code itself, as it seems utterly incomprehensible that even the most basic issues within the system appear to be beyond LL’s grasp to fix.
In his comment on the matter of listing enhancements, Darrius concludes:
Communication from your team to us is a major issue. I’ve no doubt why this is the case. Most people have a very difficult time going to others with the need to say “We’re sorry, we screwed up.” With the number of times you must begin a blog post in that manner, it’s no wonder you don’t post very much at all. So here’s an idea … stop being lazy, stop short-cutting things and rushing changes into production, stop screwing up .. and STOP having to begin every post with an apology.
While I agree with his point of view, I’d go a step further.
It doesn’t matter as to whether or not these issues are only affecting a “small number” of merchants (as the Commerce Team have repeatedly stated); it also doesn’t matter as to whether LL regard L$ as “real money” or “tokens”.
What matters is that the company actively encourages people to get involved in their platform’s commerce engine, and to invest time and money in it – and they promote the Marketplace as a major means for people to do so. People have taken LL at their word, and for many of those affected by all the Marketplace screw-ups over the years, it very much is the case that real money is involved, and real stress and real upset.
As such, it is time for someone within Linden Lab to recognise this, take responsibility and step forward with a sincere apology for the manner in which the entire litany of mistakes, errors and mishaps going back as far as at least 2010 has been handled. They then need to go on to ensure issues are managed in such a way that people are kept properly informed on progress, and that issues are not exacerbated by what appears to be either flaws in internal processes – or carelessness.
Simply saying people are busy “crunching numbers” doesn’t really cut it any more.
As it is, a decent projection as to when LL will “have a fix” for Marketplace problems, would appear to be, “Around the 12th of Never”.
As some of you are aware, we have been giving away free listing enhancements to many of you for the past 2 months. We recently deployed a fix that was supposed to just start billing again. Unfortunately, it back-billed the free enhancements. We apologize for doing this without warning.
To show our appreciation for your patience with this mix-up, we will be doing the following:
Giving everyone free listing enhancement renewals* for the period between August 7, 2012 and today. This means all listing enhancements billed during this time will be refunded before end of day today (October 4, 2012).
Once the refunds have completed, we will begin billing those enhancements again.
*If you created a new listing enhancement after August 7, 2012, that listing enhancement is not eligible for this refund.
While it is interesting that the post presents the suspension of listing payments as being a part of a “free listing enhancements” offer – an offer merchants appear to have been unaware was actually being made (I also can’t remember having seen any notification, but then I could easily miss an elephant walking by unless it happened to tread on my foot in passing).
There has been some connection between my earlier post on the matter and the above announcement from LL. While I’m sincerely flattered that such connections have been made, I’m nevertheless sure the timing was little more than coincidental.
Hopefully, once the round of refunds has been completed, this matter at least can be pout to bed where the Marketplace is concerned. Hopefully, there will be more communications on the status of the remaining issues in the near future.
With thanks to Toysoldier Thor for notifying me of the LL forum post
Linden Lab depend on land tier (server space, call it what you will) for 80%(ish) of their revenue. This places them in an awkward position vis-à-vis providing any form of tier easement, even if they wanted to (as I’ve commented before).
The remaining 20% comes from the likes of Premium membership, and more particularly, the SL Marketplace (SLMP). In the case of the latter, the revenue doesn’t only come from the 5% commission on goods sold through the Marketplace, a lot of it comes via the listing enhancements merchants are encouraged to pay for. These theoretically boost sales by placing items in places such as the SLMP home page, or on the Checkout pages, and are paid for on a 30, 15, or 7-day rolling subscription basis, costing merchants between approximately $5.00 and $12.00 USD a month per item, with some merchants paying over $200 USD per month for enhancements.
Listing enhancements – can amount to a pretty penny in outlay per item
Earlier this year, the system went haywire, failing to take due subscriptions for around a two-week period (JIRA WEB-4638). It was finally resolved by Linden Lab taking a single, large payment from merchants’ accounts. While people had no issue in paying for services rendered, the problems here were that a) next to no forewarning was given that accounts were about to be so debited, leaving many merchants with a sudden and unexplained drop in their account balances; and b) many were billed in excess of the two weeks subscriptions actually owed (with some reporting being billed for up to four weeks); while c) the billing information received made it hard for merchants to actually determine which of their listing enhancements had been billed, or even if the right enhancements had been billed.
This understandably led to some confusion within the merchant’s forum, and not a little upset, particularly as some merchants had also been faced with an inability to cancel some of their listing enhancements due to an ongoing issue with many items remaining stuck in a “locked” mode, preventing them from being edited – a situation itself which at the time was some 18 months old (and is now some two years old, and still awaiting resolution – see WEB-2974).
While merchants were refunded for any overcharging on their account as a result of the billing issue, the manner in which the situation was handled by LL resulted in something of a drop in trust where the Marketplace is concerned, and a number of merchants publicly indicated they would be ceasing in their use of enhanced listings.
At the end of July, the problem started again, and was raised as a topic for discussion in mid-August, as well as having a new JIRA (WEB-4927) raised against it. Neither the discussion thread nor the JIRA drew comment from the Commerce Team. Instead, a single payment was taken from all “at fault” accounts, again without any forewarning, and again with Merchants facing issues over what, precisely, they have been charged for, and whether those listings they have made payment against are actually active.
Failed subscriptions for August – courtesy Ry0ta Exonar
Again, the problem here is not so much that things Went Wrong and broke again – that’s pretty much taken to be the standard operating condition for the Marketplace nowadays – but how the Commerce Team managed the issue. Almost nothing was said on the matter (again), with the only communications forthcoming from the Commerce Team being a terse instruction not to re-open WEB-4638 after Ry0ta Exonar attempted to do so (hence WEB-4927), and a brief Marketplace dashboard message posted on September 28th, which simply said:
We are aware of some issues with Product Listing Enhancements. Keep an eye on the Grid Status page for more details.
With neither the dashboard message or Status page message actually stating what was about to be done.
So is it little wonder that merchants are again looking at listing enhancements with a jaundiced eye? Several have re-stated the fact that they will no longer participate in the process and others have stated they are terminating – either automatically or manually – their subscriptions. Given that LL are seeking to increase their non-tier related revenues, one would think that ensuring the one service which does so is run with a level of professionalism and communication that would not undermine customers’ faith in the service, or their willingness to place money into it.
Currently, and added to the rest of the ongoing litany of issues and problems with the Marketplace, this doesn’t appear to be the case.
On September 20th, Commerce Team Linden provided a terse update to the ongoing problems related to direct delivery. The Update reads in full:
The transition from Magic Boxes to Direct Delivery has been extended indefinitely. We will be providing a 30-day warning before any shutdown actions are taken and will avoid doing this before peak holidays for the Marketplace (Halloween, Christmas/New Year’s, Valentine’s Day).
We are aware that some Merchants are still having problems with the Merchant Outbox. We are are working with TPVs and our internal development teams to address this issue.
The comment relating to TPV involvement possibly relates to WEB-4600, VWR-28630/VWR-28631 and VWR-28629, although it has caused some confusion, as there appears to have been little or no recent discussion on matters between LL and at least one TPV team.
The update gives no further information on the status of the range of JIRA items related to the Marketplace which have been under investigation now for the greater part of 2012. These items comprise:
Limited Quantity Support Merchant does not have rights to copy the items for sale (no JIRA number supplied)
WEB-2974 (Listing enhancement stuck in “Charging, cannot edit right now” state)
WEB-4696 (Deleted listings appearing in search results)
Meanwhile, in the Marketplace release for September 26th, 2012, the Commerce Team did successfully change the “sent from” from email address to service@mail.Secondlife.com for all sales notification e-mails sent to merchants (this was previously no-reply@marketplace.secondlife.com). However, as no formal notification was apparently given on this change ahead of time, many merchants only discovered the change as a result of their e-mail applications treating incoming sales notifications with the new “from” address as spam …
Inara Pey: Living in a Modemworld 