On Saturday, September 5th, Bright Canopy held an in-world meeting and their island in Second Life to discuss recent events regarding the Bright Canopy service (you can read the background here).
In particular, the aim of the meeting was for the Bright Canopy team to share what they’ve learned since moving to launch the service on August 29th, and discuss the options needed to make the service sustainable going forward.
Both Bill and Jerri Glover (Chaos Priestman and Beth (Bethsael) Robbiani in SL) were present at the meeting, which was held in text, and a transcript of the chat log is available on the Bright Canopy website. What follows here is a high-level summary.
The meeting opened with Chaos providing some historical context of how Bright Canopy came into being, paying particular attention to how the service is structured, as this is important to grasp. In summary:
- Bright Canopy manage the service and take the viewer and tweak it to run as a part of a cloud service
- The Bright Canopy service is delivered to subscribers using Amazon’s Elastic Compute Cloud (EC2) g2.2xlarge server instances (1 per user)
- This delivery via Amazon is facilitated by Frame, a company with considerable experience in provisioning optimised Windows applications to users via the cloud.
All of this obviously involves costs – most notably with both Amazon and with Frame. In order to minimise the costs with Amazon, the most efficient means to provision Bright Canopy is using Amazon’s Spot Instance pricing mechanism. Since its introduction, this has generally pitched at around US $0.12-$0.15 per hour for g2.2xlarge server instances.
Unfortunately, at the start of August 2015, the Spot Instance pricing for the server instances started spiking, first in Ireland, then in both California and Virginia – the three Amazon POPs Bright Canopy would be using via Frame. These spikes meant that instance costs ballooned from under US $0.25 an hour to anywhere between $1.00 and $8.00 depending on the location.
“Our business model was based on Amazon’s Spot Instance prices remaining below $0.25 as they had since they were introduced,” Chaos explained. “That’s just the cost of the instances. That doesn’t include Frame being paid or Bright Canopy being paid … This [spiking] broke our business model, but it looked like a temporary spike. We decided to continue with the planned launch. We believed the prices would come back down.”
To try to counter the unpredictability of the Spot Instance prices, Bright Canopy moved to Amazon’s On Demand pricing. This is far more predictable than spot Instance, but comes at a premium – US $0.80 an hour – leaving the service losing money.
“We hoped that usage would even out in such a way that we would lose money slowly enough to maintain our course until we could build out a solution that cost less on the back-end,” Chaos said of the move. “In the meantime we also hoped the Spot prices would come back down and give us some relief.”
Following launch, however, user behaviour changed quite dramatically. People were spending much longer periods logged-in, both increasing costs and forcing the use of even more server instances.
“It became clear that we could not sustain the losses,” Chaos said. “Usage was just not the same as we had seen in Pre-release. We expected a difference, but we didn’t expect such a huge difference. We agreed to pull the plug and rethink things.”
More recently, the Spot Instance prices in the USA have showed signs of settling down once more. However, it is still too soon to know whether this is an indication that prices are resuming their pre-August levels, and Ireland has certainly remained volatile.
So what does this mean for the service?
Most immediately, it means that the service will not be back up for Monday, September 7th, the date Bright Canopy had indicated as being the earliest by which it might be resumed. Instead, things remain in what Bright Canopy is calling a holding pattern until such time as a consensus has been reached on the best, most sustainable means of moving the service forward for the benefit of those needing it.
“We have worked with Frame on a proposed plan that we would be able to offer to a limited number of people at first,” Chaos said. “We have not come to an agreement yet on all of the details of that plan. If and when we do, please understand that this is just a stopgap so that the people who most need the service will have an option.”
This approach is intended to meet at least some of Frame’s costs (who up until now, as with Bright Canopy, haven’t received any income from the venture), as well continuing to meet Amazon’s charges. Bright Canopy will continue to work on the service unpaid, but will have to step back from 24/7 support and response and for the timing being to providing responses to questions and support requests within 24 hours.
The main presentation was followed by a Q&A session, which covered a number of topic areas, and I recommend those interested read the transcript in full to see both questions and answers.