In case you weren’t playing at the time, we had some downtime last night that lasted for 2-3 hours, and we’d like to explain what happened to anyone that’s interested.
Our cloud servers live inside what’s called an "environment", and we periodically create new environments in order to test things. In this case we were working on making our monitoring more sensitive; this is the tool that wakes our dev team up at night when something breaks.
The test code we were working on is close to what we use for the actual games, and when we shut down the test environment we accidentally shut down the regular one as well.
Even though we were able to bring servers back up quickly, it look some frantic work to put the environment back together because that system isn’t designed to change so quickly.
From all of us at Kano, we hope you will accept our sincere apologies for this interruption in service. We goofed, and we’re doing our best to make sure it never happens again.
In the short term we’re building some new automatic systems, so that if our environment ever goes down again, we can put it back together much faster. This will also help protect the games in case a big datacenter goes down because of a natural disaster or power failure.
In the long term, we’re moving to a whole new infrastructure system for our servers that is more reliable. Our current system is designed to be simple and easy to use, but as we’ve built more systems and settings into it we’ve grown past its capabilities.
Once we migrate to this new system, the option to turn everything off will thankfully be much more difficult to do by accident.
We hope that helps shed some light on what happened and we appreciate your patience while we put this back together. If you haven’t claimed them yet, I've posted some free Full Stamina Refills in the General Discussions forum:
http://forums.kanoapps.com/threads/full-stamina-refill-expires-nov-12.17200/Click to expand...
