As you are probably aware of, our server provider Amazon Web Services suffered failures in the US East availability zone (where our servers are stored) yesterday at an unprecedented level. Other major web services impacted include Reddit, Pinterest, Foursquare, Airbnb and the list goes on... Service was unreliable for about +16 hours, during this time we were faced with a locked up database server and had the choice of either keeping LCN down for a potential 8+ hours or revert to previous backup. We decided to revert to the most recent backup and bring the game back online (taken before the 10:30am outage). It was critical to get the game back online for time based events like the arena and calendar, there was also uncertainty that the database could have been recovered. Unfortunately with this there was some data loss in a DB that contains "lower priority" data as follows: 1) Invites - people may be missing invites sent after the backup, may need to resend 2) Gifts - missing gifts sent after the backup, may need to resend 3) Faction member join requests - missing requests sent after the backup, may need to rejoin 4) Private messages - missing private messages sent after the backup 5) Boss attacks - may not get credit for bosses that they helped on after the backup For the Spooktober calendar, because of the downtime and community suggestions we increased the end time by an additional 12 hours. If you are experiencing any issues with this please write into support and we can look into your account. We apologize for the inconvenience, it's our priority to have these games online and stable and work very hard on stability and uptime even during unplanned outages from Amazon. We made the decision that with the impacted database that the priority was to get the game up as quickly as possible with minimal data loss on the data mentioned above as opposed to running a +12-hr recovery for potentially a few hours of data on data mentioned above. We place a high importance on uptime and data integrity, have invested a lot in this area and with unforeseen issues due to Amazon yesterday we will continue to do so. We would also like to mention how much we appreciate our volunteers and members of the community who helped us spread information yesterday, and worked towards keeping everyone calm while we worked towards getting the games back up. We know it was an incredibly hectic time, and it was awesome to see everyone working together. We can't tell you enough how great you all were.
