Database Issues & Fallout

Database Corruption Issues

This quick newsletter brings sad tidings. A couple of days ago our host had a hard RAID failure on our box. Once recovered, we found that we had massive database corruption. We were able to restore a very recent backup of the drupal (content side) application. We unfortunately found that our phpBB (forum) application backup was quite old. We initially thought that we would have to roll back all the way to the Drupal From > phpBB migration (early November). However, we found that we did have a phpBB backup from mid-January.

What Happened

Why didn't you have real backups, you might ask? This is a great question, and I have an answer. We've had regular backups running on the site (Drupal) for quite some time. The Drupal Advanced Forums were a part of this install and db tables. All was good. When we migrated from Drupal AF to phpBB, the forum data moved to its own mySQL tables and structure. In all of the weeks-long migration work, the task of expanding the backup routine escaped the team.

This explanation is no excuse, and we should have caught the gap. The reality is that everyone on the WSGF staff is volunteer, and has their own day jobs and IRL responsibilities. We all have households, wives, and many of us have kids. We all love the WSGF, and we pour our hears and souls into it. However at the end of the day we are a virtual team (spread across three continents and two hemispheres), and sometimes mistakes happen even in the best of situations and organizational structures.

Fallout

We didn't lose any Drupal data (DRs, articles, benchmarks, blogs, etc.). We have lost forum posts back to Jan 15th. The unfortunate thing is that tomorrow we will have to wipe the user records back to Jan 15th a well. As it stands now we have a whole host of users in the Drupal tables that are not in the phpBB tables. Any user in this "orphan" group who logs into the site will have their UserIDs end up out of sync between the two systems. And, any new user who signed up after the recovery (and before we turned off new user reg) will have their UID of out sync. For the Drupal>phpBB Bridge to work properly, UserIDs must be in sync.

Once we get the user tables purges and synced, we will give the all clear and anyone will be able to re-register. We apologize for the issues, and the inconvenience. We work hard to gain and maintain your trust, and this instance we've let you down. Please know that we have learned from our mistakes and already fixes the forum backup issues. We are also working to provide better site performance and to minimize downtime and outages.

Keeping up to Date

As we make this last recovery step, the best way to keep updated will be through our Facebook, Twitter and Steam accounts. Also, the staff (and a growing number of users) are always in our IRC channel, and would love to see you join in the conversation. All of our social media outlets are listed below - just click on the icon.