Just after a late night deployment I received this error message while running queries against the production database I just deployed to:

Got error 28 from storage engine query

I could connect to the database fine, but any database query returned that error. To make things worse, that error had also prevented some of the deployment process I just ran from completing successfully, so our game was down until I found a fix. Luckily that didn’t take long. Some quick Googling revealed that the database server was out of disk space:

From http://drupal.org/node/44160
—
This happened to me this morning =)

The error come when MySQL doesn’t have any free hard disk space to write to. Check your /tmp directory, that’s where I had run into problems. I had forgot to make my backup script delete older backups and it kept writing backups until the disk was full.
—

That post, and most others I found, suggest to check and clear the server’s /tmp directory, but ours was fine. It had 20 – 30 session files in it and little else – that wasn’t the problem. Instead, our deployment process backs up the database on each publish, and doesn’t clean out old backups. Needless to say, w/multiple publishes happening w/in the same day at times, w/each creating a new database backup, we clearly hit a point where our backups were consuming too much disk space. Once those were deleted, we were back up.

It’s worth noting that whether it be the /tmp directory, or removing other files, the problem is disk space.  Free some up, regardless of how you do it, and you’ll get past this error.

I hope this saves someone some time and panic. Now to see about flushing old backups. 🙂