We had a hard disk crash on 9/28 around 11:15pm. The good news is that our backup process worked and we only lost about 5 minutes worth of updates.
Our fancy hardware raid controller didn't help us a single bit however, as the raid "stopped working" quite a while ago and the hosting company didn't notice.
What saved us is our off-site, remote database replication. We rely on hardware raid on our server as first line of defense. However, we also do remote database replication to another hosting company off-site in Florida. Without this last night we would have been in big trouble.
What does this mean for you
1. Outbound email ship notification will be delayed for 9/28 shipments.
2. Please check your transactions, especially your PayPal transactions, from 11:00pm last night (9/28) to 5am this morning to make sure nothing is missing.
If you see any missing transactions, we recommend you turn on PayPal alternative API integration for one day so it can catch the missing orders.
Please contact us if you have any issues.