Replication is never a proper replacement for backups

Today I almost had that special moment that makes you glad you did backups -- was going to drop the database and noticed that I'm on the wrong server less than a second before my finger finally reached that "Enter" key =) I wonder if MongoDB should have some built-in measures against this, maybe some kind of database/collection setup versioning that prevents actual data loss. Then again, I think that's too much for a general-purpose database, but still, having some kind of schema versioning with ability to do a rollback would be a nice feature for something that is already complex and proven to be robust enough for critical applications, like PostgreSQL. Still, the (learned before) lesson for today was that you should never count on replication alone for protection, software failures (both client- and server-side) still may destroy your data.

Homegrown replication

Recently, I was working on improving our homegrown file replication system that we use to allow redundant image storage. Currently it serves tens of millions of files that occupy about 4TiB of storage, about 8-10 gigabytes are added per day. In reality, it's not a "replication system" by itself, it's just a system that delays writes to unavailable targets until they become available. It turned out to be very efficient and resilient even to the "we just lost two disks" cases, without any centralized authority.
We considered both custom filesystems for both Linux and Windows, but all of them required some kind of a central management server or servers, which we rather not have. Also, we considered storing files in MongoDB GridFS, but a simple session with calculator told us that replacing a node (taking down, adding another one, syncing before oplog gets exhausted) for such volumes and large items would be prohibitive, copying a virtual disk image is much simpler and faster than doing it though the database layer, so while the idea of specialized file storage in database is very appealing, MongoDB GridFS deployment for terabyte-scale files with intensive write load requires considerable amount of preplanning that defeats the main (well, for me) feature of MongoDB -- simplicity.