mod_gridfs is dead, long live gridfs_server

Long time no write, but mod_gridfs is no longer going to be developed, it's being superseded by gridfs_server. It's written in C#, based on ASP.NET Core, Kestrel, runs on .NET Core 2.0 Preview 2, and successfully serves about 25 million files per day from our MongoDB GridFS installation, as a backend to a front-end cache (that serves about 450 million files per day).

mod_gridfs v0.4: tag sets

Short news: mod_gridfs is now v0.4, supporing read preference tag sets along with having some performance and robustness improvements. Addition of tag sets allows many interesting things, for example fair 1-to-1 load balancing between shard replicas.

mod_gridfs v0.3

Well, it was long time since my last post, but at least now I have something to tell about. mod_gridfs, an Apache 2.2+ module that serves files from MongoDB GridFS is now v0.3, supporting authentication, custom prefixes, read preference (currently only mode string is supported, tags will be added later), improved memory footprint and error logging.

UPDATE: Custom collection names are also supported now.

mod_gridfs performance

In my previous post, I announced mod_gridfs. Now, it's time for some numbers. Serving a 3KiB file over a gigabit network on modern hardware, 100 concurrent requests, MongoDB replica set of 3 machines as a backend:

• NGINX + nginx-gridfs: 1.3krps
• Apache + mod_gridfs: 6.6krps
• Apache + mod_gridfs with SlaveOk and one slave: 12.2krps

Not testing with larger files, because this way I'll be benchmarkng OS I/O performance instead of user-mode code.

mod_gridfs

As we were planning to move our terabytes of files into MongoDB GridFS, it occured to us that there is no readily available way to efficiently serve these files over the web, without resorting to using an ASP.NET GridFS IHttpHandler we implemented for local debugging some time ago.

After much hassle while developing GridFS handler for G-WAN (it is certainly fast, but keeps crashing even on Ubuntu 10.04.4 LTS, both x86 and x64 versions, even when I removed all non-boilerplate code from the module itself), performance measuring https://github.com/mdirolf/nginx-gridfs (until there is an asynchronous MongoDB driver, any GridFS module for NGINX is doomed), I decided to write an Apache 2.x (actually tested on 2.4) module to serve files from GridFS.

I decided to release it as open source here: https://bitbucket.org/onyxmaster/mod_gridfs/. It runs faster than the NGINX one, even with multiple workers, so we're going to use it as a backend (with NGINX as a caching frontend of course).

Configuration example:
GridFSConnection rsTest/db1,db2
GridFSDatabase my_database

Looks like 2.6.32.36 kernel on Xen has some timer problems, see https://jira.mongodb.org/browse/SERVER-4709 (we ran into it when using MongoDB).

Basic SSD tuning for MongoDB

We explored several options when using MongoDB on an SSD and came to following conclusions:

1. Don't turn off MongoDB journaling unless you really 100% sure -- the degree of comfort it provides you after your server restarts non-gracefully, is enough to warrant it's usage even within a replica set.
2. Use ext4 file system, mounted with "noatime,data=writeback,nobarrier" (nobarrier didn't give measurable differences on our workload, but others say it's still a good thing). ext4 is fast when allocating files (see 3. below), along with allowing you to delay file metadata updates (reliability is already covered by MongoDB journals).
   Correction from the year 2019: just use XFS.
3. Enable MongoDB options smallfiles and noprealloc (unless you're writing an application that is very heavy on inserts that is going to push the SSD to it's limits). SSDs still cost a lot of money and if you're installing 120GB or 160GB ones as we do, you don't want five empty databases occupy a gigabyte of that precious space (we run with directoryperdb=true also, it's handy for management). With smallfiles=true, noprealloc=true works just fine -- 512MiB files that get created by MongoDB are allocated in abour 300-600ms even under load, thus saving you even more space.

Improving robustness for C# MongoDB clients

I wonder if I should publish a set of tools and patches that make easier to write close-to-zero-downtime-without-users-noticing-that-half-servers-are-gone applications. Guess I'll put in a bit of effort to make it better suited for public release, like translating all the documentation comments from Russian to English :)
The basic idea for the tools is to provide a side-attached layer that gracefully handles failure and retries the operation if the tool decides that it still might succeed. While the idea is easy, it really works for something as crude as pulling the plug for half of the servers with users noticing only a slight (several seconds max) delay with their web pages load times for a few seconds.

Replication is never a proper replacement for backups

Today I almost had that special moment that makes you glad you did backups -- was going to drop the database and noticed that I'm on the wrong server less than a second before my finger finally reached that "Enter" key =) I wonder if MongoDB should have some built-in measures against this, maybe some kind of database/collection setup versioning that prevents actual data loss. Then again, I think that's too much for a general-purpose database, but still, having some kind of schema versioning with ability to do a rollback would be a nice feature for something that is already complex and proven to be robust enough for critical applications, like PostgreSQL. Still, the (learned before) lesson for today was that you should never count on replication alone for protection, software failures (both client- and server-side) still may destroy your data.

MongoDB connection affinity

When using MongoDB via C# driver (might apply to other drivers as well), if you're queuing modifications (i.e. using SafeMode.False) and expect to see the first modification done before the second (I was Remove-ing one item and Insert-ing another one with the same key), never forget to use the same connection (via MongoDatabase.RequestStart), unless would like to get unpleasant surprises, like unexplainable intermittent failures that only occur when your application gets enough load. While after having a close look it is obvious that write ordering is not preserved by default, it wasn't obvious enough when I wrote the initial code for our message queue.

Homegrown replication

Recently, I was working on improving our homegrown file replication system that we use to allow redundant image storage. Currently it serves tens of millions of files that occupy about 4TiB of storage, about 8-10 gigabytes are added per day. In reality, it's not a "replication system" by itself, it's just a system that delays writes to unavailable targets until they become available. It turned out to be very efficient and resilient even to the "we just lost two disks" cases, without any centralized authority.
We considered both custom filesystems for both Linux and Windows, but all of them required some kind of a central management server or servers, which we rather not have. Also, we considered storing files in MongoDB GridFS, but a simple session with calculator told us that replacing a node (taking down, adding another one, syncing before oplog gets exhausted) for such volumes and large items would be prohibitive, copying a virtual disk image is much simpler and faster than doing it though the database layer, so while the idea of specialized file storage in database is very appealing, MongoDB GridFS deployment for terabyte-scale files with intensive write load requires considerable amount of preplanning that defeats the main (well, for me) feature of MongoDB -- simplicity.

MongoDB as an MQ persistence solution

After reviewing several message queues (ZeroMQ and RabbitMQ looked fine, but ZeroMQ has no durability and is in fact more a protocol than an MQ, and RabbitMQ has that "solutionness" all around it), we decided to roll our own simple MQ, using MongoDB (multiple publishers/multiple subscribers with implicit configuration). After I completed the implementation, a reference to a nice article about MQ in MongoDB popped up on mongodb-user or mongodb-masters list, so I got assured that the idea itself was O.K. Too bad I found another article on the same topic late enough -- while I had the idea of using capped collections (and rejected it because write performance isn't that important for our MQ, while persistence is), using tailable cursors just didn't even appear in my mind, although I knew of the feature's existence. Many thanks to the author =)