mod_gridfs

As we were planning to move our terabytes of files into MongoDB GridFS, it occured to us that there is no readily available way to efficiently serve these files over the web, without resorting to using an ASP.NET GridFS IHttpHandler we implemented for local debugging some time ago.

After much hassle while developing GridFS handler for G-WAN (it is certainly fast, but keeps crashing even on Ubuntu 10.04.4 LTS, both x86 and x64 versions, even when I removed all non-boilerplate code from the module itself), performance measuring https://github.com/mdirolf/nginx-gridfs (until there is an asynchronous MongoDB driver, any GridFS module for NGINX is doomed), I decided to write an Apache 2.x (actually tested on 2.4) module to serve files from GridFS.

I decided to release it as open source here: https://bitbucket.org/onyxmaster/mod_gridfs/. It runs faster than the NGINX one, even with multiple workers, so we're going to use it as a backend (with NGINX as a caching frontend of course).

Configuration example:
GridFSConnection rsTest/db1,db2
GridFSDatabase my_database

NGINX and keepalives to backend

I wonder when NGINX will have keepalive connections to backend? Patch by Maxim Dounin was floating around for about half a year, then it made into beta, but it's still in beta now, while our media frontend server is busy creating/destroying several thousands of connections to backends instead of reusing existing ones, there are only about 20 connections needed to serve ~3.5kreq/s (current load). Well, maybe G-WAN will be better, although AFAIK it doesn't come with a prebuilt proxy caching.