Looks like 2.6.32.36 kernel on Xen has some timer problems, see https://jira.mongodb.org/browse/SERVER-4709 (we ran into it when using MongoDB).

Windows Azure, [not grand] finale

It appears that while Windows Azure has a lot of good points (for example, the idea of PaaS is pretty good, since Web Roles look not that much different from the fabled Heroku deployment), the built-in limitations (like 20 cores limitation per account) and the inability to run different OSes (custom Windows images via VM Role are fine, but we need Linux also) will make our migration much harder. In fact, we only need Windows for application servers, Linux runs everything other just fine and is much easier to staff. So, while Azure looked fine in the first place, it looks like Amazon Web Services would be our choice (yes, it has its own bad limits like 2Gbps EC2->EBS bandwidth cap, yet it appears to be more flexible).

7 million hits and transfer billing

A week ago we had 7 million daily hits on one of our websites and today we have it another time, another small, yet important milestone for us =)
I'm happy we aren't growing exponentially though, even linear growth with limited resources wasn't that easy to handle in terms of performance and (unsurprisingly) staffing.
The biggest headache this far though was media storage, which currently has several terabytes of images stored on our hardware with over a hundred TiB served per month (yes we aren't THAT big yet), and that we're looking forward to move somewhere like Windows Azure, but the estimated transfer bills are, well, shocking: ten times more expensive than we pay now. I wonder if transfer prices will drop soon. While storage is really getting cheaper (hard disks are getting cheaper, well they were until that flood), there is no visible driving force that could cause transfer rates go down.

Windows Azure, part 2

When I first checked out Windows Azure, I was glad to find it has root containers. Unfortunately, they are almost unusable for us, since they do not allow subdirectories (see the docs for that). I learned that the semi-hard way, stumbling upon the very helpful StorageClientException: "The requested URI does not represent any resource on the server." So, although root containers technically exist, with this limitation they are of no use.

Also, while the retry policy and timeout handling in Windows Azure .NET SDK is fine, exception handling is not. While getting StorageClientException and StorageServerException is expected, the WebException is not expected at all (I thought that one should be wrapped in StorageServerException).

Other than that, though, Windows Azure .NET SDK is pretty straightforward and easy to use.

Windows Azure first experience

Windows Azure looks to be a fine platform, but the toolset installation could be more streamlined. First, Azure Tools for Visual Studio complained about "Error 0x80070643", which was resolved by installing Azure SDK, Libraries and Emulator before. Then, emulator told me that I do not have SQL Server 2008 installed by popping up with a helpful message that there is a "possible security problem, see here", which leaded me to a page that had nothing to do with real cause of the problem.
After that, though, everything went smoothly. Since we're mostly interested in storage side of cloud services, I'll explore Windows Azure storage and probably will write a post or two about it.

Basic SSD tuning for MongoDB

We explored several options when using MongoDB on an SSD and came to following conclusions:

1. Don't turn off MongoDB journaling unless you really 100% sure -- the degree of comfort it provides you after your server restarts non-gracefully, is enough to warrant it's usage even within a replica set.
2. Use ext4 file system, mounted with "noatime,data=writeback,nobarrier" (nobarrier didn't give measurable differences on our workload, but others say it's still a good thing). ext4 is fast when allocating files (see 3. below), along with allowing you to delay file metadata updates (reliability is already covered by MongoDB journals).
   Correction from the year 2019: just use XFS.
3. Enable MongoDB options smallfiles and noprealloc (unless you're writing an application that is very heavy on inserts that is going to push the SSD to it's limits). SSDs still cost a lot of money and if you're installing 120GB or 160GB ones as we do, you don't want five empty databases occupy a gigabyte of that precious space (we run with directoryperdb=true also, it's handy for management). With smallfiles=true, noprealloc=true works just fine -- 512MiB files that get created by MongoDB are allocated in abour 300-600ms even under load, thus saving you even more space.

Improving robustness for C# MongoDB clients

I wonder if I should publish a set of tools and patches that make easier to write close-to-zero-downtime-without-users-noticing-that-half-servers-are-gone applications. Guess I'll put in a bit of effort to make it better suited for public release, like translating all the documentation comments from Russian to English :)
The basic idea for the tools is to provide a side-attached layer that gracefully handles failure and retries the operation if the tool decides that it still might succeed. While the idea is easy, it really works for something as crude as pulling the plug for half of the servers with users noticing only a slight (several seconds max) delay with their web pages load times for a few seconds.