Puppet was slow under Windows for us, now it isn't

In short, I've found a bug in Puppet running on Windows that, when fixed, made our catalogs apply about 10 times faster. See PUP-9092 for details and puppet#7033 for the fix.
It appears it was an oversight instead of an error, but still... having catalog applied in 31 seconds instead of 320 is very satisfying. Hope it being fixed in official Puppet releases soon, so we don't have to keep patched version installed.

P.S. I had to take a look at it sooner, I swear, it only took about 10 minutes to find the root cause, but it always seemed less important than other things.

Do not use "downlink delay" on Cisco Nexus if vPC peer-keepalive is done through the access ports

The Cisco Nexus 3000 series switches with 1GE copper interfaces support the "downlink delay" feature, that looks really helpful in the first place, since it blocks traffic flow until the switch is connected to the core. But, you should be very careful when combining it with vPC if peer-keepalive is built either over the access copper ports or the downlink ones (a non-existent scenario, since usually you can't spare even one downlink port with the usual 4-port configuration), instead of default recommendation for mgmt0.
With downlink delay configured, the access ports come up with a specified delay (30s default), leading to peer-keepalive being down. When one of the switches comes down and then up, this leads the second (vPC peer) switch to believe that since peer-keepalive is down and peer-link is up, it should not become primary and, in fact should shut down all local vPCs. So, whenever you reload any of the vPC peers, all your vPCs are down on both switches for the downlink delay.
The solution is simple -- either disable downlink delay (we went this way and didn't encounter any problems we anticipated when enabling this setting in the first place), or use mgmt0 ports for vPC-keepalive.

You might want to run your .NET Core ping tool with superuser rights on Linux

ICMP echo (ping) on .NET Core on Linux may be too slow if the .NET process is not running with superuser rights. On Windows there is an IcmpSendEcho2 function (from IP Helper library) that allows ICMP echo/reply even for non-superusers. On Linux, it requires working with raw sockets, and requires superuser rights. To work around this limitation, the .NET Core uses a trick to implement System.Net.NetworkInformation.Ping that runs the system ping tool (that can access raw sockets due to SUID bit being set), when superuser rights are not available, see here. But, if you're pinging many hosts and do it frequently (like once per 50ms), spawning a process for each operation might become too slow, putting extra load on the machine and skewing the measurement results. To make fast pings on Linux, you should run your process with superuser rights.
Also, take note that .NET Core currently implements the synchronous Ping API as wrappers around asynchronous methods, so there is next to no benefit going with the sync version for performance reasons (as was the case for .NET Framework).

.NET Core 2.0 in AWS ElasticBeanstalk-managed environments

Update: This is no longer needed.

The .NET Core 2.0 is released now, but the AWS Elastic Beanstalk Windows AMIs do not yet support it (only 1.0/1.1 is supported), so if you're using AWS Toolkit for VS 2017 you can't (successfully) deploy .NET Core 2.0 projects.
While waiting for Amazon to update them (not sure if they are going to do it, especially since .NET Core 2.0 is not LTS for now), I devised a quick fix that requires minimal changes to the project and does not require any AWS interaction. Feel free to use it until AWS upgrades the subject AMIs.
If dotnet/core#848 is resolved, the easier way using "packages" key will be available.

Also, a small recommendation on AMI selection -- do not use Windows Server 2016 images for these kinds of deployments, you gain almost nothing, but the antimalware tool that is installed by default impacts performance too much. I recommend to use Windows Server 2012R2 Core AMI (ami-1bfa1a63 at the time of writing).

P.S. I know that Azure deployments are easier, but their "security measures" are developer-unfriendly.

mod_gridfs is dead, long live gridfs_server

Long time no write, but mod_gridfs is no longer going to be developed, it's being superseded by gridfs_server. It's written in C#, based on ASP.NET Core, Kestrel, runs on .NET Core 2.0 Preview 2, and successfully serves about 25 million files per day from our MongoDB GridFS installation, as a backend to a front-end cache (that serves about 450 million files per day).

ASP.NET browser capabilities caching gotcha

If you're doing ASP.NET (pre-vNext) development and use browser capabilities checking (e.g. Request.Browser), add the following to your config file:

<browserCaps userAgentCacheKeyLength="256" />

You'll thank me later.
In short, ASP.NET caches capabilities based on first N characters of a user agent string, where N is 64 by default. After a "mobile" Google bot visits your site (with user agent of "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)") your iPhone users will be considered as crawlers, which is probably not what you want.

mod_gridfs v0.4: tag sets

Short news: mod_gridfs is now v0.4, supporing read preference tag sets along with having some performance and robustness improvements. Addition of tag sets allows many interesting things, for example fair 1-to-1 load balancing between shard replicas.

Backups to Amazon S3 -- simple and efficient

We use Amazon S3 as a part of our backup strategy -- all of our backup servers in datacenter replicate local backup images to S3 daily. While we have at least six physical copies of each backup (3 copies, each on different machine, all backup disks are in RAID1), having an offsite backup for DR even if we'll go multi-datacenter and will replicate data in semi-realtime is very important.
During the lifetime of this installation we used different approaches. The first one was using s3cmd -- while very functional and reliable it was very slow, because there was no real way to determine what changed between local and remote "filesystem" and just copying 100s of gigabytes per host per day was very slow. We thought that something like rsync would be much better, so we moved to s3fs+rsync. Unfortunately, it was very unstable and either required a second copy of files (to cache remote attributes) or was prone to downloading parts of files to compare them with originals, to determine if the file itself should be copied. We also evaluated duplicity (it consumed large amounts of temporary space) and a couple of commercial solutions, but none of them were good enough for us.
So, I decided to write a simple utility that would do this kind of a sync -- s3backup.
Features:

• easy to use -- configure AWS credentials, specify source and destination, put it into your crontab and you're set
• resource-efficient -- never downloads remote files, compares file sizes (good if your backups are named differently for each day) and/or MD5 checksums (good for other cases, consumes a bit more CPU), does not attempt to read entire file in RAM, etc.
• works with large files -- currently, the upper limit is about 500GB (10,000 chunks, 50MB each), this can easily be increased up to 5TB if you don't need MD5 checksum comparison
• supports recycling -- locally-removed files are removed from S3 only after they reach a specified age

You're welcome to try it out and use it. Feedback (especially about file sizes you manage and RAM constraints you have on backup machines) is very welcome also.

.NET ResourceManager loading incorrect resources

Recently, we found that some of our backends were serving pages in the wrong language. A simple application domain recycling helped, but the problem manifested itself several days later after another publish. I've spent about 4 hours investigating, and found out, that we're not the only people with the same problem. It appears that there is a bug in .NET's ResourceManager class that leads to wrong resources being loaded. There is some kind of a race condition that occurs when multiple threads try to create access the same underlying resource file. Thankfully, it's was easy enough to reproduce, but it won't be easy to fix (externally), so I reported it and hoping for a fix from Microsoft. While the temporary workaround seems obvious (just access resources under some kind of a lock), the most common case of ResourceManager class usage (autogenerated *.Designer.cs files which are created when you add resources file) would be very hard to patch by hand. Maybe custom "ThreadSafeResXFileCodeGenerator" would do, if Microsoft won't come up with a quick solution.

Linux guests on Hyper-V: I/O and CPU utilization caveats

Probably this is known to experienced Hyper-V admins, but I've recently found out, that high "system" CPU utilization in Linux guest (Ubuntu 12.04.2 x64) on a Hyper-V Server 2012 might actually be an indicator of I/O capacity being exhausted.

We are running PostgreSQL on one of the VMs and it had per-core 5-minute average load greater than 5 (actually 7-10). When I checked monitoring, I was surprised that the CPU breakdown was as follows: ~20% softirq (expected since this VM has about 100Mbps in/300Mbps out and is not the only VM on the host system, while NIC's on this machine don't support SR-IOV), 30-40% user (expected, since we don't do sequential scans), ~4% iowait (unexpected since the working set of the database does not completely fit in RAM) and 30-40% system (completely unexpected and unexplainable).

I tried numerous changes including adjusting PostgreSQL settings, adding/removing memory and cores, turning off NUMA both at the guest and the host level, trying a newer OS kernel... until I decided to stop and think it over.

When the monitoring system told us that CPU load is too high, it also mentioned that time spend doing disk I/O on one of the partitions was high too (about 85-90% disk time). I disregarded this warning at first, since iowait was low, but after monitoring PostgreSQL per-table statistics using a simple tool built around the query alike to 'select relid, relname, heap_blks_read + idx_blks_read + coalesce(toast_blks_read, 0) + coalesce(tidx_blks_read, 0) from pg_statio_user_tables' and shuffling around tables to balance reads between different partitions that are mapped to different disks, the system CPU utilization dropped by half.

This is different from running on real hardware, where iowait CPU utilization was going much higher when disk performance capacity was about to be exhausted, while system CPU utilization stayed mostly the same. I believe, that this has something to do not only with virtualization, but also with I/O scheduler used (we're using deadline on hardware and noop on VMs).

P.S. Microsoft's latest hypervisor is actually better at running Linux than many Linux ones and is very good at running Windows guests (no surprises here) -- we evaluated XenServer and KVM, both of them not only had problems with Windows stability and both network and I/O performance, but also with Linux kernels newer than 2.6.x.

mod_gridfs v0.3

Well, it was long time since my last post, but at least now I have something to tell about. mod_gridfs, an Apache 2.2+ module that serves files from MongoDB GridFS is now v0.3, supporting authentication, custom prefixes, read preference (currently only mode string is supported, tags will be added later), improved memory footprint and error logging.

UPDATE: Custom collection names are also supported now.

mod_gridfs performance

In my previous post, I announced mod_gridfs. Now, it's time for some numbers. Serving a 3KiB file over a gigabit network on modern hardware, 100 concurrent requests, MongoDB replica set of 3 machines as a backend:

• NGINX + nginx-gridfs: 1.3krps
• Apache + mod_gridfs: 6.6krps
• Apache + mod_gridfs with SlaveOk and one slave: 12.2krps

Not testing with larger files, because this way I'll be benchmarkng OS I/O performance instead of user-mode code.

mod_gridfs

As we were planning to move our terabytes of files into MongoDB GridFS, it occured to us that there is no readily available way to efficiently serve these files over the web, without resorting to using an ASP.NET GridFS IHttpHandler we implemented for local debugging some time ago.

After much hassle while developing GridFS handler for G-WAN (it is certainly fast, but keeps crashing even on Ubuntu 10.04.4 LTS, both x86 and x64 versions, even when I removed all non-boilerplate code from the module itself), performance measuring https://github.com/mdirolf/nginx-gridfs (until there is an asynchronous MongoDB driver, any GridFS module for NGINX is doomed), I decided to write an Apache 2.x (actually tested on 2.4) module to serve files from GridFS.

I decided to release it as open source here: https://bitbucket.org/onyxmaster/mod_gridfs/. It runs faster than the NGINX one, even with multiple workers, so we're going to use it as a backend (with NGINX as a caching frontend of course).

Configuration example:
GridFSConnection rsTest/db1,db2
GridFSDatabase my_database

Looks like 2.6.32.36 kernel on Xen has some timer problems, see https://jira.mongodb.org/browse/SERVER-4709 (we ran into it when using MongoDB).

Windows Azure, [not grand] finale

It appears that while Windows Azure has a lot of good points (for example, the idea of PaaS is pretty good, since Web Roles look not that much different from the fabled Heroku deployment), the built-in limitations (like 20 cores limitation per account) and the inability to run different OSes (custom Windows images via VM Role are fine, but we need Linux also) will make our migration much harder. In fact, we only need Windows for application servers, Linux runs everything other just fine and is much easier to staff. So, while Azure looked fine in the first place, it looks like Amazon Web Services would be our choice (yes, it has its own bad limits like 2Gbps EC2->EBS bandwidth cap, yet it appears to be more flexible).

7 million hits and transfer billing

A week ago we had 7 million daily hits on one of our websites and today we have it another time, another small, yet important milestone for us =)
I'm happy we aren't growing exponentially though, even linear growth with limited resources wasn't that easy to handle in terms of performance and (unsurprisingly) staffing.
The biggest headache this far though was media storage, which currently has several terabytes of images stored on our hardware with over a hundred TiB served per month (yes we aren't THAT big yet), and that we're looking forward to move somewhere like Windows Azure, but the estimated transfer bills are, well, shocking: ten times more expensive than we pay now. I wonder if transfer prices will drop soon. While storage is really getting cheaper (hard disks are getting cheaper, well they were until that flood), there is no visible driving force that could cause transfer rates go down.

Windows Azure, part 2

When I first checked out Windows Azure, I was glad to find it has root containers. Unfortunately, they are almost unusable for us, since they do not allow subdirectories (see the docs for that). I learned that the semi-hard way, stumbling upon the very helpful StorageClientException: "The requested URI does not represent any resource on the server." So, although root containers technically exist, with this limitation they are of no use.

Also, while the retry policy and timeout handling in Windows Azure .NET SDK is fine, exception handling is not. While getting StorageClientException and StorageServerException is expected, the WebException is not expected at all (I thought that one should be wrapped in StorageServerException).

Other than that, though, Windows Azure .NET SDK is pretty straightforward and easy to use.

Windows Azure first experience

Windows Azure looks to be a fine platform, but the toolset installation could be more streamlined. First, Azure Tools for Visual Studio complained about "Error 0x80070643", which was resolved by installing Azure SDK, Libraries and Emulator before. Then, emulator told me that I do not have SQL Server 2008 installed by popping up with a helpful message that there is a "possible security problem, see here", which leaded me to a page that had nothing to do with real cause of the problem.
After that, though, everything went smoothly. Since we're mostly interested in storage side of cloud services, I'll explore Windows Azure storage and probably will write a post or two about it.

Basic SSD tuning for MongoDB

We explored several options when using MongoDB on an SSD and came to following conclusions:

1. Don't turn off MongoDB journaling unless you really 100% sure -- the degree of comfort it provides you after your server restarts non-gracefully, is enough to warrant it's usage even within a replica set.
2. Use ext4 file system, mounted with "noatime,data=writeback,nobarrier" (nobarrier didn't give measurable differences on our workload, but others say it's still a good thing). ext4 is fast when allocating files (see 3. below), along with allowing you to delay file metadata updates (reliability is already covered by MongoDB journals).
   Correction from the year 2019: just use XFS.
3. Enable MongoDB options smallfiles and noprealloc (unless you're writing an application that is very heavy on inserts that is going to push the SSD to it's limits). SSDs still cost a lot of money and if you're installing 120GB or 160GB ones as we do, you don't want five empty databases occupy a gigabyte of that precious space (we run with directoryperdb=true also, it's handy for management). With smallfiles=true, noprealloc=true works just fine -- 512MiB files that get created by MongoDB are allocated in abour 300-600ms even under load, thus saving you even more space.

Improving robustness for C# MongoDB clients

I wonder if I should publish a set of tools and patches that make easier to write close-to-zero-downtime-without-users-noticing-that-half-servers-are-gone applications. Guess I'll put in a bit of effort to make it better suited for public release, like translating all the documentation comments from Russian to English :)
The basic idea for the tools is to provide a side-attached layer that gracefully handles failure and retries the operation if the tool decides that it still might succeed. While the idea is easy, it really works for something as crude as pulling the plug for half of the servers with users noticing only a slight (several seconds max) delay with their web pages load times for a few seconds.