Probably this is known to experienced Hyper-V admins, but I've recently found out, that high "system" CPU utilization in Linux guest (Ubuntu 12.04.2 x64) on a Hyper-V Server 2012 might actually be an indicator of I/O capacity being exhausted.
We are running PostgreSQL on one of the VMs and it had per-core 5-minute average load greater than 5 (actually 7-10). When I checked monitoring, I was surprised that the CPU breakdown was as follows: ~20% softirq (expected since this VM has about 100Mbps in/300Mbps out and is not the only VM on the host system, while NIC's on this machine don't support SR-IOV), 30-40% user (expected, since we don't do sequential scans), ~4% iowait (unexpected since the working set of the database does not completely fit in RAM) and 30-40% system (completely unexpected and unexplainable).
I tried numerous changes including adjusting PostgreSQL settings, adding/removing memory and cores, turning off NUMA both at the guest and the host level, trying a newer OS kernel... until I decided to stop and think it over.
When the monitoring system told us that CPU load is too high, it also mentioned that time spend doing disk I/O on one of the partitions was high too (about 85-90% disk time). I disregarded this warning at first, since iowait was low, but after monitoring PostgreSQL per-table statistics using a simple tool built around the query alike to 'select relid, relname, heap_blks_read + idx_blks_read + coalesce(toast_blks_read, 0) + coalesce(tidx_blks_read, 0) from pg_statio_user_tables' and shuffling around tables to balance reads between different partitions that are mapped to different disks, the system CPU utilization dropped by half.
This is different from running on real hardware, where iowait CPU utilization was going much higher when disk performance capacity was about to be exhausted, while system CPU utilization stayed mostly the same. I believe, that this has something to do not only with virtualization, but also with I/O scheduler used (we're using deadline on hardware and noop on VMs).
P.S. Microsoft's latest hypervisor is actually better at running Linux than many Linux ones and is very good at running Windows guests (no surprises here) -- we evaluated XenServer and KVM, both of them not only had problems with Windows stability and both network and I/O performance, but also with Linux kernels newer than 2.6.x.
2012-12-19
mod_gridfs v0.3
Well, it was long time since my last post, but at least now I have something to tell about. mod_gridfs, an Apache 2.2+ module that serves files from MongoDB GridFS is now v0.3, supporting authentication, custom prefixes, read preference (currently only mode string is supported, tags will be added later), improved memory footprint and error logging.
UPDATE: Custom collection names are also supported now.
UPDATE: Custom collection names are also supported now.
2012-04-13
mod_gridfs performance
In my previous post, I announced mod_gridfs. Now, it's time for some numbers. Serving a 3KiB file over a gigabit network on modern hardware, 100 concurrent requests, MongoDB replica set of 3 machines as a backend:
- NGINX + nginx-gridfs: 1.3krps
- Apache + mod_gridfs: 6.6krps
- Apache + mod_gridfs with SlaveOk and one slave: 12.2krps
2012-04-10
mod_gridfs
As we were planning to move our terabytes of files into MongoDB GridFS, it occured to us that there is no readily available way to efficiently serve these files over the web, without resorting to using an ASP.NET GridFS IHttpHandler we implemented for local debugging some time ago.
After much hassle while developing GridFS handler for G-WAN (it is certainly fast, but keeps crashing even on Ubuntu 10.04.4 LTS, both x86 and x64 versions, even when I removed all non-boilerplate code from the module itself), performance measuring https://github.com/mdirolf/nginx-gridfs (until there is an asynchronous MongoDB driver, any GridFS module for NGINX is doomed), I decided to write an Apache 2.x (actually tested on 2.4) module to serve files from GridFS.
I decided to release it as open source here: https://bitbucket.org/onyxmaster/mod_gridfs/. It runs faster than the NGINX one, even with multiple workers, so we're going to use it as a backend (with NGINX as a caching frontend of course).
Configuration example:
GridFSConnection rsTest/db1,db2
GridFSDatabase my_database
After much hassle while developing GridFS handler for G-WAN (it is certainly fast, but keeps crashing even on Ubuntu 10.04.4 LTS, both x86 and x64 versions, even when I removed all non-boilerplate code from the module itself), performance measuring https://github.com/mdirolf/nginx-gridfs (until there is an asynchronous MongoDB driver, any GridFS module for NGINX is doomed), I decided to write an Apache 2.x (actually tested on 2.4) module to serve files from GridFS.
I decided to release it as open source here: https://bitbucket.org/onyxmaster/mod_gridfs/. It runs faster than the NGINX one, even with multiple workers, so we're going to use it as a backend (with NGINX as a caching frontend of course).
Configuration example:
GridFSConnection rsTest/db1,db2
GridFSDatabase my_database
2012-01-22
Looks like 2.6.32.36 kernel on Xen has some timer problems, see https://jira.mongodb.org/browse/SERVER-4709 (we ran into it when using MongoDB).
2011-12-11
Windows Azure, [not grand] finale
It appears that while Windows Azure has a lot of good points (for example, the idea of PaaS is pretty good, since Web Roles look not that much different from the fabled Heroku deployment), the built-in limitations (like 20 cores limitation per account) and the inability to run different OSes (custom Windows images via VM Role are fine, but we need Linux also) will make our migration much harder. In fact, we only need Windows for application servers, Linux runs everything other just fine and is much easier to staff. So, while Azure looked fine in the first place, it looks like Amazon Web Services would be our choice (yes, it has its own bad limits like 2Gbps EC2->EBS bandwidth cap, yet it appears to be more flexible).
2011-12-06
7 million hits and transfer billing
A week ago we had 7 million daily hits on one of our websites and today we have it another time, another small, yet important milestone for us =)
I'm happy we aren't growing exponentially though, even linear growth with limited resources wasn't that easy to handle in terms of performance and (unsurprisingly) staffing.
The biggest headache this far though was media storage, which currently has several terabytes of images stored on our hardware with over a hundred TiB served per month (yes we aren't THAT big yet), and that we're looking forward to move somewhere like Windows Azure, but the estimated transfer bills are, well, shocking: ten times more expensive than we pay now. I wonder if transfer prices will drop soon. While storage is really getting cheaper (hard disks are getting cheaper, well they were until that flood), there is no visible driving force that could cause transfer rates go down.
I'm happy we aren't growing exponentially though, even linear growth with limited resources wasn't that easy to handle in terms of performance and (unsurprisingly) staffing.
The biggest headache this far though was media storage, which currently has several terabytes of images stored on our hardware with over a hundred TiB served per month (yes we aren't THAT big yet), and that we're looking forward to move somewhere like Windows Azure, but the estimated transfer bills are, well, shocking: ten times more expensive than we pay now. I wonder if transfer prices will drop soon. While storage is really getting cheaper (hard disks are getting cheaper, well they were until that flood), there is no visible driving force that could cause transfer rates go down.
Subscribe to:
Posts (Atom)