I ran the 14 DaCapo tests on three different Linux machine configurations on each cloud, using the default JVM. The instances aren't perfect "apples to apples" matches, but they are roughly comparable in terms of size and price. The configurations and cost per hour are broken out in the table below.
I gathered two sets of numbers for each machine. The first set shows the amount of timethe instance took to run the benchmark from a dead stop. It fired up the JVM, loaded the code, and started to work. This isn't a bad simulation because many servers start up Java code from command lines in scripts.
To add another dimension, the second set reports the times using the "converge" option. This runs the benchmark repeatedly until consistent results appear. This sometimes happens after just a few runs, but in a few cases, the results failed to converge after 20 iterations. This option often resulted in dramatically faster times, but sometimes it only produced marginally faster times.
The results will look like a mind-numbing sea of numbers to anyone, but a few patterns stood out:
- Google was the fastest overall. The three Google instances completed the benchmarks in a total of 575 seconds, compared with 719 seconds for Amazon and 834 seconds for Windows Azure. A Google machine had the fastest time in 13 of the 14 tests. A Windows Azure machine had the fastest time in only one of the benchmarks. Amazon was never the fastest.
- Google was also the cheapest overall, though Windows Azure was close behind. Executing the DaCapo suite on the trio of machines cost 3.78 cents on Google, 3.8 cents on Windows Azure, and 5 cents on Amazon. A Google machine was the cheapest option in eight of the 14 tests. A Windows Azure instance was cheapest in five tests. An Amazon machine was the cheapest in only one of the tests.
- The best option for misers was Windows Azure's Small VM (one CPU, 6 cents per hour), which completed the benchmarks at a cost of 0.67 cents. However, this was also one of the slowest options, taking 404 seconds to complete the suite. The next cheapest option, Google's n1-highcpu-2 instance (two CPUs, 13.1 cents per hour), completed the benchmarks in half the time (193 seconds) at a cost of 0.70 cents.
- If you cared more about speed than money, Google's n1-standard-8 machine (eight CPUs, 82.9 cents per hour) was the best option. It turned in the fastest time in 11 of the 14 benchmarks, completing the entire DaCapo suite in 101 seconds at a cost of 2.32 cents. The closest rival, Amazon's m3.2xlarge instance (eight CPUs, $0.90 per hour), completed the suite in 118 seconds at a cost of 2.96 cents.
- Amazon was rarely a bargain. Amazon's m1.medium (one CPU, 10.4 cents per hour) was both the slowest and the most expensive of the one CPU instances. Amazon's m3.2xlarge (eight CPUs, 90 cents per hour) was the second fastest instance overall, but also the most expensive. However, Amazon's c3.large (two CPUs, 15 cents per hour) was truly competitive — nearly as fast overall as Google's two-CPU instance, and faster and cheaper than Windows Azure's two CPU machine.