AWS surpasses rivals in test of cloud capabilities

Cockroach Labs’ 2020 Cloud Report finds that Google Cloud Platform and Microsoft Azure are catching up with Amazon.

Top 5 things to know about open source and the cloud
Cloud software hinders open source software companies from making a profit. Tom Merritt explains the five things you need to know about open source and the cloud.

Cockroach Labs tested the speed and power of the three major cloud providers and discovered that Amazon Web Services has a head start on Google Cloud Platform and Microsoft Azure.

In the 2020 Cloud Report, Azure did the best with the CPU performance test, but AWS offered the best networking and I / O capabilities. The testers discovered that GCP had made significant improvements over last year’s report and had the best results in network throughput.

Cockroach Labs tested the three providers on a series of micro benchmarks and customer-like workloads. The goal was to understand the performance of every cloud provider in general, as well as the strength of each company’s machine types.

Cockroach Labs checked the results with the main cloud providers for an assessment of the arrangement of the machines and benchmarks. Cockroach Labs has placed the test process and the results in this public repository. Paul Bardea, Charlotte Dillon, Nathan VanBenschoten and Andy Woods of Cockroach Labs wrote the 2020 report.

The performance tests and test tools include:

  • CPU (stress ng)
  • Network transit and latency (iPerf and ping)
  • Storage I / O read and write (sysbench)
  • General workload performance (TPC-C)

CPU experiments

In this category, the best performing Azure machines achieved significantly better results on the CPU microbenchmark.

The testers discovered that “the best performing Azure machines use 16 cores with 1 thread per core, while the other clouds use hyperthreading across all instances and use 8 cores with 2 threads per core to reach 16 vCPUs.”
The authors warn that the effects of avoiding hyperthreading may have inflated the benchmark and may not represent performance on other workloads. They also said that these results are strongly correlated with the clock frequency of each instance type.

Network experiments

The reviewers changed this test setup this year by testing the load on multiple clients and observing the results of a single target server.

The throughput comparison tests showed that the GCP network outperformed AWS or Azure: “Not only do their best performing machines beat the best performing machines of any network, but also their lowest performing machines.”

The authors of the report note that last year, AWS outperformed GCP in network tests.

In the latency comparisons, GCP improved compared to last year’s report, but AWS again won the race with Azure far behind both competitors: “Even the best machine on Azure is more than five times worse than on AWS or GCP.”

Storage of I / O experiments

Cloud providers offer two types of storage hardware: locally-linked storage and network-linked storage. Each provider has a different label for these two types:

Locally connected storage Network linked storage

AWS Instant store volumes Storage volumes with elastic blocks
azure Temporary disks Managed disks
GCP Local SSDs Permanent disks

Cockroach also tested in this category for transit and latency. The testers used a “sysbench configuration that simulates small write actions with frequent synchronization for both write and read performance” and measured read and write capabilities separately.

AWS won the writing round with “superior write storage performance with the i3en machine type.”

Azure had an advantage over the other two providers in the ability to manage threads: AWS and GCP hit a bottleneck with four threads, but Azure continues to increase the number of write iOPs to 16 threads. The report states that Azure write iOPs excel in managing applications with more threads after initially falling behind smaller thread sizes.

SEE: Special function: Industry cloud (free PDF)

The storage optimized machines from AWS fulfill their invoicing as strong choices when optimizing for storage performance. Azure cannot surpass AWS reliably in read throughput and the provider’s read latency is extremely variable.

The report showed that AWS wins the combined read comparison for storage in all categories with its i3 machine type.

TPC-C performance

In this category, the testers measured the number of processed orders per minute and the total number of supported warehouses. Testers discovered that all clouds were within 5% of each other, although AWS was at the top.

The comparison showed that “the best-performing machine types from each cloud are also the same machine types that performed best on the CPU and network throughput tests.”

Both AWS’s c5n.4xlarge and GCP’s c2-standard-16 won the CPU, network transit and network latency tests, while Azure’s Standard_DS14_v2 won the CPU and network transit transit tests.

The machine types that won the read and write storage tests – AWS i3.4xlarge and i3en.6xlarge, GCPs n2-standard-16 and Azure’s Standard_GS4 – varied in their TPC-C performance.

The authors say this suggests that these tests are less influential in determining OLTP performance and that OLTP workloads such as TPC-C are often limited by compute sources.

Cloud Insights newsletter

Your knowledge base for the latest news on AWS, Microsoft Azure, Google Cloud Platform, Docker, SaaS, IaaS, cloud security, containers, the public cloud, the hybrid cloud, the industry cloud and much more.
Delivered on Mondays

Register today

Also see

Image: Cockroach Labs

Image: Cockroach Labs

Image: Cockroach Labs

Follow AsumeTech on

More From Category

More Stories Today

Leave a Reply