Async vs Sync Benchmark (.NET)

One of my favorite interview questions is “What do such words as async and await tell you?”, because it opens up an opportunity to have an interesting discussion with an interviewee… Or it doesn’t because they float in this topic. In my opinion it is drastically important to understand why we use this technique. I have a feeling that many developers prefer to rely just on “it is the best practice” and blindly use asynchronous methods.

This article shows the difference between asynchronous and synchronous methods in practice.

Tools

  • Azure Sql Database
  • Azure App Service (hosts the application)
  • Azure App Insights (to gather metrics)
  • locust framework (to simulate user load).

Configuration

Experiment schema

I will run a benchmark in the following way. There are two independent locust instances running on two machines*. Locust instances simulate a user that does the following:

  • a user from the locust host 1 hits the synchronous endpoint of the App Service 1, gets the response, stays idle during 0.5–1 seconds (the exact time delay is random). Repeats till the end of the experiment
  • a user from the locust host 2 behaves exactly the same, with the only one difference — he hits the asynchronous endpoint of the App Service 2.

Under the hood each App Service connects to its own database and executes a SELECT query that takes five seconds and returns a few rows of data. See the controller’s code below for references. I’ll use dapper to make a call to the database. I’d like to draw your attention to the fact that the asynchronous endpoint calls the database asynchronously as well (QueryAsync<T>).

App Services code

Worth adding that I deploy the same code to both app services.

During the test the number of users grows evenly up to the target number (Number of Users). The speed of growth is controlled by a Spawn Rate parameter (number of unique users to join per second) — the higher the number is, the quicker the users are being added. Spawn rate is set to 10 users/s for all experiments.

All experiments are limited to 15 minutes.

You may find machines configuration details in the Technical details section of the article.

Metrics

  • thread count — shows the number of threads the app service consumes
  • median response time, ms

The red lines refer to asynchronous, the blue lines — to the synchronous endpoint respectively.

That’s it about theory, let’s start.

Experiment #1

We can see that both endpoints perform similarly — being able to handle around 750 requests per minute with a median response time of 5200 ms.

Experiment #1. Requests per minute

The most fascinating graph in this experiment is a thread trend. You can see significantly higher numbers for the synchronous endpoint (a blue graph) — more than 100 threads!

Experiment #1. Thread count

That’s expected though and matches the theory — when a request comes in and the application makes a call to the database the thread is being blocked because it has to wait for a roundtrip to complete. Therefore when another request comes in the application has to produce a new thread to handle it.

The red graph — the asynchronous endpoint thread count— proves different behavior. When a request comes in and the application makes a call to the database then the thread returns to a thread pool instead of being blocked. Therefore when another request comes in this free thread is being reused. Despite incoming requests keep growing the application doesn’t require any new threads, so their count remains the same.

Worth to mention the 3rd metric — median response time. Both endpoints showed the same result — 5200 ms. So there is no difference in terms of performance.

Experiment #1. Summary

Now it is time to pull up stakes.

Experiment #2

We doubled the load. The asynchronous endpoint handles this task successfully — its request per minute rate floats around 1500. The synchronous brother eventually reached the comparable number of 1410. But if you look at the graph below you will see that it took 10 minutes!

The reason for that is that the synchronous endpoint reacts on a new user arrival by creating another thread, but users are being added to the system (just to remind that the Spawn Rate is 10 users/s) quicker than the web server can adapt to. That’s why it queued so many requests at the very beginning.

Experiment #2. Requests per minute

It is not a surprise that the thread count metric is still around 34 for the asynchronous endpoint while it increased from 102 to 155 for the synchronous one. The median response time degraded similarly to the request per minute rate — the synchronous response time was much higher at the beginning of the experiment. If I kept the test for 24 hours the median numbers would become even.

Experiment #2. Summary

Experiment #3

The third experiment is intended to prove trends revealed during the second one —we can see further degradation of the synchronous endpoint.

Experiment #3. Summary

Conclusion

Appendix #1. Technical details

  • Azure Sql Database: Standard S4: 200 DTUs, 500 Mb storage
  • Sql Connection settings: Max Pool Size=200

Appendix #2. Notes

Appendix #3. Bonus Experiment

Bonus Experiment. Requests per minute

My goal is to eliminate technological barriers and help businesses to grow and change the world to the best