Python is a perfect programming language unless you have to deal with asynchronous action.

Many programmers using Javascript complain about ‘callback hell’. Still, people become grateful when they have to deal with asynchronous action (simultaneous, parallel, concurrent, etc.).

Recently I needed to send several requests at the same time, instead of waiting for the previous request to finish. Moreover, all responses should be collected as one variable.

1. Ordinary request

If we call 10 requests sequentially, it took 11.285 seconds.

The more request, the longer time 🙁

Let’s see how we can solve this!

2. Concurrent GET request

First two lines are the packages that we are using.

Line 4 shows the function that we will use to request.

Line 7 is a list of 10 URLs that we want to request simultaneously.

Line 9-10 is the core part of this script. “ThreadPoolExecutor” is a pool of threads that can run asynchronously. Therefore you can specify the number of workers who can work at the same time.

However, a higher number of workers don’t always guarantee better performance. (See this answer)

In the line 10, people may get confused about arguments ‘get_url’ and ‘list_of_urls’. This statement indicates of mapping between function and each element of the list.

For example,

get_url(url1)
get_url(url2)
...
get_url(url10)

Now we run the code.

In theory, the total time spent is decided by the longest request.

If 10 threads can be completely independent (one thread per one CPU core and no context switching), there will be no difference between one request and 10 requests.

3. Concurrent POST request

However, you may wonder how we can pass more than one argument to the function.

In this case, you can set the element as a tuple like below code.

As you can see from line 11, each element (tuple) consists of URL and dict_data.

In the line 4, args[0] indicates the first element of the tuple (URL), and args[1] indicates the second element of the tuple (form_data).

Final test!

Happy coding!