Benchmarking for Python 3.11
Python is one of the most used scripting languages in data science (DS) and machine learning (ML). according to PopularitY of Programming Languages , Python is the most Googled language. In addition to being an excellent glue language for bringing together various DS/ML solutions, it has many other libraries that do all sorts of things with data.
About a month ago we got our new annual Python release - version 3.11. I'm very excited about this new version because the main feature of this version is a significant increase in speed.
On various social media, we've seen a lot of people testing the new version of the post with shocking results. But the best way to see how fast Python 3.11 really is is to run the tests yourself.
In this post, I will share my step-by-step analysis of Python 3.11. All code can be found on Github.
https://github.com/CYang828/p...
Benchmarking a programming language is not at all easy. When you read x faster than y, you should be skeptical of the result. One implementation of an algorithm may be better than x, and another implementation is better than y. For our benchmarks, we tried our best to make it as simple as possible, since we are testing Python with Python, but we may have chosen some elements from the language that have little impact. With this in mind, I would like to introduce the algorithm I use for benchmarking: the genome assembly algorithm DNA K-mers.
Genome Assembly Algorithms DNA K-mers.
The idea of this algorithm is simple, DNA is a long string of sequences called nucleotides. In DNA, there are 4 nucleotides represented by the letters A, C, G and T. Humans (or Homo sapiens more precisely) have 3 billion nucleotide pairs. For example, a small portion of human DNA might be:
ACTAGGGATCATGAAGATAATGTTGGTGTTTGTATGGTTTTCAGACAATT
In this example, if one wanted to select any 4 consecutive nucleotides (i.e. letters) from this string, it would be a k-mer of length 4 (let's call it a 4-mer). Here are some examples of 4-mers derived from the example.
ACTA, CTAG, TAGG, AGGG, GGGA, etc.
For this article, let's generate all possible 13-mers. Mathematically speaking, this is a permutation problem. So we have 4¹³ (=67108864) possible 13-mers. I use a simple algorithm to generate results in C++ and Python. Let's see how different Python versions and C++ compare.
Comparing results in Python and C++
Testing different versions of Python
def K_mer(k: int) -> None:
def convert(c):
if c == "A":
return "C"
if c == "C":
return "G"
if c == "G":
return "T"
if c == "T":
return "A"
print("start")
opt = "ACGT"
s = ""
s_last = ""
len_str = k
for i in range(len_str):
s += opt[0]
for i in range(len_str):
s_last += opt[-1]
pos = 0
counter = 1
while s != s_last:
counter += 1
# You can uncomment the next line to see all k-mers.
# print(s)
change_next = True
for i in range(len_str):
if change_next:
if s[i] == opt[-1]:
s = s[:i] + convert(s[i]) + s[i + 1 :] # type: ignore
change_next = True
else:
s = s[:i] + convert(s[i]) + s[i + 1 :] # type: ignore
break
# You can uncomment the next line to see all k-mers.
# print(s)
print("generate k-mers quantity: {}".format(counter))
print("Finish!")The script above is ready to run, but we want to use it to test various versions of Python, not just the currently installed (or activated) version. The easiest way to test multiple Python versions is to use Docker. Python maintains many docker images. Of course, all supported versions, including some end-of-life (EOL) versions like 2.7 or 3.2. To use Docker, you need to install it. It's relatively easy on Linux and Mac, on Windows I'm not so sure, but probably not difficult either. I recommend just installing the docker CLI, the desktop is too bloated for me. To run a local script in a containerized Python environment, run:
docker run -it --rm \ -v $PWD/your_script.py:/your_script.py \ python:3.11-rc-slim \ python /yourscript.py
To automate the testing of various versions, we will of course use Python as well. This script simply starts a subprocess to start a container with a specific Python version, and collects the results. It is nothing special:
import time
import argparse
def K_mer(k: int) -> None:
def convert(c):
if c == "A":
return "C"
if c == "C":
return "G"
if c == "G":
return "T"
if c == "T":
return "A"
print("start")
opt = "ACGT"
s = ""
s_last = ""
len_str = k
for i in range(len_str):
s += opt[0]
for i in range(len_str):
s_last += opt[-1]
pos = 0
counter = 1
while s != s_last:
counter += 1
# You can uncomment the next line to see all k-mers.
# print(s)
change_next = True
for i in range(len_str):
if change_next:
if s[i] == opt[-1]:
s = s[:i] + convert(s[i]) + s[i + 1 :] # type: ignore
change_next = True
else:
s = s[:i] + convert(s[i]) + s[i + 1 :] # type: ignore
break
# You can uncomment the next line to see all k-mers.
# print(s)
print("generate k-mers quantity: {}".format(counter))
print("Finish!")
def run_test(k: int) -> None:
start_time = time.time()
K_mer(k)
print(f"{(time.time() - start_time):.4f}second")
def main():
"""Main loop in arg parser."""
parser = argparse.ArgumentParser()
parser.add_argument(
"-k",
"--k_mer",
help="DNA sequence length",
type=int,
default=13,
)
args = parser.parse_args()
run_test(args.k_mer)
if __name__ == "__main__":
main()
When running these tests, the absolute value will vary from machine to machine depending on the processor (it is CPU intensive). Here are the results for the last 7 major Python versions:
new version Python 3.11 cost 31.9501 second. Python 3.5 cost 63.3502 second.(Python 3.11 98 faster than it.3% ) Python 3.6 cost 62.9635 second.(Python 3.11 97 faster than it.1% ) Python 3.7 cost 57.5806 second.(Python 3.11 80 faster than it.2% ) Python 3.8 cost 59.0129 second.(Python 3.11 84 faster than it.7% ) Python 3.9 cost 54.4991 second.(Python 3.11 70 faster than it.6% ) Python 3.10 cost 47.7407 second.(Python 3.11 faster than it 49.4% )
The Python 3.11 benchmark took an average of 6.46 seconds. Compared to the previous version (3.10), this is almost 37% faster. Very impressive! The differences between versions 3.9 and 3.10 are about the same, making 3.11 almost 70% faster! Draw all the points in the image below.
Testing for C++
When it comes to speed, we always have someone who says: if you want speed, why not use C.
C is much faster that Python! — that one guy
Below we use C++ to implement the above K-mers algorithm.
#include <iostream>
#include <string>
using namespace std;
char convert(char c)
{
if (c == 'A') return 'C';
if (c == 'C') return 'G';
if (c == 'G') return 'T';
if (c == 'T') return 'A';
return ' ';
}
int main()
{
cout << "start" << endl;
string opt = "ACGT";
string s = "";
string s_last = "";
int len_str = 13;
bool change_next;
for (int i=0; i<len_str; i++)
{
s += opt[0];
}
for (int i=0; i<len_str; i++)
{
s_last += opt.back();
}
int pos = 0;
int counter = 1;
while (s != s_last)
{
counter ++;
// You can uncomment the next line to see all k-mers.
// cout << s << endl;
change_next = true;
for (int i=0; i<len_str; i++)
{
if (change_next)
{
if (s[i] == opt.back())
{
s[i] = convert(s[i]);
change_next = true;
} else {
s[i] = convert(s[i]);
break;
}
}
}
}
// You can uncomment the next line to see all k-mers.
// cout << s << endl;
cout << "generate k-mers quantity: " << counter << endl;
cout << "Finish!" << endl;
return 0;
}As we all know, C++ is a compiled language, so we need to compile the source code before we can use it. After installing the base software for C++, you can type:
g++ -o k_mer k_mer.c
After compiling, just run the build executable. The output should look like this:
generate k-mers quantity: 67108864 Spend time: 8.92218 second generate k-mers quantity: 67108864 Spend time: 9.01176 second generate k-mers quantity: 67108864 Spend time: 8.88735 second generate k-mers quantity: 67108864 Spend time: 8.9754 second
Each loop takes an average of 8.945 seconds to calculate.
We have to agree with this because it's really fast. It only took 8.945 seconds to complete the same loop we previously programmed in Python. Let's add it as a line to the previous plot as shown in the image below.
Now, after a long analysis of the preceding numbers, we can clearly see Python's momentum. Python is about 35% faster since version 3.9. The Python developers mentioned that the next two versions will have significant speed improvements, so we can assume that this speed will remain the same (yes, super safe assumption).
The question now is, when this momentum is corrected, when will Python surpass the C++ era. For this, we can of course use inference to predict the runtime of the next Python version. These can be seen in the image below:
The results were fantastic! If you keep this iteration speed, Python 3.14 will be faster than C++~
statement
While these Python3.5 to Python3.11 benchmarks are valid, this extrapolation is of course a joke. XKCD-style plotting is another reminder ;-) If you want to run these tests or your own tests on various Python versions, you can download the code from Github. Github: https://github.com/CYang828/p...
This article is by mdnice Multi-platform release