To accelerate the execution of some performance critical code, we can write the code in C++ with the help of pybind11 and export the C++ code as shared library¹. Then we can import the shared library as a module and enjoy speed boost.

Install pybind11:

python3 -m pip install pybind11

A simple code using pybind11#

A simple example using pybind11 is shown below:

// include pybind11 header files so that we can use PYBIND11_MODULE macro
#include <pybind11/pybind11.h>

namespace py = pybind11;

int sum(int start, int end){
    // calculate sum from i to j
  if (start > end) return 0;

  int sum = 0;
  for (int i = start; i <= end; i++){
    sum += i;
  }

  return sum;
}

PYBIND11_MODULE(demo, m) {
    m.doc() = "pybind11 demo plugin"; // optional module docstring

    m.def("sum", &sum, "calculate sum from start to end",
        py::arg("start") = 1, py::arg("end") = 1000);
}

In the above code, we use macro PYBIND11_MODULE to define a module, whose name is demo and represented as m. m.def() is used to register function to this module:

• The first parameter is the function name you want to use. It does not need to be the same with the C++ function name.
• Second parameter is the function address (&sum).
• Third parameter is the function documentation.
• py:arg() is used to add keyword argument and its default values to the function.

Compile the C++ code#

On the command line#

We can directory compile the code on the command line:

c++ -O3 -Wall -shared -std=c++11 -fPIC -I$(python3 -m pybind11 --includes) demo.cc -o demo$(pyton3-config --extension-suffix)

In the above command, python3 -m pybind11 --includes is used to get the relevant include path for pybind11. On my system, the output is like the following:

-I/Users/jdhao/tools/miniconda3/include/python3.9 -I/Users/jdhao/tools/miniconda3/lib/python3.9/site-packages/pybind11/include

The command python3-config --extension-suffix is used to get the proper suffix for shared library based on current system.

Compile using Makefile#

It would be tedious to type the above command each time to compile the code. We can write a simple Makefile to simplify the work:

INCLUDE := $(shell python3 -m pybind11 --includes)
FLAG := -O3 -Wall -shared -std=c++11 -fPIC
SUFFIX := $(shell python3-config --extension-suffix)
demo:
    c++ $(FLAG) $(INCLUDE) demo.cc -o demo$(SUFFIX)

Then use make demo to compile.

Import and run the code#

Open a Python interpreter and we can use the module just like other Python modules:

import demo

print(demo.sum())  # use the default parameter value

print(demo.sum(start=1, end=100))

References#

• Official doc: https://pybind11.readthedocs.io/en/latest/basics.html