When your Python data analysis script slows to a crawl—or runs out of memory—what’s actually happening under the hood? What does “making Python faster” really mean, and what tools or techniques will give you the biggest payoffs for real-world problems?
This session breaks down what really limits Python’s performance and how to escape those traps. We’ll start by looking at concrete scenarios where performance or memory matter—batch computations, lunch-break explorations, and the interactivity users expect.
The core section is a practical anatomy of Python’s speed bumps, with simple benchmarks and visualizations:
- Dynamic features: Why Python’s automatic types, reference counting, and runtime polymorphism make it much slower than C or compiled languages.
- How interpreters work: Step through bytecode, memory models, and the real reason why tight Python loops are slow (hint: type checks, reference count updates, and generic dynamic behavior).
Next, we’ll look at “escape hatches”:
- NumPy for fast array operations by working in compiled extensions rather than the Python interpreter.
- Awkward Array for high-performance computations on irregular and nested data—addressing cases where Pandas and NumPy break down.
- JIT compilation with Numba and hand-written C++ bindings, showing the dramatic speedup when you batch away dynamic features.
Special topics cover the big gotchas for power users:
- Garbage collection: How Python’s GC is triggered by allocation count, not memory usage, and what that means when you hit memory limits.
- The GIL: Python’s global interpreter lock and why pure-Python threads don’t use multiple cores—plus what’s (finally) changing with Python 3.13’s no-GIL mode and subinterpreters.
Speeding up Python is really about escaping dynamic features—by pushing hot loops to compiled code (NumPy/Numba/etc.) or sidestepping, not optimizing, the interpreter. With the right tools, order-of-magnitude improvements are routine. But understanding Python’s internals is essential: it helps you recognize when to reach for C extensions, shape your data for efficient computation, and know when configuration details like memory limits or thread models will matter.
Presented as a Jupyter notebook, shared in a publicly accessible Docker container: see instructions here. You don’t need to install anything but Docker to run all the interactive examples!