Python is fast enough for most tasks — until it isn't. When a data pipeline that takes 3 seconds starts handling 100x more data, that 3 seconds becomes 300 seconds, and suddenly you have a production incident. The real mistake isn't writing slow Python; it's optimizing blind without first understanding where time is actually going.

Measure Before You Touch Anything

The cardinal rule: never optimize without profiling first. Human intuition about bottlenecks is wrong roughly 80% of the time. The function you think is slow rarely is — the actual culprit is usually somewhere you weren't looking.

Python ships with two useful built-in profilers:

  • cProfile: deterministic, low overhead, good for finding the top time consumers
  • line_profiler: line-by-line timing, surgical precision when you know roughly where the problem is

Start with cProfile:

import cProfile import pstats import io from pstats import SortKey def run_with_profiling(func, *args, **kwargs): pr = cProfile.Profile() pr.enable() result = func(*args, **kwargs) pr.disable() stream = io.StringIO() stats = pstats.Stats(pr, stream=stream) stats.sort_stats(SortKey.CUMULATIVE) stats.print_stats(20) print(stream.getvalue()) return result 

The cumulative sort shows functions consuming the most total time, including subcalls — more useful than tottime for initial triage. It tells you which call chain is burning time, not just which individual function.

If you want something with less friction, py-spy is a sampling profiler that attaches to a running process without code changes:

pip install py-spy py-spy top --pid 12345 py-spy record -o profile.svg --pid 12345 

The SVG flamegraph makes hot paths immediately obvious.

The Most Common Python Performance Mistakes

Once you have a profile, you're usually staring at one of these patterns:

Quadratic loops: a nested loop over a list of n items is O(n²). Convert the inner lookup to a dict or set.

Unnecessary object creation: string concatenation with += in a loop creates a new string object on every iteration. Use "".join(list_of_parts) instead.

Calling the same function repeatedly inside a loop: if you call len(my_list) inside a loop condition, Python evaluates it every iteration. Cache it outside.

# Before — O(n³) worst case def find_duplicates(items): duplicates = [] for i in range(len(items)): for j in range(len(items)): if i != j and items[i] == items[j]: if items[i] not in duplicates: duplicates.append(items[i]) return duplicates # After — O(n) def find_duplicates_fast(items): seen = set() duplicates = set() for item in items: if item in seen: duplicates.add(item) seen.add(item) return list(duplicates) 

On 10,000 items, the difference is seconds vs microseconds. The algorithmic fix always outperforms any amount of micro-optimization on the slow version.

Leveraging Built-ins and Standard Library Tools

CPython's built-in functions run at C speed. Using them instead of pure Python loops is usually the cheapest performance win available, and it requires no dependencies.

Compare sorting:

import timeit data = list(range(10000, 0, -1)) def bubble_sort(arr): n = len(arr) for i in range(n): for j in range(0, n - i - 1): if arr[j] > arr[j + 1]: arr[j], arr[j + 1] = arr[j + 1], arr[j] return arr t1 = timeit.timeit(lambda: bubble_sort(data.copy()), number=10) t2 = timeit.timeit(lambda: sorted(data), number=10) print(f"Bubble sort: {t1:.3f}s") print(f"sorted(): {t2:.5f}s") 

The result: sorted() runs 1000x faster because Timsort is implemented in C. No surprises there, but the same principle applies in less obvious places:

  • collections.Counter over manual counting dicts
  • itertools.chain over manual list concatenation
  • functools.lru_cache for memoizing expensive pure functions
  • bisect for maintaining sorted lists instead of re-sorting on every insert

NumPy and Vectorization

For numerical work, the biggest lever is vectorization — replacing Python loops with NumPy operations that run in C with SIMD instructions.

import numpy as np import time n = 1_000_000 data = list(range(n)) # Pure Python start = time.perf_counter() result_py = [x ** 2 for x in data] py_time = time.perf_counter() - start # NumPy arr = np.array(data) start = time.perf_counter() result_np = arr ** 2 np_time = time.perf_counter() - start print(f"Python list comprehension: {py_time:.3f}s") print(f"NumPy vectorized: {np_time:.4f}s") print(f"Speedup: {py_time / np_time:.0f}x") 

Typical output: 20-50x speedup. No multiprocessing, no C extensions — just letting NumPy use the CPU efficiently.

For Pandas: avoid iterrows(). It iterates row by row in Python, discarding all the performance advantages of the library. Write the transformation as a column operation directly, or use apply() with a vectorized function.

Concurrency: Threads, Processes, and asyncio

Python's GIL means threads don't help CPU-bound code. But they do help I/O-bound work — API calls, database queries, file reads.

The correct tool depends on the bottleneck:

  • I/O-bound (waiting on network, disk): asyncio or ThreadPoolExecutor
  • CPU-bound (computation): ProcessPoolExecutor
from concurrent.futures import ProcessPoolExecutor, as_completed import os def heavy_computation(n): factors = [] for i in range(2, int(n ** 0.5) + 1): if n % i == 0: factors.append(i) factors.append(n // i) return sorted(set(factors)) numbers = [999983, 999979, 999961, 999931, 999917] * 20 # Sequential results_seq = [heavy_computation(n) for n in numbers] # Parallel across CPU cores with ProcessPoolExecutor(max_workers=os.cpu_count()) as executor: futures = {executor.submit(heavy_computation, n): n for n in numbers} results_par = [f.result() for f in as_completed(futures)] 

On a 4-core machine, the parallel version runs 3-4x faster. The overhead of spawning processes makes this worthwhile only for tasks that take at least a few milliseconds each — otherwise the inter-process overhead dominates.

For security tooling specifically, parallelizing I/O-bound work such as scanning a list of URLs against a threat intelligence checklist routinely produces 10-50x throughput improvements with asyncio, since you're almost entirely waiting on network responses rather than computing anything.

The Takeaway

Performance optimization in Python follows a strict order: measure, identify the actual bottleneck, apply the cheapest fix first.

  1. Profile with cProfile or py-spy — find where the time actually goes
  2. Fix algorithmic issues first (O(n²) → O(n) beats every other optimization)
  3. Replace Python loops with built-ins and NumPy where possible
  4. Parallelize I/O-bound work with asyncio, CPU-bound work with ProcessPoolExecutor
  5. Only reach for C extensions (Cython, ctypes) or PyPy when nothing else is enough

Skipping step 1 and jumping straight to parallelism is the most common mistake teams make. You'll add complexity without fixing the real problem. Profile first, every time.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.