How to process files that outgrow memory and explain your approach in an interview.

Python interview illustration of a huge CSV passing one row at a time through a small RAM tray.

The file can outgrow RAM without the working set doing the same.

Your script finishes on the sample file and now your output looks sensible, and you mentally move on to the next task.

But then someone sends the actual export.

Suddenly, the useful little script is competing with everything else on your laptop for memory. You add a chunk size, try again, and discover that Python can still run out of room while politely processing one chunk at a time.

Honestly, this is my favorite part of the problem to unpack, because the first answer sounds so reassuring. Read the file in smaller pieces and everything should be fine.

Except reading less at once does not help enough if you keep everything afterward.

Let’s build a small CSV pipeline around that distinction. By the end, you’ll have a practical approach for an oversized export and a much stronger answer when an interviewer changes the memory limit.

Before choosing a library, decide what survives each row

Imagine an orders export with four columns, and a task that sounds pleasantly boring. Keep paid orders, retain their IDs and amounts, and write one JSON object per line.

Nothing in that result requires remembering an earlier order. There is no global sort, no deduplication, and no running dictionary of customers.

That detail gives us permission to release each parsed row after writing its result. If someone adds a requirement to remove duplicate order IDs, we have a different memory problem, even though the input file hasn’t changed.

I prefer to sketch three places where data can accumulate before touching the implementation.

  • Input includes parser buffers and the current row or chunk.
  • Retained state includes anything kept between iterations, such as a set of IDs or a list of results.
  • Output includes serialization and anything waiting to be written.

Memory budget divided into input buffer, retained state and output buffer, with a growing pile of retained keys.

Account for all three buckets, plus process overhead.

Those three buckets sit on top of the interpreter, libraries, and other process overhead. So a 512 MB process limit does not mean we can comfortably choose a 512 MB chunk.

For this task, the goal is to keep the working set tied to a bounded record size, rather than the total number of records. We still need enough disk space for the output, which is a separate budget.

Give the tiny script a clear contract

Let’s keep the exercise small enough to explain without turning it into a data platform.

  • Input is a stable, local UTF-8 CSV with exactly order_id,status,amount_cents,note as its header.
  • Each record has four fields. IDs must be nonempty, status must be paid, pending, or refunded, and amounts are nonnegative ASCII integers of at most 12 digits.
  • Fields are limited to 100,000 characters. We assume a trusted producer also bounds total record size and field count; this is not a parser for arbitrary hostile uploads.
  • Every valid paid record produces an output object containing only its ID and integer amount. Other valid statuses are skipped, input order is preserved, and duplicate IDs remain duplicates.
  • Malformed input stops the run. A partially written file must not be presented as a completed export.

That last point is worth agreeing on early. Silently skipping bad rows can make a job look successful while quietly changing the answer.

We also need a real CSV parser. A note can contain a comma or a quoted newline, so neither splitting on commas nor treating every physical line as a complete record is enough. Python’s CSV documentation explains the parser behavior and recommends opening file objects with newline=''.

Let one row travel all the way to disk

For our example, order A17 has status paid and an amount of 1250 cents. Its note can disappear because the output does not need it. Order A18, with status pending, produces no output record.

Here’s a reference implementation for that contract. It uses the standard library and writes to a new partial file before promoting the finished result.

import csv import json from pathlib import Path HEADER = ['order_id', 'status', 'amount_cents', 'note'] STATUSES = {'paid', 'pending', 'refunded'} def export_paid(source, destination): source = Path(source) destination = Path(destination) partial = destination.with_name(destination.name + '.partial') if source.resolve() in {destination.resolve(), partial.resolve()}: raise ValueError('Input and output paths must differ') if destination.exists(): raise FileExistsError(destination) old_limit = csv.field_size_limit(100_000) try: with source.open(encoding='utf-8', newline='') as src: reader = csv.reader(src, strict=True) if next(reader, None) != HEADER: raise ValueError('Unexpected or missing header') with partial.open('x', encoding='utf-8', newline='\n') as out: for record_no, row in enumerate(reader, start=2): if len(row) != 4: raise ValueError(f'Wrong field count at record {record_no}') order_id, status, amount, _note = row valid_amount = ( 1 <= len(amount) <= 12 and amount.isascii() and amount.isdigit() ) if not order_id or status not in STATUSES or not valid_amount: raise ValueError(f'Invalid values at record {record_no}') if status == 'paid': result = {'order_id': order_id, 'amount_cents': int(amount)} out.write(json.dumps(result, ensure_ascii=False) + '\n') partial.replace(destination) finally: csv.field_size_limit(old_limit) export_paid('orders.csv', 'paid-orders.jsonl')

A paid order passes through CSV parsing, validation, filtering and JSONL writing; invalid rows stop the run.

One accepted record becomes one output object. Illustrated values; the code supplies the exact serialization.

Notice what never appears here. We don’t call list(reader), append every result to a list, or wait until the end to serialize the entire export.

Each iteration replaces the previous row. The output file uses ordinary bounded buffering, and JSON serialization briefly creates a string for the current result. With the stated record size assumptions, memory does not need to grow in proportion to the file’s row count.

The record number in an error is a logical record number, with the header counted as record one. It is not necessarily the physical line number, because quoted fields can span lines.

One practical boundary matters here, too. This example assumes one process owns these output paths. Checking whether a destination exists is not a concurrency lock, and replace can overwrite a destination created by another process in the meantime. Concurrent jobs need unique run paths and a coordinated completion step. CSV field size setting is also process-wide, so don’t change it casually inside competing threads.

Chunking is useful until the chunks start collecting

Maybe pandas is already the right tool for your transformation. Wonderful, keep it. Just inspect what happens between reading a chunk and asking for the next one.

This pattern is the trap.

pieces = [] for chunk in pd.read_csv('orders.csv', chunksize=50_000): pieces.append(transform(chunk)) result = pd.concat(pieces)

Assuming the transformed chunks remain substantial, the list keeps their data alive. Concatenating them may require additional allocation. We’ve delayed the big memory bill instead of removing it.

Keeping every chunk grows memory, while writing and releasing chunks keeps only the working set.

Chunk size helps only when retained data stays within the budget.

For an independent per-row transformation, write each processed chunk to the output and release it before continuing. Read only the columns you need, choose suitable types, and leave room for temporary arrays created by the transformation. The pandas scaling guide covers these options.

Also, 50,000 rows is a row count, not a memory guarantee. Fifty thousand short numeric records and fifty thousand long text records are very different guests at the same party.

Measure peak process memory on representative data, including unusually wide records and the actual transformation. A DataFrame’s own memory estimate is useful, but it doesn’t describe every allocation in the process.

When the follow-up changes, change the plan

Now imagine the interviewer asks for total spend per customer.

A dictionary feels natural, and it may be perfectly reasonable when the number of distinct customers fits the budget. But its size follows distinct keys, which could grow almost as fast as the file.

We can still read sequentially while using too much memory. The input path is streaming; the retained state is growing.

If that state cannot fit, move the aggregation into a disk-backed database or a processing engine that can spill intermediate work. A hand built partitioning approach needs bounded partitions and a plan for skew, because one oversized partition can bring the same problem back.

A global sort changes the task again. Sorted chunks alone are not a globally sorted result; an external sorting plan needs sorted runs, a merge, and enough temporary disk space.

Decision sheet for per-row transforms, grouping by key, and global sorting or deduplication.

Save this before adding a new requirement to your file processing script.

You don’t need to bring distributed computing into every large file conversation. Start with the required result, estimate the state it demands, and choose the smallest approach that fits the actual limits.

Make the answer checkable before making it bigger

Before scaling this example, I want a tiny fixture whose expected output I can explain without trusting the script. These are the cases to include.

  1. Mix paid and pending orders. Only paid records should appear, in their original order.
  2. Put a comma and a quoted newline inside a note. Each should remain part of one parsed field and must not create extra output records.
  3. Include a paid order with zero cents. Zero is valid under our contract, so it must survive.
  4. Add a duplicate paid ID. Both records should remain because deduplication was never requested.
  5. Try an invalid amount, a wrong field count, and a field over the size limit. Each should fail the run instead of yielding a supposedly complete destination.
  6. Use a header only file. It should produce an empty completed output like a completely empty file should fail the header check.

Those are proposed checks for this reference implementation than just reported test results or a benchmark.

Next, increase the amount of representative data while tracking elapsed time, peak memory, output size, and whether the output remains correct. If memory climbs with every batch, look for references that keep previous data alive before reaching for a larger machine.

Leave the next person a trustworthy output

Our partial file is deliberately boring. If processing fails halfway through, the final destination is never created by this run. Inspect the error and the leftover partial file, then remove or archive that partial file before restarting from the beginning with the same unchanged source.

That is a simple restart policy, not resumable processing. Renaming after success also isn’t a promise of crash durability or exactly once delivery to another system.

For very expensive jobs, design checkpoints around stable input identity and valid parser boundaries, with a clear policy for already-written output. A random byte offset can land inside a quoted CSV record or a multibyte character, so saving the last offset is not automatically a safe resume strategy.

For a nice transfer exercise, try the large-log-file section of this practice question. Focus on Part 3 and explain how your approach changes when you need a timestamp range or the end of a huge log, rather than a complete CSV transformation.

Try answering that before looking for another tool. You’ll quickly discover whether you’re choosing an access pattern or just naming a library.

Keep this explanation in your interview notes

I’ll first clarify whether the output needs information from previous records. For this filter-and-transform task, I can parse one bounded record, validate it, write its result, and release it. I’ll budget input buffers, retained state, serialization, and process overhead. If a new requirement introduces global grouping or sorting, I’ll account for that state separately and use disk-backed work when it cannot fit.

That answer gives the interviewer something concrete to explore, and it gives you a design you can actually defend.

Small samples are lovely for checking logic.
They just can’t tell you what your program will keep when the real file arrives.

Save the decision sheet for your next export, and follow along if you like turning ordinary engineering problems into interview practice that feels useful at work, too.