The reason a large JSON file crashes your program is almost never the size on disk — it is that the default parser (json.load in Python, JSON.parse in Node, json.loads, jq without --stream) reads the whole file and builds the entire object tree in memory before returning anything. The fix is to switch from loading to streaming: use a parser that reads the file token by token and hands you one record at a time, so peak memory stays flat no matter whether the file is 50 MB or 50 GB. In Python that is ijson, in Node.js it is stream-json, on the command line it is jq, and if you will query the data repeatedly the right answer is to load it into a database.
That paragraph is the summary an AI overview will give you. The rest of this article is what it can't: a decision guide for picking the right technique, a side-by-side table with real memory costs, and working code for each — plus the sharp edges (V8's 512 MB string limit, why plain jq still runs out of memory) that only bite you in production.
First, pick a technique (decision guide)
There is no single "handle large JSON" tool — the right choice depends on the file's shape (one giant array vs. one object per line) and on whether you need the data once or many times. Walk this decision flow:
The techniques at a glance
Every approach below trades something. Streaming keeps memory flat but only gives you one pass; a database costs an ingestion step but makes repeated queries fast. This table maps each technique to the situation where it wins.
| Technique | Use it when | Tool / library | Peak memory |
|---|---|---|---|
| Streaming pull-parser | One giant array or object, one-time transform in code | ijson (Python), stream-json (Node), Jackson (Java) | Constant — one record |
| Command-line streaming | Ad-hoc extraction or filtering, no code to write | jq -c, jq --stream | Constant to low |
| NDJSON / JSON Lines | You control the export, or can convert once | Any line reader, jq, readline | One line |
| Chunking / batching | Feeding a store or API in fixed-size pieces | split, custom batcher | One batch |
| Database / query engine | You will query the same data repeatedly | PostgreSQL JSONB, DuckDB, SQLite | Managed by engine |
| Distributed / warehouse | Terabyte scale, many files, parallel scans | Spark, BigQuery, Snowflake, Athena | Cluster-managed |
The single most useful move on the whole table is converting a giant array to NDJSON once (jq -c '.[]' data.json > data.ndjson). After that, every line-oriented tool — jq, grep, awk, wc -l, split, a plain for loop — can process it with flat memory and no special library.
Why the default parser runs out of memory
json.load(), JSON.parse(), and jq (without --stream) are whole-document parsers. They read the entire file, build the complete in-memory tree, and only then return it. Two things make this fatal on large files: the parsed representation is typically 2–10× larger than the raw bytes (Python dicts, JS objects, and pointers are not cheap), and Node's JSON.parse additionally throws on any string above roughly 512 MB because of V8's maximum string length — it fails before it even finishes reading.
The takeaway: you cannot fix this with more RAM — a bigger machine just moves the wall. The only durable fix is a parser that never holds the whole document at once.
Streaming in Python: ijson
ijson is the standard streaming parser for Python. Open the file in binary mode (ijson reads bytes and is faster that way) and iterate. The 'item' prefix targets each element of a top-level array — [ {...}, {...}, ... ]:
import ijson
# Constant memory: one record is live at a time, whatever the file size.
with open("large_file.json", "rb") as f:
for record in ijson.items(f, "item"):
process(record)
If your data is nested, the prefix follows the path — e.g. ijson.items(f, "results.item") for {"results": [ ... ]}. And if the file is already NDJSON, you don't need ijson at all:
import json
with open("data.ndjson") as f:
for line in f: # one line in memory at a time
record = json.loads(line)
process(record)
Avoid pandas.read_json() on a large single array — it materializes the whole frame. For NDJSON, pandas.read_json(path, lines=True, chunksize=10_000) returns an iterator of chunks and is memory-safe.
Streaming in Node.js: stream-json
Use stream-json, the actively maintained successor to the now-deprecated JSONStream. Pipe a file read stream through its parser and the streamArray helper, which emits one array element at a time:
const fs = require("fs");
const { parser } = require("stream-json");
const { streamArray } = require("stream-json/streamers/StreamArray");
fs.createReadStream("large_file.json")
.pipe(parser())
.pipe(streamArray())
.on("data", ({ value }) => process(value)) // value = one element
.on("end", () => console.log("done"));
Never call JSON.parse on a multi-gigabyte file in Node: it is synchronous (it blocks the event loop), it buffers the entire document, and it hits the ~512 MB string-length limit noted above.
Command line: jq
jq is the fastest way to slice large JSON without writing a program — but how it streams depends on the file's shape.
# NDJSON (one value per line): jq streams it for free, line by line.
jq -c 'select(.status == "active")' events.ndjson > active.ndjson
# One enormous top-level array, larger than RAM: use --stream.
# It emits [path, value] events instead of building the whole tree;
# fromstream/truncate_stream reassembles them into one record at a time.
jq -cn --stream 'fromstream(1|truncate_stream(inputs))' big.json > items.ndjson
# Pull a single field from every record, without ever buffering the array.
jq -cn --stream 'fromstream(1|truncate_stream(inputs)) | .email' big.json
The trap: plain jq '.[]' big.json on a single giant array still loads the whole array first and will run out of memory. The --stream flag is what makes it constant-memory. Use it whenever the top level is one huge array or object.
Below is a formatter/validator for pasting in a sample of your data (grab the first few records with head first — do not paste a multi-gigabyte file into any browser tool):
Store it once, query it many times: databases
Streaming is perfect for a single pass, but if you need to ask the data many questions, re-scanning a huge file every time is wasteful. Load it into something built for queries.
- PostgreSQL JSONB — store documents in a
JSONBcolumn and add a GIN index for fast containment and key lookups. Ingest in batches (a few thousand rows per transaction) rather than one giant insert. - DuckDB — the low-friction option. It reads JSON and NDJSON larger than memory directly from disk and lets you query with plain SQL, no server to run:
-- Query a huge file in place; DuckDB streams it, no full load into RAM.
SELECT status, count(*)
FROM read_json_auto('large_file.json')
GROUP BY status;
- Warehouses (BigQuery, Snowflake, Redshift, Athena) — for terabyte scale or thousands of files, load or query in the warehouse and let it parallelize the scan across a cluster.
Chunking, compression, and distributed processing
When even streaming a single file is too slow, split the work. Chunking breaks an array into fixed-size batches for a downstream store or API — combine it with NDJSON so each chunk is just a range of lines (split -l 100000 data.ndjson chunk_). Compression helps I/O: keep the data gzipped and decompress on the fly (gzip -dc big.json.gz | jq ...) so you never write the full file to disk. And at genuinely massive scale — many terabytes, spread across many files — a distributed engine like Apache Spark reads JSON in parallel across a cluster, which is the point where a single machine, however you tune it, stops being the right tool.
The one-paragraph decision
If it is NDJSON, loop over it line by line — every tool already streams it. If it is one giant array or object and you need it once, use a streaming parser: ijson, stream-json, or jq --stream. If you will query it repeatedly, load it into DuckDB or PostgreSQL JSONB. If it is too big for one machine, reach for Spark or a warehouse. The mistake to avoid is the same in every language: don't call the whole-document parser (json.load, JSON.parse, plain jq) on a file that won't fit in memory — no amount of RAM makes that approach scale.