How to open a large JSON file

What to use at each size

Why large JSON files freeze your tools

A size in megabytes does not explain the freeze. Three things do.

• The whole file must be read first. JSON has no index. A program cannot know where a value ends until it reads what comes before it.
• Parsed data is bigger than the text. In a browser, the objects take about 4 to 6 times the size of the text. A 200 MB file can need about 1 GB of memory.
• Drawing every row is slow. A viewer that makes one page element for each value makes millions of them, and the browser must lay out every one.

Chrome and Node also have a hard limit. One string can hold 536,870,888 characters, so file.text() fails on a file of about 512 MB, whatever memory you have.

Code editors have a similar problem. VS Code turns off features such as folding and highlighting on very large files, so you can open the file but you cannot explore it well.

Files up to 200 MB: use a viewer that draws only what you see

A virtual list draws the rows on your screen and nothing else, so the number of rows in the file stops mattering. A list of five million items opens as pages of 10,000, and the page stays responsive.

1. Open the large JSON viewer.
2. Choose your .json, .jsonl or .zst file. The file is read in your browser and is not uploaded.
3. Look at the size on each closed branch to find the part that makes the file big.
4. Search by text, or run a short query to count and group records.

In our test, a 108 MB file with 600,000 records opened in a few seconds on a laptop. jsonloupe refuses a document over about 200 MB and says so, because a browser tab cannot hold more as live objects.

Files over 200 MB: stream with jq

The plain form of jq reads the whole file into memory. The --stream option reads it piece by piece.

This prints the first 2 items of the tasks list, without loading the rest:

jq -cn --stream '
  limit(2; fromstream(2 | truncate_stream(
    inputs | select(.[0][0] == "tasks"))))
' big.json

If the top of the file is one big list, this writes one record per line. That format is JSONL, which is the next section:

jq -cn --stream 'fromstream(1 | truncate_stream(inputs))' big.json > big.jsonl

Many records: convert to JSONL and split

JSONL holds one JSON record on each line, so normal command-line tools can cut it. Take a sample, or split the file into parts:

head -n 1000 big.jsonl > sample.jsonl
split -l 100000 big.jsonl part-

Each part is small enough to open in the JSONL viewer.

Pull data out with Python

The ijson package reads a file as a stream, so memory use stays low. This prints the first 5 tasks:

from itertools import islice
import ijson

with open('big.json', 'rb') as f:
    for task in islice(ijson.items(f, 'tasks.item'), 5):
        print(task)

Python's own json.load reads the whole file into memory, so it can run out of memory on the same file.

Common questions

How big can a JSON file be?

The JSON format has no size limit. Each tool sets its own limit, usually from memory.

Can I open a compressed JSON file?

jsonloupe opens Zstandard .zst files directly and decodes them in your browser. The 200 MB limit applies to the decoded document.

Why do the IDs in my large file look wrong?

Many viewers round numbers past 15 or 16 digits. This guide explains why, and how to keep every digit.

Is my file uploaded?

No. jsonloupe has no server to send it to. It keeps working with your wifi off.

Open your file now. It stays on your computer, and there is no account to make. Open the viewer →

More guides

• Why JSON.parse rounds long IDs, and how to fix it
• How to convert nested JSON to Excel with multiple sheets