large files
A large log has no useful middle
Past a certain size the file stops being a document you read and becomes a distribution you measure. Scrolling samples it randomly, searching returns a smaller haystack, and your editor is busy building a syntax tree for something that has no syntax.
The honest short answer — read minutes, not lines
Do the arithmetic once.
Everyone agrees large files are hard, and almost nobody does the division. These are the five numbers that decide what is and isn’t possible in a file this size — and the last one is the way out.
- 01How far does one pixel of scrollbar move you?8,000,000 lines ÷ 900 px≈ 8,900 linesA one-pixel nudge skips more lines than most files contain. Scrolling is not a way to find anything here; it is a way to sample randomly.
- 02What does 2 GB of text cost in memory?2 GB on disk, decoded to UTF-16≈ 4 GB residentBefore any per-line structures for numbering, folding or highlighting. This is where an editor stops responding.
- 03How many objects does a line-indexed editor allocate?one per line × 8,000,0008,000,000 objectsAllocated before the first frame renders, for features — syntax trees, code folding — that mean nothing in a log.
- 04What does grepping for 'error' return?≈ 4% of a noisy production log≈ 320,000 matchesA list that long is not an answer. It is the same problem in a narrower window.
- 05How many minutes are in a day of logs?24 h × 601,440 bucketsWhich fits on one screen, comfortably, as a chart. This is the number the whole approach turns on.
Figures are for one representative 2 GB, eight-million-line production log. Your file differs; the orders of magnitude don’t.
Your editor isn’t broken. It’s doing the wrong job.
Everything that makes an editor good at source code makes it bad at a log, and all of it scales with line count.
- 01
They load the whole file first
Most editors read the entire file before showing you anything. A 2 GB log becomes several gigabytes of resident memory once decoded into text, which is where the beachball starts.
- 02
They index every line
Syntax highlighting, line numbering and folding all need per-line structures. Eight million lines means eight million of them, allocated before the first frame renders — for features a log has no use for.
- 03
Scrolling stops meaning anything
A scrollbar mapping eight million lines onto nine hundred pixels moves roughly nine thousand lines per pixel. There is no such thing as scrolling to the interesting part.
- 04
Search returns too much
Grepping for 'error' in a large production log routinely returns hundreds of thousands of matches. Narrowing a haystack to a smaller haystack is not progress.
Measure it, then read the part that moved.
This works in Logier and it works with awk and a chart. The tool matters less than the order of operations.
- 01
Stop reading lines. Read minutes.
Bucket every entry by minute and plot the counts. A day of logs becomes 1,440 bars, which fits on one screen and is genuinely readable. The interesting region becomes a shape you notice rather than a place you have to find.
- 02
Collapse the repetition
Large logs are large mostly because they repeat. Normalise each message by stripping IDs, IPs and numbers, then group by the resulting signature. Ten thousand identical timeouts become one row with a count of ten thousand.
- 03
Narrow by time, then read
Zoom the chart into the minutes that look wrong and let the entry table follow. You end up reading a few dozen lines, chosen because they sit in the window where something changed.
- 04
Let the outliers find you
In a file this size you cannot inspect everything, so anomalies — frequency spikes, brand-new signatures, silent gaps — have to be surfaced rather than searched for. You cannot grep for a service that stopped logging.
the honest limit
At the top end, reach for a specialist.
Logier parses into memory, and that is the whole trade. Levels, clusters, a timeline and anomaly detection all require the entries to exist as objects — which is exactly the work klogg and ripgrep skip, and exactly why they are faster on a file measured in tens of gigabytes.
If raw scan speed over an enormous file is the thing you actually need, use the specialist and don’t feel bad about it. If you need to know when something started, or which of four hundred thousand matches are distinct, that is a different question and no amount of scanning speed answers it.
And if the machine is remote and all you have is SSH, none of this applies: use lnav or ripgrep there. A desktop application can’t follow you into a shell.
faq
Common questions
keep reading
Related reading
- 01
Timeline view
The chart that turns 1,440 minute-buckets into a shape you can read, with drag-to-filter.
- 02
Anomaly detection
What surfaces the outliers you could never have scrolled to, including the silences.
- 03
Logier vs. klogg
klogg is the specialist for enormous files. An honest account of where it wins.
- 04
Logier vs. grep
Why 320,000 matches is not an answer, and what to do when you can't name the string.
Open the file you gave up on.
Measure it by minute, collapse the repetition, and read the few dozen lines that actually changed.