▌Logier

Clustering Repeated Messages

Collapse a noisy log into the handful of message shapes it is actually made of. Logier groups lines that differ only in ids, numbers and timestamps, counts them, and lets you filter the whole view down to one pattern.

Most large log files are not made of thousands of different messages. They are made of a few dozen messages that happen thousands of times, each one carrying a different id, duration or IP address.

Clustering shows you that view. Logier reduces every line to its shape, groups the lines that share one, and sorts the groups by how often they occur. A 200,000-line file usually collapses into something you can read in a few seconds.

Finding the Clusters panel#

Open a log file and look just above the timeline. Alongside the Anomalies summary you will find a bar reading something like 34 message patterns, with a badge showing what share of the file those patterns account for. Click it to expand the list.

The panel only appears when there is something to show — a file where nothing repeats has no patterns to report.

Reading a pattern#

Each row in the list is one message shape:

  • —The pattern — the message with the parts that change on every occurrence replaced by placeholders
  • —Level and services — the level it occurs at most often, and which services produced it
  • —Time span — first and last time this pattern appeared
  • —A sparkline — when in the file those occurrences landed
  • —A count and a percentage — how many times it occurred, and what share of the file that is
Every sparkline is drawn against the whole file's time span, not the pattern's own. That is what makes them comparable: a burst confined to the last five minutes looks obviously different from a steady trickle across the entire file.

What the placeholders mean#

To decide that two lines are the same message, Logier masks the parts that make every occurrence unique. Those masks are left visible in the pattern so you can see what was taken out:

  • —<n> — any number, whole or decimal
  • —<ip>, <url>, <email>, <path> — addresses and locations
  • —<uuid>, <id>, <hex> — identifiers, session tokens, hashes and container names
  • —<time> — a timestamp written inside the message text itself
  • —"<str>" — anything in quotes, which is usually a value rather than part of the message

So connection refused to 10.0.4.19 after 3000ms and connection refused to 10.0.7.2 after 250ms both become connection refused to <ip> after <n>ms — one pattern, counted twice.

Masking numbers means two messages that differ only by a status code — "returned 200" and "returned 500" — land in the same cluster. That is usually what you want when counting repetition, but when the number is the point, filter by level or search for the specific code instead.

Filtering down to one pattern#

Click any row and the entire view narrows to just those lines — the table, the timeline and the level breakdown all follow. This is the fastest way to answer "when did this actually start?" for a message you have spotted.

Click the same row again to clear it, or use Show all on the Clusters bar.

A pattern selection is a filter in its own right and stacks with the others — you can select a pattern and narrow by level, service or search term at the same time. See Filtering Log Entries for the rest of them.

Selecting a pattern is not the same as searching for its text. The pattern contains placeholders where the volatile data was, so no search term could express it — Logier matches on the shape itself.

What makes the list#

  • —A pattern needs at least two occurrences. A line that happened once is not a pattern, it is just a line — and including them would bury the real repetition.
  • —The 50 largest patterns are listed. The percentage badge on the bar tells you how much of the file they cover between them, so you can see what is left over.
  • —The result does not depend on line order. The same content always produces the same clusters, so what you see is reproducible and safe to quote in a report.

Clustering runs over the whole file, off the main thread, so it does not slow the viewer down on large logs. Very long messages are shortened in the pattern text — the grouping still uses the full line.

Clusters and anomalies#

The two panels sit side by side and use the same idea of "the same message". A repeat storm anomaly is simply a cluster that grew large enough to be worth flagging, and a new signature anomaly is a pattern that had never appeared before. See Understanding Anomalies.

Use Anomalies when you want Logier's opinion on what is unusual, and Clusters when you want the plain shape of the file with no judgement attached.

Asking the Copilot#

Ask the Copilot "what is repeating in this log?" or "what are the most common errors?" and it uses the same clustering engine through its cluster_logs tool — so what it describes and what you can see on screen are the same groups. See MCP Tools.

This is one of the most useful things to ask about a large file. The Copilot only reads an excerpt of your log directly, but this tool runs across the complete file — so a message occurring 40,000 times is reported as one dominant pattern rather than drowning the excerpt.

Hiding the panel#

If you would rather not see it, right-click anywhere in the log viewer and untick Clusters. The same menu toggles the Filters, Anomalies, Chart and Table sections.

A quick workflow#

  1. Open the file and expand Clusters to see what it is mostly made of.
  2. Scan the sparklines for a pattern that is bursty rather than steady — that is usually where the incident is.
  3. Click it to filter the view, then use the timeline to see exactly when it started.
  4. Press Show all, and repeat with the next suspicious pattern.
←Back to Log Viewer