My Dashboard Looked Beautiful and Had Exactly One Operation In It
59 of 59 traces were the same 250ms heartbeat. My dashboards were lying to me.
I stood up what looked like a production-grade observability stack on Tacavar: Tempo for traces, Grafana for panels, OpenTelemetry for instrumentation. Three dashboards—Bailian Team Overview, Paperclip Swarm Observability, Tacavar Ops—all rendering beautifully. Tight latency heatmaps. Populated bar charts. Traces flowing in real time. Any founder would have looked at that screen and said, "Ship it."
Then I probed the queries behind the panels. What I found wasn't just embarrassing. It was a masterclass in observability anti-patterns.
A Dashboard That Looked Perfect
Every panel was styled, labeled, and arranged like a product screenshot. The Bailian Team Overview had a row for agent calls, another for token usage, another for cost. The Paperclip Swarm dashboard showed trace throughput across every worker. The Tacavar Ops view had error rates, p95 latencies, and a heatmap that looked like a healthy heart rhythm.
Nothing was red. Nothing was flashing. No alerts fired. That was the problem.
We're trained to trust green. A dashboard with no alerts is a dashboard that says "all clear." But a dashboard is only as honest as the queries beneath it. If the queries return nothing, the green is just a painted backdrop.
The Query Behind the Pretty Panels
I started by opening the raw query for each panel. Not the rendered graph—the actual PromQL or TraceQL. The Bailian Team Overview dashboard was querying Prometheus metrics called agent_calls_total, agent_tokens_total, and agent_cost_usd_total.
Those metrics didn't exist. Not a single series. I checked the metric catalog and found zero entries with those names.
Still, Grafana rendered them as beautiful empty timeseries. Styled axes. Subtle gridlines. A flat line that looked like "low but real" traffic. Classic grafana dashboard empty pattern—not an error, just an absence dressed up as a signal.
That's the trap. A missing series renders identically to a healthy zero. Unless you know what to look for, the dashboard doesn't tell you it's blind.
59 Traces, One Heartbeat: What Was Actually Running
The Tempo data was worse. I queried the last hour of traces from the Paperclip Swarm and got 59 results. 59 out of 59 were the same operation: paperclip_handle_heartbeat, a 60-second cron that polls its own work queue and returns a 250ms no-op.
Zero agent runs. Zero LLM calls. Zero tool calls. Zero task outcomes. The entire swarm had been idle for an hour, and the dashboard showed a lively trace stream.
The trace heatmap wasn't lying—it was faithfully rendering a single repeated operation. But the story it told was fiction. I'd built an observability stack that could detect a heartbeat but couldn't tell me whether the system was doing anything that mattered.
This is the heart of opentelemetry trace debugging: you can't just collect traces, you have to ask what they represent. A trace is not a task. A heartbeat is a trace. A cron poll is a trace. But neither tells you if value was produced.
Graceful Degradation Is an Anti-Feature
Graceful degradation sounds like a feature. In observability, it's the enemy.
A broken dashboard—one with a red error, a blank panel, or a query failure—makes people investigate. They dig in, find the root cause, fix it. That's healthy.
A beautifully empty dashboard makes people think their system is working. There's nothing to investigate. No red flags. No missing data warnings. The system is running, panels are updating, and the only lie is the one you can't see.
We've built an entire culture around graceful degradation in software. A service that fails silently is considered robust. But for observability, silent failure is the disaster. The moment a dashboard stops telling the truth, it should scream, not whisper.
Missing Data Looks Identical to Healthy Data
This is the deepest problem. Missing data doesn't look like missing data—it looks like a low-traffic period.
If the agent_calls_total metric doesn't exist, the panel renders a flat line at zero. If the system actually ran zero agent calls, the panel renders the same flat line. Identical pixels. One is a healthy absence, the other is complete blindness.
The only way to differentiate is to count rows. Query the series and ask: how many distinct metric names exist? How many unique trace operations appear in the last hour? If the answer is zero or one, you don't have low traffic—you have an observability black hole.
This is why missing metrics detection matters more than any fancy visualization. It's the difference between "the panel is empty because nothing happened" and "the panel is empty because the query is broken."
Auditing Observability by Counting Rows, Not Looking at Graphs
Stop looking at graphs. Start looking at queries.
Here's a simple audit you can run today. For every dashboard you trust, open the query and execute it against the raw data source. Count the rows returned. Ask three questions:
- Are the metric names real? Check the actual series catalog.
- Are the trace operations varied? Count distinct operation names in the last hour.
- Does the query reference fields that exist in the schema?
In my audit, the Bailian Team Overview had zero real metrics. The Paperclip Swarm had one real trace operation. The Tacavar Ops dashboard had healthy metrics, but they were measuring infrastructure, not agent behavior.
That's the other trap: measuring the wrong thing. A dashboard full of CPU and memory panels is technically accurate. It just doesn't tell you whether the system is doing useful work.
Run the counts. If a dashboard returns one row where you expect thousands, that's not a quiet day. That's a lie.
Building Dashboards That Fail Loudly
The fix isn't more panels. The fix is making dashboards fail loudly when the underlying data is empty or meaningless.
Here's what that looks like in practice:
- Every panel should include a row-count indicator. If a query returns zero series, the panel should render an error state, not a flat line.
- Every metric name should be validated against the catalog. Unknown names should trigger a warning.
- Every trace-based panel should show the number of distinct operations, not just total trace volume. If 59 traces are all the same heartbeat, that's a signal, not noise.
- Alerts should fire on missing data, not just on thresholds. Absence is a value.
The industry has accepted observability anti-patterns for too long. We ship dashboards that look like product demos and call it monitoring. We hire SREs to stare at green screens while the system silently does nothing.
Tacavar's approach is different. The monitoring stack doesn't just render whatever query you give it—it audits the query results, checks for missing metrics, and flags when your data is too thin to trust. It treats an empty panel as a bug, not a style choice.
The lesson from my 59 heartbeats is simple: never trust the rendered output alone. Always run the query and count the rows. If you can't do that manually, use a tool that does it for you.
Before you trust another dashboard, audit the queries behind it. Tacavar's monitoring stack does this automatically. Explore at https://tacavar.com/observability.