Metrics
Most important commands to remember
curl --fail— fetch the metrics exposition from an existing endpoint.promtool check metrics— check Prometheus text exposition and lint findings.
Commands and flags
| Command or syntax | Meaning |
|---|---|
metrics_sample=$(mktemp) |
Create a private temporary file and save its path. |
--fail --silent --show-error |
Report HTTP errors without a progress meter. |
--max-time 5 --output FILE |
Bound the transfer to five seconds and save its body. |
cat FILE |
Display the captured sample. |
promtool check metrics < FILE |
Read the exposition from standard input for syntax and lint checks. |
rm FILE |
Remove the downloaded sample. |
Quotes preserve the temporary path. < is shell input redirection. The endpoint is the server’s own instrumentation, not a query returning every metric it has collected.
The concepts that matter
1. A metric measures a defined quantity over time
A metric records a numeric observation with a name and context. Repeated samples form a time series that can show changes in traffic, latency, saturation, or errors.
The unit and measurement point matter. Requests accepted by a proxy and requests completed by an application are different quantities even if both are called requests. Define what the value counts before using it to judge health.
2. Counters, gauges, and histograms answer different questions
A counter accumulates events and can reset when its process restarts. Its increase over time is useful for rates. A gauge represents a value that can rise or fall, such as current queue depth.
A histogram records a distribution of observations, such as request durations, using a supported histogram representation. It can reveal a slow tail that an average hides. A single latest counter value is not requests per second, and a mean latency does not tell you the worst experience.
3. Labels identify separate series
Labels attach dimensions such as method, status class, or service. Each distinct label combination creates another time series. This makes comparison possible but also increases storage and query work.
Unbounded identifiers such as request IDs or arbitrary user input can create excessive cardinality. Those details usually fit event records or traces better than metric labels. Choose dimensions that answer useful aggregate questions with a bounded number of combinations.
4. Collection and interpretation are separate stages
An exporter exposes values; a monitoring system collects and stores samples; queries and alerts interpret them. A working metrics endpoint does not prove that collection, storage, and alert evaluation are all healthy.
Rates need an observation window and correct reset handling. Histogram quantiles are estimates with representation-dependent resolution. Missing samples, changing labels, and process restarts can all affect interpretation. Compare the metric’s semantics and collection history before attributing a graph change to the application.
One small example
Optional: fetch the existing local server’s metrics. Run each line individually and continue to inspection only after curl succeeds. The example neither starts Prometheus nor changes its scrape configuration.
metrics_sample=$(mktemp)
curl --fail --silent --show-error --max-time 5 --output "$metrics_sample" http://localhost:9090/metrics
cat "$metrics_sample"
promtool check metrics < "$metrics_sample"
rm "$metrics_sample"Read HELP and TYPE lines before interpreting samples. Compare a metric name, its labels, and its numeric value. Some families include multiple related series rather than one standalone number.
Promtool reports format or lint findings; success does not prove sensible units, correct instrumentation, successful scraping, or alert quality. The file contains one capture, so it cannot establish a rate or trend. The last command removes it. If downloading fails, inspect that error instead of treating an empty file as healthy instrumentation.
Keep this idea: Metrics summarize defined quantities over time; type, unit, labels, and observation window determine what a number means.