The Thanos Query UI provides robust querying capabilities across multiple Prometheus instances, offering a comprehensive analysis of your metrics. Using the Thanos Interactive example dataset, I experimented with various PromQL queries to explore metrics. Here’s a summary based on frequent use cases:

Available Metrics in the Metrics Explorer:

• continuous_app_metric0
• continuous_app_metric1
• continuous_app_metric2
• continuous_app_metric3
• continuous_app_metric4

PromQL Query Examples:

1. Sum of Metrics:sum(continuous_app_metric0) - Sum across all clusters and replicas.
2. Sum by Cluster:sum(continuous_app_metric0) by (cluster) - Grouped by cluster.
3. Sum by Replica:sum(continuous_app_metric0) by (replica) - Grouped by replica.
4. Average by Cluster:avg(continuous_app_metric0) by (cluster) - Average grouped by cluster.
5. Maximum Value by Replica:max(continuous_app_metric0) by (replica) - Max value grouped by replica.
6. Count of Instances:count(continuous_app_metric0) - Count of reporting instances.
7. Specific Cluster and Replica:continuous_app_metric0{cluster="eu-1", replica="0"} - Current value for a specific cluster and replica.
8. Alerting:continuous_app_metric0 > 100 - Create an alert if the metric exceeds a threshold.

Understanding Empty Query Results:

• Histogram Quantile Query: May return empty if_bucket suffix is missing, required forhistogram_quantile.
• Comparison Between Clusters: Returns empty if data points are missing or unmatched in timestamps.

PromQL Info Messages:

• Rate and Increase Queries: Indicate thatcontinuous_app_metric0 might not be a counter as it does not end in_total,_sum,_count, or_bucket.

Metric Types Explained:

• Metric Types in Prometheus
  • Gauge: Tracks current values that can go up or down (e.g., memory usage).
  • Counter: Tracks cumulative totals that only increase (e.g., total HTTP requests).
  • Histogram: Tracks distributions across buckets (e.g., request latencies).
  • Summary: Tracks distributions as quantiles/percentiles (e.g., request latencies).
• Metric Types in Instrumentation
  • Gauge: Use methods likeSet(),Inc(),Dec(),Add(),Sub().
  • Counter: Use methods likeInc(),Add().
  • Histogram: Configure buckets, useObserve() to categorize values.
  • Summary: Specify quantiles, useObserve() to track values.