K8sGPT - Giving Kubernetes Superpowers to Everyone

Overview

K8sGPT provides comprehensive observability features that allow you to monitor and analyze your Kubernetes clusters effectively. This tutorial will guide you through setting up Prometheus and Grafana integration with K8sGPT.

Prerequisites

K8sGPT installed in your cluster
Prometheus and Grafana installed in your cluster
Access to your cluster's monitoring stack
Basic understanding of Kubernetes metrics

Metrics and Monitoring

K8sGPT exposes several metrics that can be collected by Prometheus:

# HELP k8sgpt_analysis_duration_seconds Duration of K8sGPT analysis
# TYPE k8sgpt_analysis_duration_seconds histogram
k8sgpt_analysis_duration_seconds_bucket{le="0.1"} 0
k8sgpt_analysis_duration_seconds_bucket{le="0.5"} 0
k8sgpt_analysis_duration_seconds_bucket{le="1"} 0
k8sgpt_analysis_duration_seconds_bucket{le="+Inf"} 0
k8sgpt_analysis_duration_seconds_sum 0
k8sgpt_analysis_duration_seconds_count 0

# HELP k8sgpt_issues_total Total number of issues found
# TYPE k8sgpt_issues_total counter
k8sgpt_issues_total{severity="critical"} 0
k8sgpt_issues_total{severity="warning"} 0

Setting Up Prometheus

Configure ServiceMonitor: Create a ServiceMonitor resource to tell Prometheus to scrape K8sGPT metrics.
Verify Metrics Collection: Check that Prometheus is collecting K8sGPT metrics.
Set Up Alerting Rules: Configure alerting rules based on K8sGPT metrics.

Grafana Dashboards

Import the following Grafana dashboard to visualize K8sGPT metrics:

{
  "dashboard": {
    "title": "K8sGPT Metrics",
    "panels": [
      {
        "title": "Analysis Duration",
        "type": "graph",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "rate(k8sgpt_analysis_duration_seconds_sum[5m]) / rate(k8sgpt_analysis_duration_seconds_count[5m])"
          }
        ]
      },
      {
        "title": "Issues by Severity",
        "type": "piechart",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "k8sgpt_issues_total"
          }
        ]
      }
    ]
  }
}

Logging Integration

K8sGPT logs can be integrated with your existing logging stack:

Configure log levels and output format
Set up log aggregation with Fluentd or similar tools
Create log-based alerts for critical issues

Best Practices

Regularly review and update alerting thresholds
Set up proper retention policies for metrics and logs
Implement proper access controls for monitoring data
Regularly backup your monitoring configuration

Observability