{
  "schema": "https://ai-atoms.com/schemas/skill-v1.json",
  "type": "skill",
  "id": "skill/service-mesh-observability",
  "version": "1.0.0",
  "name": "Service Mesh Observability",
  "description": "Implement comprehensive observability for service meshes including distributed tracing, metrics, and visualization. Use when setting up mesh monitoring, debugging latency issues, or implementing SL...",
  "system_prompt_fragment": "# Service Mesh Observability\n\nComplete guide to observability patterns for Istio, Linkerd, and service mesh deployments.\n\n## Do not use this skill when\n\n- The task is unrelated to service mesh observability\n- You need a different domain or tool outside this scope\n\n## Instructions\n\n- Clarify goals, constraints, and required inputs.\n- Apply relevant best practices and validate outcomes.\n- Provide actionable steps and verification.\n- If detailed examples are required, open `resources/implementation-playbook.md`.\n\n## Use this skill when\n\n- Setting up distributed tracing across services\n- Implementing service mesh metrics and dashboards\n- Debugging latency and error issues\n- Defining SLOs for service communication\n- Visualizing service dependencies\n- Troubleshooting mesh connectivity\n\n## Core Concepts\n\n### 1. Three Pillars of Observability\n\n```\n┌─────────────────────────────────────────────────────┐\n│                  Observability                       │\n├─────────────────┬─────────────────┬─────────────────┤\n│     Metrics     │     Traces      │      Logs       │\n│                 │                 │                 │\n│ • Request rate  │ • Span context  │ • Access logs   │\n│ • Error rate    │ • Latency       │ • Error details │\n│ • Latency P50   │ • Dependencies  │ • Debug info    │\n│ • Saturation    │ • Bottlenecks   │ • Audit trail   │\n└─────────────────┴─────────────────┴─────────────────┘\n```\n\n### 2. Golden Signals for Mesh\n\n| Signal | Description | Alert Threshold |\n|--------|-------------|-----------------|\n| **Latency** | Request duration P50, P99 | P99 > 500ms |\n| **Traffic** | Requests per second | Anomaly detection |\n| **Errors** | 5xx error rate | > 1% |\n| **Saturation** | Resource utilization | > 80% |\n\n## Templates\n\n### Template 1: Istio with Prometheus & Grafana\n\n```yaml\n# Install Prometheus\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: prometheus\n  namespace: istio-system\ndata:\n  prometheus.yml: |\n    global:\n      scrape_interval: 15s\n    scrape_configs:\n      - job_name: 'istio-mesh'\n        kubernetes_sd_configs:\n          - role: endpoints\n            namespaces:\n              names:\n                - istio-system\n        relabel_configs:\n          - source_labels: [__meta_kubernetes_service_name]\n            action: keep\n            regex: istio-telemetry\n---\n# ServiceMonitor for Prometheus Operator\napiVersion: monitoring.coreos.com/v1\nkind: ServiceMonitor\nmetadata:\n  name: istio-mesh\n  namespace: istio-system\nspec:\n  selector:\n    matchLabels:\n      app: istiod\n  endpoints:\n    - port: http-monitoring\n      interval: 15s\n```\n\n### Template 2: Key Istio Metrics Queries\n\n```promql\n# Request rate by service\nsum(rate(istio_requests_total{reporter=\"destination\"}[5m])) by (destination_service_name)\n\n# Error rate (5xx)\nsum(rate(istio_requests_total{reporter=\"destination\", response_code=~\"5..\"}[5m]))\n  / sum(rate(istio_requests_total{reporter=\"destination\"}[5m])) * 100\n\n# P99 latency\nhistogram_quantile(0.99,\n  sum(rate(istio_request_duration_milliseconds_bucket{reporter=\"destination\"}[5m]))\n  by (le, destination_service_name))\n\n# TCP connections\nsum(istio_tcp_connections_opened_total{reporter=\"destination\"}) by (destination_service_name)\n\n# Request size\nhistogram_quantile(0.99,\n  sum(rate(istio_request_bytes_bucket{reporter=\"destination\"}[5m]))\n  by (le, destination_service_name))\n```\n\n### Template 3: Jaeger Distributed Tracing\n\n```yaml\n# Jaeger installation for Istio\napiVersion: install.istio.io/v1alpha1\nkind: IstioOperator\nspec:\n  meshConfig:\n    enableTracing: true\n    defaultConfig:\n      tracing:\n        sampling: 100.0  # 100% in dev, lower in prod\n        zipkin:\n          address: jaeger-collector.istio-system:9411\n---\n# Jaeger deployment\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: jaeger\n  namespace: istio-system\nspec:\n  selector:\n    matchLabels:\n      app: jaeger\n  template:\n    metadata:\n      labels:\n        app: jaeger\n    spec:\n      containers:\n        - name: jaeger\n          image: jaegertracing/all-in-one:1.50\n          ports:\n            - containerPort: 5775   # UDP\n            - containerPort: 6831   # Thrift\n            - containerPort: 6832   # Thrift\n            - containerPort: 5778   # Config\n            - containerPort: 16686  # UI\n            - containerPort: 14268  # HTTP\n            - containerPort: 14250  # gRPC\n            - containerPort: 9411   # Zipkin\n          env:\n            - name: COLLECTOR_ZIPKIN_HOST_PORT\n              value: \":9411\"\n```\n\n### Template 4: Linkerd Viz Dashboard\n\n```bash\n# Install Linkerd viz extension\nlinkerd viz install | kubectl apply -f -\n\n# Access dashboard\nlinkerd viz dashboard\n\n# CLI commands for observability\n# Top requests\nlinkerd viz top deploy/my-app\n\n# Per-route metrics\nlinkerd viz routes deploy/my-app --to deploy/backend\n\n# Live traffic inspection\nlinkerd viz tap deploy/my-app --to deploy/backend\n\n# Service edges (dependencies)\nlinkerd viz edges deployment -n my-namespace\n```\n\n### Template 5: Grafana Dashboard JSON\n\n```json\n{\n  \"dashboard\": {\n    \"title\": \"Service Mesh Overview\",\n    \"panels\": [\n      {\n        \"title\": \"Request Rate\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"sum(rate(istio_requests_total{reporter=\\\"destination\\\"}[5m])) by (destination_service_name)\",\n            \"legendFormat\": \"{{destination_service_name}}\"\n          }\n        ]\n      },\n      {\n        \"title\": \"Error Rate\",\n        \"type\": \"gauge\",\n        \"targets\": [\n          {\n            \"expr\": \"sum(rate(istio_requests_total{response_code=~\\\"5..\\\"}[5m])) / sum(rate(istio_requests_total[5m])) * 100\"\n          }\n        ],\n        \"fieldConfig\": {\n          \"defaults\": {\n            \"thresholds\": {\n              \"steps\": [\n                {\"value\": 0, \"color\": \"green\"},\n                {\"value\": 1, \"color\": \"yellow\"},\n                {\"value\": 5, \"color\": \"red\"}\n              ]\n            }\n          }\n        }\n      },\n      {\n        \"title\": \"P99 Latency\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"histogram_quantile(0.99, sum(rate(istio_request_duration_milliseconds_bucket{reporter=\\\"destination\\\"}[5m])) by (le, destination_service_name))\",\n            \"legendFormat\": \"{{destination_service_name}}\"\n          }\n        ]\n      },\n      {\n        \"title\": \"Service Topology\",\n        \"type\": \"nodeGraph\",\n        \"targets\": [\n          {\n            \"expr\": \"sum(rate(istio_requests_total{reporter=\\\"destination\\\"}[5m])) by (source_workload, destination_service_name)\"\n          }\n        ]\n      }\n    ]\n  }\n}\n```\n\n### Template 6: Kiali Service Mesh Visualization\n\n```yaml\n# Kiali installation\napiVersion: kiali.io/v1alpha1\nkind: Kiali\nmetadata:\n  name: kiali\n  namespace: istio-system\nspec:\n  auth:\n    strategy: anonymous  # or openid, token\n  deployment:\n    accessible_namespaces:\n      - \"**\"\n  external_services:\n    prometheus:\n      url: http://prometheus.istio-system:9090\n    tracing:\n      url: http://jaeger-query.istio-system:16686\n    grafana:\n      url: http://grafana.istio-system:3000\n```\n\n### Template 7: OpenTelemetry Integration\n\n```yaml\n# OpenTelemetry Collector for mesh\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: otel-collector-config\ndata:\n  config.yaml: |\n    receivers:\n      otlp:\n        protocols:\n          grpc:\n            endpoint: 0.0.0.0:4317\n          http:\n            endpoint: 0.0.0.0:4318\n      zipkin:\n        endpoint: 0.0.0.0:9411\n\n    processors:\n      batch:\n        timeout: 10s\n\n    exporters:\n      jaeger:\n        endpoint: jaeger-collector:14250\n        tls:\n          insecure: true\n      prometheus:\n        endpoint: 0.0.0.0:8889\n\n    service:\n      pipelines:\n        traces:\n          receivers: [otlp, zipkin]\n          processors: [batch]\n          exporters: [jaeger]\n        metrics:\n          receivers: [otlp]\n          processors: [batch]\n          exporters: [prometheus]\n---\n# Istio Telemetry v2 with OTel\napiVersion: telemetry.istio.io/v1alpha1\nkind: Telemetry\nmetadata:\n  name: mesh-default\n  namespace: istio-system\nspec:\n  tracing:\n    - providers:\n        - name: otel\n      randomSamplingPercentage: 10\n```\n\n## Alerting Rules\n\n```yaml\napiVersion: monitoring.coreos.com/v1\nkind: PrometheusRule\nmetadata:\n  name: mesh-alerts\n  namespace: istio-system\nspec:\n  groups:\n    - name: mesh.rules\n      rules:\n        - alert: HighErrorRate\n          expr: |\n            sum(rate(istio_requests_total{response_code=~\"5..\"}[5m])) by (destination_service_name)\n            / sum(rate(istio_requests_total[5m])) by (destination_service_name) > 0.05\n          for: 5m\n          labels:\n            severity: critical\n          annotations:\n            summary: \"High error rate for {{ $labels.destination_service_name }}\"\n\n        - alert: HighLatency\n          expr: |\n            histogram_quantile(0.99, sum(rate(istio_request_duration_milliseconds_bucket[5m]))\n            by (le, destination_service_name)) > 1000\n          for: 5m\n          labels:\n            severity: warning\n          annotations:\n            summary: \"High P99 latency for {{ $labels.destination_service_name }}\"\n\n        - alert: MeshCertExpiring\n          expr: |\n            (certmanager_certificate_expiration_timestamp_seconds - time()) / 86400 < 7\n          labels:\n            severity: warning\n          annotations:\n            summary: \"Mesh certificate expiring in less than 7 days\"\n```\n\n## Best Practices\n\n### Do's\n- **Sample appropriately** - 100% in dev, 1-10% in prod\n- **Use trace context** - Propagate headers consistently\n- **Set up alerts** - For golden signals\n- **Correlate metrics/traces** - Use exemplars\n- **Retain strategically** - Hot/cold storage tiers\n\n### Don'ts\n- **Don't over-sample** - Storage costs add up\n- **Don't ignore cardinality** - Limit label values\n- **Don't skip dashboards** - Visualize dependencies\n- **Don't forget costs** - Monitor observability costs\n\n## Resources\n\n- [Istio Observability](https://istio.io/latest/docs/tasks/observability/)\n- [Linkerd Observability](https://linkerd.io/2.14/features/dashboard/)\n- [OpenTelemetry](https://opentelemetry.io/)\n- [Kiali](https://kiali.io/)",
  "applicable_domains": [
    "other"
  ],
  "category": "other",
  "invocation": [
    "/service-mesh-observability"
  ],
  "authored_by": "claudeskills.in community",
  "source_url": "https://claudeskills.in/skill/service-mesh-observability",
  "provenance": {
    "source": "claudeskills.in",
    "source_url": "https://claudeskills.in/skill/service-mesh-observability",
    "license": "unknown",
    "imported_at": "2026-09-03",
    "notes": "Aggregated by claudeskills.in from community GitHub lists."
  },
  "tags": [
    "claudeskills",
    "other"
  ],
  "lifecycle": "draft"
}