Book a Demo!
CoCalc Logo Icon
StoreFeaturesDocsShareSupportNewsAboutPoliciesSign UpSign In
gitpod-io
GitHub Repository: gitpod-io/gitpod
 Path: blob/main/operations/observability/mixins/meta/rules/public-api.yaml
2500 views
1
apiVersion: monitoring.coreos.com/v1

2
kind: PrometheusRule

3
metadata:

4
  labels:

5
    prometheus: k8s

6
    role: alert-rules

7
  name: public-api-monitoring-rules

8
spec:

9
  groups:

10
  - name: public-api

11
    rules:

12
    - alert: PublicAPI_NoMetrics

13
      expr: absent(up{job="public-api-server"}) == 1

14
      for: 15m

15
      labels:

16
        severity: warning

17
        team: webapp

18
      annotations:

19
        runbook_url: https://github.com/gitpod-io/runbooks/blob/main/runbooks/PublicAPI_NoMetrics.md

20
        summary: We have not been able to collect metrics from the Public API. This can indicate an issue with the instances, or with metrics collection. Investigation required.

21
        description: Metrics for Public API are not available. Either the public-api-server pods are down, or there is a problem with metric collection and we are flying blind. Investigate.

22


23
    - alert: PublicAPI_ServiceReturningServerErrors

24
      expr: sum(increase(connect_server_handled_seconds_count{code=~"unknown|internal|unavailable|data_loss"}[1m])) by (package, call) > 1

25
      for: 15m

26
      labels:

27
        severity: warning

28
        team: webapp

29
      annotations:

30
        runbook_url: https://github.com/gitpod-io/runbooks/blob/main/runbooks/PublicAPI_ServiceReturningServerErrors.md

31
        summary: PublicAPI serves multiple different Services and RPC. There have been failing requests due to server errors. Investigation required.

32
        description: Service {{ $labels.package }}.{{ $labels.call }} has returned {{ printf "%.2f" $value }} server errors in the last 10 minutes.

33


34
    - alert: GitpodStripeWebhookFailures

35
      expr: sum(increase(gitpod_http_request_duration_seconds_count{handler="/stripe/invoices/webhook", code=~"5.*"}[30m])) > 0

36
      for: 10m

37
      labels:

38
        severity: warning

39
        team: webapp

40
      annotations:

41
        runbook_url: https://github.com/gitpod-io/runbooks/blob/main/runbooks/GitpodUsageStripeWebhookFailures.md

42
        summary: Detected {{ printf "%.2f" $value }} errors handling Stripe webhook.

43
        description: Stripe is sending us webhooks but we are failing to handle them. Inconsistent usage data very likely.

44


45