Ir al contenido
1337skills 1337skills - Hojas de Trucos

incident.io Cheat Sheet

Overview

incident.io is a modern incident management platform designed to streamline the entire incident lifecycle from declaration through resolution and post-mortem. It operates natively within Slack, allowing teams to declare incidents, assign roles, track actions, and communicate status updates without leaving their collaboration tool. The platform automates many tedious aspects of incident response like creating channels, notifying stakeholders, and generating timelines.

Beyond real-time response, incident.io provides powerful analytics, catalog management for services, and on-call scheduling capabilities. Its catalog feature lets organizations model their technical estate—services, teams, features, and dependencies—providing rich context during incidents. The platform’s post-incident workflows automatically generate incident reviews, track follow-up actions, and surface trends across incidents to drive systemic improvements.

Installation

Slack Integration Setup

# incident.io is primarily a SaaS platform
# Installation involves connecting to your Slack workspace

# 1. Sign up at https://incident.io
# 2. Install the Slack app via OAuth flow
# 3. Configure the bot in your Slack workspace

# The Slack bot requires these permissions:
# - channels:manage (create incident channels)
# - chat:write (post updates)
# - users:read (identify responders)
# - commands (slash commands)

Terraform Provider

# Install the incident.io Terraform provider
terraform {
  required_providers {
    incident = {
      source  = "incident-io/incident"
      version = "~> 3.0"
    }
  }
}

provider "incident" {
  api_key = var.incident_io_api_key
}

CLI / API Setup

# Set API key for CLI usage
export INCIDENT_IO_API_KEY="your-api-key"

# Base URL for all API calls
export INCIDENT_API="https://api.incident.io/v2"

# Verify connectivity
curl -s -H "Authorization: Bearer $INCIDENT_IO_API_KEY" \
  "$INCIDENT_API/incidents" | jq '.incidents | length'

Core Commands — Incident Management

Declaring Incidents via Slack

# Declare an incident from Slack
/incident new

# Quick declare with details
/incident new "Payment processing failures" --severity critical

# Declare with initial summary
/incident new "Elevated error rates on checkout service" \
  --severity major \
  --type "service_outage"

Managing Incidents via API

# Create an incident
curl -X POST "$INCIDENT_API/incidents" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "idempotency_key": "unique-key-123",
    "visibility": "public",
    "incident_type_id": "incident-type-id",
    "severity_id": "severity-id",
    "name": "Database connection pool exhausted",
    "summary": "Production database connection pool at 100%",
    "mode": "real",
    "incident_role_assignments": [
      {
        "incident_role_id": "lead-role-id",
        "assignee": {"email": "oncall@company.com"}
      }
    ]
  }'

# List active incidents
curl -s "$INCIDENT_API/incidents?status_category[one_of]=active" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY" | jq '.incidents[] | {id, name, severity}'

# Update incident status
curl -X POST "$INCIDENT_API/incidents/INCIDENT_ID/actions/update" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "incident": {
      "summary": "Identified root cause — deploying fix",
      "severity_id": "new-severity-id"
    }
  }'

# Close / resolve an incident
curl -X POST "$INCIDENT_API/incidents/INCIDENT_ID/actions/close" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"postmortem_required": true}'

Incident Roles and Assignments

# List available incident roles
curl -s "$INCIDENT_API/incident_roles" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY" | jq '.incident_roles[] | {id, name}'

# Assign a role during an incident
curl -X POST "$INCIDENT_API/incidents/INCIDENT_ID/actions/update" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "incident": {
      "incident_role_assignments": [
        {
          "incident_role_id": "comms-lead-id",
          "assignee": {"email": "comms@company.com"}
        }
      ]
    }
  }'

# Get incident timeline
curl -s "$INCIDENT_API/incidents/INCIDENT_ID/events" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY" | jq '.incident_events[] | {type: .action, at: .occurred_at}'

Core Commands — Catalog Management

Managing the Service Catalog

# List catalog types
curl -s "$INCIDENT_API/catalog/types" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY" | jq '.catalog_types[] | {id, name}'

# Create a catalog entry (e.g., a service)
curl -X POST "$INCIDENT_API/catalog/entries" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "catalog_type_id": "service-type-id",
    "name": "checkout-service",
    "attribute_values": {
      "team": {"value": {"catalog_entry_id": "team-id"}},
      "tier": {"value": {"literal": "tier-1"}},
      "slack_channel": {"value": {"literal": "#checkout-service"}}
    }
  }'

# Search catalog entries
curl -s "$INCIDENT_API/catalog/entries?catalog_type_id=service-type-id&name_contains=checkout" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY"

# Update a catalog entry
curl -X PUT "$INCIDENT_API/catalog/entries/ENTRY_ID" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "checkout-service",
    "attribute_values": {
      "tier": {"value": {"literal": "tier-0"}}
    }
  }'

Configuration

Terraform Configuration

# Define severities
resource "incident_severity" "critical" {
  name        = "Critical"
  description = "Complete service outage affecting all users"
  rank        = 1
}

resource "incident_severity" "major" {
  name        = "Major"
  description = "Significant degradation affecting many users"
  rank        = 2
}

# Define incident roles
resource "incident_incident_role" "lead" {
  name         = "Incident Lead"
  description  = "Coordinates the incident response"
  required     = true
  shortform    = "lead"
  instructions = "You are responsible for coordinating the response"
}

resource "incident_incident_role" "comms" {
  name         = "Communications Lead"
  description  = "Manages stakeholder communication"
  required     = false
  shortform    = "comms"
  instructions = "Post regular updates to #incidents-public"
}

# Define custom fields
resource "incident_custom_field" "affected_customers" {
  name        = "Affected Customers"
  description = "Estimated number of affected customers"
  field_type  = "numeric"
  required    = "before_closure"
}

# Define workflows
resource "incident_workflow" "auto_create_channel" {
  name           = "Auto-create incident channel"
  trigger        = "incident.created"
  condition_type = "all"

  steps {
    action = "slack.create_channel"
    params = {
      channel_name_template = "inc-{{ .incident.reference }}"
    }
  }
}

Alert Routing Configuration

{
  "alert_routes": [
    {
      "name": "PagerDuty critical to incident",
      "source": "pagerduty",
      "conditions": [
        {"field": "severity", "operator": "equals", "value": "critical"}
      ],
      "escalation_path": "critical-response",
      "auto_create_incident": true,
      "incident_template": {
        "severity": "critical",
        "type": "service_outage"
      }
    },
    {
      "name": "Datadog high priority",
      "source": "datadog",
      "conditions": [
        {"field": "priority", "operator": "in", "value": ["P1", "P2"]}
      ],
      "escalation_path": "platform-team-oncall",
      "auto_create_incident": false
    }
  ]
}

Advanced Usage

Workflows and Automation

# List all workflows
curl -s "$INCIDENT_API/workflows" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY" | jq '.workflows[] | {id, name, trigger}'

# Create an automated workflow
curl -X POST "$INCIDENT_API/workflows" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Notify executives on critical incidents",
    "trigger": {
      "type": "incident.updated",
      "condition": {
        "severity_id": {"one_of": ["critical-severity-id"]}
      }
    },
    "steps": [
      {
        "action": "slack.send_message",
        "params": {
          "channel": "#exec-incidents",
          "message_template": "🔴 Critical incident: {{ .incident.name }}\nLead: {{ .incident.lead.name }}"
        }
      }
    ]
  }'

Post-Incident Reviews

# List post-incident reviews
curl -s "$INCIDENT_API/post_incident_tasks?incident_id=INCIDENT_ID" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY"

# Create a follow-up action
curl -X POST "$INCIDENT_API/actions" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "incident_id": "INCIDENT_ID",
    "description": "Add circuit breaker to payment service",
    "assignee": {"email": "engineer@company.com"},
    "status": "outstanding"
  }'

# Export incident data for analysis
curl -s "$INCIDENT_API/incidents?created_at[gte]=2026-01-01&page_size=100" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY" \
  | jq '[.incidents[] | {name, severity: .severity.name, duration_seconds: .duration, created: .created_at}]'

On-Call Schedules

# List on-call schedules
curl -s "$INCIDENT_API/schedules" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY" | jq '.schedules[] | {id, name}'

# Get current on-call for a schedule
curl -s "$INCIDENT_API/schedules/SCHEDULE_ID/entries?entry_time=now" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY"

# Create an on-call override
curl -X POST "$INCIDENT_API/schedule_overrides" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "schedule_id": "SCHEDULE_ID",
    "user_id": "override-user-id",
    "start_at": "2026-05-20T09:00:00Z",
    "end_at": "2026-05-21T09:00:00Z"
  }'

Troubleshooting

IssueCauseSolution
Slack bot not respondingOAuth token expiredReinstall the Slack app from incident.io settings
Incident channel not createdMissing Slack permissionsEnsure bot has channels:manage and groups:write
Alerts not auto-creating incidentsRouting rules misconfiguredCheck alert route conditions and escalation paths
Catalog sync failingAPI rate limits exceededReduce sync frequency or use batch endpoints
Terraform plan shows driftManual UI changesImport existing resources: terraform import incident_severity.critical ID
Workflow not triggeringCondition mismatchVerify trigger conditions match incident field values
Timeline missing eventsSlack message permissionsGrant channels:history permission to the bot
API returning 403Insufficient API key scopeGenerate a new key with required scopes in Settings
# Debug: check API key permissions
curl -s "$INCIDENT_API/utilities/identity" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY" | jq '.'

# Debug: list recent webhook deliveries
curl -s "$INCIDENT_API/webhooks/deliveries?per_page=10" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY"

# Verify Slack connectivity
curl -s "$INCIDENT_API/integrations/slack/status" \
  -H "Authorization: Bearer $INCIDENT_IO_API_KEY"