Connector task failed

Use this when change data stops flowing: reports are stale, integrations are delayed, or the Connect log shows "This connector will be stopped". If the connector is up and some events are missing, use runbook B.

Things to know before you start:

• Connect and Debezium logs are in UTC. IST is UTC plus 5 hours 30 minutes.
• A connector can show RUNNING while its task is FAILED or still loading. Always check the task.
• blc- in a log line is the binlog client's thread name, not part of the database hostname.
• A negative number in "Failed to read next byte from position" is a counter overflow and means nothing.
• All commands are read-only except step 6.

Step 1: check task state

kubectl -n prod-kafka exec kafka-prod-connect-connect-0 -- \
  curl -s "http://localhost:8083/connectors?expand=status" \
  | jq -r 'to_entries[] | .value.status as $s | $s.tasks[]? | [$s.name, .state, .worker_id] | @tsv'

If the exec fails, use kafka-prod-connect-connect-1 instead.

Step 2: read the failure reason

kubectl -n prod-kafka exec kafka-prod-connect-connect-0 -- \
  curl -s "http://localhost:8083/connectors/<CONNECTOR>/status" \
  | jq -r '.tasks[] | .trace // "no trace"' | head -20

Step 3: find out whether the database restarted

Take the failure time from the log and use a window around it.

Maintenance history (the most direct answer):

az mysql flexible-server maintenance list \
  --subscription c9599934-9aad-4b75-958c-3a9b48faf5d5 \
  --resource-group rg-zwing-g1-common-prod \
  --server-name zwingproddb -o json

Server metrics (shows a restart whether or not Azure recorded one):

az monitor metrics list \
  --resource "/subscriptions/c9599934-9aad-4b75-958c-3a9b48faf5d5/resourceGroups/rg-zwing-g1-common-prod/providers/Microsoft.DBforMySQL/flexibleServers/zwingproddb" \
  --metric memory_percent cpu_percent active_connections aborted_connections \
  --start-time <UTC-START> --end-time <UTC-END> \
  --interval PT1M --aggregation Maximum Total -o table

Reading the metrics:

• Memory drops to near zero and there are blank rows: the server restarted.
• Memory near 100% just before: the server ran out of memory.
• Memory flat and connections dip: a network cut, not a restart.

Activity log (shows a restart, scale or parameter change made by a person or automation):

az monitor activity-log list \
  --subscription c9599934-9aad-4b75-958c-3a9b48faf5d5 \
  --resource-id "/subscriptions/c9599934-9aad-4b75-958c-3a9b48faf5d5/resourceGroups/rg-zwing-g1-common-prod/providers/Microsoft.DBforMySQL/flexibleServers/zwingproddb" \
  --start-time <UTC-START> --end-time <UTC-END> \
  --query "[].{time:eventTimestamp, op:operationName.localizedValue, status:status.value, caller:caller}" -o json

Resource Health did not record the 2 October maintenance, so an empty Resource Health history does not rule out a restart.

Step 4: confirm the database is back

az mysql flexible-server show \
  --subscription c9599934-9aad-4b75-958c-3a9b48faf5d5 \
  --resource-group rg-zwing-g1-common-prod --name zwingproddb \
  --query "{state:state}" -o json

The state should be Ready, and the applications should be healthy.

Step 5: check that a restart is safe

Binlog retention. The failure must be more recent than this many seconds, otherwise the position may be gone:

az mysql flexible-server parameter show \
  --subscription c9599934-9aad-4b75-958c-3a9b48faf5d5 \
  --resource-group rg-zwing-g1-common-prod --server-name zwingproddb \
  --name binlog_expire_logs_seconds --query "{value:value}" -o json

Pod memory headroom:

kubectl -n prod-kafka top pod | grep -E "NAME|connect"
kubectl -n prod-kafka get kafkaconnect kafka-prod-connect -o json | jq '.spec.resources'

If a pod is within a few hundred MiB of its limit, raise the limits before restarting anything. Restarting a connector loads its whole schema history into memory.

Step 6: restart one connector at a time (this changes state)

kubectl -n prod-kafka exec kafka-prod-connect-connect-0 -- \
  curl -s -o /dev/null -w "%{http_code}\n" -X POST \
  "http://localhost:8083/connectors/<CONNECTOR>/restart?includeTasks=true&onlyFailed=true"

It should print 202 or 204.

• Restart one connector, complete step 7 for it, then do the next.
• Go from the smallest schema history to the largest: item master, GIP, outbound, then reporting last.
• Before restarting the outbound connector, check with its owner that downstream systems can take the backlog in a burst.
• Do not restart a connector that is still replaying its schema history. It will start again from zero.

Step 7: verify it really resumed

Find which pod the task is on (step 1), then:

kubectl -n prod-kafka logs <POD> --since=30m \
  | grep -E "Found previous offset|INFO: Connected to zwingproddb|ERROR" \
  | sed -E 's/.*(restartBinlogFilename=[^,]+, restartBinlogPosition=[0-9]+).*(\[task-thread-[^]]+\]).*/\2 \1/'

You need both of these for the connector:

• An offset line with the binlog file and position it resumed from.
• A "Connected to zwingproddb" line with the same position.

If there is an offset line but no connection line, the connector is still loading its schema history. Watch its progress using the connector's topic prefix:

kubectl -n prod-kafka logs <POD> --since=15m \
  | grep "<TOPIC-PREFIX>" \
  | grep -E "Finished database schema history|recovery in progress|Starting streaming|ERROR" \
  | cut -c1-170 | tail -3

"Starting streaming" means it is connected. A record count that barely moves means the pod is short of memory or CPU; go to step 8.

Step 8: check for memory kills

kubectl -n prod-kafka get pods | grep -E "NAME|connect-connect"

kubectl -n prod-kafka get pods kafka-prod-connect-connect-0 kafka-prod-connect-connect-1 -o json \
  | jq '.items[] | {pod: .metadata.name, lastTermination: (.status.containerStatuses[0].lastState.terminated | {reason, exitCode, finishedAt})}'

A restart count that has gone up, with reason OOMKilled, means the pod cannot hold its connectors. Stop restarting tasks and escalate to Platform to raise the limits through the Helm values.

Step 9: confirm downstream and close

• The connector owner confirms consumers have caught up and data is current.
• Record the start time, end time, cause and actions in an incident note.

Related

• Runbook B: events missing while the connector was running
• 2 October 2026 CDC outage