Events missing while the connector was running

Use this when someone reports that a change made in Zwing never reached a topic, a report or an integration, and the connector was RUNNING the whole time with no failed task.

Most reports of missing events turn out to be a delay, a table that was never captured, or a consumer problem. Steps 1 to 3 separate those from real loss in the connector. All commands are read-only; the repair in step 7 is not.

Step 0: get one concrete example

Do not start without these. One specific row is enough.

• Tenant database and table.
• Primary key of the row.
• When the change was made, in IST and UTC.
• Which connector and which consumer should have received it.
• What was expected and what is there instead.

Step 1: is the connector really streaming, and how far has it read

Check the task state with step 1 of runbook A. Then read the connector's committed offset:

kubectl -n prod-kafka exec kafka-prod-connect-connect-0 -- \
  curl -s "http://localhost:8083/connectors/<CONNECTOR>/offsets" \
  | jq '.offsets[] | {file: .offset.file, pos: .offset.pos, last_event_utc: (.offset.ts_sec | todate)}'

• last_event_utc is earlier than the change in your example: the connector has not reached it yet. This is a delay, not a loss. Check whether it is still replaying schema history or catching up (runbook A, step 7).
• last_event_utc is later than the change: the connector has read past it. Continue.
• The call returns an error: the Connect version does not offer this endpoint. Use the resume lines from runbook A, step 7, or the latest message time on the topic in Kafka UI.

Step 2: is the table captured at all

kubectl -n prod-kafka exec kafka-prod-connect-connect-0 -- \
  curl -s "http://localhost:8083/connectors/<CONNECTOR>/config" \
  | jq -r '.["database.include.list"], .["table.include.list"]' | tr ',' '\n' | grep -i "<DATABASE-OR-TABLE>"

• No output: the database or table is not in the connector's include lists, so its changes were never captured. This is the usual cause for a newly onboarded tenant or a new table. Go to step 7.
• The lists can contain patterns, so if the name is not there literally, read the list for a pattern that covers it.

Step 3: did the event reach Kafka

The topic name is <topic-prefix>.<database>.<table>. In Kafka UI, open that topic, set the time filter to a window around the change, and search for the primary key.

• The event is in the topic: the connector did its job. The problem is on the consumer side. Hand it to the consumer's owner to check consumer lag, error logs, any dead-letter topic and any filtering.
• The event is not in the topic: continue.

Step 4: look for signs that the connector skipped something

Search the Connect logs in Loki for the hour around the change:

{namespace="prod-kafka", pod=~"kafka-prod-connect-connect-.*"} |~ "Trying to restore lost connection|No previous offsets found|schema isn't known to this connector|Error while processing event|Stopped reading binlog|Connected to zwingproddb"

For the last day or so, the pods themselves are enough:

for p in 0 1; do
  echo "== connect-$p"
  kubectl -n prod-kafka logs kafka-prod-connect-connect-$p --since=24h \
    | grep -E "Trying to restore lost connection|No previous offsets found|schema isn't known to this connector|Error while processing event" \
    | cut -c1-200 | tail -20
done

Then check whether the connector is configured to skip things:

kubectl -n prod-kafka exec kafka-prod-connect-connect-0 -- \
  curl -s "http://localhost:8083/connectors/<CONNECTOR>/config" \
  | jq '{snapshot_mode: .["snapshot.mode"], failure_mode: .["event.processing.failure.handling.mode"], schema_mismatch_mode: .["inconsistent.schema.handling.mode"], skipped_operations: .["skipped.operations"], transforms: .transforms, keepalive: .["connect.keep.alive"], signal_table: .["signal.data.collection"]}'

• failure_mode or schema_mismatch_mode set to warn or skip: the connector drops events it cannot process instead of failing. null means the default, which is to fail.
• skipped_operations: operation types that are never emitted.
• transforms: any filter in this chain can drop events by design.

Step 5: check what else happened at that time

• Pod restarts and memory kills: runbook A, step 8. A restart by itself only causes duplicates.
• Database restart or maintenance: runbook A, step 3.
• A connector that was deleted and recreated, or renamed, loses its stored offsets. That shows up as No previous offsets found in step 4.

Step 6: was the change written to the binlog

If steps 1 to 5 explain nothing, confirm the server is logging changes the way Debezium needs:

for p in binlog_format binlog_row_image gtid_mode; do
  az mysql flexible-server parameter show \
    --subscription c9599934-9aad-4b75-958c-3a9b48faf5d5 \
    --resource-group rg-zwing-g1-common-prod --server-name zwingproddb \
    --name $p --query "{name:name, value:value}" -o tsv
done

binlog_format must be ROW and binlog_row_image should be FULL. If both are right, ask Platform to read the binlog around the change time with database access. That is the only way to prove whether the change was in the binlog.

Step 7: scope the gap and repair it (this changes state)

Agree the repair with the connector owner and Platform before doing anything.

1. Write down the scope: which connector, which tables, and the time window from the first to the last missing change.
2. Pick the repair:
   • Table not captured (step 2): add it to the include list, then backfill it.
   • Rows skipped by the connector (step 4): backfill the affected tables for the window.
3. Backfill options:
   • An incremental snapshot of the affected tables through Debezium's signal mechanism. It re-emits the current rows, which repairs missing inserts and updates. It does not re-emit deletes. It needs a signal table or signal topic configured (signal_table in step 4).
   • A resync on the consumer side, reading the affected rows from the source for the window. For reporting this is the reporting team's backfill.
4. Do not rewind or edit connector offsets to replay a window unless Platform has agreed it. It replays every table the connector captures.
5. After the repair, recheck the example from step 0.

Step 8: record it and prevent it

• Add the cause and the scope to an incident note.
• Keepalive reconnect: this is PLT-10 in the 2 October 2026 CDC outage. Bring forward connect.keep.alive=false and the upgrade.
• Table not captured: add the include-list change to the tenant and table onboarding checklist.
• Skip modes in the config: decide with the owner whether the connector should fail instead of skipping.

Related

• Runbook A: connector task failed
• 2 October 2026 CDC outage