Zwing CDC outage after Azure MySQL maintenance
Draft incident report for 2 October 2026. Azure maintenance restarted zwingproddb and the Debezium MySQL connectors stayed failed.
Draft, recovery in progress
State as of 13:42 IST on 2 October 2026. GIP, item master, and outbound have recovered. Reporting recovery is still in progress.
| Date of incident | 2 October 2026, 04:31 IST (1 October, 23:01 UTC) |
| Report status | Draft. State is as of 13:42 IST on 2 October. |
| Incident status | GIP, item master and outbound: recovered. Reporting: recovery in progress. |
| Prepared by | Rishabh |
| Owners | Mukesh (GIP), Chetan (Reports and item master), Chandramani (Platform) |
| Severity | To be assigned |
1. Summary
At 04:31 IST on 2 October, Azure ran a routine maintenance on the Zwing production MySQL server zwingproddb and restarted it. The database was unavailable for about five minutes and the applications recovered without help.
The four Debezium MySQL connectors did not recover. Their tasks went to FAILED and stayed there for eight to nine hours, because the Debezium version we run has a bug that treats every connection error as permanent. No alert fired. When the connectors were restarted by hand, the Kafka Connect pods turned out to be too small to reload them: one pod was killed for memory, and the reporting connector was still reloading its schema history more than 20 minutes later.
No data was lost. Every connector resumed from the binlog file that was live when the maintenance hit, and the server keeps binlogs for seven days. Change events were delayed, not dropped. A separate check for silent row loss while the connectors were running also came back clean (see Other findings).
2. Impact
| Service | Owner | Effect | Duration |
|---|---|---|---|
Zwing applications (measured on zwing-adminconsole) | Platform | HTTP 500 | 5 min 1 s (04:31:44 to 04:36:45 IST) |
GIP_ZWING_CONNECTOR_4 | Mukesh | No change events to GIP | About 8 h (04:31 to about 12:40 IST) |
ZWING_CONNECT_ITEM_MASTER_CONNECTOR_21 | Chetan | No change events | About 8 h 50 min (04:31 to about 13:23 IST) |
ZWING_CONNECT_OUTBOUND_CONNECTOR_22 | To be confirmed | No change events | About 8 h 50 min (04:31 to about 13:23 IST) |
ZWING_REPORTING_CONNECTOR_20 | Chetan | Reporting data stale | More than 9 h 10 min and still open at 13:42 IST |
Not affected: GIP_CRM_CONNECTOR_2 (MongoDB). ZWING_CONNECT_OFFLINE_CONNECTOR_1 was already in STOPPED state; whether that is intentional needs confirming.
Business impact (tenants, integrations and reports affected, tickets raised) is not yet recorded. Each owner should add it for their service.
3. Timeline
| IST (2 Oct) | UTC | Event |
|---|---|---|
| 04:30:00 | 1 Oct 23:00:00 | Azure maintenance window DH1Q-4GG opens (type RoutineMaintenance, system-managed schedule) |
| 04:30:58 | 23:00:58 | Azure starts executing the maintenance |
| 04:31:33 | 23:01:33 | MySQL is stopped. All Debezium binlog streams receive end-of-stream. Tasks for reporting, item master and outbound fail on connect-0; GIP_ZWING_CONNECTOR_4 stops at the same binlog file |
| 04:31:44 | 23:01:44 | Hyperping: zwing-adminconsole DOWN (HTTP 500) |
| 04:32 | 23:02 | No server metrics emitted |
| 04:33 to 04:35 | 23:03 to 23:05 | New MySQL process starting (memory 0.01% to 0.3%) |
| 04:36:45 | 23:06:45 | Hyperping: zwing-adminconsole UP. Applications reconnect without help |
| 04:38:01 | 23:08:01 | Azure marks the maintenance complete |
| 04:38 to about 12:30 | Four connector tasks remain FAILED. No alert | |
| About 12:30 | About 07:00 | Failure noticed and investigation starts (how it was noticed: to be added) |
| About 12:40 | About 07:10 | GIP_ZWING_CONNECTOR_4 restarted by hand; resumes from mysql-bin.002631 / 36931606 |
| 13:22 | 07:52 | Reporting, item master and outbound restarted by hand, together |
| 13:22:21 | 07:52:21 | connect-0 container killed for memory (OOMKilled, restart 128). Tasks rebalance to connect-1 |
| 13:22:42 to 13:23:26 | 07:52:42 to 07:53:26 | Outbound task stopped and started four times during the rebalance; the last attempt connects at mysql-bin.002631 / 25641031 |
| About 13:23 | About 07:53 | Item master connects at mysql-bin.002631 / 30244013. Reporting starts replaying its schema history (resume point mysql-bin.002631 / 38094028) |
| 13:24:25 | 07:54:25 | GIP_ZWING_CONNECTOR_4 reconnects on connect-0 after the rebalance |
| About 13:35 | About 08:05 | connect-1 at 1,496 MiB of a 1,536 MiB limit and 501m of a 500m CPU limit |
| 13:42 | 08:12 | Reporting has replayed 140,516 schema records and is still replaying. Last known state |
4. Root cause
Trigger
Azure routine maintenance restarted zwingproddb. The server has no high availability, so a restart is a full outage, and the maintenance window is system-managed, so Azure chose the time. No advance notice was found.
Why the connectors stayed down
Debezium 3.2.0.Final has a bug in the MySQL connector's error handling: every error is classified as non-retriable, so a dropped binlog connection stops the task permanently instead of triggering a reconnect. Our connectors use the default errors.max.retries (unlimited retries), but the bug bypasses it. The bug is tracked as debezium/dbz#2611. It was introduced in 3.1.0.CR1 and is fixed in two releases: 3.6.3.Final (18 September 2026) and 3.7.0.Final (29 September 2026). No release in the 3.2 to 3.5 series has the fix. The log line "This connector will be stopped" is its signature.
Until this is addressed, every MySQL connector on this Connect cluster will stop at every database restart.
Why nobody noticed for eight hours
There is no alert on failed connector tasks. The Connect REST API reported each connector as RUNNING with its task FAILED, which is easy to misread.
Why recovery was slow and fragile
- The Connect pods are undersized: 1,536 MiB memory and 0.5 CPU each, with no JVM heap settings. Restarting three connectors together pushed
connect-0over its limit. The pods have restarted 128 and 135 times in 100 and 92 days, and the last termination of both wasOOMKilled. - Both Connect replicas run on the same node (
aks-system-25971204-vmss000001, system pool, 81% memory used). - The reporting connector stores schema for every table on the server (
schema.history.internal.store.only.captured.tables.ddlisfalse). Its history is over 140,000 records. The replay rate fell from about 240 records a second to about 11 a second as the pod approached its memory and CPU limits.
Other findings
GIP_ZWING_CONNECTOR_4andZWING_CONNECT_OUTBOUND_CONNECTOR_22share the topic prefixprod. Debezium expects a unique prefix per connector. If their table lists overlap, both publish the same table to the same topic.- Azure Resource Health did not record the outage and the activity log was empty. Only the maintenance history and the server metrics showed it.
- Separate from this incident, the Debezium version we run has a known bug that can silently drop rows (debezium/dbz#2359). When a connector is blocked by backpressure and its binlog connection drops in the middle of a transaction, the keepalive thread reconnects from inside that transaction and the remaining rows of the interrupted statement are skipped with no error. It applies to servers without GTID and connectors with keepalive reconnect enabled. Both are confirmed for our setup:
gtid_modeis OFF and all four MySQL connectors use the default keepalive, which is on. The undersized Connect pods make the backpressure condition more likely. It did not cause this incident: here the tasks failed and resumed from their stored offsets, which is the safe path. We also checked for it directly. The reconnect logs the line "Trying to restore lost connection", and that line appears zero times in the current and previous container logs of both Connect pods, which together span the maintenance, the failed period and the recovery on 2 October. Pod logs only hold recent history, so the weeks before the incident still need a Loki search. See PLT-10 under Chandramani (Platform) and runbook B.
Ruled out
- Database overload: CPU was at 2% before the restart.
- Database memory exhaustion: memory was flat at 75.59% until the restart.
- A restart by a person or automation: the activity log has no operation in the window.
- A network fault between AKS and the database: the server's own metrics show the restart.
- Duplicate server IDs: the four connectors use 10000003, 20000003, 30000003 and 40000003.
- Silent row loss from a keepalive reconnect during the incident or recovery: zero occurrences of the reconnect in the current and previous logs of both Connect pods.
5. Environment reference
Database
| Item | Value |
|---|---|
| Server | zwingproddb (Azure Database for MySQL flexible server) |
| Subscription | G1-Production, c9599934-9aad-4b75-958c-3a9b48faf5d5 |
| Resource group | rg-zwing-g1-common-prod |
| Region | Jio India West |
| SKU | Standard_D8ds_v4 |
| High availability | Disabled |
| Maintenance window | System-managed (no custom window) |
| Binlog retention | 604,800 seconds (7 days) |
Kafka Connect
| Item | Value |
|---|---|
| Resource | KafkaConnect/kafka-prod-connect, namespace prod-kafka, 2 replicas |
| Managed by | Helm chart kafka-0.1.0, ArgoCD application kafka-prod |
| Limits | 500m CPU, 1,536 MiB memory (requests 200m, 512 MiB) |
| JVM options | None set |
| Debezium | 3.2.0.Final |
Connectors on zwingproddb
| Connector | Owner | Topic prefix | Server ID | Only captured tables in history | Databases / tables |
|---|---|---|---|---|---|
GIP_ZWING_CONNECTOR_4 | Mukesh | prod | 40000003 | Yes | 38 / 201 |
ZWING_REPORTING_CONNECTOR_20 | Chetan | reporting_3 | 30000003 | No | 53 / 2,173 |
ZWING_CONNECT_ITEM_MASTER_CONNECTOR_21 | Chetan | item_master | 20000003 | Yes | 53 / 53 |
ZWING_CONNECT_OUTBOUND_CONNECTOR_22 | To be confirmed | prod | 10000003 | Yes | 48 / 672 |
6. Actions
"Now" means today. "This week" items are the prevention work. "Planned" items need design or a maintenance slot.
Chandramani (Platform)
| ID | When | Action |
|---|---|---|
| PLT-1 | Now | Raise the Connect pod resources through the Helm values for ArgoCD app kafka-prod. Starting point: memory request and limit 4Gi, CPU request 1 and limit 2, jvmOptions -Xms 2g and -Xmx 3g. This is what unblocks the reporting connector. The change rolls both pods and every connector replays its history once |
| PLT-2 | Now | After the roll, confirm all five tasks are streaming (runbook A, step 7) and record kubectl top pod for both pods |
| PLT-3 | This week | Alert on any connector task in FAILED state and on Connect container restarts with reason OOMKilled. Route to the connector owner and the platform channel |
| PLT-4 | This week | Decide whether to bridge the gap until the upgrade with the Debezium workaround on the four MySQL connectors: the custom retriable exception setting, with a pattern that matches the binlog disconnect message, for example .*Failed to read next byte from position.*. The 3.7 release notes describe this as an internal property, so the real key is probably internal.custom.retriable.exception and it is not a supported setting. Prove the exact key and behaviour in staging by killing a binlog connection on the database before using it in production. Applying it restarts each connector, so do it after PLT-1. If PLT-6 is in place first, the workaround may not be needed |
| PLT-5 | This week | Plan the upgrade from Debezium 3.2.0.Final to a release with the fix for dbz#2611: 3.6.3.Final or 3.7.0.Final. Suggested target: the 3.7 line, because it is the only one that also fixes the silent row loss in PLT-10. With PLT-6 and PLT-10 in place as stop-gaps there is no rush, so it is reasonable to wait for a 3.7 patch release; 3.6.3.Final is an acceptable interim step only if connect.keep.alive=false stays set. Points to weigh: (1) in the 3.6 series the only MySQL breaking change is the removal of the never snapshot mode, which none of our connectors use; (2) 3.7 additionally changes the MySQL connector's integer type mapping, which can alter event schemas for consumers such as GIP and reporting; (3) 3.7 also has a schema history memory optimisation and a fix for a MySQL bug that could silently lose rows after a keepalive reconnect under backpressure, neither of which is listed for 3.6.3; (4) 3.6 already adds memory management for table history and schemas and a fix for a schema history recovery race. Before upgrading: read the breaking changes for 3.3, 3.4 and 3.5, which have not been reviewed; confirm our Kafka Connect version against Debezium's tested range (3.1 and later for both series); run it in staging with all four connectors and their consumers |
| PLT-6 | This week | Add an automatic restart for failed tasks, for example a CronJob that calls restart?includeTasks=true&onlyFailed=true for each connector. Do this only after PLT-1; on the current limits it can cause a memory-kill loop. Strimzi's autoRestart does not apply because these connectors are created through the REST API |
| PLT-7 | This week | Set a custom maintenance window on zwingproddb, and create a Service Health alert for planned maintenance on G1-Production that goes to the platform team. Check first whether any such alert already exists; this was not verified |
| PLT-8 | This week | Spread the two Connect replicas across nodes with pod anti-affinity and move them off the system node pool |
| PLT-9 | Planned | With Rishabh, decide on high availability for zwingproddb. Without it, every maintenance is about five minutes of application errors |
| PLT-10 | This week | Close the silent row-loss risk from dbz#2359. (1) Exposure is confirmed: gtid_mode is OFF, net_write_timeout is 240 seconds, and all four MySQL connectors run with the default keepalive (on, one-minute interval). The reconnect can fire once the reader has been blocked for longer than the keepalive interval. (2) The pod logs for 2 October show zero keepalive reconnects. Search Loki for "Trying to restore lost connection" over the full retention period to cover the weeks before; every hit is a reconnect to audit with the connector owner using runbook B. (3) Until the upgrade, set connect.keep.alive=false on the four connectors so a dropped connection fails the task and restarts from a transaction boundary, which gives duplicates and never loss. This depends on PLT-3 and PLT-6, because with keepalive off every drop needs a restart and a connection that dies without closing is no longer detected by the connector. (4) The permanent fix is Debezium 3.7.0 or later; enabling GTID on the server is an alternative that also makes the reconnect safe |
Mukesh (GIP)
| ID | When | Action |
|---|---|---|
| GIP-1 | Now | Confirm GIP_ZWING_CONNECTOR_4 is streaming and that GIP consumers have worked through the backlog from 04:31 to about 12:40 IST |
| GIP-2 | Now | Check the integrations for effects of the delayed and replayed events. The connector resumed slightly before its stop point, so a small number of events were delivered twice |
| GIP-3 | Now | Record the business impact for GIP in section 2 |
| GIP-4 | This week | Apply the retry workaround to the connector with Platform (PLT-4) |
| GIP-5 | This week | Compare the table list of GIP_ZWING_CONNECTOR_4 with ZWING_CONNECT_OUTBOUND_CONNECTOR_22. If any table appears in both, the two connectors are writing the same topic. Agree with the outbound owner which connector moves to a unique prefix; that renames its topics, so it is a planned change for consumers |
Chetan (Reports and item master)
| ID | When | Action |
|---|---|---|
| REP-1 | Now | Treat reporting data as stale since 04:31 IST and tell report users. ZWING_REPORTING_CONNECTOR_20 shows RUNNING but is still replaying schema history and has not streamed any data |
| REP-2 | Now | Do not restart the reporting connector again before PLT-1 is applied. Each restart begins the replay from zero |
| REP-3 | Now | Once it is streaming, confirm the reporting store has caught up, check the "last refreshed" values, and spot-check reports for the window from 04:31 IST |
| REP-4 | Now | Record the business impact for reporting in section 2 |
| REP-5 | This week | Apply the retry workaround to the reporting and item master connectors with Platform (PLT-4) |
| REP-6 | This week | Add a freshness alert on the reporting data itself, so staleness is detected even when the connector status looks healthy |
| REP-7 | Planned | Rebuild the connector's schema history with schema.history.internal.store.only.captured.tables.ddl set to true, on a new history topic. This cuts memory use and startup time to roughly what the other connectors have. Validate the procedure in staging with Platform first |
| REP-8 | Now | Confirm ZWING_CONNECT_ITEM_MASTER_CONNECTOR_21 is streaming and that its consumers have worked through the backlog from 04:31 to about 13:23 IST. Record the business impact for item master in section 2 |
Owner to be confirmed (outbound connector)
| ID | When | Action |
|---|---|---|
| ZC-1 | Now | Confirm ZWING_CONNECT_OUTBOUND_CONNECTOR_22 is streaming and downstream systems have processed the backlog. It replayed about nine hours of events in a burst |
| ZC-2 | This week | Apply the retry workaround to the connector with Platform (PLT-4) |
| ZC-3 | This week | Work with Mukesh on the shared prod prefix (GIP-5) |
Rishabh
| ID | When | Action |
|---|---|---|
| CTO-1 | Now | Name the owner of the outbound connector. Item master is with Chetan |
| CTO-2 | Now | Approve PLT-1 |
| CTO-3 | Planned | Decide on high availability for zwingproddb (PLT-9) |
7. Open items in this report
- Whether the reporting connector has finished its replay and is streaming.
- The total size of the reporting schema history topic
prod_reporting_3_schemahistory_v2, which would give a finish time at the current rate. - Whether any Service Health alert exists on G1-Production.
- Which Debezium release to upgrade to (3.6.3.Final or 3.7.0.Final), and the breaking changes in 3.3, 3.4 and 3.5, which have not been reviewed.
- The Loki search for keepalive reconnects before 2 October (PLT-10).
- The owner of
ZWING_CONNECT_OUTBOUND_CONNECTOR_22. - Business impact for each service.
- How and when the failure was first noticed.
8. Who to call
| Area | Owner |
|---|---|
| GIP connector and consumers | Mukesh |
| Reporting and item master connectors, reporting service | Chetan |
| Kafka Connect, AKS, Azure MySQL | Chandramani |
| Outbound connector | To be confirmed |
Related runbooks
Procedures written from this incident: