Skip to content

fix: time-limit default response de-duplication - #1926

Open
Dukakz wants to merge 5 commits into
Koenkk:masterfrom
Dukakz:default-response-dedup-window
Open

Dukakz wants to merge 5 commits into
Koenkk:masterfrom
Dukakz:default-response-dedup-window

Conversation

@Dukakz

@Dukakz Dukakz commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

fix: time-limit default response de-duplication

Fixes #1873

What

sendDefaultResponse skips the response when the incoming frame's ZCL transaction sequence
number equals _lastDefaultResponseSequenceNumber. The check has no time limit, so the
sequence number is treated as "already answered" forever. This PR limits it to 1 s.

Why

The check exists because the same frame is sometimes handed up twice. Those duplicates arrive
back to back (3 ms apart, measured in #1873), so a 1 s window keeps that protection fully
intact — the existing Send default response once to frame received twice test passes
unchanged.

What the permanent form breaks:

It is a device firmware bug, but the cost of the two failure modes is not symmetric: a
redundant default response is dropped by the sender, a missing one deadlocks the device.

Field data

Tuya TS011F socket _TZ3210_cehuw1lw (TS011F_plug_1), Zigbee2MQTT 2.14.2, ember / EmberZNet
7.4.4:

  • ~5 frames/s on cluster 0x0B04 haElectricalMeasurement, command report (0x0A), byte-identical
    payload, ZCL transaction sequence number fixed at 182 while the APS sequence kept incrementing,
    disableDefaultResponse: false. So these are new transmissions, not stack-level retries.
  • That one socket was 83 % of all frames the coordinator received; MQTT traffic for the whole
    network went from ~700 to ~115 publishes per minute once it stopped.
  • /configure and configure_reporting do not get the device out of it, which fits the
    mechanism: the device's own readRsp / configReportRsp carry disableDefaultResponse: 1,
    so those frames produce no default response and the stored sequence number never changes.
  • Restarting Zigbee2MQTT — the only thing that clears this in-memory field for a router that
    never re-announces — stopped the flood immediately, every time. It came back later on its own.

Four other users in #1873 report the direct form of the recovery on Z-Stack and ember, on
mains- and battery-powered devices (_TZE204_laokfqwu, _TZE284_o3x45p96, _TZE284_locansqn):
sending one more default response for the repeated sequence number releases the device within a
second, after hours or days of looping.

Change

  • DEFAULT_RESPONSE_DUPLICATE_WINDOW = 1000.
  • The time of the last default response is stored next to the sequence number, in a # private
    field so it stays out of the database entry and out of the device clones the tests compare.
  • A frame is a duplicate only if the sequence number matches and the previous response went
    out less than 1 s ago.

Test

New test in the existing block: the same frame, repeated 1 s later, is answered a second time.
It fails on master (expected "defaultResponse" to be called 2 times, but got 1 times) and
passes with this change. Full suite green, coverage still 100 %.

🤖 Generated with Claude Code

@Nerivec

Nerivec commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator

Don't think we want that low a time frame.
A measured sample of one cannot be used for any kind of global behavior decision, too many devices, too many variables.
This behavior is not covered by spec, Z2M adds it to prevent spamming from some brands that retransmit heavily for no good reason...

I've never had either of the two opposite scenario, so, not sure what's a good value here... Most tx-related spec timeouts are around 8sec, we may want to start closer to that; we can adjust later from more feedback.

Note that none of this will fix a device stuck in retransmit loop even after receiving a default response.

Also, Zigbee 4.0 enforces duplicate rejection at APS layer, so, it may not matter much, in the medium-to-long term, these devices may become entirely broken.

@Dukakz

Dukakz commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

Fair enough — one measured device is not a basis for a global timing decision, and the suppression of heavy retransmitters is worth keeping. Raised the window to 8 s, in the order of the spec's transmit timeouts rather than the few milliseconds stack-level duplicates actually need. Test moved to 8 s with it.

Worth separating the two things, though, because the failure this fixes does not depend on the window being short: the problem is that the check has no time limit. A device whose firmware does not increment the ZCL transaction sequence number on retransmit gets exactly one default response, ever — every later copy is dropped forever. _lastDefaultResponseSequenceNumber lives in memory and is only cleared in resetTransient (device announce / join), which a mains-powered router never triggers, so the only ways out are a Zigbee2MQTT restart or a power cycle of the device. The same deadlock hits a perfectly healthy device if that single default response happens to be lost over the air.

With 8 s, a device retransmitting at 3–5 frames/s is answered once per window instead of never: the flood is still suppressed by ~97 %, and the device can get out of the loop. Happy to go higher if you prefer — the point is that it should be finite.

@WueF

WueF commented Nov 2, 2026

Copy link
Copy Markdown

Field data in support of this change, from a production mesh where the deadlock hit most of the routers at once.

Setup: Z2M 2.13.0 / zigbee-herdsman 10.8.0, ember, SLZB-MR1U (EFR32MG21, EmberZNet 8.0.2) over TCP. 46 nodes, 35 of them mains Tuya DIN breakers TS0601 _TZE204_fhvdgeuh (TS0601_din_4), all routers.

What we saw:

  • 22 of 31 meters were each repeating a single frame at about 4.5 frames/s, and sent nothing else for days. The mesh carried about 60 msg/s, and the adapter logged The adapter has run out of buffers in bursts (156 in one day).
  • Example from a 180 s debug capture: one device sent the same manuSpecificTuya commandDataReport (DP 15, Tuya seq 23552) 804 times. The ZCL transaction sequence number stayed at 248, while the APS sequence kept incrementing (179, 180, …). disableDefaultResponse: false.
  • No defaultRsp was sent to any of the looping devices during the capture. Default responses were going out only to the healthy devices.
  • Outbound unicasts to some of these routers were lossy: about 25 % FAIL / ZIGBEE_DELIVERY_FAILED in the same window. So "the single response was lost and is never re-sent" is a realistic path on a congested mesh, not only the "device doesn't increment the TSN" case. I can't prove which one started each loop.

What a re-sent default response does:

  • A Z2M restart cleared _lastDefaultResponseSequenceNumber. The looping devices were answered once, moved on to their next report, and most of them looped again on that one (new TSN, new payload). That fits a lossy return path.
  • A time-bounded re-answer, equivalent to this PR, fixed it. We deployed a small external extension: if the same TSN repeats for more than 1 s, it clears _lastDefaultResponseSequenceNumber so herdsman answers the next copy, at most once per 2 s per device.
    • Within about 5 minutes the mesh dropped from ~60 msg/s to ~5 msg/s.
    • The previously frozen meters resumed normal reporting: voltage and energy moving again.
    • Some devices needed several re-answers before one got through. The worst took about 20 in the first minute; most needed fewer than 10.
    • It has stayed at about 5 msg/s since, on 2.14.2 and now 2.14.3.

So on this hardware, a device stuck in the loop does recover once a default response actually reaches it. The deadlock is the permanent dedupe, not the device. With an 8 s window these devices would get one answer per 8 s instead of none. On a lossy path, recovery takes longer than with our 1–2 s workaround (worst case a few minutes), but it does recover, and that is the important part.

@Dukakz

Dukakz commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for taking the time to capture all this — 22 devices on different hardware is a far better basis than my single breaker.

Two things in your data answer the open questions here better than my one sample could:

  • The frozen ZCL transaction sequence number (248 across 804 frames) while the APS sequence keeps incrementing is exactly the mechanism, now confirmed on a different adapter and a different herdsman version.
  • Your ~25 % ZIGBEE_DELIVERY_FAILED on the outbound path shows the second route into the deadlock: the firmware does not have to misbehave at all, the single default response only has to be lost over the air. After that the device is stuck for good, because _lastDefaultResponseSequenceNumber is cleared only in resetTransient, which a mains-powered router never reaches.

And your deployment addresses the "this won't fix a device stuck in a retransmit loop" concern directly: it did recover, as soon as an answer actually reached it — ~60 msg/s down to ~5, and the frozen meters reporting voltage and energy again.

On the window size: your workaround re-answered every 1–2 s and recovery took under a minute, with up to ~20 re-answers for the worst device. At 8 s that worst case becomes a few minutes instead. I kept 8 s because it is the value @Nerivec suggested (in the order of the spec's transmit timeouts), and because the property that matters is that the window is finite at all rather than its exact length — but your numbers are a fair argument that something lower recovers faster on a lossy mesh. Happy to adjust if the maintainers prefer.

@Nerivec

Nerivec commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

22 same devices is still just 1 sample in this regard.
Also an AI analysis from the looks of it...

This PR is a global behavior, so the goal is to be "fine for all devices" (as opposed to targeting a few). We can always lower the number in the future if we see no issue crop up.

The stuck in retx loop scenario I mentioned was for devices that are actually stuck, no matter if a default rsp is sent or not.

@Dukakz

Dukakz commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

Ok, thanks! By the way, I'm using a translator because English is not my first language :)

@WueF

WueF commented Nov 2, 2026

Copy link
Copy Markdown

22 same devices is still just 1 sample in this regard. Also an AI analysis from the looks of it...

This PR is a global behavior, so the goal is to be "fine for all devices" (as opposed to targeting a few). We can always lower the number in the future if we see no issue crop up.

The stuck in retx loop scenario I mentioned was for devices that are actually stuck, no matter if a default rsp is sent or not.

I agree, that single device is not a wide sample, and as stated, problem was solvable by using custom converter, although it took me a lot of time to investigate the issue and notice the sympthoms.

Nevertheless, I found it surprising, that herdsman didn't repeat defaultRsp by itself, as message in regular scenario can get lost, (even if this Tuya meters are very specific and indeed, maybe don't follow the standard themselves), but got stuck anyway on one lost answer.

@Nerivec

Nerivec commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

Correcting bad behavior is always tricky, you fix it for one, break it for another... It's whac-a-mole! 🥵

@Dukakz can you reduce the size of the comments to be more to-the-point before we merge this?

@andreacasarin

Copy link
Copy Markdown

Observed the same issue with a Tuya 2CT (_TZE204_dhotiauw), Z2M 2.6.3 / herdsman 6.3.2, EmberZNet 8.0.2: about five identical reports per second, fixed ZCL TSN, advancing APS sequence, and no requested default responses.

One manually re-sent default response immediately restored all measurements. I’ve backported the eight-second change; readings remain normal after restart. Waiting for a natural recurrence to verify automatic recovery.

@andrei-lazarov andrei-lazarov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we please move this forward?

I am affected as well image

Comment thread src/controller/model/device.ts Resolved
Comment thread src/controller/model/device.ts Resolved
Dukakz and others added 3 commits November 13, 2026 21:59
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A single measured device is not a basis for a global timing decision, and the
existing check also guards against devices that retransmit heavily for no good
reason. Keep that suppression by sizing the window in the order of the spec's
transmit timeouts instead of the few milliseconds duplicates actually need.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Andy <andrei_lazarov@yahoo.com>
@Nerivec
Nerivec force-pushed the default-response-dedup-window branch from 8e47505 to b923090 Compare November 13, 2026 20:59
@j4m3s

j4m3s commented Nov 16, 2026 •

Copy link
Copy Markdown

Field data from an Ember setup, on the window size and on whether a stuck device recovers once it gets a response.

Setup: Zigbee2MQTT 2.14.3, zigbee-herdsman 11.2.0, ember adapter, ZBDongle-E (EmberZNet 7.4.4), ~54 devices. Floods started after upgrading to 2.14.3. All the affected devices are Tuya mains devices.

1. The duplicate check left a device stuck (herdsman_garage_excerpt.txt). A 3-gang Tuya switch module sent 392 copies of a genBasic report, all TSN 239, in about 80 s (~5/s). Each had a new APS sequence and disableDefaultResponse: false. herdsman sent Default Responses to 7 other devices in the same window and none to this one. The flood had been running for about 6 hours, and only power-cycling the device stopped it.

2. Re-answering alone wasn't enough here (herdsman_towel_excerpt.txt). A TS0001 _TZ3000_kqvb5akv was stuck on TSN 149. An external extension re-sent the Default Response every 5 s, which is effectively this PR with a 5 s window. Every one ended ZIGBEE_DELIVERY_FAILED: 13/13 in the debug window and about 200 over 14 minutes. The device's own frames arrived fine. The Default Response goes out without APS RETRY (options 4352), and the device's link evidently couldn't carry a single attempt.

3. With APS RETRY, the device recovered at once (herdsman_rescue_excerpt.txt). The same extension, changed to send with disableDefaultResponse: false so the ember adapter keeps RETRY, cleared the stuck report on the first reply. The device then cleared its backlog and settled. So a device stuck in this loop does recover once a response actually reaches it.

The window looks right to me, but on Ember it may also need the Default Response sent with RETRY. I've opened #1957 for that part.

herdsman_towel_excerpt.txt
herdsman_rescue_excerpt.txt
herdsman garage excerpt.txt

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Default response de-duplication by TSN can deadlock mains-powered devices that retransmit after a lost defaultRsp (network flood, BUFFER_FULL)

6 participants