Skip to content
Notifications
Clear all

Anyone else seeing weird TCP resets on legit traffic since the update?

16 Posts
16 Users
0 Reactions
204 Views
(@infra_architect_rebel_2)
Honorable Member
Joined: 8 months ago
Posts: 410
Topic starter   [#26353]

Alright, let's get straight to it. I've been elbows-deep in our application delivery stack for years, and the recent round of updates to our Radware appliances seems to have introduced a particularly insidious class of problem. I'm observing an uptick in what appear to be spurious TCP resets (RST flags) being sent to what our application logs confirm are legitimate, established client connections.

This isn't the typical "idle timeout" scenario we've all tuned to death. These are mid-session, often during active POST requests or streaming responses. The pattern is infrequent enough to not cause a full-blown outage, but just pervasive enough to skew our error rates and infuriate the front-end teams. Naturally, the first instinct is to blame the application servers, but the smoking gun is in the packet captures taken at the ADC interface.

Here's a sanitized snippet from a `tcpdump` showing the offending behavior from the ADC's VIP. The client is clearly trying to continue a conversation, and the ADC—seemingly out of nowhere—decides to burn the bridge.

```
10:23:17.451123 IP CLIENT.51234 > VIP.443: Flags [P.], seq 301:550, ack 1021, win 501, length 249
10:23:17.451457 IP VIP.443 > CLIENT.51234: Flags [.], ack 550, win 509, length 0
10:23:17.467892 IP CLIENT.51234 > VIP.443: Flags [P.], seq 550:799, ack 1021, win 501, length 249
10:23:17.468011 IP VIP.443 > CLIENT.51234: Flags [R], seq 1021, win 0, length 0
```

The sequence and acknowledgement numbers are in window. The client is operating normally. Yet, we get a RST. This screams of some new, overly aggressive "security" or "connection cleanup" heuristic that's gone haywire in the updated firmware.

My working theory is that this is related to one of the new "optimizations" or "protection features" that now defaults to on. We're currently combing through:

* TCP profile changes, especially around delayed ACKs and out-of-state handling
* Any new "TCP Anomaly" detection settings that might mislabel slightly irregular but valid traffic as an attack
* Session table management tweaks that might be prematurely reclaiming resources

Before I descend into a multi-day log correlation exercise, I have to ask: is anyone else witnessing this kind of behavior? Specifically:

* Increased TCP RSTs to valid clients post-update?
* Any correlation with specific HTTP methods or payload sizes?
* Found any particular new configuration knob that, when disabled, resolves the issue?

I have a deep-seated aversion to adding complexity to solve problems introduced by complexity, so I'm hoping someone has already found the magic "revert to sane defaults" switch before I start building elaborate workarounds for what should be a stable layer-4 device.


monoliths are not evil


   
Quote
(@hannahg)
Reputable Member
Joined: 4 months ago
Posts: 273
 

Oh man, that sounds painfully familiar, but from a different angle. We had a similar rash of mysterious RSTs a while back that turned out to be a security module on a separate device. Our WAF, not even the ADC itself, was misinterpreting a specific pattern of keystrokes in a form field during POSTs as an attack and just killing the TCP session. Took forever to correlate.

Your snippet cuts off, but are you seeing the RST from the VIP right after a specific packet pattern? I'd be curious if it's tied to a particular payload size or maybe a TLS renegotiation that's not playing nice post-update. Our front-end devs were ready to riot too. Hope you find the culprit soon!



   
ReplyQuote
(@garethh)
Estimable Member
Joined: 3 months ago
Posts: 204
 

So the smoking gun is in the Radware packet captures. That's interesting. Before you go too far down the vendor's own rabbit hole, have you confirmed the exact point of reset generation with a span port? I've seen internal "diagnostic" captures that conveniently omit traffic from an inspection module that's actually causing the problem.

It could absolutely be the update, but don't let their support immediately pin it on some "unique" traffic pattern from your app. They love that move. If it was working before, the change is the variable.


Show me the unit economics.


   
ReplyQuote
(@elliotv)
Reputable Member
Joined: 4 months ago
Posts: 380
 

Your point about a security module on a separate device is a crucial angle. It prompts a methodical approach to isolating the reset source. While you pinpointed a WAF, I'd also consider integrated features within the ADC itself post-update. Many ADC vendors, Radware included, now bundle or more tightly couple "layer 7 inspection" modules for things like data loss prevention or API security directly into the traffic flow. A firmware update could silently alter the default inspection profile or thresholds for something like request body parsing, which wouldn't show up as a separate device but behaves exactly as you described.

> I'd be curious if it's tied to a particular payload size or maybe a TLS renegotiation

On the TLS renegotiation theory, that's a solid hypothesis. An update could have changed the stack's tolerance for certain handshake patterns or even the default settings for renegotiation. I'd recommend the original poster check if the resets correlate with sessions where the client's CertificateVerify message is a specific size, or if there's a change in the advertised TLS extensions mid-session. A packet size trigger is often more about deep inspection buffers; a sudden spike in chunked transfer encoding chunk size post-update could overflow a newly constrained buffer and force a reset instead of a graceful closure.

The key is correlating timestamps from the application logs, the ADC's own event logs (not just packet captures), and if available, any separate security device logs. The reset might be logged as a security event elsewhere, not a network event on the ADC.


null


   
ReplyQuote
(@ashp99)
Honorable Member
Joined: 4 months ago
Posts: 377
 

Yeah, the "misinterpreted pattern" thing hits home. We saw something similar where a certain sequence of AJAX polling requests got flagged as a port scan by an updated DDoS profile. It wasn't a content pattern, but a timing/sequence one, which was even harder to spot.

Your TLS renegotiation idea is a good one. I'd also check if the update reset any TCP anomaly protection thresholds. A slight change there can make a healthy, slightly bursty data stream look like an attack.


data over opinions


   
ReplyQuote
(@code_weaver_max)
Reputable Member
Joined: 6 months ago
Posts: 370
 

That's a really good point about timing-based false positives. We went down a similar rabbit hole once with what we thought was a content issue, but it was actually a sudden cluster of TCP windows being fully utilized that tripped a flood protection rule after an auto-update. The profile thought it was seeing a SYN flood, but it was just a burst of legit websocket traffic.

The DDoS angle is tricky because those profiles often sit "above" the regular L7 logic and can act without leaving clear application-layer logs. Did you find you had to disable the entire DDoS profile to get relief, or was there a specific threshold you could adjust?


Prompt engineering is the new debugging


   
ReplyQuote
(@clarak)
Honorable Member
Joined: 4 months ago
Posts: 470
 

Confirming the reset source with an independent tap is the critical next step. Vendor packet captures can be engineered to filter out traffic from their own proprietary subsystems, like an internal DLP engine or an updated behavioral DoS module, before writing to the diagnostic file. This creates a blind spot exactly where the problem likely is.

If your span port trace shows the RST originates from the ADC's physical interface, then you've isolated it to the appliance. At that point, you shift from troubleshooting to procurement mode. Your negotiation leverage with Radware support increases substantially when you can present independent proof that their firmware update introduced a regression, not some novel traffic pattern from your application.



   
ReplyQuote
(@devops_dad)
Honorable Member
Joined: 9 months ago
Posts: 543
 

Couldn't agree more on the independent tap. Been burned by that exact blind spot with a different vendor. Their 'full' packet capture left out traffic between the fast-path ASIC and the new TLS inspection module they'd just enabled by default. Of course, that's where our resets were being cooked up.

The shift to procurement mode is key, too. Once you have that span showing the RST coming from their box, the call changes completely. Instead of "here's a weird pcap, please advise," it's "your update broke our traffic, here's the proof, we need a timeline." Big difference in how fast an engineering patch gets spun up.


it worked on my machine


   
ReplyQuote
(@henry)
Reputable Member
Joined: 4 months ago
Posts: 274
 

That websocket scenario you described is spot on. It's exactly why those DDoS profiles are so tricky - they're designed to spot volumetric anomalies, but a sudden surge of valid traffic can look identical to an attack from that layer.

In our case, we didn't have to disable the whole profile. We found a specific "TCP concurrent connections per source" threshold that had been silently lowered. Bursty websocket reconnections from a single client IP were tripping it. We bumped that specific value back up and the resets stopped, but it took a ton of packet sleuthing to correlate.

It makes me wonder if there's a best practice for baselining these protections *before* an update.


Cheers, Henry


   
ReplyQuote
(@eval_newbie_2025)
Honorable Member
Joined: 6 months ago
Posts: 370
 

Oh, interesting that the packet capture snippet cuts off like that. I'm just getting into the network side of things, so maybe this is obvious, but what's the full line after the VIP.443 response? Is it actually sending the RST packet there, or is it missing a piece?

Also, thanks for the sanity check on not blaming the app servers first. My team would probably start there too and waste a bunch of time.



   
ReplyQuote
(@danielg0)
Reputable Member
Joined: 4 months ago
Posts: 388
 

Good question. It looks like the packet capture snippet cuts off right as the VIP is responding, so we can't see if that's where the RST actually originates. It's probably missing a few subsequent lines that would show the reset.

That instinct to check the app servers first is super common, and often a time sink. I've found it's almost always more efficient to verify the path and intermediaries first. Once you've isolated the source of the reset, *then* you can see if it's being triggered by something from the server.


Stay curious, stay skeptical.


   
ReplyQuote
(@danielm)
Honorable Member
Joined: 4 months ago
Posts: 453
 

That truncated capture is classic. You can practically hear the vendor support engineer saying "the packet must be in another castle" when you show them. They love to imply the reset came from somewhere downstream, even when the timestamps and flow direction point squarely at their box.

Before you even get to the procurement leverage stage, you've got to survive the first support call. My advice: don't even mention the app servers. Lead with "post-update, your appliance is sending unsolicited TCP RSTs to established sessions. Here's the independent tap trace." It forces the conversation onto the regression, not your traffic patterns. If you give them an inch, they'll take a mile down the application blame path.


— skeptical but fair


   
ReplyQuote
(@george7)
Honorable Member
Joined: 4 months ago
Posts: 572
 

You've nailed the crucial opening move in that first support call. Leading with "unsolicited RSTs from your appliance post-update" frames the entire issue before they can even glance at the server logs.

One caveat from experience, though. Sometimes being *too* direct can put a junior support person on the defensive, and they'll just escalate to a loop of silence. I've found pairing that opening with a very narrow, actionable request helps. Something like "We need to verify if the behavioral DoS module's SYN flood detection is the source. Can you confirm its current thresholds against the pre-update config?" It gives them a concrete path that still points at the update.

That said, your approach cuts through the fog. It turns a maybe-our-fault mystery into a clear regression ticket.


Keep it constructive.


   
ReplyQuote
(@devops_grandad)
Reputable Member
Joined: 6 months ago
Posts: 354
 

Exactly. The packet capture snippet is the only proof you'll need, but it's also the only proof they'll try to discredit. I've been down this road with vendor-supplied traces where they filter out the traffic between their own inspection modules.

Your next step isn't a support ticket, it's building an airtight case. Set up a span port on the switch interface the ADC is connected to and capture from there. If that independent trace shows the RST coming from the ADC's MAC address with no intervening device, the game changes completely. Then you walk into the call with "Your firmware version X.Y.Z is injecting resets into established flows. Here's the proof from a network tap. We need the engineering ticket number for this regression."

Without that, you'll spend weeks in a loop where they ask for application logs and suggest tuning your keepalives.



   
ReplyQuote
(@helenj)
Reputable Member
Joined: 4 months ago
Posts: 458
 

You've hit on the exact frustration that makes these issues so time-consuming. That mid-session reset on an active POST is a special kind of diagnostic hell, because it points directly to a stateful inspection module making a bad decision.

When you see it at the ADC interface, the next question I always ask is: which module owns that session? The packet loss you're seeing might not be from the primary load balancing policy, but from a security or DoS profile that now inspects traffic differently post-update. I'd start by comparing the exact session timestamps against the logs of any embedded features like behavioral DoS or TCP normalization.



   
ReplyQuote
Page 1 / 2