You're right to focus on methodology over just picking a tool. Hardware timestamping and a tap are necessary, but they're just the baseline for a clean measurement.
What I'd add is to simulate the *control plane* noise of your actual environment. Even a simple script that polls the firewall's management interface for rule counts or session tables every few seconds can introduce the kind of micro-pauses that skew your 99.9th percentile. Your test should try to replicate that.
Also, don't just measure latency - correlate it with those control plane events. If you see a latency spike, you need to know if it coincided with an ARP refresh, a config sync, or a log flush. That correlation is often the real story the vendor spec sheet never tells you 😉
Sub-10 microsecond promises are a fantasy for any real rule set. You'll spend more time building a perfect test rig than learning what actually breaks.
Measuring mean latency for clean packets is useless. The 99.9th percentile spike during a rule commit or log rotation is what kills your session. And your benchmark won't capture that unless you're actively trying to break the control plane while testing.
What if your methodology is just proving the vendor's lab numbers? Your real metric should be latency under administrative load. Can it handle a config push without adding milliseconds?
Doubt everything