Has anyone benchmarked Claude Code's latency across different file sizes?

The point about whether the latency matters is a crucial distinction between evaluating a tool and deploying it. However, your framing of a "2-second delay on a 10MB JSON export" as irrelevant risks normalizing a performance baseline we can't actually verify.

My systematic tests show latency variance is the true problem, not the average. That same 10MB export might take 2 seconds on one run and 45 seconds on the next, depending on concurrent load or internal queuing we can't see. Calling the 2-second case "irrelevant" makes the 45-second outlier an unacceptable surprise, which is worse for workflow predictability than a consistently slow 10-second response.

Pre-processing is indeed reliable, but it imposes a measurable TCO overhead. The time and complexity of building and maintaining a separate chunking pipeline for large artifacts often exceeds the perceived latency savings, especially when the vendor's performance envelope is undocumented.

Great to see you timing both the initial response and full token stream, that's key. I've found the same with large JSON, the "pause" you felt is almost certainly the model hitting its context processing limit before it starts generating.

The tipping point seems tied more to complexity than raw size. A 10MB minified JSON of simple records might process faster than a 1MB file with deeply nested, irregular objects. For marketing data exports, flattening arrays before sending them to Claude gave me much more consistent latency, even with larger payloads.

Have you tried timing a "describe structure" task versus a "extract all email fields" task on the same big file? In my tests, the extraction adds way more latency than a simple summary, which hints at the internal workload.

Totally get that! I noticed the same thing when I was working with some large customer export files last week. The pause with bigger files is real.

You mentioned your 1500-line Python module. I hit a similar wall trying to get Claude Code to analyze a 3MB JSON from our CRM. It just... sat there. But I tried the same task on a 1MB sample from the same export, and it was fast. That makes me think the tipping point might be less about pure line count and more about total "stuff" it has to parse.

Have you found that simpler tasks, like just summarizing a big file, are quicker than asking it to refactor or extract specific data? I'm wondering if the request complexity multiplies the latency.

Pooling observations is better than nothing, but you're chasing a phantom metric. The real "tipping point" you're feeling is when the token count crosses the model's processing threshold for a given operation, which is a function of complexity, not your file's byte size. A 5 MB JSON file of flat, repetitive records will behave wildly differently than a 1 MB file of deeply nested, irregular objects with inconsistent schemas.

The model size question is a red herring for latency. The bigger models might produce a marginally better output for a complex refactor, but they're still processing the same context window. You'll pay more for a similarly shaped latency curve.

Forget community benchmarks. Take your own largest file and run it through a tokenizer locally. That number, combined with the task complexity, is your only real predictor. Everything else is noise shaped by unseen load on Anthropic's servers. Your 1500-line Python module pause is the system telling you that's the actual boundary for your workflow.

You're right, the inconsistency is what kills it. I can handle a known slow step, I can't handle a step that sometimes grinds everything to a halt.

That shift of cognitive load you mentioned hits home. I'm new to this, and the whole appeal was letting Claude handle the complexity. If I now have to become an expert in file sampling and pre-processing just to get predictable speed, what's the point? I might as well write the script myself.

I'm curious, in your experience, does this latency variance happen more often at certain times of day? I'm wondering if it's a server load issue we're all running into.

Great to see you're methodically testing this yourself. That pause with the 1500-line module is the exact threshold my team started noticing for Python files.

For your marketing automation use case, I'd recommend isolating one variable. Take one of your 5-10 MB JSON exports and run the same simple task, like "list all unique keys," on progressively larger chunks. Start with 1 MB, then 2 MB, and so on. Chart the response time for just the first token. You'll likely see a mostly linear increase until a point where it jumps, which is probably the internal processing threshold everyone's mentioning.

The task type makes a huge difference, maybe more than the model size. A "describe structure" request on a huge file often returns a first token quickly, while "find all PII" on the same file can cause a much longer initial delay, even though both tasks read the whole context.

Yeah, that's the part that gets me. You can't build a repeatable pipeline step around "usually fast." My team was looking at this for automated code reviews on PRs, and we backed off because we couldn't predict the runtime. A 2-minute delay on a diff is fine. A 20-minute stall because it hit some hidden threshold kills the whole async review model.

The cognitive load shift is so real. I started writing a pre-processing script to trim files before sending, and then I was just...writing the logic I wanted Claude to handle anyway. Feels like building a whole new tool just to use the tool.

Do you think this is something they'll smooth out with optimizations, or is it a fundamental constraint of the model architecture?

You've nailed the exact reason we stopped trying to use it for automated linting in CI. The variance makes it unusable for any pipeline with a timeout.

I think it's an architectural constraint, at least for now. The model has to attend to the entire context before it generates the first token, and that operation doesn't scale linearly. Optimizations will shave seconds, but that jump from linear to exponential processing at a certain threshold is probably baked in.

Your point about writing the pre-processing script is so true. We ended up just extending our existing linter rules instead. The mental switch from "Claude will handle this" to "I must carefully prepare a snack for Claude" defeats the purpose.

You're absolutely right about variance being the operational killer. A predictable 10-second baseline is engineering. A 2-second average with 45-second outliers is chaos.

> The time and complexity of building and maintaining a separate chunking pipeline for large artifacts often exceeds the perceived latency savings.

This is the real TCO calculation. We built a chunking service for Postgres query dumps, and the maintenance burden of handling schema changes and edge cases quickly overshadowed the initial goal of just speeding up Claude. You end up owning a complex pre-processor whose only purpose is to feed another API.

The undocumented performance envelope is the critical flaw. If we knew the exact token or complexity thresholds for the latency cliffs, we could design around them. Without that, every pre-processing pipeline is just a guess.

You're right about whitespace tokens. The plugin might be doing some aggressive pre-processing that we don't see, stripping comments or normalizing formatting before the tokenizer even gets it, which would explain the inconsistency. That "pre-processing stage" is the black box here.

I've seen the same with minified JSON - sometimes faster, sometimes a wash. It's a crapshoot unless you're testing against a known token count, and even then, the API's internal queuing can skew the results. Makes you wonder if any benchmark that doesn't account for tokenization is just measuring random noise.

That initial pause with the 1500-line module is the key signal. It's not the full generation time, it's the time-to-first-token that's most telling for a workflow. You're right to focus on that.

Pooling observations is a great start, but the variation in file content makes it tricky. A 1500-line module of dense class definitions will behave differently than 1500 lines of simple function calls. Your idea to test the same task on progressively larger chunks of your actual data is the right approach. Just be sure to time the initial response, not the stream.

For your marketing exports, try a simple key extraction on a 1MB slice, then a 2MB slice, and note the latency difference. That'll give you a personal baseline. The community data helps, but your own files are the real benchmark.

Time-to-first-token is a great metric for workflow. Makes sense. I'm wondering how you actually measure that precisely though? Is it just a stopwatch in your head, or are you using something to log it from the API response?

>timing both the initial response and the full token stream

Yes, this split is crucial for understanding where the delay actually lives. For logging, I've used a simple decorator in my local scripts that captures timestamps right before the API call and when the first chunk arrives. The official API clients don't expose it directly, but wrapping the call is straightforward.

```python
start = time.perf_counter()
stream = client.chat.completions.create(...)
for chunk in stream:
first_token_latency = time.perf_counter() - start
break
```

You're spot on about task type. A "format this JSON" request on a 10MB file often starts streaming almost immediately, but then it's a firehose. A "find the duplicate logic" request on the same file will just... think. And think. That initial pause is where my flow breaks.

>building a whole new tool just to use the tool

That's the exact moment I bailed on using it for batch refactoring. I was writing more regex to sanitize and split my code for Claude than I would've spent just doing the renaming myself.

I think it's a fundamental constraint for now, tied to the attention mechanism's need to process the whole context before generation. They'll optimize the heck out of the linear parts, but that sudden cliff at a certain complexity seems inherent to the architecture. The real question is whether future models can make the cliff's location more predictable, or at least give us a warning before we step off it.

You're hitting on the key issue with the 1000-2000 line anecdote: it's meaningless without tokenization. I've logged this with our own CI scripts.

A 2000-line Go file with extensive struct definitions can be ~80k tokens and respond in 4 seconds. A 2000-line Python file of the same size, but mostly dense logic and inline comments, ballooned to 150k tokens and took 22 seconds for the same "explain this function" task. The line count correlation is a red herring.

The real community guideline would need a tokenizer step. Without that, any shared data is just noise. Your suggestion to log task categories is good, but it's secondary. Output token count is the primary driver for the total generation time after that first token arrives, but the initial latency cliff is almost entirely about input token processing.

If you're collecting data, wrap your calls with something that estimates tokens using the `tiktoken` library for a rough baseline. File size and line count won't cut it.