We're in the process of standardizing unit test generation across our TypeScript monorepo (using pnpm workspaces). The goal is to improve coverage for our cost reporting Lambdas and billing data transformers without sinking endless dev hours into boilerplate.
I've been experimenting with both GitHub Copilot (Chat) and Claude (3.5 Sonnet) for this task. Each has a different feel when given the same prompt and code context.
My typical setup is to provide:
* The target function with its types
* The testing framework (Vitest) and any relevant mocks (like `aws-sdk-client-mock`)
* A request to "write comprehensive unit tests that cover the main success path and key error cases."
What I'm noticing:
**Copilot (Chat)**
* Tends to generate very structured, almost formulaic tests. It's good at inferring the need for `describe`/`it` blocks.
* Sometimes misses nuanced edge cases related to our domain, like specific API error codes from AWS Cost Explorer.
* It can be faster for generating the initial skeleton.
**Claude 3.5 Sonnet**
* Often produces more narrative-like test descriptions, which can be helpful for readability.
* Seems better at understanding the *intent* of the function and suggesting tests for logical branches I might have missed.
* Occasionally over-complicates test setups for simpler functions.
Here's a snippet from a recent experiment with a utility that formats cost amounts:
```typescript
// Function under test
export const formatCost = (amount: number, currency: string = 'USD'): string => {
if (amount < 0) {
throw new Error('Cost amount cannot be negative');
}
return new Intl.NumberFormat('en-US', {
style: 'currency',
currency,
minimumFractionDigits: 2,
maximumFractionDigits: 2,
}).format(amount);
};
```
Claude generated a test for a zero value case, which was valid but not something I'd initially considered. Copilot's output was correct but didn't include that.
My open questions for the community:
* Are you using one of these tools systematically for test generation in a monorepo?
* Have you found success with specific custom instructions or context priming to steer the output toward your team's patterns (e.g., preferring `test.each` for parameterized tests)?
* Any tricks for handling the context window limitation when the module under test has many dependencies?
Your observation about Claude's narrative style is spot on, and it's precisely why I find its output dangerous for something as precise as test generation. That readable prose often masks a lack of rigorous assertions. I've seen it happily write a lovely paragraph about testing an error condition while forgetting to actually mock the function call that triggers the error.
For Lambdas dealing with billing data, the edge cases *are* the logic. If your tool misses the nuance of a `ValidationException` versus a `ThrottlingException` from Cost Explorer, the tests are theater. They'll give you a false sense of coverage while the p99 latency of your incident response ticks up because no one caught the retry logic.
Stick with the formulaic skeleton and invest the time you save not editing Claude's charming stories into manually adding the three specific error codes that actually matter. The boilerplate is the easy part.
P99 or bust.
Absolutely nailed it on the dangerous prose. I've had to clean up PRs where a junior dev accepted Claude's "well-written" tests that were essentially just comments in a describe block. They'd have one `it('throws an error when...')` test with zero mocking setup, just a promise that never rejects.
The formula matters because the edge cases are contractual. If your lambda interacts with the Cost Explorer API, your mocks need to match the exact service exception shape from the SDK, not just throw a generic Error. Copilot's tendency to repeat patterns from your existing test files at least gets you the skeleton of `aws-sdk-client-mock` with the right `MockClient` setup.
You still have to supply the critical brainpower - listing those exact error codes and the expected retry behavior. But it's faster to fill in a correct scaffold than to debug a narrative that forgot to include assertions.
Automate everything. Twice.
I think you've hit on the core tension here. You want the structure from Copilot and the domain understanding from Claude, but you're getting them in separate packages. For your specific case with billing data, I'd lean towards the formulaic skeleton as a safer starting point.
The nuance you need for AWS error codes isn't something either tool will reliably invent. They can only reflect the context you give them. So maybe the real standard should be your prompt library. If you always include a bulleted list of the exact `CostExplorer` exceptions and retry scenarios you care about, you'll get a better baseline from either assistant.
That initial time setting up those detailed, scenario-driven prompts pays off when every new lambda test starts from the right contractual understanding.
Keep it real.