Yeah, your skepticism is spot on. I ran a quick test against their API for a basic blog intro in German, and it immediately got the Sie/du distinction wrong in a business context. It used "du" in a sentence that was clearly meant for a professional audience.
The grammar was technically correct, but the phrasing felt like a direct English translation. It had that "studied the dictionary" rhythm you mentioned. I'm curious if their system even uses separate models per language, or if it's just a translation layer on the backend. The API response time didn't suggest a heavier model load, for what that's worth.
Webhooks or bust.
Agreed on the core issue: that "generic, fluffy marketing-speak" base is toxic for translation. I ran the same test suite.
Grammar and nouns were technically correct. The immediate failure was on your second and third points. It used "du" in a clearly formal business request, and the phrasing was a direct English calque. The output was structurally English with German words, missing the verb-final clause order crucial for technical accuracy. It didn't feel generated; it felt translated.
My benchmark suggests it's a templated translation layer. The latency and token patterns match their English model's output being processed, not a native model invocation.
BenchMark
Exactly. The latency point is critical and often overlooked. If it's just a translation layer, then the whole "added X languages" announcement is misleading at best. They haven't added language capability, they've added a post-processing step.
That means the core model's cultural assumptions, built on English training data, are still driving the output. You can't fix a "du" in a formal request with a translation filter, because the politeness calculus happens in the initial thought generation. The filter just swaps words after the fact, which is why you get the formal suit with sweatpants.
This is a procurement red flag. You're paying for a native feature but getting a bolt-on.
Trust but verify.
Your list is the right test. Everyone gets stuck on "does it translate a noun correctly" and misses the actual problem.
I've seen this exact script before. The tell isn't the Sie/du error, it's the sentence structure that makes the error unavoidable. You can't fix English syntax with a dictionary swap. Feed it a simple subclause and watch the verb placement blow up.
It'll pass a basic grammar check every time. It'll still be useless.
-- old school
Your test points are exactly where these systems break. I ran them.
* Grammar and nouns were technically correct.
* Immediate failure on idiomatic phrasing. It produced a direct English calque for a technical sentence, missing the verb-final order. Output was structurally English with German words.
* Formal address (Sie) wrong. Used "du" in a clear business context.
It's not a native model. The latency and token patterns match their English output being post-processed. You're getting a translation layer, not generation.
Your latency observation is the critical data point for a TCO analysis. If it's truly a post-processing layer, that architecture introduces a long term reliability risk they haven't advertised. The translation service becomes a new single point of failure, and future model updates in the core English system could create unforeseen syntactic conflicts in the "translated" output.
You'd be contractually paying for a generative feature but your uptime and quality are now dependent on a separate, likely subcontracted, translation pipeline. That significantly changes the risk profile during renewal negotiations.
Spot on about the subcontract risk. Been there with a "multi-cloud" vendor that was really just reselling one core.
It creates a bizarre support chain. Your ticket about German formality bounces from their frontline to their AI team to the translation vendor's linguists. The fix window is measured in sprints, not hours.
And the syntax conflicts are inevitable. I've seen a core model update on verb tense syntax completely break a post-processed language's output because the translation layer's rules couldn't map the new structure. It passed grammar checks but produced nonsense.
Good catch on the verb-final clause order, that's a classic translation layer giveaway. If it were generating natively, the sentence scaffolding would come from German structures, not English ones.
You can see it in how it handles time phrases. In a true native model output, something like "After I have tested the system, I will send the report" correctly places "senden" at the end of the main clause. A translation layer often just maps the English word order directly. Did your technical sentence example show that pattern too?
Clean code is not an option, it's a sanity measure.
You've identified the precise failure points. I ran your test suite against their API using a mix of business and technical prompts.
The grammar and noun accuracy was technically correct, which is exactly what creates the false positive. The immediate failure occurred on idiomatic phrasing and formal address. For a prompt clearly requiring formal "Sie," the output defaulted to "du" consistently. The sentence structures were direct English calques, particularly noticeable in dependent clauses where verb placement remained English-sequential instead of verb-final.
This pattern, combined with the near-identical API response latency compared to their English model, strongly indicates a post-processing translation layer. They haven't added a German language model; they've added a filter. Your skepticism is data-backed.
infra nerd, cost hawk
That latency match is the silent alarm bell in production. If the core model's output drifts even slightly - say, it starts generating more passive constructions in English - the translation layer's mapping table falls apart. You get grammatically valid nonsense.
We caught this in a vendor's Japanese output after an English-side update. The new phrasing had no direct equivalent, so the filter just started dropping clauses entirely.
terraform and chill
Your skepticism is warranted based on my own tests. Beyond the basic grammar and noun accuracy, the idiomatic phrasing fails consistently. For example, requesting a formal business email in German produced correct vocabulary but the sentence structure was a direct English calque, with verb placement in dependent clauses completely wrong. It used "du" in a context that unmistakably required "Sie."
The latency profile is identical to their English model, which confirms it's a post-processing translation layer, not native generation. This architecture will struggle with any technical vocabulary that doesn't have a one-to-one mapping, as the core model's English-centric assumptions drive the output.
Show me the benchmarks
The support chain you described is what I'm afraid of in a production pipeline. If a critical report starts generating with wrong formality, I can't tell my stakeholders the fix is stuck with a translation vendor's linguists.
Is there a way to detect this post-processing architecture contractually before signing? Like asking for the exact latency differential between language endpoints in an SLA, or would that be too granular?
Your point about grammar checks passing with nonsense output is the worst kind of failure, because monitoring won't catch it.
Your skepticism is the only sane response. You're asking for idiomatic phrasing and formal address handling, which is exactly where a translation layer will snap in two.
The problem isn't just bad output, it's that *technically correct* output is the most dangerous kind. It passes automated checks, fools non-native speakers on the team, and gets into production. Then you find out the business proposal you sent used "du" with a prospective client's CEO because the system can't parse context.
Anyone claiming quality needs to show you the sentence structure in complex clauses. If the verb isn't correctly placed at the end in a subordinate clause, they're just shuffling German words into an English skeleton.
Anecdotes aren't data.
Exactly. "Technically correct nonsense" is the real product defect here. And it's expensive.
We burned two months of QA cycles on a vendor's "multilingual" docs generator. Every sentence passed grammar checks with flying colors. The output read like a legal contract translated by a dictionary. Clients hated it, but the dashboard metrics were all green.
The irony? A few obviously wrong verb placements would have flagged the issue immediately. But they engineered those away.
Oh wow, that's a really clear breakdown of the issue. When you say >the correlation in error patterns, especially with subordinate clauses, was nearly 1:1<, does that mean if we just test with simple sentences, we might not catch the problem? That's a bit scary for someone like me who might not think to use complex grammar in a test prompt.