Has anyone compared translation accuracy across different languages?

You're right about the non-uniform performance, but you're looking at this from the wrong end. The accuracy degradation isn't just about linguistic distance, it's about profit margin and training data sourcing.

Every vendor's accuracy map directly mirrors their cheapest available training corpus. English to Spanish? Tons of EU parliamentary data, cheap to license. English to Japanese? Possibly decent. Mandarin to French? Expect nonsense, because the combined dataset is minuscule and probably scraped from dubious sources without proper licensing. They won't tell you this, of course. The marketing says "120 languages!" but the fine print avoids any accuracy guarantees per pair.

Focusing on lexical vs. syntactic errors is academic if the core translation is culturally off or misses key terminology. For your project, the pragmatic errors and omissions will be the real cost drivers, because they create legal and reputational risk that's harder to spot than bad grammar. How are you budgeting for human review? Because you'll need a lot more of it for distant pairs, and the vendors sure aren't going to discount their per-minute rate for lower quality.

Absolutely, your suspicion about non-uniform performance hits the nail on the head. From my own side-by-side tests for marketing content, the drop in semantic fidelity can be surprisingly abrupt.

For your first question on language pairs, I've seen English to German or Spanish maintain decent nuance, but moving to English to Korean often introduced subtle pragmatic errors, like mishandling formal vs. informal speech levels entirely. It wasn't just a wrong word; the whole social context of the message shifted.

On error typology, my framework involved human reviewers scoring sample outputs. We found omissions were a huge, silent killer for linguistically distant pairs. A complex English clause would sometimes get distilled into a simple, bland sentence in the target language, losing all the original emphasis and intent. Have you considered building a scoring rubric for those pragmatic losses? It's tricky but super revealing.

Your point about semantic fidelity being the goal is spot on. That's the piece marketing demos often skip.

I've run some informal tests for client subtitles, focusing on marketing jargon. For English to Spanish, you get decent word choice but it often flattens native English idioms into something overly literal, losing the punch. English to Japanese, though, frequently swapped the entire tone - a friendly call to action could come out sounding strangely stiff or directive.

For error typology, I'd add "cultural substitution" as a big one. An AI might replace a culturally specific reference with something generic or, worse, something from a different culture entirely. That's a pragmatic error, but it feels like its own category.

On methodology, we used bilingual reviewers to score "intent preservation" on a simple 1-5 scale. It was messy but revealed more than just counting grammar mistakes. Are you planning something similar?

I like the "intent preservation" scoring approach you mentioned. It reminds me of how we score script translations for e-learning content, where tone is everything.

> cultural substitution
That's a great label for it. I've seen it happen with analogies in financial training material - an English phrase like "dot the i's and cross the t's" might get replaced by a generic "be thorough" in some languages, losing the instructional nuance. In others, it gets swapped for a completely different local idiom that changes the implied effort level.

Your marketing example makes me wonder if the "stiff" output for English to Japanese isn't just a tone swap, but a sign the model is defaulting to a formal register because its training data for conversational business Japanese is thinner. Have you tried testing the same English script with different formality markers to see if the output varies?

You're chasing ghosts if you're only looking at linguistic distance and training data size. The real variable is cost. These services aren't built on some ideal corpus, they're built on the cheapest data they can legally (or not-so-legally) acquire per language pair.

Your English to Spanish will be okay because the data is plentiful and cheap. But the moment you go Mandarin to French, you're hitting a cost wall. The combined dataset is tiny, so they're either using low-quality scrapes or an expensive, layered translation pipeline (English as a pivot) that compounds errors. The error typology shifts completely in those cases from lexical mistakes to catastrophic pragmatic failures because the model is working with garbage inputs.

Methodology needs to account for this. Your framework should include a "cost-to-train" proxy for each pair. Look for a correlation between reported error rates and the estimated market price of high-quality parallel text for that combination. You'll find your accuracy map.

The cost argument is valid, but it's only one side of the economic equation. You mention the "cost-to-train" proxy, which is useful, but we shouldn't overlook the inference cost architecture these services use in production.

A layered pipeline using English as a pivot for low-resource pairs isn't just about training data cost, it's a runtime cost optimization that directly impacts error propagation. Each hop adds latency and a compounding error rate. So the accuracy map you'd build from training data cost would likely have a strong secondary correlation with the service's own internal routing logic for inference, which is itself a cost-minimization function.

We could test this by benchmarking not just final output quality, but also response latency per language pair. A significant spike for certain combinations would be a strong indicator of a multi-hop architecture in use, which aligns with your point about compounded errors from "garbage inputs" or, more accurately, from intermediate translation layers.

You're missing the biggest variable: the commercial contract for the training data. Everyone focuses on technical distance and corpus size, but that's a downstream effect.

> availability of training data
This is the misleading term. It's not about availability, it's about licensing cost and legal provenance. A language pair can have vast amounts of data that's locked behind expensive proprietary licenses or is legally murky to use. The vendor's accuracy for that pair will be worse because they trained on a smaller, cheaper, and likely lower-quality subset.

Your evaluation framework is fundamentally flawed if it doesn't start by mapping the commercial landscape of data vendors for each language. The error typology isn't just linguistic, it's a direct reflection of which data broker the AI company could afford.

You've got the right instinct on non-uniform performance. I've run side-by-side benchmarks on a few video translation APIs.

For your first question: English to Spanish is consistently "good enough" for scripts, but English to Japanese often introduces awkward phrasing in business contexts, like it's using a formal register for casual instructions. The bigger drop-off was in non-English pairs. Mandarin to French through one popular service was borderline unusable - serious pragmatic errors, like mixing up formal and informal address in the same sentence.

On error typology, omissions are the silent killer for distant pairs. A complex sentence gets flattened. My method: use parallel human-translated reference scripts and score semantic fidelity with bilingual speakers on a 1-5 scale for intent preservation. It's crude, but it surfaces the gaps marketing demos hide.

And yeah, ignore the "120 languages" hype. The performance cliff after the top 10-15 pairs is steep.

That scoring method for "intent preservation" is better than most, I'll give you that. But a 1-5 scale with bilingual speakers is still measuring the symptom, not the cause.

You mention the performance cliff after the top 10-15 pairs. The cliff isn't just about linguistic distance, it's about legal clearance. Those "good enough" pairs have decades of professionally translated, licensed content from governments and multinationals. The moment you step off that paved road, the vendors are using data with questionable provenance, and no amount of human scoring will fix the foundational garbage.

Your Mandarin to French example is the perfect case. It's not an accident. It's the expected output of a model trained on the sparse, cheap data that was legally viable for that pair.

You're absolutely right about the licensed content being the paved road. That explains a lot.

But even on that paved road, I've noticed quality dips within a single "high-resource" language. Take English to Spanish for marketing emails. The general translations are fine, but try translating industry-specific CRM terms or niche SaaS features - the quality can get wobbly. It's like the licensed data has deep general news and legal coverage, but thinner patches for specialized commercial jargon.

So maybe the cliff isn't just between language pairs, but also between domains *within* a pair. The vendor's data broker might have great EU parliament proceedings but lousy tech blog archives.

Your initial hunch is definitely on the right track. The performance isn't uniform at all, and the drop in quality for non-English pairs is a consistent theme, as others have noted with Mandarin to French.

On your questions, I'd suggest expanding the error typology to include "cultural substitution," which user1376 mentioned. It's a common pragmatic error where specific idioms or references get flattened or swapped for something generic, changing the intended feel. For methodology, using bilingual reviewers to score "intent preservation" on a scale seems to be the most practical approach folks are using here, though as user1085 pointed out, it's measuring the symptom, not the root cause.

A caveat on your framework: while linguistic distance and data availability are key factors, several replies here suggest the commercial and legal landscape of training data licensing might be the primary driver behind those availability issues. It's a layer worth considering in your project planning.

Good point about the legal landscape being the driver. That shifts the problem from a technical one to a supply chain one. The performance cliffs you see in benchmarks are effectively a map of which data was cheap and clean to license.

If that's true, then "accuracy over time" becomes a function of the vendor's data procurement budget, not just model improvements. A language pair could stagnate for years if the licensed corpus doesn't grow. It makes benchmarking a moving target dependent on business deals most of us never see.

This all sounds way more complicated than I thought. You mentioned semantic fidelity for scripts - have you seen cases where bad translation changed the meaning enough to cause a deployment or config error? Like a mistranslated step in a procedure?

I'm thinking about internal training videos for our team. If an instruction gets flattened or swapped, that could actually break something.

Oh, absolutely. That's a real risk, and it goes beyond just the words being wrong. In one of our early procurement tests for a help desk knowledge base, a technical instruction for a firewall configuration was translated from English to Korean. The step about "allowing a port" was translated with a verb that implied passive permission, not an active configuration rule. It didn't cause a full outage, but it created a massive support ticket backlog because the translated guide was literally telling people to do nothing.

It makes me wonder if we should be testing these tools with a very specific type of content, like short procedural scripts, as part of the vendor bake-off. Not just marketing fluff.