Having recently integrated several machine translation services into our internal developer portal for localized documentation, I was curious to see how Notion AI's often-marketed translation capability holds up under a more rigorous, technical evaluation. While many reviews focus on the user experience or speed, I am particularly interested in the measurable quality of the output, especially for technical or structured content common in our fields.
To establish a baseline, I propose we consider several key dimensions for benchmarking:
* **Translation Accuracy & Fidelity:** Does it correctly handle technical jargon (e.g., "pod," "stateful set," "provisioner") and preserve the meaning of complex sentences?
* **Context Retention:** Can it maintain consistency across a document when translating terms that have multiple meanings? For example, does it distinguish between a Terraform "module" and a software "module" based on surrounding text?
* **Formatting Preservation:** A critical aspect for Notion's block-based system. Does translated text maintain lists, code blocks, callouts, and inline formatting?
* **Idiomatic Correctness:** Does the output sound natural in the target language, or is it a stilted, literal translation?
I conducted a preliminary, informal test using a paragraph from a Terraform configuration guide. The source English text included a code block and some bullet points.
```hcl
# Example: This module manages an AWS EKS cluster.
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "~> 19.0"
cluster_name = "my-cluster"
}
```
Initial observation: Notion AI translated the surrounding explanatory prose into Spanish (es-ES) adequately for general understanding. However, the translation of the comment within the code block was less ideal, altering the technical specificity. The code block itself, correctly, was left untouched.
My open questions for the community are these:
* Has anyone performed a comparative analysis against dedicated translation services (e.g., DeepL, Google Cloud Translation API) using a standardized set of technical documents?
* Are there noticeable quality differences between language pairs? I would hypothesize that translation between major European languages might be more robust than for languages with significantly different syntactic structures.
* How does the feature handle the translation of entire database entries, including property values? This is a unique Notion-centric challenge.
Without reproducible metrics—such as BLEU scores or human evaluation scores on a technical corpus—it is difficult to move beyond anecdotal evidence. I am interested in collaborating on or reviewing any structured tests the community may have undertaken.
infra nerd, cost hawk