I've been evaluating AI-powered development tools for integration into our CI/CD and developer workflow tooling, with a specific focus on their performance outside the mainstream JavaScript/Python/Java ecosystems. Our teams work with a diverse set of technologies, including Elixir for backend services and R for data science pipelines, so vendor claims of "broad language support" require rigorous testing.
I conducted a systematic test of Codeium (using the VS Code extension) against several non-trivial codebases. The core question was whether it could provide meaningful completions and suggestions for syntax patterns unique to these languages, or if it would simply fall back to generic, often incorrect, snippets.
**Elixir Findings:**
Codeium handles basic Elixir syntax adequately. It correctly suggests common Phoenix framework constructs and standard library function names. However, it struggles significantly with the more idiomatic, macro-heavy parts of the language and complex pipe (`|>`) chains. For example, when working on a GenServer implementation:
```elixir
def handle_call(:get_state, _from, state) do
# Codeium suggested a generic reply, but missed the standard {:reply, state, state} pattern
# It also failed to correctly anticipate the structure of a `handle_cast` for a custom message.
end
```
Its understanding of Ecto query compositions was surface-level; it would propose the initial `from` macro but couldn't reliably continue a dynamic `where` or `join` built at runtime.
**R Findings:**
The experience with R was more pronounced in its limitations. Codeium recognized basic `dplyr` verbs like `mutate()` or `filter()`, but its context window seemed to lose the dataframe structure quickly. In a typical data manipulation sequence:
```r
df_processed %
filter(date > as.Date("2023-01-01")) %>%
group_by(category) %>%
summarise(
avg_value = mean(value, na.rm = TRUE),
# Here, it suggested unrelated column names not present in the inferred schema
total = sum(incorrect_column)
)
```
It consistently failed to suggest appropriate column names from the prior steps in the pipe, which is a critical feature for productive use. Furthermore, its suggestions for ggplot2 layer construction were often syntactically valid but semantically nonsensical for the plot being built.
**General Observations:**
* **Latency:** Completions for these languages were noticeably slower compared to Python/TypeScript, suggesting less optimized underlying models.
* **Context Awareness:** The tool's ability to maintain context across multiple files (e.g., an Elixir `context.ex` and its corresponding `schema.ex`) was poor, limiting its usefulness for real project work.
* **Configuration:** No pipeline-specific configuration options were evident to tailor behavior for these languages, which is a gap compared to the fine-tuning available for mainstream languages in other tools.
For teams strictly in the Node/Python ecosystem, Codeium may be sufficient. However, for polyglot environments with legitimate niche language usage, its current capabilities introduce a risk of generating plausible but incorrect code, which could negatively impact code review burden and test automation outcomes. I would be interested to hear if others have conducted similar integration tests, particularly concerning the tool's behavior within Docker-based development environments or CI jobs where the context might be even more constrained.
-jf
Interesting that you're seeing the Elixir macro limitations. I've run into similar gaps when trying to use these tools for dbt Jinja templating - the pattern is identical. The model gets the basic syntax but fails on the nested context and custom macros, which is precisely where you need the help.
Your point about the pipe chains is critical. In data pipelines (which are conceptually similar to Elixir's `|>`), the tool often breaks down after two or three transformations. It loses the semantic thread of the data's shape. I'd be curious if your R testing showed analogous issues with the tidyverse `%>%` operator or with non-standard evaluation patterns in ggplot2. That would point to a fundamental weakness with functional, chained syntaxes in these models.
Garbage in, garbage out.
Your test is interesting but I'm skeptical about the methodology. How did you quantify "struggles significantly" - was it a blind evaluation with a scoring rubric, or just a gut feel? Without a reproducible set of prompts and a way to measure completion accuracy against a known correct solution, this is just anecdotal. I'd also want to know what the model's training cutoff was and whether you checked if your test cases accidentally matched examples from the training data. The GenServer example you mentioned is a Phoenix staple, so it's exactly the kind of pattern the model might have seen a million times. The real test would be a custom macro that only exists in your private codebase. Did you test that?
Data skeptic, not a data cynic.