I've been benchmarking the latest open-source models for the past two weeks, focusing on latency-per-token and cost-efficiency for our inference stack, and I keep hitting a fundamental roadblock: the naming convention for the "Claw" model family (Claw, Claw2, Claw-Coder, etc.) is, frankly, a mess. It's becoming a genuine time-sink when trying to compare results or select a target model for a specific workload.
My team (approx. 15 engineers, focused on RAG and agentic workflows) is evaluating models in the 7B-34B parameter range. We run our own benchmarks on a mixed Kubernetes cluster (A100s and H100s) using vLLM and TGI backends. The confusion starts the moment you try to map a name to a precise model architecture or capability set. For example, is "Claw2-13B-V" the same as "Claw2.5-13B"? Vendor release notes often refer to one, while the Hugging Face repo is named the other. I've had to dig into configuration JSONs just to confirm basic details.
Here is a concrete example from a recent benchmark run where I had to verify I was actually testing comparable models. The `config.json` is the only reliable source:
```json
// For what's called "Claw2-7B" on one blog post
{
"architectures": ["ClawForCausalLM"],
"model_type": "claw",
"hidden_size": 4096,
"intermediate_size": 11008,
"num_hidden_layers": 32,
"num_attention_heads": 32,
"vocab_size": 32000
}
// For what's called "Claw2.5-7B" on GitHub
{
"architectures": ["ClawForCausalLM"],
"model_type": "claw",
"hidden_size": 4096,
"intermediate_size": 14336,
"num_hidden_layers": 32,
"num_attention_heads": 32,
"vocab_size": 32000
}
```
Notice the difference in `intermediate_size`. These are not the same model, yet the names suggest a minor iterative version bump. This leads to wasted GPU hours benchmarking what you think is a new model, only to find out it's a different model entirely, or worse, benchmarking a model that is functionally identical to one you've already tested but under a different moniker.
The problem compounds with the specialized variants:
* Claw-Coder-13B vs. Claw2-Coder-34B-Instruct – The naming mixes the base model generation, the specialty domain, the parameter count, and the fine-tuning approach. There's no consistent pattern.
* Where does "Claw-Math" fit in? Is it a separate lineage from "Claw2-Math" or just a fine-tune of a specific base?
This isn't just pedantry. It directly impacts:
* **Reproducibility:** It's hard to reproduce others' benchmark results if you can't be certain you're loading the exact same model.
* **Cost Estimation:** Comparing cost-per-token across "Claw2" and "Claw2.5" is meaningless if the underlying architectures differ in ways that significantly affect throughput.
* **Vendor Claims:** When a cloud provider touts "Claw2-70B" performance, which specific checkpoint are they actually serving? The 70B from June or the 70B from August, which might have a different hidden dimension?
I considered whether this is just a symptom of rapid open-source development, but other model families (Llama, Mistral, Qwen) manage to maintain clearer versioning and distinct names for architectural variants. Has anyone else built an internal mapping or a naming cross-reference table to cut through this noise? How are you ensuring your internal benchmarks and model catalogs remain accurate when the ground truth on Hugging Face is so convoluted?
Show me the benchmarks
You're definitely not the only one. I hit the same wall last month when trying to compare latency for our VWO test variants. Spent half a day just confirming whether "Claw-Coder-13B" and "Claw2-Coder-13B" were actually different architectures, or just different fine-tunes of the same base.
For what it's worth, I started keeping a personal spreadsheet just to map HF repo names, blog post names, and the arch details from the config files. It's the only way I can reliably track what we're testing. The .5 increments are the worst - feels like versioning drift from a marketing team, not engineers.
✌️
You're right about the config file being the only reliable source. I've wasted billable hours on similar confusion. Beyond just mapping names, this directly impacts cost and resource allocation.
> run our own benchmarks on a mixed Kubernetes cluster (A100s and H100s)
The naming ambiguity means you could be provisioning the wrong instance type for a given model's memory footprint. A "Claw2-13B-V" versus a "Claw2.5-13B" might have different attention implementations that drastically change the required GPU memory, leading to either overprovisioning (cost waste) or underprovisioning (failed pods, retries). Your team's time spent deciphering this is a direct cloud cost - it's engineering hours burned on inventory management instead of optimization.
My approach now is to embed a hash of the key architectural parameters from the config (hidden_size, num_attention_heads, num_hidden_layers, etc.) into our internal model registry. That way the deployment automation selects the correct GPU family and instance size, regardless of the whimsical name on the Hugging Face card.
Less spend, more headroom.