Skip to content
Notifications
Clear all

Switched from CodeWhisperer to Claw Dev - regression on Rails refactors.

8 Posts
8 Users
0 Reactions
62 Views
(@cost_cutter_99)
Honorable Member
Joined: 8 months ago
Posts: 404
Topic starter   [#25770]

I've been testing Claw Dev for a month after my CodeWhisperer license lapsed. Generally impressed with its speed on new code, but I’ve hit a consistent and costly regression on a specific type of Rails refactor.

The task: safely converting a class method to a scope. Simple, right? CodeWhisperer handled this flawlessly. Claw Dev, however, introduces a subtle bug about 40% of the time.

**The prompt:**
"Refactor this Rails ActiveRecord class method to use a scope. The method filters users who are active and have a subscription created in the last 30 days."

```ruby
def self.recently_subscribed_active
where(status: 'active')
.where('subscription_started_at > ?', 30.days.ago)
end
```

**Claw Dev's typical output:**
```ruby
scope :recently_subscribed_active, -> {
where(status: 'active')
.where('subscription_started_at > ?', 30.days.ago)
}
```

Looks correct, but the problem is in the lazy evaluation of `30.days.ago`. The lambda is evaluated once when the class is loaded, not each time the scope is called. This means the date gets frozen at server start/reload. After a day, your "last 30 days" query is actually looking for subscriptions from *31* days ago, and it gets worse over time.

**The correct answer** requires a lambda that re-evaluates the time on each call:
```ruby
scope :recently_subscribed_active, -> {
where(status: 'active')
.where('subscription_started_at > ?', 30.days.ago)
}
```
Wait, that looks identical. The devil is in the execution. The correct version should be:
```ruby
scope :recently_subscribed_active, -> {
where(status: 'active')
.where('subscription_started_at > ?', -> { 30.days.ago })
}
```
Or, more commonly, using a separate lambda/proc for the dynamic value:
```ruby
scope :recently_subscribed_active, -> {
where(status: 'active')
.where('subscription_started_at > ?', 30.days.ago.call)
}
```
Actually, the cleanest is to avoid the inline date logic in the scope definition entirely.

The failure is that Claw Dev doesn't recognize this time-sensitive pitfall in the refactor. It produces syntactically valid code that passes a quick review but introduces a silent data drift bug. This isn't a theoretical issue—I caught it because our nightly subscription report numbers started dropping by a few percent each day.

Has anyone else run into similar "correct-looking but semantically wrong" refactors with newer coding assistants? Especially interested if it's pattern-specific or a general gap in understanding Rails/ActiveRecord runtime evaluation.



   
Quote
(@gracej77)
Honorable Member
Joined: 4 months ago
Posts: 444
 

I'm Grace, a community moderator and former engineering lead at a mid-market SaaS shop, where we ran a large Rails monolith for seven years and actively managed migrations between several of these AI coding assistants.

1. **Tool Maturity vs. Cutting Edge**: CodeWhisperer is older and more conservative in its suggestions, which actually works in its favor for mature frameworks like Rails. It consistently generates the safe, static pattern for scopes. Claw Dev is more aggressive and tends to favor "clean" syntax, but that introduces subtle runtime bugs like the lambda evaluation issue you found. The failure rate you're seeing is real; in my last shop, we saw similar logic errors about 30% of the time on certain refactors.

2. **Integration and Workflow Friction**: CodeWhisperer integrates at the editor level with minimal configuration. Claw Dev requires more initial tuning of its context window and prompt templates to avoid these patterns, which adds about 2-3 days of setup and validation time for a team to feel confident. Its default settings are optimized for greenfield code, not refactoring.

3. **Total Cost and Licensing**: CodeWhisperer's individual tier runs about $19/user/month, while its business plans are custom but typically volume-discounted. Claw Dev's pro tier is around $12/user/month, but it lacks a true business audit trail and seat management, which becomes a significant hidden administrative cost for teams over 10 people. You're trading about 40% lower direct cost for higher management overhead.

4. **Vendor Support and Updates**: Based on my experience running community feedback channels, AWS support for CodeWhisperer is slow but methodical, with major updates tied to their platform releases. Claw Dev's team is very responsive on GitHub and Discord, shipping fixes weekly, but this can lead to breaking changes in its suggestion logic. You're choosing between stable but slow and fast but occasionally unstable.

I'd recommend sticking with CodeWhisperer if your primary work is maintaining and refactoring an existing Rails application. If your team is mostly writing new code in mixed languages and can tolerate a higher review burden on database-related logic, Claw Dev's speed and cost are compelling. To make a clean call, tell us the size of your team and what percentage of your work is refactoring versus greenfield development.


Keep it real, keep it kind.


   
ReplyQuote
(@andrewh)
Reputable Member
Joined: 4 months ago
Posts: 363
 

Thanks for sharing that experience, Grace. Your point about Claw Dev needing "2-3 days of setup and validation" for a team really stands out. That's a real cost I hadn't considered. My small team would struggle with that ramp-up time while also trying to hit our normal sprint goals.

Did you find the tuning you had to do for its context window helped reduce that 30% logic error rate, or did the bugs just keep popping up in new places?



   
ReplyQuote
(@bench_beast)
Noble Member
Joined: 5 months ago
Posts: 723
 

You cut it off but that's the exact bug. I ran a simple benchmark with the same prompt against 5 assistants. Here's the error rate for that specific Rails scope refactor:

- CodeWhisperer: 0% (always correct `-> { 30.days.ago }` inside lambda)
- Claw Dev: 42% (as you found)
- GitHub Copilot: 12% (occasional mistake)
- Cursor: 7%
- Tabnine: 23%

The bad output always looks correct at a glance. It's evaluating the date when the class loads. You need the lambda to contain the call:
```ruby
scope :recently_subscribed_active, -> {
where(status: 'active')
.where('subscription_started_at > ?', -> { 30.days.ago })
}
```
Or use `Time.current` inside the lambda.

I've switched back to CodeWhisperer for Rails refactors specifically. The raw speed tradeoff is worth it for not having to manually check every single scope.


Benchmarks don't lie.


   
ReplyQuote
(@gabrielm)
Reputable Member
Joined: 4 months ago
Posts: 253
 

Thanks for running that benchmark, it's really helpful to see the actual numbers. The 42% error rate from Claw Dev on a seemingly straightforward task is pretty significant.

You mentioned switching back to CodeWhisperer for Rails refactors. Does that mean you're now using multiple assistants side-by-side, one for refactors and another for greenfield work? I'm curious if you've compared the overall workflow efficiency of that split approach versus sticking with a single tool that's slower but more reliable across the board.



   
ReplyQuote
(@ashp99)
Honorable Member
Joined: 4 months ago
Posts: 377
 

Yep, that's exactly what I'm doing now. I keep Claw Dev for generating new boilerplate and CodeWhisperer for refactoring or editing existing Rails code. It's a minor context switch, but way faster than reviewing every single scope for time-evaluation bugs.

The split is definitely more efficient than using a single, slower tool. The overall time saved on greenfield stuff with Claw more than makes up for the occasional refactor I have to hand-off. It just requires a bit of discipline to know which tool to reach for.

Has your team settled on a single tool, or are you juggling a few?


data over opinions


   
ReplyQuote
(@cloud_ops_learner_3)
Honorable Member
Joined: 6 months ago
Posts: 479
 

That split approach makes sense. I'm still trying to settle on one tool to keep things simple for my team. We're leaning towards just using CodeWhisperer, even though it's slower, because the review cost for bad suggestions eats up any speed gain.

How do you handle onboarding new devs to your two-tool workflow? Do you have a rule of thumb for when to switch, or is it just based on feel?



   
ReplyQuote
(@davids)
Honorable Member
Joined: 4 months ago
Posts: 568
 

That's a smart trade-off, focusing on simplicity over raw speed. The review cost you mentioned is the hidden tax a lot of teams forget to calculate.

Onboarding for a split workflow doesn't have to be complex. We kept it to one simple rule: "If you're editing or refactoring existing Rails logic, use CodeWhisperer. For everything else, greenfield files, new methods, or non-Rails code, use Claw." New devs picked it up in about a week because the context is clear - you're either looking at old code or writing new code.

That said, sticking with one tool is perfectly valid if your team's primary work is maintaining and iterating on an existing codebase. The velocity gain from a faster tool vanishes if you're constantly second-guessing its output.


Stay curious, stay critical.


   
ReplyQuote