Skip to content
Notifications
Clear all

We compared five models for log analysis. The winner wasn't the most expensive.

32 Posts
30 Users
0 Reactions
9 Views
(@devops_journeyman)
Estimable Member
Joined: 3 months ago
Posts: 71
 

We used a base prompt template but absolutely had to tweak it for Haiku, and a bit for Sonnet. The syntax was the same, but Haiku responds better to extremely explicit, almost rigid formatting instructions in the system prompt. Think "Return a JSON object with exactly these keys" rather than "Please provide the data in JSON format."

GPT-4 was more forgiving with vague prompts, but you pay for that flexibility. The tweaking time for Haiku was maybe an hour upfront, and it's been solid since.



   
ReplyQuote
(@danag)
Estimable Member
Joined: 2 weeks ago
Posts: 98
 

Spot on about the nuance gap. It's that exact tipping point where you go from parsing to reasoning. I've seen Haiku do brilliantly on extracting error codes and timestamps from a messy stream, but ask it "which of these five warnings is most likely to cause a user-facing delay?" and it starts guessing.

For us, the sweet spot became a hybrid. We use Haiku as a first-pass filter to extract and structure the raw logs, then we have a separate, smaller set of rules (sometimes even just heuristics) that run on the structured output to do the actual prioritization and flag the "need a human" cases. That way you're only paying the big model tax on the 2% of logs that actually require understanding.



   
ReplyQuote
Page 3 / 3