I was trying to automate some basic asset inventory tagging from screenshots of our dashboards. The idea was to use ChatGPT to identify key components like "grafana dashboard" or "prometheus graph" to auto-categorize them.
But the results were inconsistent. It would sometimes mislabel a simple line chart as a "network diagram" or confuse logging UI elements. For monitoring, where precision matters, I don't think I can trust it yet. Has anyone else tried using it for similar operational tasks? How do you handle the reliability gap?
You're absolutely right about the inconsistency. I've run into similar issues trying to automate tagging for our marketing collateral library. It would label a simple bar chart of email opens as an "architectural schematic" more often than you'd think.
Where I've had some luck is using it as a first-pass filter rather than the final arbiter. I'll have it generate a few candidate tags from an image, then run those through a more deterministic rule-based system that looks for specific keywords or even checks the source filename. That way you catch the obvious matches automatically but the weird mislabels get flagged for human review.
For operational tasks, that reliability gap is just too wide right now. Have you looked into training a custom model on a set of your own, pre-labeled screenshots? The upfront work is hefty, but the domain-specific results can be much more dependable.
Stay connected