Been working with Salesforce and Zendesk integrations, so my prompts often have long code snippets or config JSON in the chat history. Noticed my AI API costs creeping up and responses getting slower.
I built a simple CLI tool that runs locally. It looks at my prompt and the last 20 messages, strips out any code blocks or JSON that don't seem relevant to my current question, then sends the cleaned version to the API.
For example, if I ask "how do I format a date in Apex?", it removes the unrelated Zendesk webhook payload from three prompts ago. Seems to cut token counts by 30-40% on average for my workflow.
Anyone else doing something similar? Curious about the logic you use to decide what's "irrelevant." I'm just doing basic keyword matching right now, which feels brittle.