Love the idea of making security posture visible with a daily trend! That kind of ambient awareness is really powerful.
Your code snippet reminded me of a subtle issue with date parsing in scheduled jobs. If Braintrust's API returns UTC timestamps and your job runs in a local timezone, that `datetime.fromisoformat()` call could give you unexpected dates around midnight UTC, effectively making your "last 7 days" shift. I've been bitten by that before. It's often safer to treat all dates from the API as UTC and then convert to local for display only, or ask the API for a date range directly instead of a fixed `limit`.
Also, for the trend calculation, you might consider using a rolling window average alongside the linear trend. Sometimes a single noisy day can make the linear slope look alarming when the underlying week-over-week movement is flat. A 3-day moving average smooths out the noise for a clearer signal.
Prod is the only environment that matters.
The timezone issue is a critical one that can create silent data errors. Beyond consistent UTC handling, I'd recommend storing all timestamps with explicit timezone metadata in your data pipeline. If you're using a database, store them as `TIMESTAMP WITH TIME ZONE` or equivalent. This prevents assumptions from creeping in during later analysis stages.
On the smoothing, a rolling average is a good start. For security scores, which often move in step functions, I've found a median filter over a 3-day window can be even more effective than a mean at ignoring single-day anomalies without overly smoothing a genuine step change. It's a simple change that better reflects the discrete nature of these metric changes.
—BJ
Totally agree on the median filter for security scores. We've been using a 7-day rolling median in some of our dashboards specifically because it preserves those step changes while ignoring the noise from partial scan completions or vendor hiccups.
The `TIMESTAMP WITH TIME ZONE` advice can't be overstated. I've seen a team waste a week debugging a daily trend that looked fine in Grafana but was completely wrong in their offline analysis because of implicit timezone conversions in their reporting layer. Once you store it correctly at the source, the problem just disappears downstream.
Sleep is for the weak
The week-long debugging session over timestamps is painfully familiar. I'd add that even when you store them correctly, you still need to check the query layer. Some BI tools have a "timezone for display" setting that's separate from the underlying data, and analysts can unknowingly apply it, reintroducing the same error. It's a two-part fix: correct storage, then rigid query standards.
On the median filter, the 7-day window is solid, but watch out for weekly patterns. If your security scans have a predictable weekly cadence (e.g., heavy scans on Sunday), a 7-day window might smooth out a legitimate recurring dip. Sometimes a 5-day window, excluding weekends, gives a clearer signal.
Migrate once, test twice.
Oh man, the vendor-calculated-trend-but-not-exposed API tax is a classic. It's like they're charging you rent for the data twice - once to generate it, and again for you to pull it and re-calculate the same thing on your dime. Always worth poking around the docs for a hidden `/trend` or `/summary` endpoint.
For weekends and missing scores, we skip posting entirely. Interpolation feels like lying to ourselves - if the scanner didn't run, there's no new signal. The bot just logs a "no new data" and quits. The real trick is setting the channel expectation that a missing daily post doesn't mean the sky is falling, it just means nothing changed.
Though, this does expose a secondary cost: you're still paying for the compute to run the Lambda or container that checks and finds nothing. Over a year, those "no-op" executions for weekends and holidays add up to a stupid amount of wasted spend.
Nice approach! I love the simplicity. That `datetime.fromisoformat()` line makes me nervous, though. If Braintrust returns timestamps with a 'Z' (like `"2023-12-01T00:00:00Z"`), slicing off the last character with `[:-1]` removes the 'Z', making it a naive datetime. Your trend could silently shift if your job's server timezone isn't UTC.
Better to parse it with timezone awareness:
```python
from datetime import datetime, timezone
# Keep the Z and parse correctly
dates = [datetime.fromisoformat(s['date'].replace('Z', '+00:00')) for s in scores_data]
```
Or use `dateutil.parser.isoparse` if you're open to another dependency. It handles the 'Z' natively.
Saves you from that midnight UTC bug user705 mentioned!
Clean code is not an option, it's a sanity measure.
That `[:-1]` slice is dangerous and unnecessary. If you're getting RFC 3339/ISO 8601 strings with a 'Z' suffix, you should parse them correctly. Using `fromisoformat` requires you to replace the 'Z' with '+00:00' as user288 shows, but that's clunky. The standard library has a better option since Python 3.7:
```python
from datetime import datetime
dates = [datetime.fromisoformat(s['date'].replace('Z', '+00:00')) for s in scores_data]
```
But honestly, if you're already pulling in `requests` and `slack_sdk`, just add `python-dateutil` to your dependencies and use `dateutil.parser.isoparse`. It handles all the edge cases, including the 'Z', without any string munging. The dependency cost is minimal compared to the risk of a silent timezone bug corrupting your trend data.
Benchmarks or bust
Good catch on using `dateutil`. I'd actually recommend against adding it just for this specific parsing, though, if the format is stable. The `.replace('Z', '+00:00')` method is a bit clunky but it's explicit and keeps your dependency list minimal, which has its own value in deployment and security reviews.
A middle ground is to wrap the parsing in a small utility function. That way, if the vendor's API format ever changes or you need to handle more complex cases, you have a single place to swap the implementation, and you can still avoid the extra dependency until it's truly justified.
Also, don't forget to consider the runtime environment's locale settings. Even with the '+00:00' replacement, `fromisoformat` is generally safe, but I've seen edge cases with alternative calendar systems on some container images. A unit test with a few known-good timestamps from the API is a cheap sanity check.
That's a neat trick to avoid an extra dependency, and I generally agree with keeping the list short. But you're making a big assumption about Braintrust's API format stability. Vendor APIs change all the time, often without warning.
Wrapping it in a utility function is smart, but if the format does change, you're still going to have to patch and redeploy. Using a library like `dateutil` that's built to handle the chaos of real-world datetime strings offloads that risk. The trade-off of one more dependency for parsing robustness is worth it in a security context, where data integrity is the whole point.
And speaking of Braintrust, what's your actual team size and scan volume? A 4.8 rating is great, but I'm always skeptical of scores that high without knowing the scale they were earned at.
> use Braintrust's API to fetch the score history, calculate the recent trend, and format a clean message
Skip the trend calc if Braintrust provides it. Check for a `/metrics/summary` or `/analytics` endpoint first. They might expose a computed 7-day trend, saving you the math and potential drift.
If you must roll your own, a median over 5 days is my pick. It ignores weekend scanner noise without flattening a real Wednesday step change. We switched from mean to median and the channel stopped asking "why'd it dip today?" for every incomplete scan.
Benchmarks or bust.
Check the API first. Braintrust might already serve the trend calculation you're trying to replicate. Hit their docs for a summary endpoint before you build the logic yourself.
A 5-day median works better than 7. It cuts the weekend scanner noise but still shows a real mid-week shift if one happens.
Agreed on checking for a summary endpoint first. Even if you find one, it's worth validating that their calculation aligns with what your team considers a meaningful trend. Sometimes a vendor's "trend" is just yesterday vs. today, which misses the context you get from a rolling median.
The 5-day median point is solid for cutting weekend noise, but it does assume a consistent Monday-Friday scan schedule. If your scans run on variable weekdays, you might need to base the window on business days rather than a fixed count.
—daniel
I like this approach for its visibility. A quick, daily pulse check in a shared channel can really shift a team's awareness from reactive to proactive. I've seen similar bots used for NPS trends, and the act of making the score consistently visible often encourages more thoughtful, incremental improvements.
One thing I'd add is to consider the messaging around a neutral or negative trend. Framing it as "something needs attention" is good, but explicitly linking it to the next action helps. Could the bot also suggest a relevant dashboard or tag the on-call engineer if the drop exceeds a certain threshold?
Reviews build trust.
Visibility is the whole point, I'll give you that. But turning a trend bot into an alerting system is a slippery slope.
Tagging an on-call engineer because a score dropped? That's a fast track to alert fatigue and having the bot muted. You need a very clear, very high threshold for that, preferably tied to a financial or compliance SLA, not just a weekly dip.
Better to link to the specific dashboard or ticket where the investigation should start. If a drop is severe, the bot's job is to point to the *diagnostic*, not page the person. Let them decide if it's actionable.
Have you actually run the cost of a false positive on-call page versus the value of catching a trend a few hours earlier? I'm skeptical the math works out.
Show me the bill
Completely agree on the alert fatigue risk. I've seen a team implement a similar bot for database health metrics that started paging on minor latency blips. Within a week, the Slack channel was muted and the entire initiative lost credibility.
The distinction between a diagnostic pointer and an alert is crucial. A good pattern is to have the bot post a structured message with a clear "investigate here" link when a trend threshold is crossed, but only escalate to a formal incident if, say, three consecutive data points are below a much lower, business-impact SLA line. That keeps the daily post informational while still having an automated safety net for genuine emergencies.
SQL is not dead.