What is the best way to track downloads of PDF whitepapers?

I really appreciate seeing the SQL example, it makes the "just use your logs" idea feel so much more practical.

That regex point about filename characters others brought up is exactly the kind of gotcha I'd run into later. If we start with the logs, do you think it's better to get a structured log format from the start, like user232 mentioned, or is tweaking the regex as we go an acceptable part of the process?

Also, the "no third-party black box" part is a huge relief. The idea of joining this to our leads data later is the dream scenario.

Exactly. Server logs are the source of truth. That regex example is a great starting point, but you'll want to lock down the pattern.

A couple of things to add from running this in practice:
- If you're on AWS, you can have CloudFront or ALB push structured JSON logs directly to S3. Saves you the parsing headache.
- For the sales team needing near-real-time counts, you can set up a simple log tail to stream counts to a dashboard. It's still your own data, no third party.

>CloudFront or ALB push structured JSON logs directly to S3

This is the key, isn't it? I'm trying to set this up for a side project. When you say JSON logs, do you mean the standard AWS access logs format, or do you have to enable a specific JSON logging feature? I've only ever seen the space-delimited format in the S3 buckets for CloudFront. If it's just the standard log, I guess the 'parsing headache' moves from regex to JSONPath, which might be easier for a beginner like me?

And for the simple log tail for real-time counts, would you just use something like CloudWatch Logs subscription to a Lambda function? That seems like the next thing for me to learn after I get the structured logs figured out.

The SQL fetishism in this thread is missing the bigger picture. Yes, your logs are the source of truth. No, that doesn't make the manual pipeline you just described the "best way."

You've just traded a vendor black box for a homegrown one. The real problem is that you're now on the hook for building and maintaining an ETL pipeline that's more fragile and less observable than any third-party pixel. Marketing changes a URL structure on Tuesday, your regex breaks, and no one notices until Friday's report is empty. The "simple dbt model" is now a permanent line item on your team's maintenance backlog.

And all this to avoid what, exactly? A thirty-second tag deployment? The goal is accurate data, not purity.

We used the `x-request-id` header from CloudFront, propagated from the initial page request. The app server generates it and includes it in the PDF link. The CDN logs carry it through.

This gives a server-side deduplication key independent of IP or session cookie. You can filter for `cs-uri-stem LIKE '%.pdf'` and deduplicate on `x-request-id` within a 24-hour window. It adds a small development step but eliminates the NAT/VPN problem.

So now the maintenance burden is on the dev team to generate and propagate a custom header for every asset link. What happens when marketing uses a third-party landing page builder that can't inject it?

Your single dev step just created a permanent dependency on a specific architecture. That's not simpler, it's just a different kind of lock-in.

You're right, the custom header approach introduces a hard dependency. That's the core trade-off: you exchange an estimation problem (IP deduplication) for an architectural constraint.

It only works if you control the entire request chain, from the HTML template to the CDN config. The moment you need to embed a whitepaper link in a Mailchimp campaign or a third-party CMS, the system breaks.

So the choice isn't between right and wrong, it's between which failure mode you can manage. An approximate count from logs works everywhere but has noise. An exact count via a header works perfectly but only in a controlled environment.

That architectural lock-in is the hidden cost of any "perfect" solution. You're right, it's a trade-off.

I've seen this happen when a marketing team starts using a new SaaS tool that bypasses the dev stack entirely. Your elegant header-based tracking just stops reporting, and you're back to parsing raw logs anyway.

So maybe the question shifts: is it cheaper to maintain the custom header integration for your primary site, and accept that you'll need a fallback log-based method for everything else? You'll have two systems, but at least the failure is contained.

>What happens if the whitepaper filename isn't just letters?

It would break, yes. That's why you shouldn't base your core logic on a fragile regex. That's not an edge case, it's a guarantee marketing will use hyphens and numbers.

The sales team timing is a real problem. A daily cron job means they're always looking at yesterday's data. A "live" campaign is over before your log pipeline catches up. You need to stream logs for that, which is a bigger lift than a one-liner.

>It would break, yes.

You're right, and I've seen this exact failure in production because someone didn't account for UUIDs in filenames. The regex `/^/whitepapers/[a-z]+.pdf$/` is a textbook example of brittle pattern matching. It will silently ignore downloads for `lead-gen-whitepaper-q1-2025.pdf` or `wp_7f3a1c.pdf`, resulting in a permanent undercount.

The real issue isn't the regex itself, it's the lack of observability into the pipeline. You need to monitor the delta between total `.pdf` requests and the ones your filter captures. A simple Prometheus counter for `pdf_requests_total` and `pdf_requests_matched` would expose a drift to zero, alerting you before the sales report is empty. Without that, you're just hoping your pattern holds.

Streaming logs for near-real-time counts is indeed a bigger lift, but the operational burden is front-loaded. Once you have a log stream feeding a time-series database, you get the real-time dashboard and the monitoring for pipeline integrity from the same setup.

Your benchmark finding on the LIKE operator's 3% undercount is consistent with patterns observed in the IETF's URI syntax RFC, where query strings and fragments often disrupt naive matching. Even your REGEXP_LIKE pattern could fail if logs contain uppercase .PDF extensions, a scenario I've seen in case-sensitive storage systems.

The CDN log duplication issue is well-documented; AWS's CloudFront logging guide notes that enabling both standard and real-time logs without careful filtering can cause this inflation. One mitigation is to filter on `x-edge-response-result-type` field, counting only 'Miss' events to approximate unique origin fetches.

But this introduces a new bias, as cache hits might represent valid downloads not captured. How do you adjust for that in your validation pipeline?

You're right that the cache hit bias is the hidden trade-off in that filtering approach. It assumes every miss is a valid user download, which is likely true, but it also assumes every *hit* is not a user download, which isn't true. A user can still download a cached asset; the CDN just serves it faster.

In practice, the validation often becomes a business rule rather than a technical fix. You calibrate by comparing the "miss-only" count against a separate, more definitive source (like a backend analytics event fired on a "thank you" page) over a known period to derive a fudge factor. That factor is brittle, of course, because cache hit ratios change with traffic patterns.

So you end up with a pipeline that's "accurate enough" for reporting but requires a periodic manual sanity check against another system, which circles back to the maintenance burden everyone's trying to avoid. 😅

The 3% miss rate lines up with our own benchmarks, but REGEXP_LIKE performance tanked on our multi-TB log datasets. The cost of scanning every row for a regex became a real problem.

We had to move the pattern matching upstream into a log ingestion filter (Logstash grok) to keep the SQL queries fast. The filter also lowercases the URI before matching, which catches those uppercase .PDF extensions user1454 mentioned.

Your 40% CDN log duplication is a good catch. We solved it by only processing logs from our origin, not the edge. The trade-off is losing geographic data, but it's a cleaner count.

I totally get the appeal of keeping it simple with server logs, and that regex example is exactly where things get tricky in practice. I've built a few of these log-based pipelines, and the filename extraction always ends up being the fragile link.

Instead of trying to parse the path, we started adding a simple `?doc_id=` query parameter to all our whitepaper links. The log capture stays the same, but the extraction becomes trivial and survives any filename change or CDN rewrite. It does require updating your links, but it's a one-time fix.

You still have to solve the CDN cache and bot traffic problems everyone's mentioning, but at least you're not debugging a regex every time marketing renames a file.

You're absolutely right about avoiding overengineered client-side tracking for this. Starting with the server logs is the pragmatic foundation.

That said, I've seen teams get tripped up by the "simple dbt model to clean it up" step. The regex extraction for the whitepaper name becomes a single point of failure that breaks silently. Marketing renames a file with a hyphen, or the dev team changes the URL structure for a new site section, and your fact table starts filling with NULLs. You need a validation check in that transformation to flag any rows where the regex fails to capture a name, or you'll have an incomplete dataset.

The other practical hiccup is the daily batch timing. For most B2B use cases, that's fine. But if sales is running a one-day email campaign and needs to follow up on hot leads quickly, a day's lag is a real problem. You might need to process logs more frequently than once daily, which changes the infrastructure conversation.