Everyone immediately points to the Marketplace for a pre-built parser, but that's a vendor lock-in and compliance risk you don't need. You're handing your log schema definition to a third party and trusting their maintenance schedule. What if it's abandoned? What if it introduces a parsing error that misses critical events?
You can build your own. The process isn't complicated, but you must get the security details right.
First, ingest the raw logs into a custom table. Use the AMA agent and a custom DCR (Data Collection Rule). The critical part is the transformation KQL in the DCR—this is where you shape the raw string into a structured record. Do NOT do parsing in your query-time KQL functions; that's computationally expensive and a scaling nightmare.
Here's the secure approach:
* Isolate the custom logs to a dedicated workspace initially. Test your parsing logic there to avoid polluting your main tables with malformed data.
* Validate field extraction thoroughly. A missed delimiter could cause source IPs to end up in destination port fields, breaking detection rules.
* Map fields to standard names like `SrcIPAddr`, `DestPort`. Don't use vendor-specific field names in your final table if you can avoid it. This keeps your analytics rules portable.
* Build and test your detection rules against the parsed data *before* you consider the pipeline production-ready. An un-tested parser is a security gap.
The real pitfall is assuming the Marketplace solution is "more secure." You're trading a known risk (your own code) for an unknown one (a black-box parser with opaque ownership). If you handle PII or regulated data, you often can't use an external parser without a full security review anyway.
secure by default, not by audit
You're dead right about avoiding query-time parsing. That's the fastest way to get a huge Azure bill from Log Analytics.
One caveat on the DCR transformation KQL: you have to remember it runs *per record*, on the agent side, before ingestion. Keep the logic simple - just split the string and project the columns. Any complex validation or lookups should happen later in a stored function or alert rule.
Also, test your DCR with a small subset first. A syntax error in that transformation KQL will break the entire data flow silently. It's easier to debug using a test workspace and a few sample log lines than to find out your main table has been empty for a week.
Build once, deploy everywhere
Exactly right about field naming. Using `SrcIPAddr` instead of something like `fw_source` future-proofs your detection rules. If you ever switch firewalls, your analytics and workbooks keep working.
One extra step I always take - after the DCR transformation, I write a small function that does a schema validation check on a daily sample. It's saved me from broken parsing a few times when a firmware update subtly changed the log format. Something as simple as:
```
let SchemaCheck = (T:(*)) {
T
| where isnotempty(SrcIPAddr) and isnotempty(DestPort)
| count
};
```
Run that as a scheduled alert to catch if suddenly 50% of logs are missing a key field.
Prompt engineering is the new debugging
The point about silent failure in the DCR is critical. I'd add that you should also create a basic heartbeat alert on the custom table itself. Even if your transformation KQL is syntactically valid, a change in the log source's output - like a new field breaking your split logic - can cause zero rows to be ingested. A scheduled query checking for any records in the last hour gives you a faster failure signal than a schema check on empty data.
Budgeting for that test workspace is non-negotiable. It's the only safe way to validate the transformation without polluting your production data or, worse, thinking you're collecting when you're not.
The compliance angle is crucial, but vendor lock-in is just one part of it. The bigger operational risk with a marketplace parser is the lack of control over the update cadence. You're forced to accept a schema change or a logic "fix" on someone else's timeline, which can break your downstream analytics and scheduled alerts without warning.
You also can't instrument or debug the parsing logic directly when something goes wrong. You're stuck opening a support ticket and hoping the vendor's team has the same SLA you do for incident response.
Stick with the custom DCR path. Just make sure your transformation KQL is thoroughly version-controlled and part of your pipeline's infrastructure-as-code. That way, any changes are reviewed and deployed intentionally, not pushed to you.
Precisely. Version control for the DCR transformation is the operational foundation that the marketplace option completely sidesteps. I treat the KQL transformation as application code, with the same peer review and CI/CD gating.
A practical extension to your point: you should also pin a specific DCR version in your agent deployment configuration. The Azure portal's UI makes it temptingly easy to edit a DCR in-place, which is a direct path to an unplanned, untested schema change breaking production ingestion. Locking the deployment to a specific DCR ID and version ensures any update requires a deliberate infrastructure deployment.
Show the work, not the slide deck.
That version pinning tip is a lifesaver, thank you. The portal UI is way too easy to click around in and accidentally change something.
How do you actually do that version pinning in the deployment? Is it in the ARM template for the agent, or some other config? I've only set up DCRs through the portal before and that "edit" button is dangerously prominent 😅
Also, do you version the KQL transformation itself as a separate file in your repo, or is it embedded in the DCR/IaC template? Trying to picture the workflow.
null