Just tried out the new 'synthesize' beta feature on Elicit for a cloud security research task. My prompt was something like: "What are the common root causes for public S3 bucket data breaches?" The promise of a synthesized answer from multiple papers is fantastic—in theory. In practice, my initial feeling is... it's a bit thin.
It pulled from a decent number of sources, but the synthesis felt more like a slightly reworded abstract from the top 2-3 most relevant papers, rather than a truly integrated, comprehensive overview. It missed some nuanced but critical points I know are out there, like the specific IAM misconfigurations that often lead to bucket enumeration.
For example, a true synthesis should connect:
* The prevalence of wildcard actions (`s3:*`) in bucket policies.
* The danger of combining `"Principal": "*"` with `"Effect": "Allow"`.
* How these are often found in Terraform modules without proper `condition` blocks.
Instead, I got a high-level "misconfigured permissions" statement. Helpful, but not the deep, actionable insight I was hoping for from a "synthesis."
This feels a bit like when a cloud service launches a flashy new feature that's still in "preview." The foundation is there, and it's promising, but it's not quite ready to replace deep, manual analysis. I'm curious if others have had better results with different types of queries? Maybe it works better for more general topics than specific technical ones.
I'll keep testing it, because the *potential* for accelerating literature reviews is huge. But right now, I wouldn't rely on it as a single source of truth for building a threat model or a compliance checklist. You still need to go read those source papers yourself.
security by default