Relevance AI for recruitment: Screening resumes automatically. Any success stories?

Great questions. We ran a pilot with it last quarter. The setup was surprisingly straightforward, connecting via API to our Greenhouse ATS for a batch of resumes. It did a decent job on standard tech resumes, but we found its handling of non-linear career paths (career switchers, project-based roles) was a weak spot.

The time saving was real, cutting initial screening for that batch by about 70%. However, our big lesson was that you absolutely must build in a human review layer for the "maybe" pile the agent creates. We caught a fantastic candidate there that the system had undervalued because their experience keywords didn't match our JD exactly. My tip is to use the scoring as a prioritization tool, not a final gate.

I agree with the earlier comment about using it as a prioritization tool. We had a similar experience, but the setup answer depends on your stack. Since you mentioned Notion, their API connector was pretty solid for pulling in job descriptions directly. For resumes, we used the upload feature initially, which was fine for small batches. For anything larger, you'll want that direct ATS connection to avoid a manual step.

The format handling is decent, but we saw it stumble on some creative/design resumes where the visual layout threw off the text parsing. It's gotten better, but I'd still run a test batch of your own typical resumes first. The real time save came from filtering out clear mismatches, which let our recruiters focus. Just be prepared to tweak the scoring criteria a few times - our first run was way too keyword-strict.

That's a really solid breakdown, and I think your tip about the "maybe" pile is the most important takeaway here. I've seen the same pattern with other AI screening tools, not just Relevance AI. We tried a similar approach with a different tool for a customer support role, and the system actually over-penalized someone who had a ton of freelance and contract gigs even though they were clearly the best communicator in the batch. The keyword matching just couldn't see past the job titles.

Curious what you used for your scoring criteria tweaks? Did you adjust the weighting for certain skills, or did you add specific "bonus" keywords to catch those non-linear paths? I'm wondering if that's the secret sauce to making the "maybe" pile smaller and more accurate.

You've put your finger on the key operational challenge. In my experience, adjusting weightings and adding bonus keywords is necessary, but it's not a set-and-forget secret sauce. It's an iterative calibration process.

We moved beyond simple keyword weighting to implementing what I call "concept tagging." For a project management role, we added tags like "led a cross-functional team" or "managed a budget," and instructed the agent to scan for phrases demonstrating those concepts, regardless of job title. This helped surface those non-traditional candidates. However, this created a new problem - it made the scoring less transparent, as a candidate could hit a high-value concept tag but still score low overall if they were missing core hard skills.

The "maybe" pile's size is less important than its composition. Our goal was to make it contain primarily these interesting edge cases, not just parsing errors. We accepted that to do that, we had to regularly review the pile's contents and adjust the tags, which itself becomes a minor overhead.