Eval Engineer

Objective

Turn failures and adversarial behavior into release-blocking proof that agent controls still hold.

What you’ll own

• Adversarial harnesses for policy, identity, approvals, and audit evidence
• Release checks that fail when required decisions or evidence disappear
• Regression scenarios derived from production and field failures
• The eval corpus as tools, MCP servers, and sub-agents change

Evidence of success

• A known bypass cannot return without failing a named regression scenario
• Releases stop when approval, scope, containment, or evidence behavior regresses
• Eval results identify the broken control and the evidence needed to reproduce it

Constraints you should know

• A passing score cannot replace a required security invariant
• Tests must exercise the real contract boundary, not only a mock
• Release-blocking checks need deterministic reproduction and an accountable owner

Working style

• Work in person with the founding team in San Francisco
• Write production code on the same CI path the tests protect
• Treat every field failure as a candidate durable regression

What we look for

• Built adversarial or red-team infrastructure that caught a real bug
• Shipped production code on a CI path
• Comfortable in agent frameworks such as MCP and in security tooling
• Shipped at least one release-blocking check someone tried to disable