Office of Data & Innovation
2023 – present
Responsible AI · Human-in-the-loop evaluation
Evaluating generative AI for the public interest
California's Executive Order on generative AI arrived with a mandate and no playbook. What grew from it is a statewide, human-in-the-loop evaluation practice — testing frontier models (OpenAI, Anthropic, Gemini, Copilot, Amazon Q) against the real workflows of state analysts across dozens of departments, with staff trained to run their own studies so the capability outlasts any one mandate. AI is both the subject and the tool here: models help code transcripts and cluster first-pass themes, with a human judgment gate on every conclusion. Responsible AI as practice, not slogan — evidence about how these tools actually serve the people who depend on them.
validation ~50% faster · ~$200M in cost savings to date