Evaluating generative AI for the public interest
How California turned an executive order on generative AI into a repeatable, human-centered evaluation — and, along the way, taught a government to run its own research.
An executive order, and a sandbox to answer it in
In 2023, Executive Order N-12-23 asked California to pursue the benefits of generative AI for its residents while weighing the risks — a mandate short on specifics and long on stakes. To make the experiment safe, the state built what amounted to a sandbox: a space set apart from the usual procurement machinery, where two competing large language models could be trialed against each real use case, with vendors supplying their models for a nominal dollar during the evaluation.
…to realize the benefits of GenAI for the good of all California residents… while balancing the benefits and risks of these new technologies. — Executive Order N-12-23
Eight problems, five people, and no data to start from
Eight departments arrived with eight very different problems, each with its own product team behind it. The core question none of them could answer alone — would these tools actually help the people using them? — fell to a five-person team at the Office of Data & Innovation. There was no internal user-data source to lean on; the research had to be built from scratch.
- Tax & Fee — help staff answer taxpayer questions
- Transportation — predict traffic congestion
- Transportation — reduce roadway risk to pedestrians
- Public Health — and four more, department by department
Teach a government to fish
The response was less a study than a capability. Rather than run eight evaluations and leave, the team used the momentum of the moment to democratize research itself — training department staff to conduct their own studies, so the practice would outlast the pilot. It rested on four human-centered principles:
- 01Prioritize people's needs. Focus on the actual needs of Californians, not the features on offer.
- 02Define the real problem. Make sure the team is solving the right thing before building.
- 03Deliver early and often. Get business and user value into the world early, then iterate.
- 04Measure against outcomes. Evaluate progress by what changes for users, not by activity.
Each department appointed its own pilot lead, product lead, and research lead; a steering-committee cadence — monthly for executives, twice-monthly for the proof-of-concept teams — kept everyone aligned as findings formed.
Both halves of the craft — and a hard line around independence
Good evaluation needs both hands. Qualitative work asked why — what staff thought and felt, where a tool helped or got in the way; quantitative work asked how much, and how far a finding generalized. The team drew a firm line between two things that get confused during a trial: user-acceptance testing (does the feature work as built — run by the vendor) and user research (does it create value for the people using it — run independently).
Staff learned to write "people questions" — probing perceived value ("What benefits, if any, do you see in the tool?") and perceived risk ("What concerns, if any, do you have?") — and to ground usability tasks in real scenarios rather than abstractions, so what surfaced reflected the actual work.
Independent evidence — and a capability that stayed
Holy cow — I didn't realize how important independent evaluation through user research would be until we got to the end. — Round 1 GenAI Pilot Lead
When vendors' claims and users' experience diverged — and they did — the research became the state's source of truth. Independent evaluation gave executives a defensible basis for choosing between competing systems: five GenAI solutions were selected for full production on the strength of the findings. The pilot has since been formalized as a standing program for evaluating new technology while reducing risk, and requests for the research training are up 300%.