SACRAMENTO, California — SACRAMENTO, Calif. (AP) — Efforts in California to implement the nation’s first safety measures for its largest artificial intelligence systems was approved Wednesday in a key vote that could pave the way for U.S. regulation of the rapidly developing technology.
The proposal, which aims to reduce potential risks created by AI, would require companies to test their models and disclose their safety protocols to prevent the models from being manipulated to, for example, take out the state’s power grid or help build chemical weapons — scenarios that could be possible in the future, experts say, given the rapid advances in the industry.
The bill is one of hundreds of lawmakers set to vote on during the final week of the session, when Gov. Gavin Newsom has until the end of September to decide whether to sign it, veto it or let it pass without his signature.
The measure narrowly passed the Assembly on Wednesday and must go through a final vote in the Senate before it reaches the governor’s desk.
Supporters said it would establish some of the first much-needed safety rules for large-scale AI models in the United States. The bill targets systems that require more than $100 million in data to train. No current AI model has reached that threshold.
“It’s time for Big Tech to abide by some sort of regulation, not much, but something,” Republican Assemblyman Devon Mathis said Wednesday in support of the bill. “The last thing we need is an electrical grid going down, water systems going down.”
The proposal, authored by Democratic Sen. Scott Wiener, has drawn fierce opposition from venture capital firms and tech companies including OpenAI, Google and Meta, the parent company of Facebook and Instagram. They say safety regulations should be set by the federal government and that California’s law targets developers rather than those who use and exploit AI systems to cause harm.
A group of several members of the California House of Representatives also opposed the bill, with former House Speaker Nancy Pelosi calling it well-intentioned but poorly informed.”
Chamber of Progress, a left-leaning, Silicon Valley-funded industry group, said the bill is “based on science fiction fantasies about what AI might look like.”
“This bill has more in common with Blade Runner or The Terminator than it does with the real world,” Senior Tech Policy Director Todd O’Boyle said in a statement after Wednesday’s vote. “We should not be holding back California’s world-leading economic sector because of a theoretical scenario.”
The legislation has the support of Anthropic, an AI startup backed by Amazon and Google, after Wiener amended the bill earlier this month to incorporate some of the company’s suggestions. The current bill would have removed the penalty for perjury, limited the attorney general’s authority to prosecute violators and limited the responsibilities of a new AI regulatory agency. Social media platform X owner Elon Musk also threw his support behind the bill this week.
Anthropic wrote in a letter to Newsom that the bill is critical to preventing catastrophic misuse of powerful AI systems and that “its benefits likely outweigh its costs.”
Wiener said his legislation took a “light touch.”
“Innovation and safety can go hand in hand, and California is leading the way,” Weiner said in a statement after the vote.
Earlier this week, he also criticized critics who dismissed the potential catastrophic risks of powerful AI models as unrealistic: “If they really think the risks are fake, then the bill should be no problem at all.”
Wiener’s proposal is one of the dozens of AI accounts California lawmakers this year proposed measures to increase public trust, combat algorithmic discrimination and ban deepfakes that feature elections or pornography. As AI increasingly impacts Americans’ daily lives, state lawmakers have tried to strike a balance between curbing the technology and its potential risks without stifling a thriving domestic industry.
California, home to 35 of the world’s top 50 AI companies, is an early adopter of AI technologies and could soon deploying generative AI tools including to tackle traffic congestion on the highway and road safety.
Newsom, who declined to comment on the measure earlier this summer, warned against overregulation of AI.