California AI bill SB 1047 aims to prevent AI disasters, but Silicon Valley warns it will cause one

Update: California’s Appropriations Committee passed SB 1047 with significant amendments that change the bill on Thursday, August 15. You can read about them here.

Outside of sci-fi films, there’s no precedent for AI systems killing people or being used in massive cyberattacks. However, some lawmakers want to implement safeguards before bad actors make that dystopian future a reality. A California bill, known as SB 1047, tries to stop real-world disasters caused by AI systems before they happen, and it’s headed for a final vote in the state’s senate later in August.

While this seems like a goal we can all agree on, SB 1047 has drawn the ire of Silicon Valley players large and small, including venture capitalists, big tech trade groups, researchers and startup founders. A lot of AI bills are flying around the country right now, but California’s Safe and Secure Innovation for Frontier Artificial Intelligence Models Act has become one of the most controversial. Here’s why.

What would SB 1047 do?

SB 1047 tries to prevent large AI models from being used to cause “critical harms” against humanity.

The bill gives examples of “critical harms” as a bad actor using an AI model to create a weapon that results in mass casualties, or instructing one to orchestrate a cyberattack causing more than $500 million in damages (for comparison, the CrowdStrike outage is estimated to have caused upwards of $5 billion). The bill makes developers — that is, the companies that develop the models — liable for implementing sufficient safety protocols to prevent outcomes like these.

What models and companies are subject to these rules?

SB 1047’s rules would only apply to the world’s largest AI models: ones that cost at least $100 million and use 10^26 FLOPS during training — a huge amount of compute, but OpenAI CEO Sam Altman said GPT-4 cost about this much to train. These thresholds could be raised as needed.

Very few companies today have developed public AI products large enough to meet those requirements, but tech giants such as OpenAI, Google, and Microsoft are likely to very soon. AI models — essentially, massive statistical engines that identify and predict patterns in data — have generally become more accurate as they’ve grown larger, a trend many expect to continue. Mark Zuckerberg recently said the next generation of Meta’s Llama will require 10x more compute, which would put it under the authority of SB 1047.

When it comes to open source models and their derivatives, the bill determined the original developer is responsible unless another developer spends three times as much creating a derivative of the original model.

The bill also requires a safety protocol to prevent misuses of covered AI products, including an “emergency stop” button that shuts down the entire AI model. Developers must also create testing procedures that address risks posed by AI models, and must hire third-party auditors annually to assess their AI safety practices.

The result must be “reasonable assurance” that following these protocols will prevent critical harms — not absolute certainty, which is of course impossible to provide.

Who would enforce it, and how?

A new California agency, the Frontier Model Division (FMD), would oversee the rules. Every new public AI model that meets SB 1047’s thresholds must be individually certified with a written copy of its safety protocol.

The FMD would be governed by a five-person board, including representatives from the AI industry, open source community and academia, appointed by California’s governor and legislature. The board will advise California’s attorney general on potential violations of SB 1047, and issue guidance to AI model developers on safety practices.

A developer’s chief technology officer must submit an annual certification to the FMD assessing its AI model’s potential risks, how effective its safety protocol is and a description of how the company is complying with SB 1047. Similar to breach notifications, if an “AI safety incident” occurs, the developer must report it to the FMD within 72 hours of learning about the incident.

If a developer fails to comply with any of these provisions, SB 1047 allows California’s attorney general to bring a civil action against the developer. For a model costing $100 million to train, penalties could reach up to $10 million on the first violation and $30 million on subsequent violations. That penalty rate scales as AI models become more expensive.

Lastly, the bill includes whistleblower protections for employees if they try to disclose information about an unsafe AI model to California’s attorney general.

What do proponents say?

California State Senator Scott Wiener, who authored the bill and represents San Francisco, tells TechCrunch that SB 1047 is an attempt to learn from past policy failures with social media and data privacy, and protect citizens before it’s too late.

“We have a history with technology of waiting for harms to happen, and then wringing our hands,” said Wiener. “Let’s not wait for something bad to happen. Let’s just get out ahead of it.”

Even if a company trains a $100 million model in Texas, or for that matter France, it will be covered by SB 1047 as long as it does business in California. Wiener says Congress has done “remarkably little legislating around technology over the last quarter century,” so he thinks it’s up to California to set a precedent here.

When asked whether he’s met with OpenAI and Meta on SB 1047, Wiener says “we’ve met with all the large labs.”

Two AI researchers who are sometimes called the “godfathers of AI,” Geoffrey Hinton and Yoshua Bengio, have thrown their support behind this bill. These two belong to a faction of the AI community concerned about the dangerous, doomsday scenarios that AI technology could cause. These “AI doomers” have existed for a while in the research world, and SB 1047 could codify some of their preferred safeguards into law. Another group sponsoring SB 1047, the Center for AI Safety, wrote an open letter in May 2023 asking the world to prioritize “mitigating the risk of extinction from AI” as seriously as pandemics or nuclear war.

“This is in the long-term interest of industry in California and the US more generally because a major safety incident would likely be the biggest roadblock to further advancement,” said director of the Center for AI Safety, Dan Hendrycks, in an email to TechCrunch.

Recently, Hendrycks’ own motivations have been called into question. In July, he publicly launched a startup, Gray Swan, which builds “tools to help companies assess the risks of their AI systems,” according to a press release. Following criticisms that Hendrycks’ startup could stand to gain if the bill passes, potentially as one of the auditors SB 1047 requires developers to hire, he divested his equity stake in Gray Swan.

“I divested in order to send a clear signal,” said Hendrycks in an email to TechCrunch. “If the billionaire VC opposition to commonsense AI safety wants to show their motives are pure, let them follow suit.”

What do opponents say?

A growing chorus of Silicon Valley players oppose SB 1047.

Hendrycks’ “billionaire VC opposition” likely refers to a16z, the venture firm founded by Marc Andreessen and Ben Horowitz, which has strongly opposed SB 1047. In early August, the venture firm’s chief legal officer, Jaikumar Ramaswamy, submitted a letter to Senator Wiener, claiming the bill “will burden startups because of its arbitrary and shifting thresholds,” creating a chilling effect on the AI ecosystem. As AI technology advances, it will get more expensive, meaning that more startups will cross that $100 million threshold and will be covered by SB 1047; a16z says several of their startups already receive that much for training models.

Fei-Fei Li, often called the godmother of AI, broke her silence on SB 1047 in early August, writing in a Fortune column that the bill will “harm our budding AI ecosystem.” While Li is a well-regarded pioneer in AI research from Stanford, she also reportedly created an AI startup called World Labs in April, valued at a billion dollars and backed by a16z.

She joins influential AI academics such as fellow Stanford researcher Andrew Ng, who called the bill “an assault on open source” during a speech at a Y Combinator event in July. Open source models may create additional risk to their creators, since like any open software, they are more easily modified and deployed to arbitrary and potentially malicious purposes.

Meta’s chief AI scientist, Yann LeCun, said SB 1047 would hurt research efforts, and is based on an “illusion of ‘existential risk’ pushed by a handful of delusional think-tanks,” in a post on X. Meta’s Llama LLM is one of the foremost examples of an open source LLM.

Startups are also not happy about the bill. Jeremy Nixon, CEO of AI startup Omniscience and founder of AGI House SF, a hub for AI startups in San Francisco, worries that SB 1047 will crush his ecosystem. He argues that bad actors should be punished for causing critical harms, not the AI labs that openly develop and distribute the technology.

“There is a deep confusion at the center of the bill, that LLMs can somehow differ in their levels of hazardous capability,” said Nixon. “It’s more than likely, in my mind, that all models have hazardous capabilities as defined by the bill.”

But Big Tech, which the bill directly focuses on, is panicked about SB 1047 as well. The Chamber of Progress — a trade group representing Google, Apple, Amazon and other Big Tech giants — issued an open letter opposing the bill saying SB 1047 restrains free speech and “pushes tech innovation out of California.” Last year, Google CEO Sundar Pichai and other tech executives endorsed the idea of federal AI regulation.

U.S. Congressman Ro Khanna, who represents Silicon Valley, released a statement opposing SB 1047 on Tuesday. He expressed concerns the bill “would be ineffective, punishing of individual entrepreneurs and small businesses, and hurt California’s spirit of innovation.”

Silicon Valley doesn’t traditionally like when California sets broad tech regulation like this. In 2019, Big Tech pulled a similar card when another state privacy bill, California’s Consumer Privacy Act, also threatened to change the tech landscape. Silicon Valley lobbied against that bill, and months before it went into effect, Amazon founder Jeff Bezos and 50 other executives wrote an open letter calling for a federal privacy bill instead.

What happens next?

On August 15, SB 1047 will be sent to the California Senate’s Assembly floor with whatever amendments get approved. That’s where bills “live or die” in California’s Senate, according to Wiener. It’s expected to pass, given its overwhelming support from lawmakers thus far.

Anthropic submitted a number of suggested amendments to SB 1047 in late July, which Wiener says he and California’s Senate policy committees are actively considering. Anthropic is the first developer of a state-of-the-art AI model to publicly signal it’s willing to work with Wiener on SB 1047, even though it doesn’t support the bill as it stands. This was largely seen as a win for the bill.

Anthropic’s proposed changes include getting rid of the FMD, reducing the Attorney General’s power to sue AI developers before a harm occurs, and getting rid of the whistleblower protections provision in SB 1047. Wiener says he’s generally positive about the amendments, but needs approval from several Senate policy committees before adding them to the bill.

If SB 1047 passes the Senate, the bill will be sent to California Governor Gavin Newsom’s desk where he will ultimately decide whether to sign the bill into law before the end of August. Wiener says he has not spoken to Newsom about the bill, and does not know his position.

This bill would not go into effect immediately, as the FMD is set to be formed in 2026. Further, if the bill does pass, it’s very likely to face legal challenges before then, perhaps from some of the same groups that are speaking up about it now.

Correction: This story originally referenced a previous draft of SB 1047’s language around who is responsible for fine-tuned models. Currently, SB 1047 says the developer of a derivative model is only responsible for a model if they spend three times as much as the original model developer did on training.