In July 2023, Meta released Llama 2 with open weights. Within 48 hours, the model had been fine-tuned, quantized, and redistributed on Hugging Face dozens of times, including variants stripped of the safety training that Meta had built in. Within a week, researchers had demonstrated that the stripped-down model would comply with requests that the original refused. Within a month, a version of that model was being used to generate phishing emails at a scale that no human team could match. None of this was a surprise to anyone who had thought carefully about open weights. It was the predictable outcome of giving away a capability that was designed to be controlled.
That story is the open-source paradox in miniature. The same release that made the model auditable also made it weaponizable. The same transparency that allowed researchers to study its biases also allowed bad actors to strip its guardrails. The same freedom that let the community build on it also let the community dismantle it.
Here’s the claim I want to test: the open-source debate in AI is not a debate between safety and freedom. It’s a debate between two different theories of risk, and neither side has done the work to figure out which theory is right. The result is a policy conversation that is loud, polarized, and mostly unmoored from evidence. The teams that figure out how to navigate the paradox will have a structural advantage. The ones that don’t will keep arguing past each other while the models get released anyway.
The Two Theories of Risk
Let me be precise about the disagreement, because “open source is good/bad” is too vague to be useful.
Theory 1: Transparency is safety.
The argument goes like this. Closed models are black boxes. You can’t audit them, you can’t test them, you can’t find their failure modes. Open models can be audited by anyone, which means their failure modes get found faster, and the fixes get distributed faster. A closed model that is unsafe stays unsafe until the company fixes it. An open model that is unsafe gets fixed by the community, and the fix propagates.
This theory has real evidence behind it. Open-source software has a better security track record than closed-source software, on average, because more eyes find more bugs. Open models have been used to find biases, failure modes, and jailbreaks that closed models hid. The community has produced safety tooling that no single company would have built.
Theory 2: Capability is danger.
The argument goes like this. A model that can generate phishing emails, synthesize pathogens, or write persuasive disinformation is dangerous regardless of whether it’s auditable. The audit doesn’t remove the capability. It just documents it. And the documentation is public, which means the capability is public. Open weights mean that anyone, anywhere, can use the capability for anything, and no amount of auditing can prevent that.
This theory also has real evidence. The Llama 2 stripping happened. The jailbreaks happened. The phishing emails happened. The open weights made it all possible, and the safety training that Meta built in was removed within days.
Both theories are internally coherent. Both have evidence. And they point in opposite directions on the same policy question.
The problem is that neither theory has been tested at the scale that would settle the question. We don’t have data on how much open weights increase the risk of catastrophic misuse, because the misuse hasn’t happened at catastrophic scale yet. We have data on small-scale misuse, and both sides interpret it in their favor.
The Sacred Cow: “Open Source Always Wins”
Here’s the belief I want to dismantle, and it’s the default assumption in the developer community.
The claim goes like this: open source always wins. It wins on quality, it wins on adoption, it wins on security, it wins on innovation. The closed-source companies will eventually be outcompeted by the open-source community, because the community is larger, more motivated, and more innovative.
This is true for a lot of software. It’s not obviously true for AI, and the assumption that it is has led the community to systematically underweight the risks.
Here’s why AI is different.
- The capability is not in the code. It’s in the weights.
Open-source software gives you the code, and the code is auditable. Open-weight AI gives you the weights, and the weights are not auditable in the same way. You can inspect the code, but you can’t inspect the weights. The weights are a compressed representation of the training data, and the training data is not in the weights. The weights are a capability, and the capability is what’s dangerous.
- The marginal cost of replication is zero.
Open-source software has a marginal cost of replication that is close to zero, but the software still requires expertise to deploy. Open-weight AI has a marginal cost of replication that is exactly zero, and the expertise required to deploy it is dropping every month. A model that is released today can be fine-tuned and deployed by a small team within weeks. The barrier to misuse is falling faster than the barrier to safe use.
- The misuse is not visible.
Open-source software has visible failure modes. A bug in the code produces a crash, and the crash is visible. Open-weight AI has invisible failure modes. A model that has been fine-tuned to generate disinformation looks exactly like a model that has been fine-tuned for legitimate purposes. The misuse is not visible in the weights, and it’s not visible in the outputs unless you know what to look for.
The contrarian take: “open source always wins” is a claim about software, not about AI. The properties that make open source good for software (auditability, replicability, community) are the same properties that make open weights dangerous for AI (strippable safety, zero-cost replication, invisible misuse). The community has been applying a software heuristic to an AI problem, and the heuristic is wrong.
This doesn’t mean open weights are bad. It means the case for open weights has to be made on AI-specific grounds, not on software grounds. The software case is not strong enough.
What Actually Distinguishes the Risks
If the debate is between two theories of risk, the useful question is: what kind of risk are we actually talking about?
There are three different kinds of risk, and they have different implications for openness.
- Misuse risk.
A bad actor uses the model to do something harmful. Phishing, disinformation, harassment, fraud. This risk is higher for open models, because the bad actor can strip the safety training and use the model without any oversight.
The standard response is to say that misuse risk is small, because the bad actors could use closed models too. This is true for some misuse, but not for all. The bad actor who uses a closed model has to pay for it, has to work within the provider’s terms of service, and has to accept that the provider can revoke access. The bad actor who uses an open model has none of these constraints.
- Accident risk.
A well-intentioned user does something harmful because the model has a failure mode they didn’t anticipate. This risk is lower for open models, because the community can find the failure modes and fix them. The open model gets audited, and the audit reduces the accident risk.
This is the strongest argument for openness. The community finds the failure modes that the company missed, and the fixes propagate. The closed model stays vulnerable to the failure modes that the company didn’t find.
- Concentration risk.
A small number of companies control the models, and the concentration of power is itself a risk. This risk is lower for open models, because the power is distributed. The open model reduces concentration, which reduces the risk of any single company making a catastrophic decision.
This is the argument that the open-source community makes most often, and it’s the one that gets the least attention from policymakers. Concentration risk is real, and it’s the risk that open weights are best positioned to address.
The pattern: the three risks point in different directions. Misuse risk favors closed models. Accident risk and concentration risk favor open models. The right policy depends on which risk you think is most important, and that depends on your threat model.
What a Good Policy Would Look Like
I don’t want to leave you with nothing but critique. There is a policy that would address all three risks, and it’s worth being specific about it.
- Tiered openness.
Not all open weights are the same. A small model that can write poetry is different from a large model that can synthesize pathogens. The policy should distinguish between capability tiers and apply different openness requirements to each tier. Small models can be fully open. Large models should require some form of registration or audit before release.
- Capability-specific safety training.
The safety training should be specific to the capabilities that are dangerous. A model that can generate phishing emails should have specific safety training for phishing, not just general safety training. The training should be auditable, so that the community can verify it.
- Post-release monitoring.
Open weights can’t be recalled once they’re released, but they can be monitored. The policy should require some form of post-release monitoring, so that the community can detect misuse and respond. The monitoring should be transparent, so that it doesn’t become a form of surveillance.
- Liability for downstream misuse.
The people who release open weights should have some liability for downstream misuse. Not full liability, but some. The liability should be proportional to the capability of the model and the effort the releaser made to reduce the risk.
A rough sketch:def openness_policy(model):
capability = assess_capability(model)
risk = assess_misuse_risk(capability)
if risk < LOW_RISK_THRESHOLD:
return {"openness": "full", "requirements": []}
elif risk < MEDIUM_RISK_THRESHOLD:
return {
"openness": "full",
"requirements": ["safety_audit", "post_release_monitoring"],
}
else:
return {
"openness": "restricted",
"requirements": [
"pre_release_audit",
"capability_specific_safety_training",
"post_release_monitoring",
"downstream_liability",
],
}
The point is not the specific thresholds. The point is the structure: different capability tiers get different openness requirements. The structure is the policy. The thresholds can be adjusted as we learn more.
What To Actually Do
Three moves, concrete enough to start on Monday.
- Build capability assessment into your release process.
If you’re releasing open weights, assess the capability of the model before you release it. Not just the benchmark scores, but the specific capabilities that are dangerous. Document the assessment. Make it public. The public assessment is the first step toward responsible release.
- Build post-release monitoring into your infrastructure.
If you’re releasing open weights, build a way to monitor for misuse. Not surveillance of users, but detection of specific misuse patterns. The monitoring should be transparent, so that the community can verify it. The monitoring is the thing that makes open weights defensible.
- Push for tiered openness in policy.
If you’re in a position to shape policy, push for tiered openness. Not a blanket ban on open weights, not a blanket permission. A tiered structure that matches openness to capability. The tiered structure is the policy that can actually address all three risks, and it’s the policy that neither side of the debate is currently advocating.
The Mental Model
Here’s the lens I want you to keep.
The open-source paradox is not a paradox about freedom versus safety. It’s a paradox about two different theories of risk, both of which are partially right. Transparency does reduce accident risk and concentration risk. It does not reduce misuse risk, and it can increase it. The right policy is not to choose a side. It’s to build a structure that gets the benefits of transparency while managing the misuse risk.
The interesting question is not whether open weights are good or bad. It’s whether we’ll build the tiered structure that matches openness to capability. The structure is possible. The structure is not being built. The teams that build it will have a structural advantage, because they’ll be able to release open weights without being irresponsible. The teams that don’t will keep releasing models into a policy vacuum, and the vacuum will eventually be filled by people who understand the technology less than they do.
What’s your read? If you’ve worked on open-weight models, how did you handle the misuse risk? And if you’re building policy, what tier of openness do you think is defensible? Drop it in the comments. I’m genuinely curious whether the tiered structure is as achievable in practice as it looks on paper.
