{"version":1,"type":"rich","provider_name":"Libsyn","provider_url":"https:\/\/www.libsyn.com","height":90,"width":600,"title":"Open Questions On Open Weights","description":"Last month, some of Silicon Valley\u2019s biggest companies signed an open letter supporting open-weights AI. Open weights AI is like open-source software, where the creator makes the raw code publicly available for free download. It\u2019s good insofar as it\u2019s the only way an AI can truly be the user\u2019s property, as opposed to something that companies like OpenAI or Anthropic temporarily let you use subject to their corporate guidelines and increasingly-nanny-state-like restrictions. If AI becomes the linchpin of the future, open weights AI feels like the sort of thing that could be the difference between being free yeomen vs. corporate serfs. It\u2019s bad insofar as it removes the possibility of gatekeeping and lets criminals commit crimes with it. Open weights AI could be used for hacking, child pornography, harassment, or terrorism (the weights can\u2019t commit the terrorism themselves, but they could give bomb-making or bioweapon-making advice). Since AIs have gotten very good - maybe superhuman - at hacking lately, the specter of a world where anyone can hack any site has gotten people grumbling that maybe open weights should be banned. It doesn\u2019t help that China produces the best open weights AI, making the idea seem foreign and almost unpatriotic. Proponents counter that \u201cwhen AI is outlawed, only outlaws will have AI\u201d, arguing that bad people will get open weights AI regardless, and good people can use open weights AI to defend themselves. With the recent open letter, companies including Microsoft, NVIDIA, OpenAI, Intel, Amazon, Meta, Hugging Face, and over a hundred others have come out in favor of this position. Who\u2019s leading the other side? Nobody\u2019s admitted to it. Some parts of the Trump administration lean anti-open-weights on China hawk grounds, but have stopped short of explicitly asking for a full ban. Anthropic, the most notable omission on the pro-open-weights letter, made an ambiguous statement supporting \u201copen-weights models that don\u2019t have dangerous capabilities\u201d - but the industry expects open weights models to have dangerous hacking capabilities within a year, and AFAICT the letter didn\u2019t address that beyond inviting readers to draw the obvious conclusion. In the absence of a more obvious opponent, some open weights supporters suspect our conspiracy - the loose band of AI safety advocates, effective altruists, rationalists, and pause activists who worry about existential risk from superintelligence. This is a reasonable inference. By design, open weights AI is outside centralized control, and so impossible to permanently align against either human misuse (eg terrorism) or loss of control (eg AI turning against humans). Even if its creator trains it not to hack, anybody in the world can download the weights and retrain the AI to hack all day long. But in fact, most AI safety organizations have remained quietly neutral, and I don\u2019t know of any who make this a centerpiece of their activism1 (though I\u2019m not 100% up-to-date on the whole landscape; if you know of one, tell me). A few have proposed policies that are contingently incompatible with open weights AI existing, but they all frame it as collateral damage rather than something they\u2019re excited about eliminating. I\u2019m also neutral about open weights AI. I think it probably won\u2019t be long-term sustainable, but I\u2019m happy to wait for this to become clear in the normal course of things rather than expend effort and political capital to ban it immediately. https:\/\/www.astralcodexten.com\/p\/open-questions-on-open-weights ","author_name":"Astral Codex Ten Podcast","author_url":"http:\/\/sscpodcast.libsyn.com\/website","html":"