*Recently, the UK AI Safety Institute rebranded itself to the AI "Security" Institute. Reception to this has mostly been warm, but should it have been?* #### Security is a natural, if sad, response to the current climate In a world of fragmenting global coordination and cooperation, rising nationalism, and accelerating AI progress, it is unsurprising, if disappointing, that the discourse around AI has turned towards the "race" framing. The refrain of "we are in a race with China" has become more common than it once was. This wasn't necessarily inevitable - there are at least metaphysically possible worlds where global governance off the back of events like Bletchley and Seoul, in conjunction with the UN, informed by international reports by luminaries like Bengio, really took off. Nonetheless, this is not the world we actually find ourselves in. This is Leopold Aschenbrenner's world[^1]. Frontier labs are probably on a countdown, officially or unofficially, to nationalisation. This might be "hard" - where governments fully take over - or, probably more likely, "soft": where defence departments add AI to various technology export control lists, become larger and larger customers, and beef up security (adding compliance requirements, inserting personnel, you name it). We might see what Aschenbrenner and others like him simply refer to as "The Project": all the top talent goes dark, secreted away somewhere to focus on building AGI (or ASI) like they did with the nuclear bomb. But why this focus on national defence, you may ask? This is better covered elsewhere,[^6] but at a glance: - Terrorists - Hostile nations - Organised crime #### It's a politically savvy move as well Defence also provides other political cover... Now, some might say that this is overly cynical. The UK AISI says this about the rebrand[^2]: ... In some ways this is old news - Ian Hogarth, chair of the UK AISI, wrote about "AI nationalism" back in 2018.[^3] ... #### Bootleggers and Baptists In 1930s America, two key groups threw their weight behind Prohibition.[^4] Baptists got up in their pulpits and decried the evils of alcohol - a moral cause, full of noble justifications. But behind the scenes, who else was pushing for the restrictions? Bootleggers, who stood to gain a practical monopoly on the trade. This isn't to decry the Baptists as on the payroll of the bootleggers (though probably some were, in some way). Rather, there can be both good and bad reasons for a thing - so it's necessary, but not sufficient, to listen just to the most desirable arguments. #### A security institute will probably drift away from safety Let's say, however, that a safety organisation is sincere in its commitment to what we would generally think of as AI safety, and it tactically rebrands to "security". Its *vision* is safety in all but name. Its *strategy* leans into security, partly because this is *part* of safety, and partly because it's a foot in the door, a bridge position, whatever you want to call it. It starts to hire people from a security background. It starts to work with stakeholders who are interested in security, and not interested in the rest of safety. It has successes in security, so its branding becomes more tied to security. The people who are really into *safety*, and not security, leave; the people in the middle become the average of the people they interact with - so, the security people. In short order, management pause to review their strategy and think about what to do over the next X months. The organisation's unique value proposition is security. Its enablers - its people, its systems, its network - are all rooted in security. The next strategy takes advantage of this, leaning into its strengths and the opportunities - *in security*. There's a lot of concept handles for this one. - Strategic drift: the vision was X, the strategy was X', and ultimately the vision moves, rather than the strategy. - Values drift: you set out with one set of values, and over time - with exposure - your values change. - Means-end inversion: the tail wags the dog - security becomes the end, rather than the means. - Everything looks like a (security) nail when what you have is a (security) hammer. Ultimately what we're talking about is *capture* - best known in the form of 'regulatory capture', where (accidentally or deliberately) a regulator gets co-opted by the entities it's supposed to regulate. #### The purpose of a system is what it does In what would become a common heuristic in systems thinking, Stafford Beer observed there is "no point in claiming that the purpose of a system is to do what it constantly fails to do."[^5] The flipside of this is that if a system reliably produces an outcome favourable to bootleggers, then that means that's what all the incentives are set up to do. If no one's changing those incentives to change the system, then all the Baptist apologia in the world is at best moot, and at worst complicit. Unfortunately, that leaves us waiting to see what it *is* this new system does. The organisational dynamics outlined above should actually be a pretty strong prediction, but the *effects* of those dynamics remain ambiguous for now. --- Some other sources: - https://milesbrundage.substack.com/p/emergency-blog-post-reflections-on [^1]: Situational Awareness [^2]: https://www.gov.uk/government/news/tackling-ai-security-risks-to-unleash-growth-and-deliver-plan-for-change [^3]: https://www.ianhogarth.com/blog/2018/6/13/ai-nationalism [^4]: EconTalk - look up sources [^5]: https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_what_it_does [^6]: find some resources, probably