# Q1 --- *Do you agree with the definitions in this discussion paper? If not, what definitions do you prefer and why?* --- The definitions in Figure 1 are not sufficiently granular to support risk-based consideration of different kinds of AI. The definitions do not allow differentiation between AI models that provide economic benefit, models that raise significant ethical concerns, and models that raise radical safety concerns, and will not support policy that addresses the risks and opportunities of each. The foreword to ISO/IEC 22989 calls out its own limitations: it is currently lacking language for comparing AIs on dimensions like trustworthiness, robustness, resilience, reliability, accuracy, safety, security and privacy. These are exactly the properties we should be thinking about, that will have the implications that matter for policy. Critically, following human-defined objectives or parameters might be a good policy outcome, but is not a fundamental property of AIs. Tools already exist that allow AIs to perform complex multi-step procedures and generate their own recursive queries, either prompting themselves or other AIs - which immediately technically fails this definition. This presents risks that need to be addressed - definitions that overlook this will produce policy with systemic weaknesses. Overall, the the US National Science and Technology Council report “Preparing for the future of artificial intelligence” [1] may offer a better starting point (see page 6). This should be supplemented with more precise definitions relating to technical concepts like corrigibility, and observed behaviour like (seemingly human-like) intelligence and agency. I note also that this report was written more than 2-5 years prior to today, and yet holds up. [1] United States (2016) Executive Office of the President and M. Holden, J.P. and Smith. Preparing for the future of artificial intelligence. Technical report, National Science and Technology Council, Washington D.C. 20502, October 2016. https://obamawhitehouse.archives.gov/sites/default/files/whitehouse_files/microsites/ostp/NSTC/preparing_for_the_future_of_ai.pdf # Q2 --- *What potential risks from AI are not covered by Australia's existing regulatory approaches? Do you have suggestions for possible regulatory action to mitigate these risks?* --- ## Not covered: Missing risks include (1) misuse of highly capable AI systems, and (2) unintended harm from highly capable AI systems, especially systems with the capacity to plan and act as agents acting in ways that are not aligned with human directives. The actual likelihood of these risks playing out is uncertain, and could even be very low. But the harms resulting from these risks may be catastrophic and global in scope and must be acknowledged and addressed. I acknowledge this, and views about current-day harm and misuse. But to spend zero time or effort on these catastrophic risks would be grossly negligent. I am firmly convinced that analogies to pandemics or nuclear war are appropriate - even small steps to prepare for tail events present a good investment, and even a moral obligation. For example: the statement “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war” has recently been signed by the heads of leading AI labs (OpenAI, Google DeepMind, Anthropic, Stability AI) as well as many key academic and other researchers pivotal in developing this technology (Geoffrey Hinton, Yoshua Bengio, etc.) and other notable figures such as Bill Gates [1]. Similar calls date back to at least 2015 when Stephen Hawking led a group of AI experts to call attention to the risk and lay out an urgent research program [2]. Rishi Sunak, the Prime Minister of the United Kingdom [3], and Secretary-General of the United Nations, António Guterres [4], have both recognised the catastrophic and existential risks from highly capable AI systems. The policy conversation in Australia, including this discussion paper and the report from the Chief Scientist [5] are wrong to ignore the kind of risks that can occur from the development and use of highly capable AI systems. Although current work to develop these systems mainly happens outside of Australia, part of Australia’s approach to AI must include using its global influence to understand and reduce these kinds of risks. [1] https://www.safe.ai/statement-on-ai-risk#open-letter [2] https://futureoflife.org/open-letter/ai-open-letter/ [3] https://www.theguardian.com/technology/2023/may/25/no-10-acknowledges-existential-risk-ai-first-time-rishi-sunak [4] https://press.un.org/en/2023/sgsm21832.doc.htm [5] https://www.chiefscientist.gov.au/GenerativeAI ## Further initiatives But what can be done? As with other catastrophic risks, the problem could be tractable. There are three streams of work that the Government can pursue: 1. Publicly acknowledge the risk. Australia must join the growing number of countries and global governance bodies that acknowledge the possibility of catastrophic and existential risks from increasingly powerful AIs. There’s no chance of solving a problem unless we acknowledge it. This is a necessary first step. 2. Lead the establishment of global governance arrangements that are specifically tasked with tackling the possible catastrophic risks from AI. This could involve: - Fostering cultures of fairness, accountability and transparency by the companies and countries developing more advanced AIs. We need to work to avoid an ‘arms race’ that prioritises speed over safety, including making sure that risky decisions with broader implications aren’t made behind closed doors. - Monitoring the progress of AI development [1], including having the power to conduct inspections where necessary to observe for advanced indicators of increasing risks. The global community needs a clear understanding of the evolving risk of AI systems, including accidents or near misses. There is a global interest in knowing if labs produce AIs that are deceptive, aren’t aligned with human interests, can’t be effectively controlled, or demonstrate other risk factors. - Empowering the global governance body to take necessary action in the event that advanced indicators suggest AI may soon pose a catastrophic or existential risk. We need to establish what ‘triggers’ represent an unacceptable risk, and agree ahead of time that if those trigger conditions are met, national governments will freeze more advanced AIs until risks can be managed. 3. Build our national capability in AI safety, security and public decision making about AI. This could include: - Improving tech literacy in government and the public service. Australia could adopt a scheme like the US’ “TechCongress”; establish key technical roles around AI in relevant departments to ensure the safe and ethical use of AI systems in the specific sectors each department is responsible for; and ensure an ongoing dialogue between government, experts and the public as AI rapidly advances in the coming years. - Additional support for AI safety research in Australian universities, including regarding the measurement and monitoring of more advanced systems. - Developing a program to analyse and monitor deployed AI systems for robustness, corrigibility, bias, dual use capabilities and other safety relevant factors. This should build on agreement from OpenAI to provide Australia access to its models [2]. Similar agreements should be struck with other leading AI providers. Government needs to move quickly on these three core streams of work. One approach would be to establish an AI Commission, or similar body, and task it with developing the laws, regulations and policies necessary to deliver these streams. An AI Commission in Australia would bring several advantages and serve crucial functions. - First, it would offer whole-of-government leadership and engage with industry, academia, non-profits, and the community. Unlike a function within a specific department, a Commission could lead across the government and dynamically engage with the public. - Second, a Commission could address the need for timely decision-making in AI by keeping pace with the rapid advancements in technology. It would ensure that Australia doesn't fall further behind and would have the ability to guide the design and implementation of policies, prioritise AI safety, foster expertise, facilitate engagement with various stakeholders, lead on international standards, shape regulatory frameworks, drive law reform, and support research and education in AI. By expanding our focus from enhancing economic opportunities to a comprehensive approach that encompasses safety, ethics, and international leadership, an AI Commission can help Australia pursue its national interest effectively in the transformative AI era. [1] Why and How Governments Should Monitor AI Development https://arxiv.org/abs/2108.12427 [2] Government may force companies to label AI content to prevent deep fakes https://www.smh.com.au/politics/federal/government-may-force-companies-to-label-ai-content-to-prevent-deep-fakes-20230616-p5dh8r.html # Q3 --- *Are there any governance measures being taken or considered by other countries (including any not discussed in this paper) that are relevant, adaptable and desirable for Australia?* --- Australia should take note of the EU recent handling of the AI Act. For example, the European Parliament’s recognition of general-purpose AI (GPAI) is an important step in the right direction [1]. In particular, the following observations are highly valuable: - Acknowledgement that GPAI providers must comply with strict safety measures, regardless of the way it is distributed or intended to be used. - Requirements for GPAI providers to comply with various safety requirements throughout the lifecycle of their product; i.e. external audits assessing their performance, predictability, interpretability, corrigibility, safety and cybersecurity. These adjustments appear to respond to significant criticisms directed at initial drafts of the AI Act. For example, the AI Now Institute released a set of recommendations that heavily criticised the EU’s approach on some matters [2]. For example: - “GPAI models carry inherent risks and have caused demonstrated and wide-ranging harms. While these risks can be carried over to a wide range of downstream actors and applications, they cannot be effectively mitigated at the application layer.” - “GPAI must be regulated throughout the product cycle, not just at the application layer, in order to account for the range of stakeholders involved. The original development stage is crucial, and the companies developing these models must be accountable for the data they use and design choices they make. Without regulation at the development layer, the current structure of the AI supply chain effectively enables actors developing these models to profit from a distant downstream application while evading any corresponding responsibility.” In the Discussion paper, examples of “High risk” systems (e.g. medical robots or self-driving cars) were more relevant to the context in which the system was applied, rather than the system itself. This is a common error that is first encountered when reasoning about AI risks, and it is an issue which characterises the errors made by the EU. Australia must endeavour to avoid repeating these mistakes. Unfortunately, there are currently very few high-quality examples of legislation that has been enacted internationally. This is one we must instead look to the guidance from researchers in AI governance, such as those from the UK’s Centre for AI Governance, or the US’s Centre for AI Safety. [1] https://www.euractiv.com/section/artificial-intelligence/news/leading-eu-lawmakers-propose-obligations-for-general-purpose-ai/ [2] https://ainowinstitute.org/publication/gpai-is-high-risk-should-not-be-excluded-from-eu-ai-act # Target Areas (Q6-Q11) --- *Given the importance of transparency across the AI lifecycle, please share your thoughts on:         a. where and when transparency will be most critical and valuable to mitigate potential AI risks and to improve public trust and confidence in AI?         b. mandating transparency requirements across the private and public sectors, including how these requirements could be implemented.* --- The aviation industry offers a prime example of how transparency can be used to reduce risk and increase safety in a potentially dangerous industry. Transparency is embedded in every stage of the value chain, from manufacturing to operations, and extends to the reporting and investigation of risk-related incidents. The worst aviation disasters have terrible consequences, but are far from as bad as some conceivable disasters from sophisticated AI systems. Aviation safety provides a good *analogy*, but this should not be misconstrued as an *identical example*. The bar for safety needs to be much higher, but - particularly around "compute governance", notably regulating computing hardware - there are lessons to be learned. Manufacturing Stage Transparency begins at the manufacturing stage. Aircraft manufacturers, such as Boeing and Airbus, are required to adhere to stringent safety standards set by regulatory bodies like the Federal Aviation Administration (FAA) in the U.S. and the European Union Aviation Safety Agency (EASA) in Europe. These standards cover everything from the design and production of aircraft to their maintenance and operation. Manufacturers must provide detailed documentation of their processes, including design specifications, testing procedures, and quality control measures. This information is made available to regulatory bodies, airlines, and maintenance providers, ensuring that everyone involved in the aircraft's life-cycle has a clear understanding of its design and operation. This transparency allows for potential risks to be identified and mitigated early in the manufacturing process. Operational Stage Transparency continues into the operational stage. Airlines are required to maintain detailed records of their operations, including flight logs, maintenance records, and safety inspections. These records are subject to review by regulatory bodies and can be requested by other stakeholders, such as insurance companies and accident investigators. In addition, airlines are encouraged to share safety-related data with each other and with industry-wide safety organisations. This collaborative approach to safety, known as Safety Management Systems (SMS), allows airlines to learn from each other's experiences and to identify and address potential risks before they lead to accidents. Reporting and Investigation of Risk-Related Incidents Perhaps the most critical aspect of transparency in the aviation industry is the reporting and investigation of risk-related incidents. When an accident or “near miss” occurs, it is thoroughly investigated by independent bodies such as the Australian Transport Safety Bureau (ATSB) in Australia, the National Transportation Safety Board (NTSB) in the U.S. or the Air Accidents Investigation Branch (AAIB) in the UK. These investigations are conducted in a transparent manner, with regular updates provided to the public and the final report made publicly available. These investigations are conducted in a transparent manner, with regular updates provided to the public and the final report made publicly available. The goal of these investigations is not to assign blame, but to understand the underlying causes of the accident and to make recommendations for preventing similar accidents in the future. This approach encourages transparency, as airlines and other stakeholders are more likely to fully cooperate with the investigation if they are not afraid of being penalised. The Cyclical Process and the Importance of Transparency Transparency in the aviation industry is a cyclical process. The information gained from accident investigations feeds back into the manufacturing and operational stages, leading to improvements in aircraft design, maintenance procedures, and operational practices. This continuous cycle of transparency, learning, and improvement has made aviation one of the safest forms of transportation. Transparency is encouraged within the industry through a combination of regulatory requirements and voluntary initiatives. Regulatory bodies enforce transparency through mandatory reporting requirements and regular audits. At the same time, industry-wide safety organisations promote voluntary data sharing and collaboration and a broad safety culture at all levels. The aviation industry demonstrates how transparency can be used to reduce risk and increase safety. By making transparency a core value at every stage of the value chain, the industry has created a culture of safety that benefits all stakeholders, from manufacturers and airlines to passengers and the public. --- *Do you have suggestions for:         a. Whether any high-risk AI applications or technologies should be banned completely?         b. Criteria or requirements to identify AI applications or technologies that should be banned, and in which contexts?* --- In so many other technologies, in so many other industries, in so many areas of human life - I would strongly advocate for permissionless innovation and minimal government interference. But AI - like with nuclear technology, certain biological research, gain of function research, and other "dual use" technologies - is different. It has the potential, however slim or distant, to do genuinely catastrophic damage. In February 2023, OpenAI CEO Sam Altman released a statement containing the following: “As our systems get closer to AGI, we are becoming increasingly cautious with the creation and deployment of our models. Our decisions will require much more caution than society usually applies to new technologies, and more caution than many users would like.” [1] It is very reassuring to see this acknowledgement expressed by Altman, and the Australian Government would be wise to heed such warnings. Unfortunately, it may not matter how much caution OpenAI shows, because their actions have triggered the arms race that AI safety experts have been concerned about for years [2, 3]. In order to protect our nation, Australia should make an effort to impose a ban on the training of large-scale AI systems with a nontrivial expected chance of passing capabilities evaluations required to cause catastrophic risks to civilization (such as the Alignment Research Center’s tests for the capabilities required for autonomous replication [4]), until a consensus is reached among alignment researchers that AI systems at this level of capabilities can be made safe. This ban would only affect the extremely large training runs pushing the frontiers of foundation models, and would only come into effect when the chances of passing dangerous capabilities evaluations are independently deemed nontrivial (though this may be quite soon). For increased safety and simplicity, an immediate ban could be placed on the use of more than a set amount of compute (for example, roughly the size of GPT-4 or 10^25 FLOPS) in the training of any AI system. This “compute ceiling” could be raised over time as a consensus is formed around technical alignment solutions at that scale, or lowered over time as algorithmic efficiencies allow more dangerous capabilities to arise at lower compute costs. All current AI work in Australia would be completely unaffected by such a ban, but it would prevent Australia from becoming a harbour for the training of AI systems which pose catastrophic risks to humanity, and set a precedent for other countries around the world. Australia taking such an approach would also send a global signal that we should take AI safety seriously and not take risky actions until we be confident that they’re safe. [1] Altman, S (2023) Planning for AGI and beyond; https://openai.com/blog/planning-for-agi-and-beyond [2] Knight, W (2023) Google DeepMind’s CEO Says Its Next Algorithm Will Eclipse ChatGPT https://www.wired.com/story/google-deepmind-demis-hassabis-chatgpt/ [3] Bucknall et al. (2022) Current and Near-Term AI as a Potential Existential Risk Factor; https://users.cs.utah.edu/~dsbrown/readings/existential_risk.pdf [4] Alignment Research Centre (2023) Update on ARC's recent eval effortshttps://evals.alignment.org/blog/2023-03-18-update-on-recent-evals/ # Risk-based approaches (Q14-Q20) --- *Do you support a risk-based approach for addressing potential AI risks? If not, is there a better approach?* --- The regulation of AI needs to be “risk-based” in a similar way to vehicles. Extending the aviation anaology, we are right to regulate push-bikes and passenger jets in very different ways. We are right to (not) regulate consumer software in the same way we (heavily) regulate the technologies that go into nuclear weapons. I claim AI may soon become concerningly closer to the latter. A limitation of the risk management approach in the discussion paper, specifically Box 4 and Attachment C, is that it focuses only on how an AI is used and disregards the nature of the AI itself. An effective regulatory regime for a high-risk activity will regulate both the technology itself and how it is used. The Australian Human Rights Commission [1], in discussing this issue (“Human Rights and Technology”, 2021), says: Governments (appropriately) regulate high-risk activities and technologies more closely. This helps explain the comparatively strict laws that govern fields such as gene technology, aviation, healthcare and the energy industry. In these areas, regulation often applies both to the technology itself and how it is used. From a human rights perspective, the need for more prescriptive regulation will be greater where the use of a specific technology carries greater risks of harm to humans. Car manufacturers have to meet technical requirements about how safe their cars are. The technical requirements for passenger planes are much more strict, and much more lenient for push bikes. Users also have requirements, ranging from following basic road rules to advanced licencing and professional development requirements. The key gap in the Discussion Paper proposal is that it does not sufficiently focus on the safety of the AI itself, regardless of the proposed use case. That is, advanced AI will give rise to risks independent of its use case in at least three ways: 1. According to Box 4, a chatbot could range from very low risk to very high risk depending on how a company intends to use it. However, we have already seen examples of chatbots becoming uncoupled from their ‘use case’, such as by seeking to persuade a user to take actions to benefit itself [2]. Until we can solve the alignment problem and hence be confident that an AI product will adhere to a specific use case, regulation needs to be linked to the AI itself, not the use case. 2. Risk of intentional misuse of AI scale with the power of the AI, and regulating use is inherently ineffective. For instance, we have already seen “dual use” risks, where an AI intended for medical and biosecurity research purposes could be criminals or terrorists to design novel pathogens [3,4]. Where a tool is sufficiently dangerous, regulation needs to ensure it does not become widely available. 3. Sophisticated AIs that are agentic and unaligned could cause catastrophic or existential risks regardless of human intent about their usage. AI safety researchers are yet to solve the alignment problem. These future models are very high risk regardless of intended use cases. The Australian government should adopt a risk framework for AI that considers both the particular AI technology itself and proposed use cases as separate factors. Advanced AI systems with features like the ability to complete diverse ranges of reasoning tasks with human-level performance and the ability to form sophisticated plans should be considered very high risk regardless of proposed use cases. The regulatory regime will also need to provide ongoing monitoring and compliance of AIs in use.   [1] Australian Human Rights Commission (2021). Human Rights and Technology. [1] https://tech.humanrights.gov.au/sites/default/files/2021-05/AHRC_RightsTech_2021_Final_Report.pdf [2] Roose, K. (2023). “A Conversation With Bing’s Chatbot Left Me Deeply Unsettled”. https://www.nytimes.com/2023/02/16/technology/bing-chatbot-microsoft-chatgpt.html [3] Soice et al. "Can large language models democratize access to dual-use biotechnology?" June 2023. arXiv preprint arXiv:2306.03809 [4] Eshoo. “Eshoo Urges NSA & OSTP to Address Biosecurity Risks Caused by AI”. October 2022. eshoo.house.gov/media/press-releases/eshoo-urges-nsa-ostp-address-biosecurity-risks-caused-ai --- *How might a risk-based approach apply to general purpose AI systems, such as large language models (LLMs) or multimodal foundation models (MFMs)?* --- LLMs and MFMs are constantly demonstrating unprecedented general-purpose capabilities, and their emergence indicates that regulatory approaches which focus on specific use-cases or industries are becoming outdated. A modern approach requires that we focus on overall system capabilities rather than individual scenarios. Systems with advanced capabilities should face progressive levels of scrutiny. Low-risk systems can be monitored with light touch regulation. To be clear, current-generation systems (as of mid-2023) like OpenAI's GPT-4 do not seem to pose serious risks - short of being connected to critical infrastructure (which could very well occur). However, as capabilities increase, so should regulatory requirements. Precursor systems, with capabilities that can be used to build much more advanced systems, will need stricter controls to avoid proliferation of advanced systems amongst malicious actors [1]. Users who require “developer” access should require permits granting such access, and providers must guarantee responsible use via monitoring programs [2]. For advanced systems with human-level capabilities, comprehensive scrutiny is essential. Prior to deploying systems for use in Australia, developers should undertake multi-stage approval covering the full system lifecycle, from pre-development to pre-deployment. Once approved, systems would be deployed to high-security data centres operating in Australia, and accessed only via a centrally-monitored API; this would help facilitate regulatory oversight by providing an audit trail of metadata on usage patterns [3]. Once deployed, organisations who wish to access these advanced systems should be screened cautiously, and held accountable with centralised compliance programs that monitor for negligence or misuse. Establishing legal liability for users and developers is key [4]. However, this must be complemented with additional capabilities for controlling unlawful deployment or development of advanced systems; law enforcement and security agencies should leverage existing partnerships with telecommunications and cloud computing providers to monitor for suspicious activity that may indicate unlawful use of advanced AI systems. This calibrated, risk-based approach allows innovation while managing unprecedented risks. It enables beneficial use of AI where appropriate, while restricting high-risk applications pending safety assurances. With advanced AI on the horizon, it is imperative to act preemptively to protect the public. [1] Anderljung et al. (2023) Protecting Society from AI Misuse: When are Restrictions on Capabilities Warranted? https://arxiv.org/pdf/2303.09377.pdf [2] OpenAI (2022) Lessons learned on language model safety and misuse; https://openai.com/research/language-model-safety-and-misuse [3] Brundage et al. (2020) Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims; https://arxiv.org/pdf/2004.07213.pdf [4] AI Now (2023) General Purpose AI Poses Serious Risks, Should Not Be Excluded From the EU’s AI Act https://ainowinstitute.org/publication/gpai-is-high-risk-should-not-be-excluded-from-eu-ai-act --- *Should a risk-based approach for responsible AI be a voluntary or self-regulation tool or be mandated through regulation? And should it apply to:         a. public or private organisations or both?         b. developers or deployers or both?* --- In general, regulation should: 1. Be proportionate to risk. Meaning that a voluntary or self-regulation approach should only apply to low-risk or no-risk uses of AI. 2. In the case of higher-risk technologies, regulate both the technology itself and the use of the technology. Meaning that, for future and more sophisticated AIs, regulation should ensure that the technology is safe before it is created or published, as well as ensuring that it is only used in ways that are safe. 3. Place burdens on those most able to reduce potential harms. Meaning that AI that functionally operates as ‘black box’ to consumers has to be regulated at the point of the AI Lab. Applying those three principles, self-regulation may be appropriate for some technologies available today and for some participants (such as students or researchers), but a forward-looking regulatory regime must apply broadly and must be backed by Australian diplomatic efforts to ensure it is globally coordinated.