Join Our Discord (940+ Members)

International Institutions for Advanced AI

Content License: cc-by

International Institutions for Advanced AI

Papers is Alpha. This content is part of an effort to make research more accessible, and (most likely) has lost some details from the original. You can find the original paper here.

Introduction

The capabilities of AI systems have grown quickly over the last decade. Employing a growing wealth of algorithmic insights, data sources and computation power, AI researchers have created systems that can comprehend language, recognize and generate images and video, write computer programs and engage in scientific reasoning. If current trends in AI capabilities continue, AI systems could have transformative impacts on society.

Powerful AI systems will bring significant benefits and risks. These systems have the capacity to significantly improve the productivity of workers and economies as a whole, and to help us address some of our most important social and technological challenges. But these systems also present challenges including workforce dislocation, lack of transparency, biased outcomes, inequitably shared benefits and threats to national security.

Promoting AI benefits and managing AI risks both have domestic and international components. On the domestic front, governments and the private sector will need to establish rules and norms around how advanced AI systems are developed, distributed, deployed and accessed, addressing issues like security, distributive impacts, privacy, bias, and more. A number of challenges have the potential to transcend national borders and impact societies and economies worldwide. Accordingly, policymakers, technologists, and AI governance experts have recently begun to call for specific global AI governance initiatives centered on international institutions.[See, e.g.,

This paper contributes to these early conversations by discussing why AI governance may be needed on an international scale and then offering a non-exhaustive taxonomy of the institutional functions that international efforts might require. It explores four possible international institutions to perform these functions: 1) a Commission on Frontier AI that facilitates expert consensus on opportunities and risks from advanced AI; 2) an Advanced AI Governance Organization that sets international standards, supports their implementation, and could monitor compliance to future governance regimes; 3) a Frontier AI Collaborative that develops and distributes cutting-edge AI; and 4) an AI Safety Project that brings together exceptional researchers, engineers and compute to further AI safety research. Each of these approaches seeks to mitigate the societal challenges of advanced AI in different ways and each confronts significant challenges to its viability and success.

The Need for International Governance

Powerful AI systems have the potential to transform society, economics and politics in fundamental ways. Because of characteristics like its high barriers to development/utilization and the possibility of cross-border use, it is possible that harnessing AI’s potential for global benefit and managing its risks could require governance functions at the international level.

Promoting Global Benefits

Access to appropriate AI technology might greatly promote prosperity and stability, but the benefits might not be evenly distributed or focused on the greatest needs of underrepresented communities or the developing world. Inadequate access to internet services, computing power, or availability of machine learning training/expertise will also hinder certain groups’ ability to benefit fully from AI advances.

International institutions have long sought to support sustainable global development. International efforts to build consensus on AI opportunities—especially addressing barriers to their effective use globally—could support efforts to distribute and enable access to AI. On top of facilitating access, this could also include building capacity to benefit from AI through education, infrastructure, and local commercial ecosystems. In some cases, international collaborations (including public-private partnerships) to develop frontier AI systems that are suited to the needs to underserved communities may also be appropriate.

Inconsistent national regulations could also slow the development and deployment of AI, as developers of powerful AI technology may be unwilling to export to countries with inconsistent or unsuitable technology governance.[In addition to compliance costs, they may be concerned about enabling misuse, or the theft of proprietary information.] International efforts to set safety norms and standards could help coordinate governance in a way that supports innovation and serves a broad set of interests.

Managing Shared Risks

Advanced AI capabilities may also create negative global externalities. AI systems today are already capable of not just progressing drug discovery and development, but also of (re)inventing dangerous chemicalsand solving foundational problems in synthetic biology. Scientific capabilities like these could be weaponized by malicious actors for use worldwide. AI may also be used to create potent cyberweapons that can generate code, scan codebases for vulnerabilities, and engineer polymorphic malware in ways that threaten critical infrastructure. Existing AI systems already pose mis- and dis-information issues, and the introduction of more advanced systems is leading malicious actors to explore more sophisticated methods of information warfare.[See, e.g, Furthermore, building systems that act as intended in novel circumstances is a challenging problem that may only grow more difficult. As systems get increasingly capable, there will be greater incentives to deploy them in higher stakes domains where accidents could have serious global consequences.

Implementing protocols for responsible development and deployment will help address these risks of accident and misuse,[Potential protocols include: training methods that restrict the dangerous capabilities and increase the reliability of systems, subjecting AI systems to risk assessments that ascertain their propensity to cause harm before training or deployment, deployment protocols that secure systems against misuse or the exfiltration of its parameters, post-deployment monitoring to identify and respond to unforeseen risks. See.] on top of measures targeted at specific downstream issues.[Such as the Digital Services Act for disinformation, and treaties targeting chemical and biological weapons.] However, cross-border access to AI products and the cross-border effects of misuse and accidents suggests that national regulation may be ineffective for managing the risks of AI even within states. States will inevitably be impacted by the development of such capabilities in other jurisdictions.

To further the international adoption of safety protocols for advanced models, it would be useful to build consensus on risks and how they can be mitigated*,* and set safety norms and standards and support their implementation to help developers and regulators with responsible development and use. International efforts to conduct or support AI safety research may be beneficial, if it can increase the rate of safety progress or the reach of its outputs.

In the longer term, continued algorithmic and hardware progress could make systems capable of causing significant harm accessible to a large number of actors, greatly increasing the governance challenges.[According toand, the compute costs of training a model of a fixed performance level decreases approximately tenfold every 2 years.] In this case, the international community might explore measures like controlling AI inputs (although the dual-use/general purpose nature of the technology creates significant tradeoffs to doing so) and developing and/or enabling safe forms of access to AI.

The significant geopolitical benefits of AI development may disincline states to adequately regulate AI: arguments about national competitiveness are already raised against AI regulation,[See, e.g., the arguments discussed in and such pressures may strengthen alongside AI progress. We may eventually need international agreements that address these geopolitical risks, with institutions that can monitor compliance where feasible.[Monitoring can vary significantly in intrusiveness and effectiveness: while it will be highly difficult to implement adequate monitoring across major geopolitical divides, the safety of advanced systems could be a shared interest of major powers and a regime to address risk factors from smaller-scale geopolitical competition may be feasible.] Efforts to control AI inputs may be useful to enable non-proliferation of potentially dangerous capabilities and increase the technical feasibility of monitoring. More speculatively, efforts to develop frontier AI collectively or distribute and enable access and its benefits could incentivize participation in a governance regime.

The institutional functions identified above can be summarized and grouped into the following two broad categories.

  1. Science and Technology Research, Development and Diffusion
  • Conduct or support AI safety research:
    • Research and develop of measures to reduce the risks of AI misuse and accidents stemming from system characteristics like dangerous capabilities and unreliability/misalignment. This includes work on understanding and evaluating these characteristics and the threats they pose, training methods to reduce and manage risky behaviors, and examining safe deployment protocols appropriate to different system.
  • Build consensus on opportunities and risks:
    • Further international understanding of the opportunities and challenges created by advanced AI and possible strategies for mitigating the risks.
  • Develop frontier AI:
    • Build cutting-edge AI systems.
  • Distribute and enable access to cutting edge AI:
    • Facilitate access to cutting-edge systems and increase absorptive capacity through education, infrastructure, and support of the local commercial ecosystem.
  1. International Rulemaking and Enforcement
  • Set safety norms and standards:
    • Establish guidelines and standards around how AI can be developed, deployed and regulated to maximize benefit and minimize risks.
  • Support implementation of standards:
    • Provide assistance for the implementation of established guidelines and standards.
  • Monitor compliance:
    • Conduct audits /evaluations and issue certifications / licenses to ensure adherence to international standards and agreements.
  • Control AI inputs:
    • Manage or monitor models, compute, data and other ingredients of potentially dangerous technologies.

International bodies already perform some of these functions.[Seefor a more thorough overview.] The OECD’s AI Principles and AI Policy Observatory work, the ITU’s AI for Good initiative, and expert reports from the Global Partnership on AI’s Working Group on Responsible AI are early efforts at building consensus on AI opportunities and risks. Relatedly, the UK’s proposed Foundation Model Taskforceand the US’s proposed Multilateral AI Research Institute could emerge as multilateral efforts to conduct AI safety research, or potentially even develop frontier AI systems, though both are in exploratory phases.[The amount of funding required to stay on the cutting-edge of AI capabilities is significant. See, e.g.,.]

Alongside lawmaking efforts like the EU’s AI Act and the Council of Europe’s Convention on AI, Human Rights and Democracy, we have seen early norm and standard setting efforts from ISO/IEC, but little in the way of implementation support, oversight or certification. In terms of controlling dangerous inputs: computing resources have been targeted by US, Japanese and Dutch export controls that prevent the sale of certain AI chips and semiconductor manufacturing equipment to China.

International Institutions for Advanced AI

We have outlined several AI governance functions that might be needed at an international level, and shown that only a limited number of these are currently being performed by existing institutions. In this section, we discuss how functional gaps may be filled.

The functions could be split in multiple ways across institutions: drawing on existing international organizations and proposals, we describe four idealized models. We note that the models described in this section describe roles that could be filled by existing or new institutions. Participants in these institutions could include governments, non-governmental organizations, the private sector, and academia. Table 1 summarizes the previous analysis and the functions of the institutions we discuss.

Commission on Frontier AI: Fostering

Scientific Consensus

There have been several recent proposals of an intergovernmental body to develop expert consensus on the challenges and opportunities presented by advanced AI.[See, e.g.,.] Existing institutions like the Intergovernmental Panel on Climate Change (IPCC), the Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services (IPBES) and the Scientific Assessment Panel (SAS), which studies ozone depletion under the Montreal Protocol, provide possible models for an AI-focused scientific institution. Like these organizations, the Commission on Frontier AI could facilitate scientific consensus by convening experts to conduct rigorous and comprehensive assessments of key AI topics, such as interventions to unlock AI’s potential for sustainable development, the effects of AI regulation on innovation, the distribution of benefits, and possible dual-use capabilities from advanced systems and how they ought to be managed.

Functions and Motivation

International consensus on the opportunities and risks from advanced AI has the potential to facilitate effective action addressing them, for example, by engendering a shared desire for the development and adoption of effective risk mitigation strategies.

Currently, there is significant disagreement even among experts about the different opportunities and challenges created by advanced AI,[See, for example, the disagreement around whether advanced AI could pose an extinction risk:.] and this lack of consensus may worsen over time as the effects of AI systems increase in scale and number, hindering collective action on the scale necessary to ensure that AI is developed for the benefit of all. Furthermore, there are several challenges from advanced AI that may require international action before risks materialize, and the lack of a widely accepted account or even mapping of AI development trajectories makes it difficult to take such preparatory actions. Facilitating consensus among an internationally representative group of experts could be a promising first step to expanding our levels of confidence in predicting and responding to technological trends.

Challenges and Risks

Scientific challenges of understanding risks on the horizon: Understanding frontier AI risks and their mitigation is technically challenging. The nature of future AI capabilities and their impact is difficult to predict, especially given the fast rate of progress. To increase chances of success, a Commission should foreground scientific rigor and the selection of highly competent AI experts who work at the cutting edge of technological development and who can continually interpret the ever-changing technological and risk landscape.

Unfortunately, there is a relative lack of existing scientific research on the risks of advanced AI.[The recent IPCC assessment, for reference, was written by 234 scientists from 66 states and drew on 14,000 scientific papers.] To address the lack of existing scientific research, a Commission might undertake activities that draw and facilitate greater scientific attention, such as organizing conferences and workshops and publishing research agendas. It may be helpful to write a foundational “Conceptual Framework”—following the example of the IPBES—to create a common language and framework that allows the integration of disparate strands of existing work and paves the way for future efforts.

Politicization: A Commission on Frontier AI would benefit from, if not require, a clear buffer between experts charged with developing consensus narratives around the risks and opportunities of AI and policymakers acting on the political and economic interests of their states, which might push policies in different directions. The scientific understanding of the impacts of AI should ideally be seen as a universal good and not be politicized.

Membership structure can affect a Commission’s impartiality and legitimacy: ideally, there would be broad geographic representation in the main decisionmaking bodies, and a predominance of scientific experts in working groups.[If legitimacy is the primary concern, the Commission might adopt the IPCC’s innovation of writing key documents by consensus, balancing inclusion (states’ representatives and scientists review, discuss and approve the report line by line) and scientific rigor (all suggested amendments must be consisted with working group’s scientific report that is being summarized).] Unfortunately, given the uncertain and controversial nature of advanced AI risks and opportunities, representation may trade off against a Commission’s ability to overcome scientific challenges and generate meaningful consensus.[If it follows the IPCC model, experts will be nominated by member states, but there will not be a robust climate science discipline to buffer against political interests.] In addition to striking the correct balance in membership, a Commission should carefully scope the subject matter of their research—it may, for example, adopt the IPCC’s objective of being “policy-relevant” without being “policy-prescriptive.”

Assessment

The objective of a Commission on Frontier AI is worthwhile in most circumstances, but the scientific challenges and potential of politicization imply that a Commission—especially one that aims at broad political representation—may not be able to build scientific consensus effectively. The extraordinary pace of technological change may require more nimble policy responses, such as less institutionalized and politically authoritative scientific advisory panels on advanced AI.

Advanced AI Governance Organization: Promoting Norms and

Standards, Providing Implementation Support, Monitoring Compliance

As discussed above, certain misuse and accident risks of advanced AI systems may pose significant global threats, and international efforts aimed at managing these risks could be worthwhile. An intergovernmental or multi-stakeholder organization could perform a variety of governance functions furthering the regulation of such systems, in particular norm and standard setting, implementation assistance, and perhaps monitoring compliance with governance frameworks.[See, e.g.,for discussions and proposals of an institution of this type.]

Functions and Motivation

We identify two main objectives for an Advanced AI Governance Organization. How much emphasis it should place on each depends on the challenges it aims to address.

Objective 1: Internationalizing and harmonizing AI regulation. Regulatory regimes that set standards and provide implementation support may help ensure that powerful AI capabilities do not pose misuse or accident risks. Standard setting would facilitate widespread international adoption by: 1) reducing the burden on domestic regulators to identify necessary safety regulations and protocols, 2) generating normative pressure for safety protocol adoption, and 3) reducing frictions around the development of international frameworks. Implementation support would assist the establishment and maintenance of regulatory regimes meeting these frameworks. Examples of organizations that perform similar functions include the Financial Action Task Force (FATF), the International Telecommunication Union (ITU) and the International Civil Aviation Organization (ICAO).

The same functions are useful for harmonizing regulation: international standard setting would reduce cross-border frictions due to differing domestic regulatory regimes. (It is possible that future regulations will limit access to powerful AI technologies in jurisdictions with inadequate AI governance.) Implementation support would help reduce obstacles to countries meeting international standards and therefore enable greater access to advanced AI.

Objective 2: Monitoring compliance. Where states have incentives to undercut each other’s regulatory commitments, international institutions may be needed to support and incentivize best practices. That may require monitoring standards compliance. At the least intrusive end of the spectrum is self-reporting of compliance with international standards (as in the Paris Agreement—see proposals for self-reporting/registration of training runs). Organizations like the FATF, ICAO, and the International Maritime Organization (IMO) take a somewhat more intrusive approach, monitoring jurisdictions to ensure they adopt best practice regulations, and in some cases checking on the enforcement of domestic regulations embodying international standard. In the case of advanced AI, some observers have asked whether more intrusive forms of international oversight might be necessary, including detection and inspections of large data centers (partly analogous to IAEA safeguards). The more intense and intrusive any monitoring, the more challenging it may be to get to consensus.

Challenges and Risks

Speed and comprehensiveness in standard setting: One challenge for a Governance Organization is that standard setting (especially in an international and multistakeholder context) tends to be a slow process, while the rapid and unpredictable nature of frontier AI progress may require more rapid international action. A Governance Organization may need to partner with faster-moving expert bodies and expedited standard-setting approaches. The breadth of membership may also represent a trade-off between speed and diversity of perspectives. Broader membership may be important where long-term consensus is important, while urgent risks may need to be addressed at first by smaller groups of frontier AI states, or aligned states with relevant expertise.

Incentivizing participation: The impact of a Governance Organization depends on states adopting its standards and/or agreeing to monitoring. Broad agreement (or agreement among frontier AI states at least) about the risks that standards and monitoring address and financial and technical support for standards’ implementation may help induce states’ participation. Many states—even those that are not full members of the organization—adopt FATF standards because they view them as in their own interests. Other AI-specific incentives for participation include conditioning on participation access to AI technology (possibly from a Frontier AI Collaborative) or computing resources.[The cloud compute industry and the underlying semiconductor supply chain are concentrated in a small number of countries.] States might also adopt import restrictions on AI from countries that are not certified by a Governance Organization—similar, for instance, to the way states prohibit flights from jurisdictions without ICAO-certification from entering their airspace.

In the more distant case of high stakes agreements governing AI development by states (such as arms control treaties), some states may be especially reluctant to join due to fear of clandestine noncompliance by other states. They may also worry that international inspections could compromise state secrets to the benefit of adversaries (which information security protocols could address in part). Again, the current reliance of advanced AI development on significant computing resources may make it easier to track significant AI efforts.[Oversight of data centers may allow the detection of large training runs that are subject to international controls. See.] Automated (even AI-enabled) monitoring may allow closer inspection of large training runs without compromising secrets. Such measures would likely hinge on negotiated verification regimes rather than national technical means—and negotiating verification is always fraught (e.g., in the case of the Chemical Weapons Convention) and often unsuccessful (e.g., in the case of the Biological Weapons Convention).

Scoping challenges: Unlike many other technologies—from nuclear resources to aviation—AI is already broadly deployed and used by billions of people every day. To operate efficiently and at appropriate scale, a Governance Organization should focus primarily on advanced AI systems that pose significant global risks, but it will be difficult in practice to decide on the nature and sophistication of AI tools that should be broadly available and uncontrolled versus the set of systems that should be subject to national or international governance. The rapid evolution of these technologies compounds the problem, as the technological frontier is advancing quickly, and models that were “frontier” a year ago are now both outdated and widely available.

Assessment

If advanced AI poses misuse and accident risks of a global scope, and unilateral technical defenses are not sufficient to protect against them, an international Governance Organization may be valuable. However, its effectiveness will depend on its membership, governance and standard-setting processes.

It may be important for governance to apply to all countries, and particularly to those whose firms are on the frontier of AI development. Yet, aligned countries may seek to form governance clubs, as they have in other domains. This facilitates decision-making, but may make it harder to enlist other countries later in the process. It is unclear what institutional processes would satisfy the demands of legitimacy and effectiveness, and incentivize the participation of important groups of stakeholders.

Frontier AI Collaborative: Enabling International Access to AI

Policymakers and pundits have also proposed collaborations to develop and distribute cutting-edge AI systems, or to ensure such technologies are accessible to a broad international coalition. Given the significant cost of developing advanced AI systems, a Frontier AI Collaborative could take the form of an international private-public partnership that leverages existing technology and capacity in industry, for example by contracting access to or funding innovation in appropriate AI technology from frontier AI developers. Such an organization could draw inspiration from international public-private partnerships like Gavi - the Vaccine Alliance or The Global Fund to Fight AIDS, Tuberculosis and Malaria; as well as international organizations that hold and control powerful technologies, like the IAEA’s nuclear fuel bankor the Atomic Development Authority that was proposed following WWII.

Functions and Motivation

A Frontier AI Collaborative could be designed to spread beneficial technology or serve as a channel for legitimate international access to advanced AI.

Spreading beneficial technology: A Collaborative could be established to ensure the benefits of cutting-edge AI reach groups that are otherwise underserved by AI development. One motivation for this objective is that the resources required to develop advanced systems make their development unavailable to many societies. This may result in technologies being inadequately designed for and supplied to groups that may benefit most from them for a variety of reasons:

enumi.

Systems developed by private actors may not adequately cater to all societies or demographics: they may not reflect the right values, have the right language capabilities, or work efficiently in diverse geographies.

Private firms may not price their products in ways that allow for equitable or broad distribution of benefits.

In order to protect proprietary information, private AI firms may not grant deep access to their models (e.g. they may restrict API access to prevent model imitation), which could preclude the development of use cases with significant social benefit.

A Collaborative could acquire or develop and then distribute AI systems to address these gaps, pooling resources from member states and international development programs, working with frontier AI labs to provide appropriate technology, and partnering with local businesses, NGOs, and beneficiary governments to better understand technological needs and overcome barriers to use.[For example, Gavi promotes immunization e.g. by funding innovation, and negotiating bulk contracts with pharmaceutical companies (especially advanced market commitments) for vaccination programs in low-income countries.] It could enable the development of technology that better caters to the underserved,price access to AI models in a way that is equitable, provide education and build infrastructure to allow the effective utilization of AI technology, and set a paradigm for responsible and inclusive AI development. By pooling the resources of multiple parties towards these ends (including safety talent, which is currently very scarce in the AI community), one or more of the aims could potentially be pursued more quickly and effectively than under the status quo.

Facilitating legitimate international access to powerful AI: More speculatively, a sufficiently ambitious, responsible and legitimately governed AI Collaborative could further AI governance objectives and reduce geopolitical instability amidst fierce AI competition among states. For example, membership in a Collaborative and access to its safe technology could be offered as an incentive for countries to participate in a governance regime that enforces responsibility (such as agreements to enact stricter regulation, or restrict military AI development). The existence of a technologically empowered neutral coalition may also mitigate the destabilizing effects of an AI race between states, by reducing the strategic consequences of one party falling behind or moderating the power concentrated among competing powers.

In addition, the Collaborative’s technology could be used to increase global resilience to misused or misaligned AI systems by giving experts a head start in studying the kinds of threats likely to be posed by other AI systems, and by being deployed for “protective” purposes such as fixing security vulnerabilities in critical infrastructure, detecting and counteracting disinformation campaigns, identifying misuse or failures of deployed systems, or monitoring compliance with AI regulations. This would be especially useful in scenarios where sharply falling training costs (due to algorithmic progress and Moore’s law) means the ability to train dangerous models is widely spread.

Challenges and Risks

Obstacles to benefiting from AI access: It is likely to be difficult to meaningfully empower underserved populations with AI technology, as the obstacles to their benefiting from AI run much deeper than access alone. Any Collaborative whose primary objective is global benefit needs to be adequately integrated into the global development ecosystem and set up with significant capacity or partnerships for activities beyond AI development such as: understanding the needs of member countries, building absorptive capacity through education and infrastructure, and supporting the development of a local commercial ecosystem to make use of the technology. The resources required to overcome these obstacles is likely to be substantial, and it is unclear whether such a Collaborative would be an effective means of promoting development.

Diffusion of dual-use technologies: Another challenge for the Collaborative would be managing the risk of diffusing dangerous technologies. On the one hand, in order to fulfill its objectives, the Collaborative would need to significantly promote access to the benefits of advanced AI (objective 1), or put control of cutting-edge AI technology in the hands of a broad coalition (objective 2). On the other hand, it may be difficult to do this without diffusing dangerous AI technologies around the world, if the most powerful AI systems are general purpose, dual-use, and proliferate easily.[For example: it may be difficult to protect broadly-deployed models from imitation, and it may be difficult to secure the deployment pipeline from attempts to copy model weights.] This is especially the case if the Collaborative aims to deploy cutting-edge general purpose systems to manage AI risks: the kinds of systems (and their underlying source code and algorithms) capable of meaningfully protecting against dangerous AI or furthering governance objectives may pose an exceptional misuse risk, as they will likely be engineered from highly capable, general purpose models.

To address such a challenge, it would be important for the Collaborative to have a clear mandate and purpose. Members of a Collaborative would need to have a strong understanding of the risks of the models being developed now and in the future, and their implications for model distribution, organization security (especially restrictions on the movement of Collaborative model weights), and other activities that may impact their ability to benefit from the Collaborative. Only by doing this would the Collaborative be able to consistently implement the necessary controls to manage frontier systems. It may also be necessary to exclude from participation states who are likely to want to use AI technology in non-peaceful ways, or make participation in a governance regime the precondition for membership.

Assessment

A Frontier AI Collaborative may indeed be a viable way of spreading AI benefits. However, the significant obstacles to societies benefiting from AI access raise questions about its competitiveness (relative to other development initiatives) as a means of promoting the welfare of underserved communities.

The viability of a Collaborative as a site of legitimately controlled technology also unclear: it depends on whether a balance can be struck between legitimately pursuing technologies powerful enough to positively affect international stability, and managing the proliferation of dangerous systems.

AI Safety Project: Conducting Technical Safety Research

The final model we discuss is an international collaboration to conduct technical AI safety research[This could include work on understanding and evaluating characteristics of systems such as alignment/reliability and dangerous capabilities, training methods to reduce and manage these characteristics, and deployment protocols (such as system security, monitoring, accident-response) that are appropriate to different system characteristics.] at an ambitious scale.[See, e.g.,

Tthe Safety Project would be modeled after large-scale scientific collaborations like ITER and CERN. Concretely, it would be an institution with significant compute, engineering capacity and access to models (obtained via agreements with leading AI developers), and would recruit the world’s leading experts in AI, AI safety and other relevant fields to work collaboratively on how to engineer and deploy advanced AI systems such that they are reliable and less able to be misused. CERN and ITER are intergovernmental collaborations; we note that an AI Safety Project need not be, and should be organized to benefit from the AI Safety expertise in civil society and the private sector.

Functions and Motivation

The motivation behind an international Safety Project would be to accelerate AI safety research by increasing its scale, resourcing and coordination, thereby expanding the ways in which AI can be safely deployed, and mitigating risks stemming from powerful general purpose capabilities.[Being a public good, AI safety may be underfunded by default, which the Safety Project would address as a site of collective contribution.] Researchers—including those who would not otherwise be working on AI safety—could be drawn by its international stature and enabled by the project’s exceptional compute, engineers and model access. The Project would become a vibrant research community that benefits from tighter information flows and a collective focus on AI safety. The Project should also have exceptional leaders and governance structures that ensure its efforts are most effectively targeted at critical questions on the path to safer AI systems.

Because perceptions of AI risk vary around the world, such an effort would likely be spearheaded by frontier risk-conscious actors like the US and UK governments, AGI labs and civil society groups. In the long run, it would be important for membership to be broad to ensure its research is recognized and informs AI development and deployment around the world.[While safety-relevant insights should be publicized for international use, other innovations with commercial value can be collectively owned by or affordably licensed to member states to incentivize broad participation. See, e.g., CERN’s approach to this.]

Challenges and Risks

Pulling safety research away from frontier developers: One potential effect of this model is that it diverts safety research away from the sites of frontier AI development. It is possible that safety research is best conducted in close proximity to AI development to deepen safety researchers’ understanding of the processes and systems they are trying to make safe and to ensure there is adequate safety expertise in-house. This risk could be addressed by offering safety researchers within AI labs dual appointments or advisory roles in the Project, and may become less of an issue if the practice of AI safety becomes institutionalized and separated from research and development.

Security concerns and model access: In order to be effective, participants in the Project need to have access to advanced AI models, which may allow them to illegally copy the model’s weights, clone the model via access to its outputs, or understand how it could be replicated (by determining its architecture or training process). Given the importance of these assets to the business interests of frontier labs, it may be difficult to negotiate agreements where adequate model access is granted. It may also lead to the diffusion of dangerous technologies.

This issue may be addressed by restricting membership in the Safety Project and by information security measures. In particular, it may be possible to silo information, structure model access, and design internal review processes in such a way that meaningfully reduces this risk while ensuring research results are subject to adequate scientific scrutiny. Certain types of research, such as the development of model evaluations and red-teaming protocols, can proceed effectively with API access to the models, while others such as mechanistic interpretability will require access to the model weights and architectures, but may not need to work with the latest (and therefore most sensitive) models.

Assessment

Technical progress on how to increase the reliability of advanced AI systems and protect them from misuse will likely be a priority in AI governance. It remains to be seen whether—due to issues of model access and the allocation of experts between a Safety Project and sites of frontier AI development—an AI Safety Project will be the most effective way of pursuing this goal.

Combining Institutional Functions

We can imagine institutions taking on the role of several of the models above. For example, the Commission on Frontier AI and the AI Safety Project make an obvious pairing: a Commission could scale up research functions to supplement the synthesis and consensus-building efforts, or a Project could conduct synthesis work in the course of its activities and gradually take on a consensus-establishing role. A Frontier AI Collaborative would also likely conduct safety research, and could easily absorb additional resourcing to become a world-leading Safety Project.

Conclusion

This paper has outlined several reasons why the world may want to expand existing initiatives in AI governance and safety and discussed the strengths and limitations of four possible institutional models to address these needs.

To better harness advanced AI for global benefit, international efforts to help underserved societies access and use advanced AI systems will be important. A Frontier AI Collaborative that acquires and distributes AI systems could be helpful, if it can effectively enable underserved groups to take full advantage of such systems. A Commission on Frontier AI could help identify the areas where international efforts can most effectively achieve these goals, if it can prevent the politicization of its work. Relatedly, it will be important for governance approaches around the world to be coordinated, so as to reduce frictions to innovation and access: an Advanced AI Governance Organization that sets international standards for governance of the most advanced models could facilitate this.

To manage global risks from powerful AI systems, effective AI governance regimes may be needed around the world. An Advanced AI Governance Organization that establishes governance frameworks for managing global threats from advanced systems and helps with their implementation may help internationalize effective regulatory measures, but it may be difficult to establish reliable standards if AI progress continues at the present rate, and also difficult to incentivize adoption of an Organization’s standards if there is a lack of global consensus on AI risks. A Commission on Frontier AI could also support governance efforts by building scientific consensus around AI risks and their mitigation, although its task of providing a scientifically credible and internationally recognized account of a quickly changing risk landscape will be challenging, especially given the relative lack of existing scientific research on the topic. An AI Safety Project could accelerate the rate at which technical methods of mitigating AI risks are developed—provided it can overcome its efficiency and model access hurdles, and a Frontier AI Collaborative’s technology might be used to increase global resilience to misused or misaligned AI systems. More speculatively, the functions of a Governance Organization and Collaborative could in some cases counteract the geopolitical factors exacerbating AI risks.

The taxonomy of functions we have presented is not exhaustive, nor do we argue that our institutional grouping is the most promising. Given the immense global opportunities and challenges presented by AI systems that may be on the horizon, the topic of international institutions for AI governance demands much greater analytical and practical attention.

Acknowledgements

We are grateful to the following people for discussion and input: Michael Aird, Jeff Alstott, Jon Bateman, Alexandra Belias, Dorothy Chou, Jack Clark, Lukas Finnveden, Iason Gabriel, Ben Garfinkel, Erich Grunewald, Oliver Guest, Jackie Kay, Noam Kolt, Sebastien Krier, Lucy Lim, Nicklas Lundblad, Stewart Patrick, George Perkovich, Toby Shevlane, Kent Walker and Ankur Vora. We would also like to thank participants of the September 2022 and June 2023 IGOTAI Seminars, in which early work was discussed.

Bibliography

  1@misc{rees_g20_2023,
  2  file = {Snapshot:/Users/lewisho/Zotero/storage/6FPNC3R3/g20-must-set-up-an-international-panel-on-technological-change-101679237287848.html:text/html},
  3  year = {2023},
  4  month = {March},
  5  author = {Rees, Martin and Sondhi, Shivaji and VijayRaghavan, Krishnaswamy},
  6  howpublished = {Hindustan Times},
  7  urldate = {2023-07-08},
  8  language = {en},
  9  abstract = {The panel’s mandate will be to assess where we are and where we are headed in the future when it comes to “post-human technology”. Unlike IPCC, which deals with much longer time scales, IPTC will need to work much faster and nimbly. It seems to us that the G20 is the right-sized organisation for chartering such a panel and enabling its functioning.},
 10  url = {https://www.hindustantimes.com/opinion/g20-must-set-up-an-international-panel-on-technological-change-101679237287848.html},
 11  title = {G20 must set up an international panel on technological change},
 12}
 13
 14@article{chowdhury_ai_2023,
 15  file = {Snapshot:/Users/lewisho/Zotero/storage/PEE83TB7/ai-desperately-needs-global-oversight.html:text/html},
 16  keywords = {artificial intelligence, ethics, government, tech policy and law},
 17  note = {Section: tags},
 18  year = {2023},
 19  author = {Chowdhury, Rumman},
 20  journal = {Wired},
 21  urldate = {2023-07-09},
 22  language = {en-US},
 23  abstract = {As ChatGPT and its ilk continue to spread, countries need an independent board to hold AI companies accountable and limit harms.},
 24  url = {https://www.wired.com/story/ai-desperately-needs-global-oversight/},
 25  issn = {1059-1028},
 26  title = {{AI} {Desperately} {Needs} {Global} {Oversight}},
 27}
 28
 29@misc{kakkad_new_2023,
 30  file = {Snapshot:/Users/lewisho/Zotero/storage/2XY78J6T/new-national-purpose-innovation-can-power-future-britain.html:text/html},
 31  year = {2023},
 32  author = {Kakkad, Jeegar and Macon-Cooney, Benedict and Northend, Jess and Phillips, James and Rajkumar, Nitarshan and Stanley, Luke and Westgarth, Tom},
 33  urldate = {2023-07-09},
 34  language = {en-GB},
 35  abstract = {A New National Purpose: Innovation Can Power the Future of Britain},
 36  url = {https://www.institute.global/insights/politics-and-governance/new-national-purpose-innovation-can-power-future-britain},
 37  shorttitle = {A {New} {National} {Purpose}},
 38  title = {A {New} {National} {Purpose}: {Innovation} {Can} {Power} the {Future} of {Britain}},
 39}
 40
 41@misc{dubner_satya_nodate,
 42  file = {Snapshot:/Users/lewisho/Zotero/storage/U74DAPBC/satya-nadellas-intelligence-is-not-artificial.html:text/html},
 43  author = {Dubner, Stephen},
 44  urldate = {2023-07-09},
 45  language = {en},
 46  abstract = {Satya Nadella’s Intelligence Is Not Artificial - Freakonomics},
 47  url = {https://freakonomics.com/podcast/satya-nadellas-intelligence-is-not-artificial/},
 48  title = {Satya {Nadella}’s {Intelligence} {Is} {Not} {Artificial}},
 49}
 50
 51@misc{noauthor_secretary-general_2023,
 52  year = {2023},
 53  month = {June},
 54  howpublished = {United Nations},
 55  key = {un},
 56  urldate = {2023-07-09},
 57  url = {https://press.un.org/en/2023/sgsm21832.doc.htm},
 58  title = {Secretary-{General} {Urges} {Broad} {Engagement} from {All} {Stakeholders} towards {United} {Nations} {Code} of {Conduct} for {Information} {Integrity} on {Digital} {Platforms} {\textbar} {UN} {Press}},
 59}
 60
 61@misc{noauthor_elders_2023,
 62  year = {2023},
 63  month = {May},
 64  howpublished = {The Elders},
 65  urldate = {2023-07-09},
 66  key = {elders},
 67  language = {en},
 68  abstract = {The Elders today call on world leaders to work together urgently on the design of strong international governance, to allow all humanity to take advantage of the opportunities of Artificial Intelligence (AI), while limiting the enormous risks.},
 69  url = {https://theelders.org/news/elders-urge-global-co-operation-manage-risks-and-share-benefits-ai},
 70  title = {The {Elders} urge global co-operation to manage risks and share benefits of {AI}},
 71}
 72
 73@article{awokuse_stronger_2010,
 74  file = {Full Text PDF:/Users/lewisho/Zotero/storage/4IWASB5L/Awokuse and Yin - 2008 - Do Stronger Intellectual Property Rights Protectio.pdf:application/pdf},
 75  year = {2010},
 76  author = {Awokuse, Titus and Yin, Hong},
 77  journal = {World Development},
 78  number = {8},
 79  abstract = {Most of the previous studies on the effect of IPR protection on international trade have been from the perspective of major industrialized nations. However, much of the current debate on the effects of IPR protection involves large developing countries with high threat of imitation. This study contributes to the literature by analyzing the impact of the strengthening of patent laws in China on its bilateral trade flows. We estimate the effects of patent rights protection on China’s imports at the aggregate and detailed product categories for both OECD (developed) and non-OECD (developing) countries. The empirical results suggest that increased patent rights protection stimulate China’s imports, particularly in the knowledge-intensive product categories. Furthermore, while the evidence in support of the market expansion effect is significant for imports from OECD countries, it is rather weak and mostly insignificant for imports from non-OECD countries.},
 80  shorttitle = {Do {Stronger} {Intellectual} {Property} {Rights} {Protection} {Induce} {More} {Bilateral} {Trade}?},
 81  volume = {38},
 82  title = {Do {Stronger} {Intellectual} {Property} {Rights} {Protection} {Induce} {More} {Bilateral} {Trade}? {Evidence} from {China}'s {Imports}},
 83}
 84
 85@article{vinuesa_role_2020,
 86  file = {Full Text PDF:/Users/lewisho/Zotero/storage/N6W8NI5A/Vinuesa et al. - 2020 - The role of artificial intelligence in achieving t.pdf:application/pdf},
 87  pages = {233},
 88  keywords = {Computational science, Developing world, Energy efficiency},
 89  note = {Number: 1
 90Publisher: Nature Publishing Group},
 91  year = {2020},
 92  month = {January},
 93  author = {Vinuesa, Ricardo and Azizpour, Hossein and Leite, Iolanda and Balaam, Madeline and Dignum, Virginia and Domisch, Sami and Felländer, Anna and Langhans, Simone Daniela and Tegmark, Max and Fuso Nerini, Francesco},
 94  journal = {Nature Communications},
 95  urldate = {2023-07-09},
 96  number = {1},
 97  language = {en},
 98  abstract = {The emergence of artificial intelligence (AI) and its progressively wider impact on many sectors requires an assessment of its effect on the achievement of the Sustainable Development Goals. Using a consensus-based expert elicitation process, we find that AI can enable the accomplishment of 134 targets across all the goals, but it may also inhibit 59 targets. However, current research foci overlook important aspects. The fast development of AI needs to be supported by the necessary regulatory insight and oversight for AI-based technologies to enable sustainable development. Failure to do so could result in gaps in transparency, safety, and ethical standards.},
 99  doi = {10.1038/s41467-019-14108-y},
100  url = {https://www.nature.com/articles/s41467-019-14108-y},
101  issn = {2041-1723},
102  copyright = {2020 The Author(s)},
103  volume = {11},
104  title = {The role of artificial intelligence in achieving the {Sustainable} {Development} {Goals}},
105}
106
107@article{urbina_dual_2022,
108  file = {Accepted Version:/Users/lewisho/Zotero/storage/23RFNGTS/Urbina et al. - 2022 - Dual use of artificial-intelligence-powered drug d.pdf:application/pdf},
109  pages = {189--191},
110  keywords = {Cheminformatics, Drug safety, Ethics, Software, Toxicology},
111  note = {Number: 3
112Publisher: Nature Publishing Group},
113  year = {2022},
114  month = {March},
115  author = {Urbina, Fabio and Lentzos, Filippa and Invernizzi, Cédric and Ekins, Sean},
116  journal = {Nature Machine Intelligence},
117  urldate = {2023-07-09},
118  number = {3},
119  language = {en},
120  abstract = {An international security conference explored how artificial intelligence (AI) technologies for drug discovery could be misused for de novo design of biochemical weapons. A thought experiment evolved into a computational proof.},
121  doi = {10.1038/s42256-022-00465-9},
122  url = {https://www.nature.com/articles/s42256-022-00465-9},
123  issn = {2522-5839},
124  copyright = {2022 Springer Nature Limited},
125  volume = {4},
126  title = {Dual use of artificial-intelligence-powered drug discovery},
127}
128
129@article{jumper_highly_2021,
130  file = {Full Text PDF:/Users/lewisho/Zotero/storage/5WYYQ9XJ/Jumper et al. - 2021 - Highly accurate protein structure prediction with .pdf:application/pdf},
131  pages = {583--589},
132  keywords = {Computational biophysics, Machine learning, Protein structure predictions, Structural biology},
133  note = {Number: 7873
134Publisher: Nature Publishing Group},
135  year = {2021},
136  month = {August},
137  author = {Jumper, John and Evans, Richard and Pritzel, Alexander and Green, Tim and Figurnov, Michael and Ronneberger, Olaf and Tunyasuvunakool, Kathryn and Bates, Russ and Žídek, Augustin and Potapenko, Anna and Bridgland, Alex and Meyer, Clemens and Kohl, Simon A. A. and Ballard, Andrew J. and Cowie, Andrew and Romera-Paredes, Bernardino and Nikolov, Stanislav and Jain, Rishub and Adler, Jonas and Back, Trevor and Petersen, Stig and Reiman, David and Clancy, Ellen and Zielinski, Michal and Steinegger, Martin and Pacholska, Michalina and Berghammer, Tamas and Bodenstein, Sebastian and Silver, David and Vinyals, Oriol and Senior, Andrew W. and Kavukcuoglu, Koray and Kohli, Pushmeet and Hassabis, Demis},
138  journal = {Nature},
139  urldate = {2023-07-09},
140  number = {7873},
141  language = {en},
142  abstract = {Proteins are essential to life, and understanding their structure can facilitate a mechanistic understanding of their function. Through an enormous experimental effort1–4, the structures of around 100,000 unique proteins have been determined5, but this represents a small fraction of the billions of known protein sequences6,7. Structural coverage is bottlenecked by the months to years of painstaking effort required to determine a single protein structure. Accurate computational approaches are needed to address this gap and to enable large-scale structural bioinformatics. Predicting the three-dimensional structure that a protein will adopt based solely on its amino acid sequence—the structure prediction component of the ‘protein folding problem’8—has been an important open research problem for more than 50 years9. Despite recent progress10–14, existing methods fall far short of atomic accuracy, especially when no homologous structure is available. Here we provide the first computational method that can regularly predict protein structures with atomic accuracy even in cases in which no similar structure is known. We validated an entirely redesigned version of our neural network-based model, AlphaFold, in the challenging 14th Critical Assessment of protein Structure Prediction (CASP14)15, demonstrating accuracy competitive with experimental structures in a majority of cases and greatly outperforming other methods. Underpinning the latest version of AlphaFold is a novel machine learning approach that incorporates physical and biological knowledge about protein structure, leveraging multi-sequence alignments, into the design of the deep learning algorithm.},
143  doi = {10.1038/s41586-021-03819-2},
144  url = {https://www.nature.com/articles/s41586-021-03819-2},
145  issn = {1476-4687},
146  copyright = {2021 The Author(s)},
147  volume = {596},
148  title = {Highly accurate protein structure prediction with {AlphaFold}},
149}
150
151@misc{noauthor_snyk_nodate,
152  file = {Snapshot:/Users/lewisho/Zotero/storage/CJJSVS4I/snyk.io.html:text/html},
153  howpublished = {Snyk},
154  urldate = {2023-07-09},
155  language = {en-US},
156  abstract = {Snyk helps software-driven businesses develop fast and stay secure. Continuously find and fix vulnerabilities for npm, Maven, NuGet, RubyGems, PyPI and more.},
157  url = {https://snyk.io/},
158  title = {Snyk},
159}
160
161@misc{noauthor_codeql_nodate,
162  file = {CodeQL:/Users/lewisho/Zotero/storage/9S7YEZND/codeql.github.com.html:text/html},
163  urldate = {2023-07-09},
164  url = {https://codeql.github.com/},
165  title = {{CodeQL}},
166}
167
168@misc{steinhart_what_2023,
169  file = {Snapshot:/Users/lewisho/Zotero/storage/4RF6YGSP/what-will-gpt-2030-look-like.html:text/html},
170  year = {2023},
171  month = {June},
172  author = {Steinhart, Jacob},
173  howpublished = {Bounded Regret},
174  urldate = {2023-07-09},
175  language = {en},
176  abstract = {GPT-4 surprised many people with its abilities at coding, creative brainstorming, letter-writing, and other skills. How can we be less surprised by developments in machine learning? In this post, I’ll forecast the properties of large pretrained ML systems in 2030.},
177  url = {https://bounded-regret.ghost.io/what-will-gpt-2030-look-like/},
178  title = {What will {GPT}-2030 look like?},
179}
180
181@article{bond_fake_2023,
182  file = {Snapshot:/Users/lewisho/Zotero/storage/NZFWWSE4/fake-viral-images-of-an-explosion-at-the-pentagon-were-probably-created-by-ai.html:text/html},
183  year = {2023},
184  month = {May},
185  author = {Bond, Shannon},
186  journal = {NPR},
187  urldate = {2023-07-09},
188  language = {en},
189  abstract = {Authorities quickly confirmed that no explosion had taken place but the faked images spread on Twitter for a short time. The incident briefly sent the stock market lower.},
190  url = {https://www.npr.org/2023/05/22/1177590231/fake-viral-images-of-an-explosion-at-the-pentagon-were-probably-created-by-ai},
191  title = {Fake viral images of an explosion at the {Pentagon} were probably created by {AI}},
192  chapter = {Untangling Disinformation},
193}
194
195@misc{goldstein_generative_2023,
196  file = {arXiv Fulltext PDF:/Users/lewisho/Zotero/storage/F6H62PNI/Goldstein et al. - 2023 - Generative Language Models and Automated Influence.pdf:application/pdf;arXiv.org Snapshot:/Users/lewisho/Zotero/storage/PJJZ2H2A/2301.html:text/html},
197  annote = {Comment: 82 pages, 26 figures},
198  keywords = {Computer Science - Computers and Society},
199  note = {arXiv:2301.04246 [cs]},
200  year = {2023},
201  month = {January},
202  author = {Goldstein, Josh A. and Sastry, Girish and Musser, Micah and DiResta, Renee and Gentzel, Matthew and Sedova, Katerina},
203  publisher = {arXiv},
204  urldate = {2023-07-09},
205  abstract = {Generative language models have improved drastically, and can now produce realistic text outputs that are difficult to distinguish from human-written content. For malicious actors, these language models bring the promise of automating the creation of convincing and misleading text for use in influence operations. This report assesses how language models might change influence operations in the future, and what steps can be taken to mitigate this threat. We lay out possible changes to the actors, behaviors, and content of online influence operations, and provide a framework for stages of the language model-to-influence operations pipeline that mitigations could target (model construction, model access, content dissemination, and belief formation). While no reasonable mitigation can be expected to fully prevent the threat of AI-enabled influence operations, a combination of multiple mitigations may make an important difference.},
206  doi = {10.48550/arXiv.2301.04246},
207  url = {http://arxiv.org/abs/2301.04246},
208  shorttitle = {Generative {Language} {Models} and {Automated} {Influence} {Operations}},
209  title = {Generative {Language} {Models} and {Automated} {Influence} {Operations}: {Emerging} {Threats} and {Potential} {Mitigations}},
210}
211
212@misc{hendrycks_unsolved_2022,
213  file = {arXiv.org Snapshot:/Users/lewisho/Zotero/storage/CE4B3F6B/2109.html:text/html;Full Text PDF:/Users/lewisho/Zotero/storage/VHVSEEQY/Hendrycks et al. - 2022 - Unsolved Problems in ML Safety.pdf:application/pdf},
214  annote = {Comment: Position Paper},
215  keywords = {Computer Science - Artificial Intelligence, Computer Science - Computation and Language, Computer Science - Computer Vision and Pattern Recognition, Computer Science - Machine Learning},
216  note = {arXiv:2109.13916 [cs]},
217  year = {2022},
218  month = {June},
219  author = {Hendrycks, Dan and Carlini, Nicholas and Schulman, John and Steinhardt, Jacob},
220  publisher = {arXiv},
221  urldate = {2023-07-09},
222  abstract = {Machine learning (ML) systems are rapidly increasing in size, are acquiring new capabilities, and are increasingly deployed in high-stakes settings. As with other powerful technologies, safety for ML should be a leading research priority. In response to emerging safety challenges in ML, such as those introduced by recent large-scale models, we provide a new roadmap for ML Safety and refine the technical problems that the field needs to address. We present four problems ready for research, namely withstanding hazards ("Robustness"), identifying hazards ("Monitoring"), reducing inherent model hazards ("Alignment"), and reducing systemic hazards ("Systemic Safety"). Throughout, we clarify each problem's motivation and provide concrete research directions.},
223  url = {http://arxiv.org/abs/2109.13916},
224  title = {Unsolved {Problems} in {ML} {Safety}},
225}
226
227@misc{amodei_concrete_2016,
228  file = {arXiv.org Snapshot:/Users/lewisho/Zotero/storage/SCY6S6WG/1606.html:text/html;Full Text PDF:/Users/lewisho/Zotero/storage/EHY55AQR/Amodei et al. - 2016 - Concrete Problems in AI Safety.pdf:application/pdf},
229  annote = {Comment: 29 pages},
230  keywords = {Computer Science - Artificial Intelligence, Computer Science - Machine Learning},
231  note = {arXiv:1606.06565 [cs]},
232  year = {2016},
233  month = {July},
234  author = {Amodei, Dario and Olah, Chris and Steinhardt, Jacob and Christiano, Paul and Schulman, John and Mané, Dan},
235  publisher = {arXiv},
236  urldate = {2023-07-09},
237  abstract = {Rapid progress in machine learning and artificial intelligence (AI) has brought increasing attention to the potential impacts of AI technologies on society. In this paper we discuss one such potential impact: the problem of accidents in machine learning systems, defined as unintended and harmful behavior that may emerge from poor design of real-world AI systems. We present a list of five practical research problems related to accident risk, categorized according to whether the problem originates from having the wrong objective function ("avoiding side effects" and "avoiding reward hacking"), an objective function that is too expensive to evaluate frequently ("scalable supervision"), or undesirable behavior during the learning process ("safe exploration" and "distributional shift"). We review previous work in these areas as well as suggesting research directions with a focus on relevance to cutting-edge AI systems. Finally, we consider the high-level question of how to think most productively about the safety of forward-looking applications of AI.},
238  url = {http://arxiv.org/abs/1606.06565},
239  title = {Concrete {Problems} in {AI} {Safety}},
240}
241
242@misc{arnold_ai_2021,
243  file = {Snapshot:/Users/lewisho/Zotero/storage/94NYZHQX/ai-accidents-an-emerging-threat.html:text/html},
244  year = {2021},
245  author = {Arnold, Zachary and Toner, Helen},
246  howpublished = {Center for Security and Emerging Technology},
247  urldate = {2023-07-09},
248  language = {en-US},
249  abstract = {As modern machine learning systems become more widely used, the potential costs of malfunctions grow. This policy brief describes how trends we already see today—both in newly deployed artificial intelligence systems and in older technologies—show how damaging the AI accidents of the future could be. It describes a wide range of hypothetical but realistic scenarios to illustrate the risks of AI accidents and offers concrete policy suggestions to reduce these risks.},
250  url = {https://cset.georgetown.edu/publication/ai-accidents-an-emerging-threat/},
251  shorttitle = {{AI} {Accidents}},
252  title = {{AI} {Accidents}: {An} {Emerging} {Threat}},
253}
254
255@misc{noauthor_statement_2023,
256  file = {Snapshot:/Users/lewisho/Zotero/storage/3V388YAB/statement-on-ai-risk.html:text/html},
257  year = {2023},
258  howpublished = {Center for AI Safety},
259  key = {center},
260  urldate = {2023-07-09},
261  abstract = {A statement jointly signed by a historic coalition of experts: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”},
262  url = {https://www.safe.ai/statement-on-ai-risk},
263  title = {Statement on {AI} {Risk}},
264}
265
266@misc{hobbhahn_trends_2022,
267  file = {Snapshot:/Users/lewisho/Zotero/storage/H2IN66TH/trends-in-gpu-price-performance.html:text/html},
268  year = {2022},
269  author = {Hobbhahn, Marius},
270  howpublished = {Epoch},
271  urldate = {2023-07-09},
272  language = {en},
273  abstract = {Using a dataset of 470 models of graphics processing units released between 2006 and 2021, we find that the amount of floating-point operations/second per \$ doubles every {\textasciitilde}2.5 years.},
274  url = {https://epochai.org/blog/trends-in-gpu-price-performance},
275  title = {Trends in {GPU} price-performance},
276}
277
278@misc{erdil_algorithmic_2023,
279  file = {Snapshot:/Users/lewisho/Zotero/storage/B8V3SDHB/revisiting-algorithmic-progress.html:text/html},
280  year = {2023},
281  author = {Erdil, Ege and Besiroglu, Tamay},
282  publisher = {arXiv},
283  urldate = {2023-07-09},
284  language = {en},
285  abstract = {We use a dataset of over a hundred computer vision models from the last decade to investigate how better algorithms and architectures have enabled researchers to use compute and data more efficiently. We find that every 9 months, the introduction of better algorithms contribute the equivalent of a doubling of compute budgets.},
286  url = {https://arxiv.org/abs/2212.05153},
287  title = {Algorithmic {Progress} in {Computer} {Vision}},
288}
289
290@article{toner_illusion_2023,
291  file = {Snapshot:/Users/lewisho/Zotero/storage/RYPKNIHC/illusion-chinas-ai-prowess-regulation.html:text/html},
292  keywords = {Artificial Intelligence, Automation, China, East Asia, Law, Politics \& Society, Science \& Technology, United States},
293  year = {2023},
294  month = {June},
295  author = {Toner, Helen and Xiao, Jenny and Ding, Jeffrey},
296  journal = {Foreign Affairs},
297  urldate = {2023-07-09},
298  language = {en-US},
299  abstract = {Regulating AI will not set America back in the technology race.},
300  url = {https://www.foreignaffairs.com/china/illusion-chinas-ai-prowess-regulation},
301  issn = {0015-7120},
302  title = {The {Illusion} of {China}’s {AI} {Prowess}},
303}
304
305@misc{noauthor_netherlands_2023,
306  file = {Snapshot:/Users/lewisho/Zotero/storage/DNLPEWUY/the-netherlands-joins-the-us-in-restricting-semiconductor-exports-to-china.html:text/html},
307  year = {2023},
308  month = {March},
309  howpublished = {Allen Overy},
310  urldate = {2023-07-09},
311  key = {allen},
312  language = {en},
313  abstract = {In January 2023, the Netherlands and Japan agreed with the United States to impose controls on the export of certain semiconductors and related products to China.  This followed a push by the Biden Administration to ensure the effectiveness of related U.S. export controls that were introduced in October 2022.},
314  url = {https://www.allenovery.com/en-gb/global/news-and-insights/publications/the-netherlands-joins-the-us-in-restricting-semiconductor-exports-to-china},
315  title = {The {Netherlands} joins the {U}.{S}. in restricting semiconductor exports to {China}},
316}
317
318@misc{noauthor_pm_2023,
319  file = {Snapshot:/Users/lewisho/Zotero/storage/NVM2G3KI/pm-urges-tech-leaders-to-grasp-generational-opportunities-and-challenges-of-ai.html:text/html},
320  year = {2023},
321  key = {govuk},
322  howpublished = {GOV.UK},
323  urldate = {2023-07-09},
324  language = {en},
325  abstract = {The UK must act quickly if we want to retain our position as one of the world’s tech capitals, Prime Minister Rishi Sunak will tell tech leaders today [Monday 12 June].},
326  url = {https://www.gov.uk/government/news/pm-urges-tech-leaders-to-grasp-generational-opportunities-and-challenges-of-ai},
327  title = {{PM} urges tech leaders to grasp generational opportunities and challenges of {AI}},
328}
329
330@misc{singh_anthropics_2023,
331  year = {2023},
332  month = {April},
333  author = {Singh, Devin Coldewey {and} Manish, Kyle Wiggers},
334  howpublished = {TechCrunch},
335  urldate = {2023-07-09},
336  language = {en-US},
337  abstract = {AI research startup Anthropic aims to raise as much as \$5 billion over the next two years to take on rival OpenAI.},
338  url = {https://techcrunch.com/2023/04/06/anthropics-5b-4-year-plan-to-take-on-openai/},
339  title = {Anthropic's \${5B}, 4-year plan to take on {OpenAI}},
340}
341
342@article{veale_ai_2023,
343  file = {Veale et al. - 2023 - AI and Global Governance Modalities, Rationales, .pdf:/Users/lewisho/Zotero/storage/4E2INQF9/Veale et al. - 2023 - AI and Global Governance Modalities, Rationales, .pdf:application/pdf},
344  year = {2023},
345  author = {Veale, Michael and Matus, Kira and Gorwa, Robert},
346  journal = {Annual Review of Law and Social Science},
347  language = {en},
348  volume = {19},
349  title = {{AI} and {Global} {Governance}: {Modalities}, {Rationales}, {Tensions}},
350}
351
352@misc{noauthor_preparing_nodate,
353  file = {Snapshot:/Users/lewisho/Zotero/storage/NDQZCNMN/preparingreports.html:text/html},
354  urldate = {2023-07-09},
355  url = {https://www.ipcc.ch/about/preparingreports/},
356  title = {Preparing {Reports} — {IPCC}},
357}
358
359@article{shaw_relevant_2004,
360  file = {Full Text PDF:/Users/lewisho/Zotero/storage/XV3TGF7A/Shaw and Robinson - 2004 - Relevant But Not Prescriptive Science Policy Mode.pdf:application/pdf},
361  pages = {84--95},
362  year = {2004},
363  month = {January},
364  author = {Shaw, Alison and Robinson, John},
365  journal = {Philosophy Today},
366  abstract = {The Intergovernmental Panel on Climate Change (IPCC) represents perhaps the largest example of ‘mandated science’ ever undertaken. It activities therefore raise a number of critical issues concerning the science/society interface. While previous studies of the IPCC have focused on the scientific credibility of its findings, this paper will examine the credibility of the process and protocols employed to assess “policy relevant but not policy prescriptive scientific information”. In particular we will examine two unique devices used in the IPCC: the Summary for Policymakers (SPM) and the Policy Relevant Scientific Questions (PRSQ). It will be argued that, despite unhappiness on the part of some of the scientific participants, the negotiation of meaning given rise to in these processes represents a credible and useful way to bridge the science/policy divide and in turn provides insights into ways to bridge the larger issues of the role of science in society.},
367  doi = {10.5840/philtoday200448Supplement9},
368  shorttitle = {Relevant {But} {Not} {Prescriptive}},
369  volume = {48},
370  title = {Relevant {But} {Not} {Prescriptive}: {Science} {Policy} {Models} within the {IPCC}},
371}
372
373@techreport{toivanen_significance_2017,
374  file = {Toivanen - 2017 - The Significance of Strategic Foresight in Verific.pdf:/Users/lewisho/Zotero/storage/3LNAJGHV/Toivanen - 2017 - The Significance of Strategic Foresight in Verific.pdf:application/pdf},
375  pages = {LLNL--TR--738786, 1502006, 892173},
376  doi = {10.2172/1502006},
377  year = {2017},
378  month = {September},
379  author = {Toivanen, Henrietta},
380  urldate = {2023-07-09},
381  number = {LLNL-TR--738786, 1502006, 892173},
382  language = {en},
383  url = {https://www.osti.gov/servlets/purl/1502006/},
384  shorttitle = {The {Significance} of {Strategic} {Foresight} in {Verification} {Technologies}},
385  title = {The {Significance} of {Strategic} {Foresight} in {Verification} {Technologies}: {A} {Case} {Study} of the {INF} {Treaty}},
386}
387
388@misc{shavit_what_2023,
389  file = {arXiv Fulltext PDF:/Users/lewisho/Zotero/storage/NDWX7245/Shavit - 2023 - What does it take to catch a Chinchilla Verifying.pdf:application/pdf;arXiv.org Snapshot:/Users/lewisho/Zotero/storage/M8DR2UAH/2303.html:text/html},
390  keywords = {Computer Science - Artificial Intelligence, Computer Science - Machine Learning},
391  note = {arXiv:2303.11341 [cs]},
392  year = {2023},
393  month = {May},
394  author = {Shavit, Yonadav},
395  publisher = {arXiv},
396  urldate = {2023-07-09},
397  abstract = {As advanced machine learning systems' capabilities begin to play a significant role in geopolitics and societal order, it may become imperative that (1) governments be able to enforce rules on the development of advanced ML systems within their borders, and (2) countries be able to verify each other's compliance with potential future international agreements on advanced ML development. This work analyzes one mechanism to achieve this, by monitoring the computing hardware used for large-scale NN training. The framework's primary goal is to provide governments high confidence that no actor uses large quantities of specialized ML chips to execute a training run in violation of agreed rules. At the same time, the system does not curtail the use of consumer computing devices, and maintains the privacy and confidentiality of ML practitioners' models, data, and hyperparameters. The system consists of interventions at three stages: (1) using on-chip firmware to occasionally save snapshots of the the neural network weights stored in device memory, in a form that an inspector could later retrieve; (2) saving sufficient information about each training run to prove to inspectors the details of the training run that had resulted in the snapshotted weights; and (3) monitoring the chip supply chain to ensure that no actor can avoid discovery by amassing a large quantity of un-tracked chips. The proposed design decomposes the ML training rule verification problem into a series of narrow technical challenges, including a new variant of the Proof-of-Learning problem [Jia et al. '21].},
398  doi = {10.48550/arXiv.2303.11341},
399  url = {http://arxiv.org/abs/2303.11341},
400  shorttitle = {What does it take to catch a {Chinchilla}?},
401  title = {What does it take to catch a {Chinchilla}? {Verifying} {Rules} on {Large}-{Scale} {Neural} {Network} {Training} via {Compute} {Monitoring}},
402}
403
404@misc{noauthor_memorandum_2008,
405  file = {House of Lords - European Union Committee - Written Evidence:/Users/lewisho/Zotero/storage/BSPYLD94/132we08.html:text/html},
406  year = {2008},
407  howpublished = {UK Parliament},
408  key = {UK},
409  urldate = {2023-07-09},
410  url = {https://publications.parliament.uk/pa/ld200809/ldselect/ldeucom/132/132we08.htm},
411  title = {Memorandum by the {Financial} {Action} {Task} {Force} ({FATF}) {Secretariat}},
412}
413
414@book{gallagher_politics_1999,
415  file = {The Politics of Verification - Nancy W. Gallagher - Google Books:/Users/lewisho/Zotero/storage/9VN3U3UR/The_Politics_of_Verification.html:text/html},
416  year = {1999},
417  author = {Gallagher, Nancy},
418  publisher = {Johns Hopkins University Press},
419  urldate = {2023-07-09},
420  title = {The {Politics} of {Verification}},
421}
422
423@article{spirling_why_2023,
424  file = {Full Text PDF:/Users/lewisho/Zotero/storage/G8HM658L/Spirling - 2023 - Why open-source generative AI models are an ethica.pdf:application/pdf;Snapshot:/Users/lewisho/Zotero/storage/ECXTQYTH/d41586-023-01295-4.html:text/html},
425  pages = {413--413},
426  keywords = {Ethics, Machine learning, Scientific community, Technology},
427  note = {Bandiera\_abtest: a
428Cg\_type: World View
429Number: 7957
430Publisher: Nature Publishing Group
431Subject\_term: Ethics, Machine learning, Technology, Scientific community},
432  year = {2023},
433  month = {April},
434  author = {Spirling, Arthur},
435  journal = {Nature},
436  urldate = {2023-07-09},
437  number = {7957},
438  language = {en},
439  abstract = {Researchers should avoid the lure of proprietary models and develop transparent large language models to ensure reproducibility.},
440  doi = {10.1038/d41586-023-01295-4},
441  url = {https://www.nature.com/articles/d41586-023-01295-4},
442  copyright = {2023 Springer Nature Limited},
443  volume = {616},
444  title = {Why open-source generative {AI} models are an ethical way forward for science},
445}
446
447@misc{rauf_atoms_2015,
448  file = {From ‘Atoms for Peace’ to an IAEA Nuclear Fuel Bank | Arms Control Association:/Users/lewisho/Zotero/storage/MRN8K829/‘atoms-peace’-iaea-nuclear-fuel-bank.html:text/html},
449  year = {2015},
450  month = {October},
451  author = {Rauf, Tariq},
452  howpublished = {Arms Control Association},
453  urldate = {2023-07-09},
454  url = {https://www.armscontrol.org/act/2015-10/features/%E2%80%98atoms-peace%E2%80%99-iaea-nuclear-fuel-bank},
455  title = {From ‘{Atoms} for {Peace}’ to an {IAEA} {Nuclear} {Fuel} {Bank} {\textbar} {Arms} {Control} {Association}},
456}
457
458@article{mohamed_decolonial_2020,
459  file = {Full Text PDF:/Users/lewisho/Zotero/storage/2WQWENKJ/Mohamed et al. - 2020 - Decolonial AI Decolonial Theory as Sociotechnical.pdf:application/pdf},
460  pages = {659--684},
461  keywords = {Affective community, Artificial intelligence, Coloniality, Critical technical practice, Decolonisation, Intercultural ethics, Sociotechnical foresight},
462  year = {2020},
463  month = {December},
464  author = {Mohamed, Shakir and Png, Marie-Therese and Isaac, William},
465  journal = {Philosophy \& Technology},
466  urldate = {2023-07-09},
467  number = {4},
468  language = {en},
469  abstract = {This paper explores the important role of critical science, and in particular of post-colonial and decolonial theories, in understanding and shaping the ongoing advances in artificial intelligence. Artificial intelligence (AI) is viewed as amongst the technological advances that will reshape modern societies and their relations. While the design and deployment of systems that continually adapt holds the promise of far-reaching positive change, they simultaneously pose significant risks, especially to already vulnerable peoples. Values and power are central to this discussion. Decolonial theories use historical hindsight to explain patterns of power that shape our intellectual, political, economic, and social world. By embedding a decolonial critical approach within its technical practice, AI communities can develop foresight and tactics that can better align research and technology development with established ethical principles, centring vulnerable peoples who continue to bear the brunt of negative impacts of innovation and scientific progress. We highlight problematic applications that are instances of coloniality, and using a decolonial lens, submit three tactics that can form a decolonial field of artificial intelligence: creating a critical technical practice of AI, seeking reverse tutelage and reverse pedagogies, and the renewal of affective and political communities. The years ahead will usher in a wave of new scientific breakthroughs and technologies driven by AI research, making it incumbent upon AI communities to strengthen the social contract through ethical foresight and the multiplicity of intellectual perspectives available to us, ultimately supporting future technologies that enable greater well-being, with the goal of beneficence and justice for all.},
470  doi = {10.1007/s13347-020-00405-8},
471  url = {https://doi.org/10.1007/s13347-020-00405-8},
472  shorttitle = {Decolonial {AI}},
473  issn = {2210-5441},
474  volume = {33},
475  title = {Decolonial {AI}: {Decolonial} {Theory} as {Sociotechnical} {Foresight} in {Artificial} {Intelligence}},
476}
477
478@misc{noauthor_what_2020,
479  file = {Snapshot:/Users/lewisho/Zotero/storage/WAGIS4LX/what-advance-market-commitment-and-how-could-it-help-beat-covid-19.html:text/html},
480  year = {2020},
481  howpublished = {Gavi, the Vaccine Alliance},
482  urldate = {2023-07-09},
483  key = {Gavi},
484  language = {en},
485  url = {https://www.gavi.org/vaccineswork/what-advance-market-commitment-and-how-could-it-help-beat-covid-19},
486  shorttitle = {What is an {Advance} {Market} {Commitment} and how could it help beat {COVID}-19?},
487  title = {What is an {Advance} {Market} {Commitment} and how could it help beat {COVID}-19? {\textbar} {Gavi}, the {Vaccine} {Alliance}},
488}
489
490@inproceedings{cihon_should_2020,
491  file = {Cihon et al. - 2020 - Should Artificial Intelligence Governance be Centr.pdf:/Users/lewisho/Zotero/storage/NP3Z6SLX/Cihon et al. - 2020 - Should Artificial Intelligence Governance be Centr.pdf:application/pdf},
492  pages = {228--234},
493  year = {2020},
494  month = {February},
495  author = {Cihon, Peter and Maas, Matthijs M. and Kemp, Luke},
496  publisher = {ACM},
497  booktitle = {Proceedings of the {AAAI}/{ACM} {Conference} on {AI}, {Ethics}, and {Society}},
498  urldate = {2023-07-09},
499  language = {en},
500  abstract = {The invention of atomic energy posed a novel global challenge: could the technology be controlled to avoid destructive uses and an existentially dangerous arms race while permitting the broad sharing of its benefits? From 1944 onwards, scientists, policymakers, and other t echnical specialists began to confront this challenge and explored policy options for dealing with the impact of nuclear technology. We focus on the years 1944 to 1951 and review this period for lessons for the governance of powerful technologies, and find the following: Radical schemes for international control can get broad support when confronted by existentially dangerous technologies, but this support can be tenuous and cynical. Secrecy is likely to play an important, and perhaps harmful, role. The public sphere may be an important source of influence, both in general and in particular in favor of cooperation, but also one that is manipulable and poorly informed. Technical experts may play a critical role, but need to be politically savvy. Overall, policymaking may look more like “muddling through” than clear-eyed grand strategy. Cooperation may be risky, and there may be many obstacles to success.},
501  doi = {10.1145/3375627.3375857},
502  url = {https://dl.acm.org/doi/10.1145/3375627.3375857},
503  shorttitle = {Should {Artificial} {Intelligence} {Governance} be {Centralised}?},
504  isbn = {978-1-4503-7110-0},
505  title = {Should {Artificial} {Intelligence} {Governance} be {Centralised}?: {Design} {Lessons} from {History}},
506  address = {New York NY USA},
507}
508
509@misc{taori_alpaca_2023,
510  year = {2023},
511  month = {March},
512  author = {Taori, Rohan and Gulrajani, Ishaan and Zhang, Tianyi and Dubois, Yann and Li, Xuechen and Guestrin, Carlos and Liang, Percy and Hashimoto, Tatsunori},
513  howpublished = {Stanford CRFM},
514  urldate = {2023-07-09},
515  url = {https://crfm.stanford.edu/2023/03/13/alpaca.html},
516  title = {Alpaca: {A} {Strong}, {Replicable} {Instruction}-{Following} {Model}},
517}
518
519@article{nast_maori_nodate,
520  keywords = {artificial intelligence, data, internet culture},
521  note = {Section: tags},
522  author = {Nast, Condé},
523  howpublished = {Wired UK},
524  urldate = {2023-07-09},
525  language = {en-GB},
526  abstract = {Te Hiku Media gathered huge swathes of Māori language data. Corporates are now trying to get the rights to it},
527  url = {https://www.wired.co.uk/article/maori-language-tech},
528  issn = {1357-0978},
529  title = {Māori are trying to save their language from {Big} {Tech}},
530}
531
532@misc{noauthor_unfccc_nodate,
533  file = {Technology Mechanism:/Users/lewisho/Zotero/storage/66Y4EXPH/technology-mechanism.html:text/html},
534  howpublished = {UNFCCC},
535  key = {UNFCCC},
536  urldate = {2023-07-09},
537  url = {https://unfccc.int/ttclear/support/technology-mechanism.html},
538  title = {{UNFCCC} {Technology} {Mechanism}},
539}
540
541@misc{hammond_opinion_2023,
542  year = {2023},
543  month = {May},
544  author = {Hammond, Samuel},
545  howpublished = {POLITICO},
546  urldate = {2023-07-09},
547  language = {en},
548  abstract = {AI presents an enormous threat. It deserves an enormous response.},
549  url = {https://www.politico.com/news/magazine/2023/05/08/manhattan-project-for-ai-safety-00095779},
550  title = {Opinion {\textbar} {We} {Need} a {Manhattan} {Project} for {AI} {Safety}},
551}
552
553@misc{shevlane_structured_2022,
554  file = {arXiv Fulltext PDF:/Users/lewisho/Zotero/storage/7QQZEKMD/Shevlane - 2022 - Structured access an emerging paradigm for safe A.pdf:application/pdf;arXiv.org Snapshot:/Users/lewisho/Zotero/storage/ZMVHS2DK/2201.html:text/html},
555  annote = {Comment: 28 pages},
556  keywords = {68T99, Computer Science - Artificial Intelligence, Computer Science - Human-Computer Interaction, Computer Science - Software Engineering},
557  note = {arXiv:2201.05159 [cs]},
558  year = {2022},
559  month = {April},
560  author = {Shevlane, Toby},
561  publisher = {arXiv},
562  urldate = {2023-07-09},
563  abstract = {Structured access is an emerging paradigm for the safe deployment of artificial intelligence (AI). Instead of openly disseminating AI systems, developers facilitate controlled, arm's length interactions with their AI systems. The aim is to prevent dangerous AI capabilities from being widely accessible, whilst preserving access to AI capabilities that can be used safely. The developer must both restrict how the AI system can be used, and prevent the user from circumventing these restrictions through modification or reverse engineering of the AI system. Structured access is most effective when implemented through cloud-based AI services, rather than disseminating AI software that runs locally on users' hardware. Cloud-based interfaces provide the AI developer greater scope for controlling how the AI system is used, and for protecting against unauthorized modifications to the system's design. This chapter expands the discussion of "publication norms" in the AI community, which to date has focused on the question of how the informational content of AI research projects should be disseminated (e.g., code and models). Although this is an important question, there are limits to what can be achieved through the control of information flows. Structured access views AI software not only as information that can be shared but also as a tool with which users can have arm's length interactions. There are early examples of structured access being practiced by AI developers, but there is much room for further development, both in the functionality of cloud-based interfaces and in the wider institutional framework.},
564  doi = {10.48550/arXiv.2201.05159},
565  url = {http://arxiv.org/abs/2201.05159},
566  shorttitle = {Structured access},
567  title = {Structured access: an emerging paradigm for safe {AI} deployment},
568}
569
570@techreport{schmidt_nscai_2021,
571  file = {Snapshot:/Users/lewisho/Zotero/storage/RSRCPCBP/2021-final-report.html:text/html},
572  year = {2021},
573  author = {Schmidt, Eric and Work, Robert and Catz, Safra and Horvitz, Eric and Chien, Steve and Jassy, Andrew and Clyburn, Mignon and Louie, Gilman and Darby, Chris and Mark, William and Ford, Kenneth and Matheny, Jason and Griffiths, Jose-Marie and McFarland, Katharina and Andrew Moore},
574  institution = {National Security Commission on Artificial Intelligence},
575  urldate = {2023-07-09},
576  language = {en-US},
577  url = {https://www.nscai.gov/2021-final-report/},
578  title = {{NSCAI} {Final} {Report}},
579}
580
581@article{diaz_ipbes_2015,
582  file = {ScienceDirect Full Text PDF:/Users/lewisho/Zotero/storage/WGI2L4BQ/Díaz et al. - 2015 - The IPBES Conceptual Framework — connecting nature.pdf:application/pdf;ScienceDirect Snapshot:/Users/lewisho/Zotero/storage/IKQJMX29/S187734351400116X.html:text/html},
583  pages = {1--16},
584  year = {2015},
585  month = {June},
586  author = {Díaz, Sandra and Demissew, Sebsebe and Carabias, Julia and Joly, Carlos and Lonsdale, Mark and Ash, Neville and Larigauderie, Anne and Adhikari, Jay Ram and Arico, Salvatore and Báldi, András and Bartuska, Ann and Baste, Ivar Andreas and Bilgin, Adem and Brondizio, Eduardo and Chan, Kai MA and Figueroa, Viviana Elsa and Duraiappah, Anantha and Fischer, Markus and Hill, Rosemary and Koetz, Thomas and Leadley, Paul and Lyver, Philip and Mace, Georgina M and Martin-Lopez, Berta and Okumura, Michiko and Pacheco, Diego and Pascual, Unai and Pérez, Edgar Selvin and Reyers, Belinda and Roth, Eva and Saito, Osamu and Scholes, Robert John and Sharma, Nalini and Tallis, Heather and Thaman, Randolph and Watson, Robert and Yahara, Tetsukazu and Hamid, Zakri Abdul and Akosim, Callistus and Al-Hafedh, Yousef and Allahverdiyev, Rashad and Amankwah, Edward and Asah, Stanley T and Asfaw, Zemede and Bartus, Gabor and Brooks, L Anathea and Caillaux, Jorge and Dalle, Gemedo and Darnaedi, Dedy and Driver, Amanda and Erpul, Gunay and Escobar-Eyzaguirre, Pablo and Failler, Pierre and Fouda, Ali Moustafa Mokhtar and Fu, Bojie and Gundimeda, Haripriya and Hashimoto, Shizuka and Homer, Floyd and Lavorel, Sandra and Lichtenstein, Gabriela and Mala, William Armand and Mandivenyi, Wadzanayi and Matczak, Piotr and Mbizvo, Carmel and Mehrdadi, Mehrasa and Metzger, Jean Paul and Mikissa, Jean Bruno and Moller, Henrik and Mooney, Harold A and Mumby, Peter and Nagendra, Harini and Nesshover, Carsten and Oteng-Yeboah, Alfred Apau and Pataki, György and Roué, Marie and Rubis, Jennifer and Schultz, Maria and Smith, Peggy and Sumaila, Rashid and Takeuchi, Kazuhiko and Thomas, Spencer and Verma, Madhu and Yeo-Chang, Youn and Zlatanova, Diana},
587  journal = {Current Opinion in Environmental Sustainability},
588  urldate = {2023-07-09},
589  language = {en},
590  abstract = {The first public product of the Intergovernmental Platform on Biodiversity and Ecosystem Services (IPBES) is its Conceptual Framework. This conceptual and analytical tool, presented here in detail, will underpin all IPBES functions and provide structure and comparability to the syntheses that IPBES will produce at different spatial scales, on different themes, and in different regions. Salient innovative aspects of the IPBES Conceptual Framework are its transparent and participatory construction process and its explicit consideration of diverse scientific disciplines, stakeholders, and knowledge systems, including indigenous and local knowledge. Because the focus on co-construction of integrative knowledge is shared by an increasing number of initiatives worldwide, this framework should be useful beyond IPBES, for the wider research and knowledge-policy communities working on the links between nature and people, such as natural, social and engineering scientists, policy-makers at different levels, and decision-makers in different sectors of society.},
591  doi = {10.1016/j.cosust.2014.11.002},
592  url = {https://www.sciencedirect.com/science/article/pii/S187734351400116X},
593  issn = {1877-3435},
594  volume = {14},
595  title = {The {IPBES} {Conceptual} {Framework} — connecting nature and people},
596  series = {Open {Issue}},
597}
598
599@article{maya_capacity_2010,
600  year = {2010},
601  author = {Maya, Shakespeare},
602  journal = {UNFCC},
603  title = {Capacity {Building} for {Technology} {Transfer} in the {African} {Context}: {Priorities} and {Strategies}},
604}
605
606@misc{hogarth_we_2023,
607  file = {We must slow down the race to God-like AI | Financial Times:/Users/lewisho/Zotero/storage/8LEX33R2/03895dc4-a3b7-481e-95cc-336a524f2ac2.html:text/html},
608  year = {2023},
609  author = {Hogarth, Ian},
610  journal = {Financial Times},
611  urldate = {2023-07-09},
612  url = {https://www.ft.com/content/03895dc4-a3b7-481e-95cc-336a524f2ac2},
613  title = {We must slow down the race to {God}-like {AI} {\textbar} {Financial} {Times}},
614}
615
616@misc{shimony_chatting_2023,
617  file = {Snapshot:/Users/lewisho/Zotero/storage/QWM6KXIK/chatting-our-way-into-creating-a-polymorphic-malware.html:text/html},
618  year = {2023},
619  author = {Shimony, Eran and Tsarfati, Omar},
620  urldate = {2023-07-09},
621  language = {en},
622  abstract = {Abstract ChatGPT took the world by storm being released less than two months ago, it has become prominent and is used everywhere, for a wide variety of tasks – from automation tasks to the...},
623  url = {https://www.cyberark.com/resources/threat-research-blog/chatting-our-way-into-creating-a-polymorphic-malware},
624  title = {Chatting {Our} {Way} {Into} {Creating} a {Polymorphic} {Malware}},
625}
626
627@misc{anderljung_frontier_2023,
628  year = {2023},
629  author = {Anderljung, Markus and Barnhart, Joslyn and Leung, Jade and Korinek, Anton and O'Keefe, Cullen and Whittlestone, Jess and Avin, Shahar and Brundage, Miles and Bullock, Justin and Cass-Beggs, Duncan and Chang, Ben and Collins, Tantum and Fist, Tim and Hadfield, Gillian and Hayes, Alan and Ho, Lewis and Hooker, Sarah and Horvitz, Eric and Kolt, Noam and Schuett, Jonas and Shavit, Yonadav and Siddarth, Divya and Trager, Robert and Wolf, Kevin},
630  publisher = {arXiv},
631  title = {Frontier {AI} {Regulation}: {Managing} {Emerging} {Risks} to {Public} {Safety}},
632}
633
634@misc{mailhe_why_2018,
635  file = {Snapshot:/Users/lewisho/Zotero/storage/7LJ6XUZX/why-we-need-an-intergovernmental-panel-for-artificial-intelligence.html:text/html},
636  year = {2018},
637  author = {Mailhe, Nicolas},
638  urldate = {2023-07-09},
639  abstract = {Global governance has a role to play in developing standards that balance the benefits and risks of deploying AI technologies, and that ensure citizens are aware of their rights and protections.},
640  url = {https://ourworld.unu.edu/en/why-we-need-an-intergovernmental-panel-for-artificial-intelligence},
641  title = {Why {We} {Need} an {Intergovernmental} {Panel} for {Artificial} {Intelligence} - {Our} {World}},
642}
643
644@article{noauthor_editorial_2023,
645  file = {Full Text PDF:/Users/lewisho/Zotero/storage/QDRA6KG4/2023 - Stop talking about tomorrow’s AI doomsday when AI .pdf:application/pdf;Snapshot:/Users/lewisho/Zotero/storage/FWNNPINZ/d41586-023-02094-7.html:text/html},
646  pages = {885--886},
647  keywords = {Authorship, Ethics, Machine learning},
648  note = {Bandiera\_abtest: a
649Cg\_type: Editorial
650Number: 7967
651Publisher: Nature Publishing Group
652Subject\_term: Machine learning, Authorship, Ethics},
653  year = {2023},
654  month = {June},
655  key = {Nature},
656  journal = {Nature},
657  urldate = {2023-07-09},
658  number = {7967},
659  language = {en},
660  abstract = {Talk of artificial intelligence destroying humanity plays into the tech companies’ agenda, and hinders effective regulation of the societal harms AI is causing right now.},
661  url = {https://www.nature.com/articles/d41586-023-02094-7},
662  copyright = {2023 Springer Nature Limited},
663  volume = {618},
664  title = {Editorial: {Stop} talking about tomorrow’s {AI} doomsday when {AI} poses risks today},
665}
666
667@misc{altman_governance_2023,
668  file = {Snapshot:/Users/lewisho/Zotero/storage/YVWQ33J3/governance-of-superintelligence.html:text/html},
669  year = {2023},
670  author = {Altman, Sam and Brockman, Greg and Sutskever, Ilya},
671  howpublished = {OpenAI},
672  urldate = {2023-07-09},
673  language = {en-US},
674  abstract = {Now is a good time to start thinking about the governance of superintelligence—future AI systems dramatically more capable than even AGI.},
675  url = {https://openai.com/blog/governance-of-superintelligence},
676  title = {Governance of superintelligence},
677}
678
679@article{marcus_world_2023,
680  file = {The Economist Snapshot:/Users/lewisho/Zotero/storage/TGX3783P/the-world-needs-an-international-agency-for-artificial-intelligence-say-two-ai-experts.html:text/html},
681  year = {2023},
682  author = {Marcus, Gary and Reuel, Anka},
683  journal = {The Economist},
684  urldate = {2023-07-09},
685  url = {https://www.economist.com/by-invitation/2023/04/18/the-world-needs-an-international-agency-for-artificial-intelligence-say-two-ai-experts},
686  issn = {0013-0613},
687  title = {The world needs an international agency for artificial intelligence, say two {AI} experts},
688}
689
690@article{brundage_computing_nodate,
691  author = {Brundage, Miles and Sastry, Girish and Heim, Lennart and Belfield, Haydn and Hazell, Julian and Anderljung, Markus and Avin, Shahar and Leung, Jade and O'Keefe, Cullen and Ngo, Richard},
692  title = {Computing {Power} and the {Governance} of {Artificial} {Intelligence}},
693}
694
695@article{jordan_international_nodate,
696  author = {Jordan, Richard and Emery-Xu, Nicholas and Trager, Robert},
697  title = {International {Governance} of {Advanced} {AI}},
698}
699
700@inproceedings{cihon_should_2020-1,
701  file = {Cihon et al. - 2020 - Should Artificial Intelligence Governance be Centr.pdf:/Users/lewisho/Zotero/storage/WTP9A7TS/Cihon et al. - 2020 - Should Artificial Intelligence Governance be Centr.pdf:application/pdf},
702  pages = {228--234},
703  year = {2020},
704  month = {February},
705  author = {Cihon, Peter and Maas, Matthijs M. and Kemp, Luke},
706  publisher = {ACM},
707  booktitle = {Proceedings of the {AAAI}/{ACM} {Conference} on {AI}, {Ethics}, and {Society}},
708  urldate = {2023-07-09},
709  language = {en},
710  abstract = {The invention of atomic energy posed a novel global challenge: could the technology be controlled to avoid destructive uses and an existentially dangerous arms race while permitting the broad sharing of its benefits? From 1944 onwards, scientists, policymakers, and other t echnical specialists began to confront this challenge and explored policy options for dealing with the impact of nuclear technology. We focus on the years 1944 to 1951 and review this period for lessons for the governance of powerful technologies, and find the following: Radical schemes for international control can get broad support when confronted by existentially dangerous technologies, but this support can be tenuous and cynical. Secrecy is likely to play an important, and perhaps harmful, role. The public sphere may be an important source of influence, both in general and in particular in favor of cooperation, but also one that is manipulable and poorly informed. Technical experts may play a critical role, but need to be politically savvy. Overall, policymaking may look more like “muddling through” than clear-eyed grand strategy. Cooperation may be risky, and there may be many obstacles to success.},
711  doi = {10.1145/3375627.3375857},
712  url = {https://dl.acm.org/doi/10.1145/3375627.3375857},
713  shorttitle = {Should {Artificial} {Intelligence} {Governance} be {Centralised}?},
714  isbn = {978-1-4503-7110-0},
715  title = {Should {Artificial} {Intelligence} {Governance} be {Centralised}?: {Design} {Lessons} from {History}},
716  address = {New York NY USA},
717}
718
719@article{zaidi_international_2019,
720  year = {2019},
721  author = {Zaidi, Waqar and Dafoe, Allan},
722  journal = {Working Paper},
723  url = {https://www.fhi.ox.ac.uk/wp-content/uploads/2021/03/International-Control-of-Powerful-Technology-Lessons-from-the-Baruch-Plan-Zaidi-Dafoe-2021.pdf},
724  title = {International {Control} of {Powerful} {Technology}: {Lessons} from the {Baruch} {Plan} for {Nuclear} {Weapons}},
725}
726
727@misc{le_gall_how_2021,
728  file = {Snapshot:/Users/lewisho/Zotero/storage/9ZQISEJH/how-cern-intellectual-property-helps-entrepreneurship.html:text/html},
729  year = {2021},
730  author = {Le Gall, Antoine},
731  howpublished = {CERN},
732  urldate = {2023-07-10},
733  language = {en},
734  abstract = {The novel technologies and expertise developed at CERN can be applied to fields other than high-energy physics. World Intellectual Property Day, observed annually on 26 April, is an opportunity to highlight how intellectual property (IP) is at the core of transferring unique CERN knowledge to its industrial and institutional partners, from large, long-standing companies to recent start-ups. In order to share its knowledge, CERN encourages the creation of spin-offs – companies based, partially or wholly, on CERN technologies – and has adopted a dedicated spin-off policy in 2018. One such company is PlanetWatch. Founded in 2020, this spin-off bases its air-quality data-analysis activities on C2MON, a data-acquisition framework developed at CERN. CERN also offers special licensing opportunities to promote the use of CERN technology in existing start-ups. These technologies range from innovative detector technologies to complex software, from radiation-hardened components to robotic platforms. As Marco Silari, section leader in the Radiation Protection group, explains “CERN technology can become much more than originally planned”. Together with his team, he developed several detector technologies now used by start-ups and companies around Europe. The complete list of current start-ups \& spin-offs using CERN technology \& know-how is available here. Depending on the nature of the technology and its application, it may benefit from Open Source licencing. This is the case for the White Rabbit technology – a tool used to provide control and data-acquisition systems with sub-nanosecond accuracy and a synchronisation precision of a few picoseconds – available on CERN’s Open Hardware Repository under the CERN Open Hardware Licence, to a large user community. Intellectual property enables successful knowledge transfer, ensuring the application of CERN technology and expertise in a way that aligns with CERN’s values, and maximises their societal impact. CERN’s policy is to disseminate its technologies as widely as possible to industrial and institutional partners within its Member States. Find out more about CERN’s management of Intellectual Property: https://kt.cern/activities-services/intellectual-property-management},
735  url = {https://home.cern/news/news/knowledge-sharing/how-cern-intellectual-property-helps-entrepreneurship},
736  title = {How {CERN} intellectual property helps entrepreneurship},
737}

Attribution

arXiv:2307.04699v2 [cs.CY]
License: cc-by-4.0

Related Posts

The Fallacy of AI Functionality

The Fallacy of AI Functionality

Introduction As one of over 20,000 cases falsely flagged for unemployment benefit fraud by Michigan’s MIDAS algorithm, Brian Russell had to file for bankruptcy, undermining his ability to provide for his two young children.

Institutionalising Ethics in AI through Broader Impact Requirements

Institutionalising Ethics in AI through Broader Impact Requirements

Introduction Growing concerns about the ethical, societal, environmental, and economic impacts of artificial intelligence (AI) have led to a wealth of governance initiatives.

Thinking Upstream: Ethics and Policy Opportunities in AI Supply Chains

Thinking Upstream: Ethics and Policy Opportunities in AI Supply Chains

Looking upstream After children were pictured sewing its running shoes in the early 1990s, Nike at first disavowed the “working conditions in its suppliers’ factories”, before public pressure led them to take responsibility for ethics in their upstream supply chain.