Introduction
In this position paper, we share our insights about AI Governance in companies, which enables new connections between various aspects and properties of trustworthy and socially responsible Machine Learning: security, robustness, privacy, fairness, ethics, interpretability, transparency, etc.
For a long time Artificial intelligence (AI) was something enterprise organizations adopted due to the huge amounts of resources they have at their fingertips. Today, smaller companies are able to take advantage of AI due to newer technologies, e.g. cloud software, which are significantly more affordable than what was available in the past* * * * . AI has been on an upward trajectory in recent years and it will increase significantly over the next several years * * *. However, every investment has its pros and cons. Unfortunately, the cons associated with AI adoption are caused by its inherent uncertainties, and the builders of such AI systems who do not take the necessary steps to avoid problems down the road* * . Note that, in this work, AI comprises modern Machine Learning (ML) and Deep Learning (DL) systems, yet not their software around them - which represents other threats and vulnerabilities by itself* * * -.
Problems within Industry
Some popular applications of AI are: anomaly detection and forecasting* * * , recommender systems*medical diagnosis* * , natural sciences* * * , and search engines* * .
However, these applications of AI in industry is still in its infancy. With that said, many problems have arisen since its adoption* * * , which can be attributed to several factors:
Lack of risk awareness and management: Too much attention is given to applications of AI and its potential success and not enough attention is given to its potential pitfalls and risks.
AI adoption is moving too fast: According to a survey by KPMG in 2021*many respondents noted that AI technology is moving too fast for their comfort in industrial manufacturing (55
AI adoption needs government intervention: According to the same survey by KPMG,*(see surveys), an overwhelming percentage of respondents agreed that governments should be involved in regulating AI technology in the industrial manufacturing (94
Companies are still immature when it comes to adopting AI: Some companies are not prepared for business conditions to change once a ML model is deployed into the real world.
Many of these problems can be avoided with proper governance mechanisms. AI without such mechanisms is a dangerous game with detrimental outcomes due its inherent uncertainty* * . With that said, adding governance into applications of AI is imperative to ensure safety in production.
AI Governance
What is AI Governance (AIG)?
AI Governance is a framework to operationalize responsible artificial intelligence at organizations. This framework encourages organizations to curate and use bias-free data, consider societal and end-user impact, and produce unbiased models; the framework also enforces controls on model progression through deployment stages. The potential risks associated with AI need to be considered when designing models, before they affect the quality of models and algorithms. If left unmonitored, AI may not only produce undesirable results, but can also have a significant adverse impact on the organization.
In order for organizations to realize the maximum value out of AI projects and develop consistency for organization-wide adoption of AI, while managing significant risks to their business, they must implement AI governance* * * * ; this enables organizations to not only develop AI projects in a responsible way, but also ensure that there is consistency across the entire organization and the business objective is front and center. With the AI governance implemented (as illustrated in teaser), the following benefits can be realized:
Alignment and Clarity: teams would be aware and aligned on what the industry, international, regional, local, and organizational policies are that need to be adhered to.
Thoughtfulness and Accountability: teams would put deliberate effort into justifying the business case for AI projects, and put conscious effort into thinking about end-user experience, adversarial impacts, public safety & privacy. This also places greater accountability on the teams developing their respective AI projects.
Consistency and Organizational Adoption: teams would have a more consistent way of developing and collaborating on their AI projects, leading to increased tracking and transparency for their projects. This also provides an overarching view of all AI projects going on within the organization, leading to increased visibility and overall adoption.
Process, Communication, and Tools: teams would have complete understanding of what the steps are in order to move the AI project to production to start realizing business value. They would also be able to leverage tools that take them through the defined process, while being able to communicate with the right stakeholders through the tool.
Trust and Public Perception: as teams build out their AI projects more thoughtfully, this will inherently build trust amongst customers and end users, and therefore a positive public perception.
AI governance requires the following:
A structured organization that gives AIG leaders the correct information they need to establish policies and accountability for AI efforts across their entire organization. For smaller organizations, this might require a more phased approach in which they will work towards the desired structural framework of AIG. For larger organizations, this process might be more attainable due to resources alone, e.g. people, IT infrastructure, larger budgets.
A concrete and specific AI workflow that collects information needed by AIG leaders will help enforce the constructed policies. This provides information to various parties in a consumable manner. Having such information can be used to minimize mistakes, errors, and bias, amongst other things.
The requirements for AI governance manifest into a framework that an organization must work towards developing. The components of this framework need to be transparent and comprehensive to achieve a successful implementation of AIG. Specifically, this should focus on organizational and use case planning, AI development, and AI `'operationalization’’, which come together to make a 4 stage AI life cycle approach.
Stages of a Governed AI Life cycle
Organizational Planning
An AI Governance Program* * * should be organized in such a way that (a) there is comprehensive understanding of regulations, laws, and policies amongst all team members (b) resources and help available for team members who encounter challenges (c) there is a light weight, yet clear process to assist team members.
Regulations, Laws, Policies
- Laws and regulations that apply to a specific entity should be identified, documented, and available for others to review and audit. These regulations, laws, and policies vary across industry and sometimes by geographical location. Organizations should, if applicable, develop policies for themselves, which reflect their values and ethical views* * * ; this enables teams to be more autonomous and make decisions with confidence.
Organization (Center of Competency)
- Establishing groups within an organization that provide support to teams with AI projects can prove to be quite beneficial. This includes a group that is knowledgeable with regulations, laws, and policies and can answer any questions that AI teams may have; a group that is able to share best practices across different AI teams within the organization; a group that is able to audit the data, model, process, etc. to ensure there isn’t a breach or non-compliance. For more information, we refer the reader to to the survey by Floridi et al.*.
Process
- Developing a light-weight process that provides guidelines to AI teams can help with their efficiency, rather than hinder their progress and velocity. This involves identifying what the approval process and incident response would be for data, model, deployments, etc.
Use Case Planning
Building use cases involves establishing business value, technology stack, and model usage. The group of people involved in this process can include: subject matter experts, data scientists/analysts/annotators and ML engineers, IT professionals, and finance departments.
Business Value Framework. The AI team should ensure that the motivation for the AI use case is documented and communicated amongst all stakeholders. This should also include the original hypothesis, and the metrics that would be used for evaluating the experiments.
Tools, Technology, Products. The AI team should either select from a set of pre-approved tools and products from the organization or get a set of tools and products approved before using in an AI user case. If tools for AI development are not governed, it not only leads to high costs and inability to manage the tools (as existing IT teams are aware), it also leads to not being able to create repeatability and traceability into AI models.
Model Usage. Once a sense of value is attached to the use case, then the next step would be to break down the use case to its sub-components which include, but are not limited to, identifying the consumer of the model, the model’s limitations, and potential bias that may exist within the model, along with its implications. Also, one would want to ensure inclusiveness of the target, public safety/user privacy, and identification of the model interface needed for their intended use case.
AI Development
Development of a machine learning model, including data handling and analysis, modeling, generating explanations, bias detection, accuracy and efficacy analysis, security and robustness checks, model lineage, validation, and documentation.
Data Handling, Analysis and Modeling
- The first technical step to any AI project is the procurement and analysis of data, which is critical as it lays the foundation for all work going forward. Once data is analyzed, then one must decipher if modeling is needed for the use case at hand. If modeling is needed, then the application of AI can take place. Such an application is an iterative process spanned across many different types of people.
Explanations and Bias
The goal of model explanations is to relate feature values to model predictions in a human-friendly manner*. What one does with these explanations breaks down to 3 personas: modeler, intermediary user, and the end user. The modeler would use explanations for model debugging and gaining understanding of the model they just built. The intermediary user would use what the modeler made for actionable insights. And finally, the end user is the person the model affects directly. For these reasons, Explainable Artificial Intelligence (XAI) is a very active research topic* * .
Bias, whether intentional (disparate treatment) or unintentional (disparate impact), is a cause for concern in many applications of AI* * . Common things to investigate when it comes to preventing bias include the data source used for the modeling process, performance issues amongst different demographics, disparate impact, identifying known limitations & potential adverse implications, and the models impact on public safety*. We refer the reader to the survey by Mehrabi et al.*for more details about bias and fairness in AI.
Accuracy, Efficacy, & Robustness
Accuracy of a machine learning model is critical for any business application in which predictions drive potential actions. However, it is not the most important metric to optimize. One must also consider the efficacy of a model, i.e. is the model making the intended business impact?
When a model is serving predictions in a production setting, the data can be a little or significantly different from the data that the project team had access to. Although model drift and feature drift can capture this discrepancy, it is a lagging indicator, and by that time, the model has already made predictions. This is where Robustness comes in: project teams can proactively test for model robustness, using “out of scope” data, to understand whether the model perturbs. The out of scope data can be a combination of manual generation (toggle with feature values) and automatic generation (system toggles feature values).
Security
- ML systems today are subject to general attacks that can affect any public facing IT system*\cite {papernot2018marauder}; specialized attacks that exploit insider access to data and ML code; external access to ML prediction APIs and endpoints* *; and trojans that can hide in third-party ML artifacts. Such attacks must be accounted for and tested against before sending a machine learning model out into the real world.
Documentation & Validation
An overall lineage of the entire AI project life-cycle should be documented to ensure transparency and understanding* *which will be useful for the AI team working on the project and also future teams who must reference this project for their own application.
Model validation* is the set of processes and activities that are carried out by a third party, with the intent to verify that models are robust and performing as expected, in line with the business use case. It also identifies the impact of potential limitations and assumptions. From a technical standpoint, the following should be considered: (i) Sensitivity Analysis. (ii) In-sample vs. Out-of-sample performance. (iii) Replication of results from model development team. (iv) Stability analysis. Model `'validators’’ should document all of their findings and share with relevant stakeholders.
AI Operationalization
Deploying a machine learning model into production (i.e. MLOps* * ) is the first step to potentially receiving value out of it. The steps that go into the deployment process should include the following:
Review-Approval Flow: Model building in an AI project will go through various stages: experimentation, model registration, deployment, and decommissioning. Moving from one stage to the next would require `'external’’ reviewer(s) who will vet and provide feedback.
Monitoring & Alerts: Once a model is deployed, it must be monitored for various metrics to ensure there is not any degradation in the model. The cause for a model degrading when deployed can include the following: feature and/or target drift, lack of data integrity, and outliers, amongst other things. In terms of monitoring, accuracy, fairness, and explanations of predictions are of interest* * .
Decision Making: The output of a machine learning model is a prediction, but that output must be turned into a decision. How to decide? Will it be autonomous? Will it involve a human in the loop? The answers to these questions vary across different applications, but the idea remains the same, ensuring decisions are made in the proper way to decrease risk for everyone involved.
Incident Response and Escalation Process: With AI models being used in production, there is always going to be a chance for issues to arise. Organizations should have an incident response plan and escalation process documented and known to all project teams.
Companies who successfully implement AI governance for AI applications will result in a highly impactful use of artificial intelligence. While those who fail to do so, risk catastrophic outcomes and an arduous road to recovery as shown in the following use-case.
AIG Use Case
In this section we describe a recent use case where - we believe - AI Governance could have avoided a terrible outcome.
In 2021, the online real estate technology giant, Zillow, shut down its AI-powered house-flipping business. At it’s core, this line of business relied heavily on forecasting from their machine learning models. Zillow found itself overpaying for homes due to overestimating the price of a property. Such overestimation of property values led to a loss of $569 million, and 28% of their valuation* * . The monetary loss and laying off of 2,000 employees also came with a reputational cost.
This AI failure begs the question ‘What mistakes did Zillow make?’ and `Could this have been prevented?’. The answer to the second question is a firm ‘yes’. However, the answer to the first question has many facets, but we try to break down the key mistakes below:
Removing `'Human in the Loop’’
In its early days, this company hired local real estate agents and property experts to verify the output of their home price prediction ML algorithm; however, as the business scaled, the human verification process around the algorithms was minimized*and the offer-making process was automated, which helped to cut expenses and increase acquisitions. However, removing human verification in such a volatile domain led to predictions taken at face value without any verification, which played a big role in overpricing of properties.
Keeping the human in the loop can help companies from relying on overestimated and biased predictions. Considering a ML ecosystem for decision-making in organizations, having a human in the loop is a safety harness and should be used in any high stake decision making.
Not Accounting for Concept Drift and Lack of Model Monitoring
Taking a look back at the timeline of events, it appears that the ML algorithms were not adjusted accordingly to the real market status*. The algorithms continued to assume that the market was still `'hot’’ and overestimated home prices.
To avoid problems of drift, specifically concept drift, companies should leverage tools for monitoring and maintaining the quality of AI models. Ideally, in this example, they should have set up an infrastructure that automatically alerts data science teams when there is drift or performance degradation, support root cause analysis, and inform model updates with humans-in-the-loop.
Lack of Model Validation
- Before organizations deploy any of their algorithms, they should have enact a set of processes and activities intended to verify that their models are performing as expected and that they are in line with the business use case. Effective validation ensures that models are robust. It also identifies potential limitations and assumptions, and it assesses their possible impact. A lack of model validation played a crucial role in this case, and led to the overestimation of home prices. It should be noted that -ideally- model validation needs to be carried out by a third party that did not take part in the model building process.
Lack of Incident Response
- It is not clear when this company started to realize that their model’s were degrading and producing erroneous results. The lack of having such an incident response plan can cost businesses an exuberant amount of money, time, and human resources. In retrospect, if proper AI Governance would had been implemented, the company could have started an incident response i.e. incident identification and documentation, human review, and redirection of model traffic. These three steps alone could have prevented a lot of damage to their business.
This industry use case is an example of what can happen without proper AI governance. The mistakes that were made could have been avoided if proper AI governance principles were taken into account from the inception of their use case. Specifically, if this company continued to rely on their subject matter experts, accounted for various types of drift, added model validation, and implemented concrete model monitoring with proper incident response planning, then this whole situation could have been avoided or the damage could have been at a much lower magnitude.
Conclusion
AI systems are used today to make life-altering decisions about employment, bail, parole, and lending, and the scope of decisions delegated by AI systems seems likely to expand in the future. The pervasiveness of AI across many fields is something that will not slowdown anytime soon and organizations will want to keep up with such applications. However, they must be cognisant of the risks that come with AI and have guidelines around how they approach applications of AI to avoid such risks. By establishing a framework for AI Governance, organizations will be able to harness AI for their use cases while at the same time avoiding risks and having plans in place for risk mitigation, which is paramount.
Social Impact
As we discuss in this paper, governance and certain control over AI applications in organizations should be mandatory. AI Governance aims to enable and facilitate connections between various aspects of trustworthy and socially responsible machine learning systems, and therefore it accounts for security, robustness, privacy, fairness, ethics, and transparency. We believe the implementation of these ideas should have a positive impact in the society.
Acknowledgements
We thank the Trustworthy and Socially Responsible Machine Learning (TSRML) Workshop at NeurIPS 2022. This work was supported by H2O.ai.
Attribution
1@misc{gill2022brief,
2 title={A Brief Overview of AI Governance for Responsible Machine Learning Systems},
3 author={Navdeep Gill and Abhishek Mathur and Marcos V. Conde},
4 year={2022},
5 eprint={2211.13130},
6 archivePrefix={arXiv},
7 primaryClass={cs.CY}
8}Bibliography
1% Intro
2@misc{IBM,
3 Title = {{IBM} {G}lobal {AI} {A}doption {I}ndex},
4 note = {URL: \url{https://www.ibm.com/downloads/cas/GVAGA3JP}},
5 author = {IBM},
6 Year = {2022}}
7@misc{PwC,
8 Title = {{PwC} 2022 {AI} {B}usiness {S}urvey},
9 note = {URL: \url{https://www.pwc.com/us/en/tech-effect/ai-analytics/ai-business-survey.html}},
10 author = {PwC (PriceWaterhouseCoopers)},
11 Year = {2022}}
12@misc{KPMG,
13 Title = {{AI} adoption accelerated during the pandemic but many say it’s moving too fast: {KPMG} survey},
14 note = {URL: \url{https://info.kpmg.us/news-perspectives/technology-innovation/thriving-in-an-ai-world/ai-adoption-accelerated-during-pandemic.html}},
15 author = {KPMG},
16 Year = {2021}}
17
18@article{hbr,
19 Author = {Joe McKendrick},
20 Journal = {Harvard Business Review},
21 Title = {{AI} {A}doption {S}kyrocketed {O}ver {t}he {L}ast 18 {M}onths.},
22 note = {URL: \url{https://hbr.org/2021/09/ai-adoption-skyrocketed-over-the-last-18-months}},
23 Year = {2021}}
24@misc{ai_incidents,
25 title={{AI} {I}ncident {D}atabase},
26 author = {Artificial Intelligence Incident Database},
27 year = {2022},
28 note = {URL: \url{https://incidentdatabase.ai/}}},
29% Security
30@article{security_of_ml,
31 title={The {S}ecurity of {M}achine {L}earning},
32 author={Barreno, Marco and Nelson, Blaine and Joseph, Anthony D and Tygar, J Doug},
33 journal={Machine Learning},
34 volume={81},
35 number={2},
36 pages={121--148},
37 year={2010},
38 publisher={Springer},
39 note={URL: \url{https://people.eecs.berkeley.edu/~adj/publications/paper-files/SecML-MLJ2010.pdf}}}
40@inproceedings{papernot2018marauder,
41 title={A {M}arauder's {M}ap of {S}ecurity and {P}rivacy in {M}achine {L}earning: {A}n overview of current and future research directions for making machine learning secure and private},
42 author={Papernot, Nicolas},
43 booktitle={Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security},
44 year={2018},
45 organization={ACM},
46 note={URL: \url{https://arxiv.org/pdf/1811.01134.pdf}}}
47@inproceedings{membership_inference,
48 title={Membership {I}nference {A}ttacks {A}gainst {M}achine {L}earning {M}odels},
49 author={Shokri, Reza and Stronati, Marco and Song, Congzheng and Shmatikov, Vitaly},
50 booktitle={2017 IEEE Symposium on Security and Privacy (SP)},
51 pages={3--18},
52 year={2017},
53 organization={IEEE},
54 note={URL: \url{https://arxiv.org/pdf/1610.05820.pdf}}}
55@inproceedings{model_stealing,
56 title={Stealing {M}achine {L}earning {M}odels via {P}rediction {A}{P}{I}s},
57 author={Tram{\`e}r, Florian and Zhang, Fan and Juels, Ari and Reiter, Michael K and Ristenpart, Thomas},
58 booktitle={25th $\{$USENIX$\}$ Security Symposium ($\{$USENIX$\}$ Security 16)},
59 pages={601--618},
60 year={2016},
61 note={URL: \url{https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_tramer.pdf}}}
62
63% Explanations
64@book{molnar,
65 title={{I}nterpretable {M}achine {L}earning},
66 author={Christopher Molnar},
67 year={2022},
68 note={URL: \url{https://christophm.github.io/interpretable-ml-book/}}}
69
70@article{gunning2019xai,
71 title={XAI—Explainable artificial intelligence},
72 author={Gunning, David and Stefik, Mark and Choi, Jaesik and Miller, Timothy and Stumpf, Simone and Yang, Guang-Zhong},
73 journal={Science robotics},
74 volume={4},
75 number={37},
76 pages={eaay7120},
77 year={2019},
78 publisher={American Association for the Advancement of Science}
79}
80
81@article{arrieta2020explainable,
82 title={Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI},
83 author={Arrieta, Alejandro Barredo and D{\'\i}az-Rodr{\'\i}guez, Natalia and Del Ser, Javier and Bennetot, Adrien and Tabik, Siham and Barbado, Alberto and Garc{\'\i}a, Salvador and Gil-L{\'o}pez, Sergio and Molina, Daniel and Benjamins, Richard and others},
84 journal={Information fusion},
85 volume={58},
86 pages={82--115},
87 year={2020},
88 publisher={Elsevier}
89}
90
91% Bias
92@misc{bias,
93 title={{F}airness and {M}achine {L}earning: {L}imitaions and {O}pportunities},
94 author={Barocas, S. and Hardt, M. and Narayanan, A.},
95 year={2022},
96 note={URL: \url{https://fairmlbook.org/pdf/fairmlbook.pdf}}}
97
98@misc{bias_nist,
99 author = {Reva Schwartz and Apostol Vassilev and Kristen Greene and Lori Perine and Andrew Burt and Patrick Hall},
100 title = {Towards a Standard for Identifying and Managing Bias in Artificial Intelligence},
101 year = {2022},
102 month = {2022-03-15 04:03:00},
103 publisher = {Special Publication (NIST SP), National Institute of Standards and Technology, Gaithersburg, MD},
104 url = {https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=934464},
105 doi = {https://doi.org/10.6028/NIST.SP.1270},
106 language = {en},
107}
108
109@article{mehrabi2021survey,
110 title={A survey on bias and fairness in machine learning},
111 author={Mehrabi, Ninareh and Morstatter, Fred and Saxena, Nripsuta and Lerman, Kristina and Galstyan, Aram},
112 journal={ACM Computing Surveys (CSUR)},
113 volume={54},
114 number={6},
115 pages={1--35},
116 year={2021},
117 publisher={ACM New York, NY, USA}
118}
119
120
121% Model validation and documentation
122@misc{SR117,
123 title = {{S}upervisory {G}uidance on {M}odel {R}isk {M}anagement, {SR} {L}etter 11-7},
124 year = {2011},
125 url = {https://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf},
126 pages={6--16},
127 author = {Board of Governors of the Federal Reserve System Office of the Comptroller of the Currency},
128}
129
130@article{landry1983model,
131 title={Model validation in operations research},
132 author={Landry, Maurice and Malouin, Jean-Louis and Oral, Muhittin},
133 journal={European journal of operational research},
134 volume={14},
135 number={3},
136 pages={207--220},
137 year={1983},
138 publisher={Elsevier}
139}
140
141
142@inproceedings{Mitchell_2019,
143 doi = {10.1145/3287560.3287596},
144 url = {https://doi.org/10.1145%2F3287560.3287596},
145 year = 2019,
146 publisher = {{ACM}},
147 author = {Margaret Mitchell and Simone Wu and Andrew Zaldivar and Parker Barnes and Lucy Vasserman and Ben Hutchinson and Elena Spitzer and Inioluwa Deborah Raji and Timnit Gebru},
148 title = {Model Cards for Model Reporting},
149 booktitle = {Proceedings of the Conference on Fairness, Accountability, and Transparency}
150}
151@misc{data_cards,
152 doi = {10.48550/ARXIV.2204.01075},
153 url = {https://arxiv.org/abs/2204.01075},
154 author = {Pushkarna, Mahima and Zaldivar, Andrew and Kjartansson, Oddur},
155 keywords = {Human-Computer Interaction (cs.HC), Artificial Intelligence (cs.AI), Databases (cs.DB), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
156 title = {Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI},
157 publisher = {arXiv},
158 year = {2022},
159 copyright = {Creative Commons Attribution 4.0 International}
160}
161
162%% Adoption
163
164@article{alsheibani2018artificial,
165 title={Artificial intelligence adoption: AI-readiness at firm-level},
166 author={AlSheibani, Sulaiman and Cheung, Yen and Messom, Chris},
167 year={2018},
168 journal = {PACIS 2018 Proceedings}
169}
170
171@article{cubric2020drivers,
172 title={Drivers, barriers and social considerations for AI adoption in business and management: A tertiary study},
173 author={Cubric, Marija},
174 journal={Technology in Society},
175 volume={62},
176 pages={101257},
177 year={2020},
178 publisher={Elsevier}
179}
180
181@article{duan2019artificial,
182 title={Artificial intelligence for decision making in the era of Big Data--evolution, challenges and research agenda},
183 author={Duan, Yanqing and Edwards, John S and Dwivedi, Yogesh K},
184 journal={International journal of information management},
185 volume={48},
186 pages={63--71},
187 year={2019},
188 publisher={Elsevier}
189}
190
191@article{floridi2018ai4people,
192 title={AI4People—An ethical framework for a good AI society: Opportunities, risks, principles, and recommendations},
193 author={Floridi, Luciano and Cowls, Josh and Beltrametti, Monica and Chatila, Raja and Chazerand, Patrice and Dignum, Virginia and Luetge, Christoph and Madelin, Robert and Pagallo, Ugo and Rossi, Francesca and others},
194 journal={Minds and machines},
195 volume={28},
196 number={4},
197 pages={689--707},
198 year={2018},
199 publisher={Springer}
200}
201
202@article{marr2018artificial,
203 title={Is Artificial Intelligence dangerous? 6 AI risks everyone should know about},
204 author={Marr, Bernard},
205 journal={Forbes},
206 year={2018}
207}
208
209
210% AI Governance
211
212@article{wang2018artificial,
213 title={Artificial intelligence: A study on governance, policies, and regulations},
214 author={Wang, Weiyu and Siau, Keng},
215 journal={MWAIS 2018 proceedings},
216 volume={40},
217 year={2018}
218}
219
220
221@article{reddy2020governance,
222 title={A governance model for the application of AI in health care},
223 author={Reddy, Sandeep and Allan, Sonia and Coghlan, Simon and Cooper, Paul},
224 journal={Journal of the American Medical Informatics Association},
225 volume={27},
226 number={3},
227 pages={491--497},
228 year={2020},
229 publisher={Oxford University Press}
230}
231
232@article{dafoe2018ai,
233 title={AI governance: a research agenda},
234 author={Dafoe, Allan},
235 journal={Governance of AI Program, Future of Humanity Institute, University of Oxford: Oxford, UK},
236 volume={1442},
237 pages={1443},
238 year={2018}
239}
240
241@article{gasser2017layered,
242 title={A layered model for AI governance},
243 author={Gasser, Urs and Almeida, Virgilio AF},
244 journal={IEEE Internet Computing},
245 volume={21},
246 number={6},
247 pages={58--62},
248 year={2017},
249 publisher={IEEE}
250}
251
252
253@article{kuziemski2020ai,
254 title={AI governance in the public sector: Three tales from the frontiers of automated decision-making in democratic settings},
255 author={Kuziemski, Maciej and Misuraca, Gianluca},
256 journal={Telecommunications policy},
257 volume={44},
258 number={6},
259 pages={101976},
260 year={2020},
261 publisher={Elsevier}
262}
263
264%% AI aplications
265
266@article{vaishya2020artificial,
267 title={Artificial Intelligence (AI) applications for COVID-19 pandemic},
268 author={Vaishya, Raju and Javaid, Mohd and Khan, Ibrahim Haleem and Haleem, Abid},
269 journal={Diabetes \& Metabolic Syndrome: Clinical Research \& Reviews},
270 volume={14},
271 number={4},
272 pages={337--339},
273 year={2020},
274 publisher={Elsevier}
275}
276
277
278%% AI uncertainty
279
280@article{bresina2012planning,
281 title={Planning under continuous time and resource uncertainty: A challenge for AI},
282 author={Bresina, John and Dearden, Richard and Meuleau, Nicolas and Ramkrishnan, Sailesh and Smith, David and Washington, Richard},
283 journal={arXiv preprint arXiv:1301.0559},
284 year={2012}
285}
286
287@incollection{zadeh1986probability,
288 title={Is probability theory sufficient for dealing with uncertainty in AI: A negative view},
289 author={Zadeh, Lotfi A},
290 booktitle={Machine intelligence and pattern recognition},
291 volume={4},
292 pages={103--116},
293 year={1986},
294 publisher={Elsevier}
295}
296
297
298% AI Operationalization
299
300@incollection{zhu2022ai,
301 title={AI and Ethics—Operationalizing Responsible AI},
302 author={Zhu, Liming and Xu, Xiwei and Lu, Qinghua and Governatori, Guido and Whittle, Jon},
303 booktitle={Humanity Driven AI},
304 pages={15--33},
305 year={2022},
306 publisher={Springer}
307}
308
309@inproceedings{ehsan2021operationalizing,
310 title={Operationalizing human-centered perspectives in explainable AI},
311 author={Ehsan, Upol and Wintersberger, Philipp and Liao, Q Vera and Mara, Martina and Streit, Marc and Wachter, Sandra and Riener, Andreas and Riedl, Mark O},
312 booktitle={Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems},
313 pages={1--6},
314 year={2021}
315}
316
317@article{amodei2016concrete,
318 title={Concrete problems in AI safety},
319 author={Amodei, Dario and Olah, Chris and Steinhardt, Jacob and Christiano, Paul and Schulman, John and Man{\'e}, Dan},
320 journal={arXiv preprint arXiv:1606.06565},
321 year={2016}
322}
323
324@article{leike2017ai,
325 title={AI safety gridworlds},
326 author={Leike, Jan and Martic, Miljan and Krakovna, Victoria and Ortega, Pedro A and Everitt, Tom and Lefrancq, Andrew and Orseau, Laurent and Legg, Shane},
327 journal={arXiv preprint arXiv:1711.09883},
328 year={2017}
329}
330
331
332@book{treveil2020introducing,
333 title={Introducing MLOps},
334 author={Treveil, Mark and Omont, Nicolas and Stenac, Cl{\'e}ment and Lefevre, Kenji and Phan, Du and Zentici, Joachim and Lavoillotte, Adrien and Miyazaki, Makoto and Heidmann, Lynn},
335 year={2020},
336 publisher={O'Reilly Media}
337}
338
339@incollection{alla2021mlops,
340 title={What is mlops?},
341 author={Alla, Sridhar and Adari, Suman Kalyan},
342 booktitle={Beginning MLOps with MLFlow},
343 pages={79--124},
344 year={2021},
345 publisher={Springer}
346}
347
348@article{mittelstadt2019principles,
349 title={Principles alone cannot guarantee ethical AI},
350 author={Mittelstadt, Brent},
351 journal={Nature Machine Intelligence},
352 volume={1},
353 number={11},
354 pages={501--507},
355 year={2019},
356 publisher={Nature Publishing Group}
357}
358
359@article{bloomberg,
360 author = {Patrick Clark},
361 journal = {Bloomberg},
362 title = {Zillow’s Algorithm-Fueled Buying Spree Doomed Its Home-Flipping Experiment},
363 note = {URL: \url{https://www.bloomberg.com/news/articles/2021-11-08/zillow-z-home-flipping-experiment-doomed-by-tech-algorithms}},
364 Year = {2021}}
365
366@article{bloomberg2,
367 author = {Patrick Clark, Sridhar Natarajan, Heather Perlberg},
368 journal = {Bloomberg},
369 title = {Zillow Seeks to Sell 7,000 Homes for \$2.8 Billion After Flipping Halt},
370 note = {URL: \url{https://www.bloomberg.com/news/articles/2021-11-01/zillow-selling-7-000-homes-for-2-8-billion-after-flipping-halt}},
371 Year = {2021}}
372
373@article{sr711,
374 author = {Board of Governors of the Federal Reserve System},
375 title = {SR 11-7 attachment: Supervisory Guidance on Model Risk Management},
376 note = {URL: https://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf},
377 Year = {2011}}
378
379%%%%%%%%%%%%%% SOFTWARE SECURITY
380@article{alhazmi2007measuring,
381 title={Measuring, analyzing and predicting security vulnerabilities in software systems},
382 author={Alhazmi, Omar Hussain and Malaiya, Yashwant K and Ray, Indrajit},
383 journal={computers \& security},
384 volume={26},
385 number={3},
386 pages={219--228},
387 year={2007},
388 publisher={Elsevier}
389}
390
391@book{dowd2006art,
392 title={The art of software security assessment: Identifying and preventing software vulnerabilities},
393 author={Dowd, Mark and McDonald, John and Schuh, Justin},
394 year={2006},
395 publisher={Pearson Education}
396}
397
398@article{mcgraw2004software,
399 title={Software security},
400 author={McGraw, Gary},
401 journal={IEEE Security \& Privacy},
402 volume={2},
403 number={2},
404 pages={80--83},
405 year={2004},
406 publisher={IEEE}
407}
408
409@article{conde2022general,
410 title={General Image Descriptors for Open World Image Retrieval using ViT CLIP},
411 author={Conde, Marcos V and Aerlic, Ivan and J{\'e}gou, Simon},
412 journal={arXiv preprint arXiv:2210.11141},
413 year={2022}
414}
415
416@article{conde2021weakly,
417 title={Weakly-Supervised Classification and Detection of Bird Sounds in the Wild. A BirdCLEF 2021 Solution},
418 author={Conde, Marcos V and Shubham, Kumar and Agnihotri, Prateek and Movva, Nitin D and Bessenyei, Szilard},
419 journal={arXiv preprint arXiv:2107.04878},
420 year={2021}
421}
422
423@article{conde2022few,
424 title={Few-shot Long-Tailed Bird Audio Recognition},
425 author={Conde, Marcos V and Choi, Ui-Jin},
426 journal={arXiv preprint arXiv:2206.11260},
427 year={2022}
428}
429
430@inproceedings{sultani2018real,
431 title={Real-world anomaly detection in surveillance videos},
432 author={Sultani, Waqas and Chen, Chen and Shah, Mubarak},
433 booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
434 pages={6479--6488},
435 year={2018}
436}
437
438@article{portugal2018use,
439 title={The use of machine learning algorithms in recommender systems: A systematic review},
440 author={Portugal, Ivens and Alencar, Paulo and Cowan, Donald},
441 journal={Expert Systems with Applications},
442 volume={97},
443 pages={205--227},
444 year={2018},
445 publisher={Elsevier}
446}
447
448@article{gordeev2020backtesting,
449 title={Backtesting the predictability of COVID-19},
450 author={Gordeev, Dmitry and Singer, Philipp and Michailidis, Marios and M{\"u}ller, Mathias and Ambati, SriSatish},
451 journal={arXiv preprint arXiv:2007.11411},
452 year={2020}
453}
454
455@article{henkel2021recognizing,
456 title={Recognizing bird species in diverse soundscapes under weak supervision},
457 author={Henkel, Christof and Pfeiffer, Pascal and Singer, Philipp},
458 journal={arXiv preprint arXiv:2107.07728},
459 year={2021}
460}
461
462@article{makridakis2018statistical,
463 title={Statistical and Machine Learning forecasting methods: Concerns and ways forward},
464 author={Makridakis, Spyros and Spiliotis, Evangelos and Assimakopoulos, Vassilios},
465 journal={PloS one},
466 volume={13},
467 number={3},
468 pages={e0194889},
469 year={2018},
470 publisher={Public Library of Science San Francisco, CA USA}
471}
472
473@article{esteva2021deep,
474 title={Deep learning-enabled medical computer vision},
475 author={Esteva, Andre and Chou, Katherine and Yeung, Serena and Naik, Nikhil and Madani, Ali and Mottaghi, Ali and Liu, Yun and Topol, Eric and Dean, Jeff and Socher, Richard},
476 journal={NPJ digital medicine},
477 volume={4},
478 number={1},
479 pages={1--9},
480 year={2021},
481 publisher={Nature Publishing Group}
482}
483
484@inproceedings{conde2021clip,
485 title={CLIP-Art: contrastive pre-training for fine-grained art classification},
486 author={Conde, Marcos V and Turgutlu, Kerem},
487 booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
488 pages={3956--3960},
489 year={2021}
490}
491
492
493@article{suzuki2017overview,
494 title={Overview of deep learning in medical imaging},
495 author={Suzuki, Kenji},
496 journal={Radiological physics and technology},
497 volume={10},
498 number={3},
499 pages={257--273},
500 year={2017},
501 publisher={Springer}
502}