Where Responsible AI Meets Reality: Practitioner Perspectives on Enablers for Shifting Organizational Practices

Introduction

While the academic discussion of algorithmic bias has an over 20-year long history, we have now reached a transitional phase in which this debate has taken a practical turn. The growing awareness of algorithmic bias and the need to responsibly build and deploy Artificial Intelligence (AI) have led increasing numbers of practitioners to focus their work and careers on translating these calls to action within their domains. New AI and machine learning (ML) responsibility or fairness roles and teams are being announced, product and API interventions are being presented, and the first public successes —and lessons learned —are being disseminated. However, practitioners still face considerable challenges in attempting to turn theoretical understanding of potential inequities into concrete action.

Gaps exist between what academic research prioritizes, and what practitioners need. The latter includes developing organizational tactics and stakeholder managementrather than technical methods alone. Beyond the need for domain-specific translation, methods, and technical tools, responsible AI initiatives also require operationalization within —or around —existing corporate structures and organizational change. Industry professionals, who are increasingly tasked with developing accountable and responsible AI processes, need to grapple with inherent dualities in their roleas both agents for change based on their own values and/or their official role, but also as workers with careers in an organization with potentially misaligned incentives that may not reward or welcome change. Most commonly, practitioners have to navigate the interplay of their organizational structures and algorithmic responsibility efforts with relatively little guidance. As Orlikowski points out, whether designing, appropriating, modifying, or even resisting technology, human agents are influenced by the properties of their organizational context. This also means that some organizations can be differentially successful at implementing organizational changes. Individuals’ strategies must adapt to the organizational context and follow what is seen as successful and effective behavior within that setting. Myerson for example describes the concept of “tempered radicals.” These are employees who slowly but surely create corporate change by pushing organizations through persistent small steps. Advocating for socially responsible business practices became part of tempered radicals’ role over time. These employees create both individual and collective action, relying on their own perceived legitimacy, influence and support built within their organizational context.

Interestingly, the tension between academic research and industry practice is visible in research communities such as FAccT, AIES, and CSCW, where people answering calls to action with practical methods are sometimes met with explicit discomfort or disapproval from practitioners working within large corporate contexts. Vice versa,practitioners affecting concrete change in practice may have achieved such results in ways that may not fit external research community expectations or norms. Within the discourse on unintended consequences of ML-driven system, we have seen both successes and very public failures —even within the same corporation—making it imperative to understand such dynamics.

This paper builds on the prior literature in both organizational change and algorithmic responsibility in practice to better understand how these still relatively early efforts are taking shape within organizations. We know that attention to the potential negative impacts of machine learning is growing within organizations, but how to leverage this growing attention to effectively drive change in the AI industry remains an open question. To this end, we present a study involving 26 semi-structured interviews with professionals in roles that involve concrete projects related to investigating responsible AI concerns or “fair-ML” (fairness-aware machine learning) in practice. We intend this to refer not only to fairness-related projects but also more broadly projects related to the work on responsible AI and accountability of ML products and services given the high degree of overlap in goals, research, and people working on these topics.

Using the data from the semi-structured qualitative interviews to compare across organizations, we describe prevalent, emergent, and aspirational future states of organizational structure and practices in the responsible AI field, based on how often respondents identified the practice during the interview and whether the practice is currently existing or is a desired future change. We investigate practitioners’ perceptions of their own role, the role of the organizational structures in their context, and how those structures interact with adopting responsible AI practices. Based on those answers, we identify four major questions that organizations must now adapt to answer as responsible AI initiatives scale. Furthermore, we describe how respondents perceived transitions occurring within their current contexts, focusing on organizational barriers and enablers for change. Finally, we present the outcome of a workshop where attendees reflected upon early insights of this study through a structured design activity.

The main contribution of our work is the qualitative analysis of semi-structured interviews about the responsible AI work practices of practitioners in industry. We found that most commonly, practitioners have to grapple with lack of accountability, ill-informed performance trade-offs and misalignment of incentives within decision-making structures that are only reactive to external pressure. Emerging practices that are not yet widespread include the use of organization-level frameworks and metrics, structural support, and proactive evaluation and mitigation of issues as they arise. For the future, interviewees aspired to have organizations invest in anticipating and avoiding harms from their products, redefine results to include societal impact, integrate responsible AI practices throughout all parts of the organization, and align decision-making at all levels with an organization’s mission and values. Preliminary findings were shared at an interactive workshop during a large machine learning conference, which yielded organizational level recommendations to (1) create veto ability across levels, (2) coordinate internal and external pressures, (3) build robust communication channels between and within levels of an organization, and (4) design initiatives that account for the interdependent nature of the responsible AI work practices we have heretofore discussed.

Literature review

Algorithmic responsibility in practice

An almost overwhelming collection of principles and guidelines have been published to address the ethics and potential negative impact of machine learning. Mittelstadt et al.discuss over sixty sets of ethical guidelines, Zeng et al.provide a taxonomy of 74 sets of principles, while Jobin et al. find 84 different sets of principles. Even if there is relative, high-level agreement between most of these abstract guidelines, how they are translated into practice in each context remains very unclear. Insight is available from how companies changed their practices in domains such as privacy and compliance in response to legislative directives. The active debate on how requirements in the EU’s GDPR are to be interpreted, however, illustrate the challenges of turning yet nascent external guidance into concrete requirements. Krafft et al.point out that even between experts, there is a disconnect between policymakers and researchers’ definitions of such foundational terms as `AI’. This makes the application of abstract guidelines even more challenging and raises the concern that focus may be put on future, similarly abstract technologies rather than current, already pressing problems.

The diverse breadth of application domains for machine learning suggests that requirements for applying guidelines in practice should be steered by the specific elements of the technologies used, specific usage contexts, and relevant local norms. Practitioners encounter a host of challenges when trying to perform such work in practice. Organizing and getting stakeholders on board are necessary to be able to drive change. This includes dealing with imperfection, and realizing that tensions and dilemmas may occur when “doing the right thing” does not have an obvious and widely agreed upon answer. It can be hard to foresee all potential consequences of systems while building them, and it can be equally difficult to identify how to overcome unwanted side-effects, or even why they occur technically. A fundamental challenge is that such assessment should not simply be about technical, statistical disparities, but rather active engagement to overcome the lack of guidance decision-makers have on what constitute “just” outcomes in non-ideal practice. Additional challenges include organizational pressures for growth, common software development approaches such as agile working that focus on rapid releases of minimal viable products, and incentives that motivate a focus on revenue within corporate environments. Taking inspiration from other industries where auditing processes are standard practice still means that auditing procedures have to be adjusted to product and organizational contexts, and require defining the goal of the audit in context. This means that wider organizational change is necessary to translate calls to action into actual process and decision-making.

Organizational change and internal/external dynamics

Current challenges faced by responsible AI efforts can be compared to a wide selection of related findings in domains such as legal compliancewhere questions arise regarding whether compliance processes actually lead to more ethical behavior, diversity and inclusion in corporate environments, and corporate privacy practices. All of these domains appear to have gone through a process that is mirrored in current algorithmic responsibility discussions: publication of high-level principles and values by a variety of actors, the creation of dedicated roles within organizations, and urgent questions about overcoming challenges and achieving “actual” results in practice and how to avoid investing in processes that are costly but do not deliver beyond cosmetic impact.

As Weaver et al. pointed out in 1999, in an analysis of the Fortune 1000 ethics practices, success relies not only on centralized principles, but also their diffusion into managerial practices in the wider organization. Interestingly, while external efforts can effectively put reputational and legislative pressure on companies, internal processes and audits are just as important, and they all interact. Internally, this is apparent in the process of legitimization of the work on tempered radicals' in Myerson's workas described in the introduction, and these radicals' internal journey. External forces can help in more or less productive ways in that process. As discussed by Bamberger and Mulligan, for corporate privacy efforts in particular, both external and internal forces are necessary for work on corporate responsibility to be effective. Internally, they suggest focusing on getting onto board-level agendas to ensure attention and resourcing, having a specific boundary-spanning privacy professional to lead adoption of work practices, and ensuring managerialization’ of privacy practices by increasing expertise within business units and integration within existing practices. Externally, they suggest that creating positive ambiguity by keeping legislation broad can push more accountability onto firms for their specific domains, which can create communities and promote sharing around privacy failures. They found that ambiguity in external privacy discussions could foster reliance on internal professionals’ judgements, and thus created autonomy and power for those professionals identified as leading in privacy protection. Thus, they illustrate how ambiguity —rather than a fully defined list of requirements —can actually help promote more reflection and ensure that efforts go beyond compliance.

A similar internal/external dynamic is visible within the algorithmic responsibility community. For example, in the Gender Shades project, Buolamwini and Gebrupresented not only an external audit of facial recognition APIs, but also reactions from the companies whose services were audited to illustrate more and less effective responses. Such external audits can result in momentum inside of companies to respond to external critique, and in selected cases, to make concrete changes to their products. Internal efforts in turn have access to more data, ensure that auditing can be completed before public releases, develop processes for companies, and allow companies to take responsibility for their impact. Successes are beginning to emerge, and have ranged from positive changes on policy and process resulting from corporate activism, tooling built for clients or internal purposes, to direct product “fixes” in response to external critique. For example, Raji et al.present an extensive algorithmic auditing framework developed by a small team within the larger corporate context of Google. They offer general methods such as data and model documentationand also tools such as metrics to enable auditing in specific contexts like image search. Implementing these methods and tools then requires corporate processes to provide the resources for such auditing and to ensure that results of audits impact decisions within the larger organizational structure.

Organizational research and structures

To situate our work in this broader context, we will briefly examine different perspectives on organizational structures. First, it is worthwhile to revisit what organizational theorist Wanda Orlikowskicalled the duality of technology in organizations. Orlikowski discusses how people in organizations create and recreate meaning, power and norms. Orlikowski’s `structurational’ model of technology comprises of these human agents, the technology that mediates their task execution, and the properties of organizations. The latter institutional properties range from internal business strategies, control mechanisms, ideology, culture, division of labor and procedures, communication patterns, as well as outside pressures such as governmental regulation, competition and professional norms, and wider socio-economic conditions. People’s actions are then enabled and constrained by these structures, which are themselves the product of previous human actions. This perspective was augmented by Orlikowskito include a practice orientation; repeated interactions with technologies within the specific circumstances also enact and form structures.

Similarly, Dawson provides an extensive review of perspectives in studies on organizational changeand discusses the process turn', where organizations are seen as ever-changing rather than in discrete states; what may appear as stable routines may in actuality be fluid. Dawson emphasizes the socially constructed process, and the subjective lived experiences: actors' collaborative efforts in organizations unfold over time and dialogue between them shapes interpretations of changes. Such dynamics are also present in what organizational theorist Richard Scottsummarized as the rational, natural, and open perspective on organizations. Rational’ organizations were seen as the machine', best suited to industries such as assembly line manufacturing where tasks are specified by pre-designed workflow processes. The natural’ organization signified a shift in organizational ideology. No longer were people seen as mere appendages to the machines, but rather as crucial learners in relationship with machines. The metaphor is that of the organization as an ‘organism’ with a strong interior vs. exterior boundary, and needs to survive'. Similar to an organism, the organisation grows, learns, and develops. As a consequence of the survival ideology, the exterior environment can be seen as a threat against which the organism must adapt to survive. Scott however describes how the notion of environment as threat’ was replaced by the realization that environmental features are the conditions for survival. The central insight emerging from open' systems thinking is that all organizations are incomplete and depend on exchanges with other systems. The metaphor became that of an ecology’. Open systems are characterized by (1) interdependent flows of information and (2) interdependent activities, performed by (3) a shifting coalition of participants by way of (4) linking actors, resources and institutions, in order to (5) solve problems in (6) complex environments. For responsible AI efforts to succeed then, organizations must successfully navigate the changes necessary within `open’ systems.

Multi-stakeholder communities as meta organizational structures

The described ecologies', particularly in open’ systems, contain formal and informal meta-organizational structures, which have been studied in other contexts and are of increasing growing importance to the field of responsible AI. Organizations often interact with each other through standards bodies, communities, processes, and partnerships. These meta-processes can have as goals (1) producing responses to proposed regulations, standards, and best practices, (2) fostering idea exchange between silos, and (3) self-regulation. Organizations participate in multi-stakeholder initiatives to achieve a number of their own goals, including advocating for business interests, keeping up to date on industry trends, and having a voice in shaping standards or regulations that they will then be subjected to.

Berkowitzdiscusses the shift towards governance in sustainability contexts, and the key role that meta-organizations can have in facilitating meta-governance of corporate responsibility beyond simply complying with legislation. She identifies six capabilities needed for sustainable innovations: (1) anticipation of changes and negative impacts of innovation, (2) resilience to changes, (3) reflexivity, (4) responsiveness to external pressures and changing circumstances, (5) inclusion of stakeholders beyond immediate decision makers, and (6) comprehensive accountability mechanisms. Meta-organizations can promote inter-organizational learning and building of these six capabilities.

Similarly, within the field of AI, multi-stakeholder organizations, standards, and self-organized projects have been created in recent years to acknowledge the need for interdisciplinary expertise to grapple with the wide reaching impacts of AI on people. Many AI researchers have been vocal proponents of expanding the number of perspectives consulted and represented, including stakeholders such as policymakers, civil society, academics from other departments, impacted users, and impacted nonusers. Reconciling perspectives from diverse stakeholders presents its own set of challenges that change depending on the structure of the organization. Participatory action offers relevant frameworks for characterizing options for decision making in multistakeholder contexts. Decision making can be centralized within a formal organization with stakeholders being informed, consulted, involved, collaboration, or else stakeholders can self-organize informally to achieve the same levels of participation. The structures present at a meta organizational level will differ and enable the application of different group-level decision making processes. For example, ad hoc groups of researchers have self-organized to create unconference events and write multi-stakeholder reports, including reports with large groups of authors (e.g.) based originally on discussions within workshops held under Chatham house rules, while others have created new formal organizations, conferences such as AIES or FAccT, or research institutes.

In a similar manner to Berkowitz, we focus here on the “how” of achieving more adoption of responsible AI work practices in industry. We further investigate how practitioners experience these changes within the context of different organizational structures, and what they see as the shifts that drive or hinder their work within their organizations.

Study and Methods

Our motivation for this work was to identify enablers that could shift organizational change towards adopting responsible AI practices. Responsible AI research has influenced organizational practices in recent years, with individuals and groups within companies increasingly tasked with implementing research into action, whether formally or informally. Our research applies theories and frameworks of organizational structure and change management to characterize the growing practice of applied responsible AI. To better understand the implications of organizational structure on day-to-day responsible AI work and outcomes, we interviewed practitioners who are actively involved in these initiatives by themselves or within a larger team.

We conducted 26 semi-structured interviews with people based in 4 continents from 19 organizations. Except for two 30 minute interviews, all other interviews lasted between 60 and 90 minutes. Participants were given a choice of whether to allow researchers to record the interview for note taking purposes. A total of 11 interviews were recorded. In cases where the interview was not recorded, we relied on writing down the respondents’ answers to the questions during the course of the interview. In several cases, participants requested to additionally validate any written notes and make necessary clarifications before their use in the study to ensure that their anonymity was not compromised.

Sampling technique

Participants were recruited through a convenience sampling technique through snowball sampling from participants who recommended other interviewees. Three recruiting criteria were used to find interviewees: (1) did they work closely with product, policy, and/or legal teams, (2) did the outputs of their work have a direct impact on ML products and services, and (3) were some aspects of their work related to the field of responsible AI. We filtered out individuals whose roles were solely research, although interviewees may also be active contributors to responsible AI research in addition to their existing work stream.

Through the ongoing conversations we had with practitioners before as well as after conducting the qualitative interviews, we aimed to establish a substantial level of trust and transparency, which we felt was necessary given the sensitive nature of the topics discussed. This allowed for more open, nuanced, and in-depth discussions where practitioners felt that there is a shared understanding between interviewers and interviewees about the often unvoiced challenges in responsible AI work.

We intentionally sought to interview as diverse a group of practitioners as possible to capture perspectives from a broad range of organizational contexts. In Table tab-roles-table we summarize the functional roles of the interviewees who participated in the project and how they describe their responsible AI work. Participants came from a wide variety of functions, including AI Strategy, Engineering, Human Resources, Legal, Marketing and Sales, Machine Learning Research, Policy, and Product Management. Among the 26 participants, ten had educational background in Social Science, eight in Computer Science, seven in Law and Policy, and one practitioner had a degree in Economics. The majority of respondents were geographically located in the US (21 out of 26), two participants were in the UK, and the rest of the respondents were based in Australia, Denmark, and Japan. The average length that interviewees have been with their organization is 5 years and 5 months, where more than one third of the practitioners have been with their company for more than five years (9 people) and 2 people spent decades with their organization. Lastly in terms of organizational sectors, 11 practitioners worked in business-to-business organizations, 2 in business-to-consumer, and 13 in organizations which were both business-to-business and business-to-consumer.

Interview protocol

The script and questions for the semi-structured interviews were reviewed by an industrial-organizational psychologist and responsible AI practitioners within three different organizations. Questions were grouped into different sections, exploring the current state of responsible AI work, the evolution of the work over time, how the work is situated within the organization, how responsibility and accountability for the work are distributed, performance review processes and incentives, and what desired aspirational future structures and processes would enable more effective work. The semi-structured nature of the interview provided standard questions that were asked of all participants while allowing interviewers the flexibility to follow up on interesting insights as they arose during interviews. The full set of questions can be found in Appendix questionnaire .

Analysis

To analyze the interview data, we utilized a standard methodology from contextual design - interpretation session and affinity diagramming. Through a bottom up affinity diagramming approach, we iteratively assigned codes to various concepts and themes shared by the interviewees. We iteratively grouped these codes into successive higher level themes and studied the relationships between them.

Workshop

In addition to semi-structured interviews, we organized a workshop at a machine learning conference attended by a highly self-selected group of people interested in responsible AI from industry, academia, government, and civil society. The first half of the workshop was a presentation of preliminary insights from the literature review and results sections of this paper. We then conducted an interactive design exercise where participants were organized into 13 groups of between 4 and 6 individuals per group. Each group was given a scenario description of an AI organization that exemplified the prevalent work practices discussed in the Results: Interviews section below. The facilitators guided groups through a whiteboard discussion of the following questions:

• What are examples of emerging responsible AI work practices in the context of the scenario?
• What are examples of structures or processes in the prevalent organizational structure which are outside of the scope of responsible AI work but which act to protect and enable emerging fair-ML practices?
• What are examples of outlier practices outside of the prevalent practices in the scenario?
• What connections exist between these practices and organizational structures?
• What practices or organizational structures could enable positive self-reinforcing outcomes through making the connections stronger? The workshop activity was designed to allow participants to (1) gain a deeper understanding of the responsible AI challenges by connecting study findings to their own experiences, (2) collaboratively explore what organizational structures could enable the hypothetical organization developing AI products and services to resolve them through, and (3) map interdependencies and feedback loops that exist between practices to identify potentially effective recommendations to address the challenges of implementing responsible AI initiatives.

Results: Interviews

We start with a high level overview of our findings followed by a discussion of the key themes that emerged from the conducted interviews.

Overview

About a quarter of the participants had initiated their responsible AI work in their current organization within the past year (7 out of 26) compared to 73% (19 out of 26) worked on efforts that had started more than an year ago. More than half of the interviewees worked on their initiatives as individuals and not as part of a team (14 out of 26). About 40% of the respondents reported that theyvolunteer time outside of their official job function to do their work on responsible AI initiatives (11 out of 26) while the remaining 15 out of 26 participants had official roles related to responsible AI. Among the 15 interview participants with official roles related to responsible AI, 8 individuals were externally hired into their current role, while 7 transitioned into it from other roles within their organization. Interviewees who changed the focus of their existing roles or transitioned into responsible AI-related role were most commonly previously in project management roles (4 out of 7), then research (2 out of 7), then legal (1 out of 7). The majority of participants who had official responsible AI-related roles reported benefiting from an organizational structure that allowed them to craft their own role in a very dynamic and context-specific way.

Since the beginning of our conversations, we noticed that practitioners used different language in the way they described their work and how it relates to responsible AI. We observed commonalities in the way practitioners from each function framed their responsible AI work (see Table 1 ). For example, while project managers described their work in terms of product life-cycles and industry trends, legal practitioners discussed the responsible AI aspects of their role in terms of comprehensive pillars and ethical governance guidelines.

We note that a few interviewees described going through stress-related challenges in relation to their responsible AI work. During some of the interviews, we saw a noticeable tone change in the interviewees’ voice when discussing questions related to ethical tensions, accountability, risk culture, and others. Furthermore, some respondents have left their organizations between when we conducted the interviews in late 2019 and when we submitted this paper in October 2020. While we acknowledge the nascent state of responsible AI functions, these observations could point to opportunities for further study.

There were various common perspectives that we heard practitioners express repeatedly. We saw the need for a multi-faceted thematic analysis which encompasses three intuitive clusters of data: (1) currently dominant or prevalent practices, (2) emerging practices, and (3) aspirational future context for responsible AI work practices in industry:

• The prevalent practices comprise what we saw most commonly in the data.
• The set of emerging practices includes practices which are shared among practitioners but less common than prevalent practices.
• The aspirational future consists of the ideas and perspectives practitioners shared when explicitly asked about what they envision for the ideal future state of their work within their organizational context.

Within the thematic analysis (see Table table-1-overview ), we found four related but distinct key questions that every organization must have processes and structures to support answering:

• When and how do we act?
• How do we measure success?
• What are the internal structures we rely on?
• How do we resolve tensions? As organizations seek to scale responsible AI practices, they will have to transition from the prevalent or emerging practices of answering these questions to the structures and processes of the aspirational future. It is important to note that not all emerging practices we found in the data will necessarily lead to the aspirational future. In what follows, we provide details about practitioners’ personal perspectives and experiences within the individual themes and questions.

  Trends in the common perspectives shared by diverse responsible AI practitioners.

  Table Label: tab-overview-table

  Download PDF to view table

When and how do we act?

One transition we identified in the data is how organizations choose when and how to act. This includes questions of who chooses to prioritize what information within which decision-making processes. We found that many organizations behave reactively, fewer are now proactive, and respondents aspire for their organizations to become anticipatory in the future.

Prevalent work practices

Most commonly, interviewees described responsible AI work in their organizations as reactive. The most prevalent incentives for action were catastrophic media attention and decreasing media tolerance for the status quo. Many participants reported that responsible AI work can be perceived as a “taboo topic” in their organizations. Raising awareness was a challenge for one interviewee who shared: “It was an organizational challenge for us, it’s hard as when something is so new - we run into ‘Whose job is this?’” when they bring up topics about algorithmic fairness or inequity in harm at work. We found that the uncertainty and unwillingness to engage in a deeper understanding of responsible AI issues may lead to unproductive discussions or outright dismissal of important but often unvoiced concerns. responsible AI work is often not compensated, as in the case of the 40% of respondents volunteering their time to work on responsible AI initiatives, or is perceived as ambiguous or too complicated for the organization’s current level of resources. In response to the question about how interviewees are recognized for their work, one interviewee shared: “many of the people volunteering with our team had trouble figuring out how to put this work in the context of their performance evaluation.” In several cases, the formation of a full-time team to conduct responsible AI work was only catalyzed by the results from volunteer -led investigations of potential bias issues within models that were en route to deployment. The volunteers for these investigations went far beyond their existing role descriptions, sometimes risking their own career progression, to take on additional uncompensated labor to prevent negative outcomes for the company. This highlights the reactive nature of organizational support for responsible AI work in prevalent practice. Legal compliance was another factor that participants said could motivate organizational action. Beyond legal concerns, some practitioners reported that being able to use reputational risk as leverage to increase investment in responsible AI work, bringing hypothetical questions like “What if ProPublica found out about …?” into decision-making meetings. Participant responses in this section illustrate how a reactive organizational stance towards responsible AI work shifts the labor and cost of identifying and addressing issues onto the individual worker.

Emerging work practices

In emerging practices on how and when to act, a feworganizations have implemented proactive responsible AI evaluation and review processes for their ML systems, with the work and accountability often distributed across several teams. For example, some respondents reported support and oversight from legal teams. In a few cases, interviewees spoke with enthusiasm about the growing number of both internal and external educational initiatives. This included onboarding and upskilling employees through internal responsible AI curricula to educate employees about responsible AI-related issues and risks as well as externally facing materials to educate consumers and customers. Respondents referred to these efforts as an organization-level proactive investment to set up the organization to better address future responsible AI issues. Furthermore, a few participants shared about the availability or their involvement in preparing externally facing materials to educate their organization’s customers or potential customers about responsible AI considerations in practice. A small number of interviewees reported that their work on responsible AI is acknowledged and explicitly part of their compensated role, in contrast to the volunteers in the prevalent practices theme, which is another organization-level difference between prevalent and emerging practices.

On the other hand, emerging practices still show how individuals rather than organizational processes or structures remain the engine of proactive practices. In a few cases, proactive champions organizing grassroots actions and internal advocacy with leadership have made responsible AI a company-wide priority, which then sometimes made it easier for people to get resourcing for responsible AI initiatives and to establish proactive organization-wide processes. Some participants reported leveraging existing internal communication channels to organize responsible AI discussions. One participant even captured screenshots of problematic algorithmic outcomes and circulated them among key internal stakeholders to build support for responsible AI work. Similar to prevalent practices, these individuals are tasked with the labor of using existing organizational structures to build organizational support for their responsible AI work in addition to doing their responsible AI work. The difference in emerging organizational practice is how these individuals are finding more success in instilling a proactive, rather than reactive, mindset for approaching algorithmic responsibility.

Mapping the aspirational future

In an ideal future, many interviewees envisioned organizational frameworks that encourage an anticipatory approach.In the future state, an individual wanting to engage with algorithmic responsibility issues would not necessarily need to do theorganizational labor of changing structures as in the prevalent and emerging practices, but rather be supported by organization-wide resources and processes to focus their efforts directly on responsible AI work. In this aspirational future, respondents envisioned technical tools to enable large-scale implementation of responsible AI evaluations both internally and externally: well-integrated technical tools would assess algorithmic models developed by product teams and feed seamlessly into organization-wide evaluation processes that identify and address risks of pending ML systems before they go live in products, while externally, customers using the algorithmic models in different contexts have oversight through explicit assessments, which feed information about identified risks back to the organization. Their organizations would utilize clear and transparent communication strategies to explain the process and results of these evaluations both internally within the entire organization and externally with customers and other stakeholders. One practitioner questioned if their team should even engage with customers who do not agree to deploy an assessment framework ex-ante, suggesting a new baseline expectation for customers to also play their role in faster feedback loops for identifying and mitigating risk. Respondents reported that in the ideal future, product managers would have an easier way to understand responsible AI concerns relevant to their products without needing to regularly read large numbers of research papers, which could be supported by organization-level teams, tools, and/or education to synthesize and disseminate relevant knowledge. Several participants expressed that the traditional engineering mindset would need to become better aligned with the dynamic nature of responsible AI issues which cannot be fixed in predefined quantitative metrics. Anticipatory responsible AI frameworks could allow organizations to respond to the responsible AI challenges in ways which uphold organizational code of ethics and society’s values at large.

How do we measure success?

Another transition we saw our respondents navigating in their work and organizations is how organizations measure success. Many responsible AI initiatives are relatively newer and aim to measure the societal impact of technology, which is a departure from traditional business metrics like revenue or profitability. Learning organizations need to make an active change to better account for this shift. Respondents reported that many challenges in their prevalent work practices arise from the inability to adequately use existing metrics to account for the goals of responsible AI work, while emerging practices aim to begin rewarding success that falls outside of pre-existing narrow definitions. In an aspirational future, organizations value responsible AI work and processes reflect that at every level.

Prevalent work practices

The majority of respondents reported that one of the biggest challenges for their responsible AI work is the lack of metrics that adequately capture its true impact. The majority of respondents also expressed at least some degree of difficulty in communicating the impact of their work. Combined, this hinders them from fully illustrating the importance of responsible AI work for the organization’s success, which in turn keeps them from being able to receive adequate credit and compensation for their true impact. The challenges of measuring the impact of responsible AI is a deeply researched topic in the field of fairness, accountability, and transparency of ML. Through our interview questions, we have tried to further disentangle the perspectives on this challenge in industry. For example, some industry practitioners reported that the use of inappropriate and misleading metrics is a bigger threat than the lack of metrics. Respondents shared that academic metrics are very different than industry metrics, which include benchmarks and other key performance indicators tracked by product teams, such as metrics related to customer retention and development (click rate, time spent using a product, etc.) Project managers reported trying to implement academic metrics in order to both leverage academic research and facilitate a collaboration between research and product teams within their organization. One of the interviewees shared that in their personal perspective, “industry-specific product-related problems may not have sufficient research merit or more specifically an ability for the researcher to publish, sometimes because of privacy reasons data used in the research experiments may not allow researchers to be recognized for their work.” This may be due to the nature of the problem or due to privacy reasons. Since data used in the research experiments may not allow researchers to be recognized for their work, this may ultimately discourage them from investigating real world responsible AI issues. Practitioners embedded in product teams explained that they often need to distill what they do into standard metrics such as number of clicks, user acquisition, or churn rate, which may not apply to their work. Most commonly, interviewees reported being measured on delivering work that generates revenue. They spoke at length about the difficulties of measuring responsible AI impact in terms of impact on the business “bottom line.” In some cases, practitioners framed their impact in terms of profitability by arguing that mitigating responsible AI risks prior to launch is much cheaper than waiting for and fixing problems that arise after launch where real-world harm and reputational risk come into play. Again, the prevalent work practices reveal individuals working on responsible AI taking on the extra labor of trying to translate their work into ill-fitting terms and metrics that are not designed to measure or motivate success on responsible AI outcomes.

The majority of respondents expressed at least some degree of difficulty in communicating the impact of their work. The metrics-related challenges they described included: (1) product teams often have short-term development timelines and thus do not consider metrics that aim to encompass long-term outcomes; (2) time pressure within fast-paced development cycles leads individuals to focus on short-term and easier to measure goals; (3) qualitative work is not prioritized because it requires skills that are often not present within engineering teams; (4) leadership teams may have an expectation for “magic,” such as finding easy to implement solutions, which in reality may not exist or work; (5) organizations do not measure leadership qualities and (6) do not reward the visionary leaders who proactively address the responsible AI issues that arise; (7) performance evaluation processes do not account for responsible AI work, making it difficult to impossible for practitioners to be rewarded or recognized for their responsible AI contributions.

Emerging work practices

A few interviewees reported that their organizations have implemented metrics frameworks and processes in order to evaluate responsible AI risks in products and services. Practitioners talked enthusiastically about how their organizations have moved beyond ethics washingin order to accommodate diverse and long-term goals aligned with algorithmic responsibility and harm mitigation, the goals of a responsible AI practice. Interviewees identified the following enablers for this shift in organizational culture: (1) rewarding a broad range of efforts focused on internal education; (2) rewarding risk-taking for the public good; (3) following up on potential issues with internal investigations; (4) creating organizational mechanisms that enable cross-functional collaboration. These emerging organizational enablers begin to set organizational scaffolding of a work environment that supports individuals working on responsible AI as they seek to change how their organization assigns value to work to better align with the societally-focused outcomes.

Mapping the aspirational future

In an aspirational future where responsible AI work is effective and fully supported by organizational structures, interviewees reported that their organizations would measure success very differently than today’s prevalent practices: (1) their organizations would have a tangible strategy incorporate responsible AI practices or issues into the key performance indicators of product teams; (2) teams would employ a data-driven approach to manage ethical challenges and ethical decisions in product development; (3) employee performance evaluation processes would be redefined to encompass qualitative work; (4) organizational processes would enable practitioners to collaborate more closely with marginalized communities, while taking into account legal and other socio-technical considerations; (5) what is researched in academic institutions would be more aligned with what is needed in practice; (6) collaboration mechanisms would be broadly utilized. Specifically, participants discussed two kinds of mechanisms to enable collaboration: (1) working with external groups and experts in the field to define benchmarks prior to deployment, and (2) working with external groups to continuously monitor performance from multiple perspectives after deployment.

What are the internal structures we rely on?

In order for individuals to better enable responsible AI work, they need to reexamine the properties of their organizations. This involves leveraging what Orlikowski called the “structurational” model of technology in a specific applied context. In the prevalent practices, organizations do not have internal structures to ensure accountability for responsible AI work, which can then be neglected due to role uncertainty without consequences. Distributed accountability on top of existing structures was reported in emerging practices, while in the aspirational future, responsible AI work would become integrated into all product-related processes to ensure accountability.

Prevalent work practices

Most commonly, participants reported ambiguity and uncertainty about role definitions and responsibilities within responsible AI work at their organization, sometimes due to how rapidly the work is evolving. Multiple practitioners expressed that their responsible AI related concerns were heard on account of their seniority in their team and organization. In response to “Do you have autonomy to make impactful decisions?”, one data science practitioner who was volunteering time with the responsible AI team shared, “More senior people are making the decisions. I saw ethical concerns but there was difficulty in communicating between my managers and the [responsible AI] team. People weren’t open for scrutinization.” This illustrates the fragility of the prevalent practice since accountability relies on the individual’s own resources, interests, and situational power rather than scalable and systemic organizational structures and processes that would ensure the desired outcomes. Several interviewees talked about the lack of accountability across different parts of their organization, naming reputational risk as the biggest incentive their leadership sees for responsible AI work, again tying accountability to individual incentives to take responsibility rather than ensuring accountability through organization-wide processes and policies.

Emerging work practices

Interviewees shared these emerging organizational structures as enablers for responsible AI work: (1) flexibility to craft their roles dynamically in response to internal and external factors; (2) distributed accountability across organizational structures and among teams working across the entire product life cycle; (3) accountability integrated into workflows; (4) processes to hold teams accountable for what they committed to; (5) escalation of responsible AI issues to management; (6) responsible AI research groups that contribute to spreading internal awareness of issues and potential solutions; (7) internal review boards that oversee responsible AI topics; (8) publication and release norms that are consistently and widely followed; (9) cross-functional responsible AI roles that work across product groups, are embedded in product groups, and/or collaborate closely with legal or policy teams. Participants also reported being increasingly cognizant of external drivers for change, such as cities and governments participating to create centers of excellence, for example, New York’s Capital District AI Center of Excellence. As before, these emerging structures begin to shift the locus of responsibility for managing organizational change away from the individual who seeks to do responsible AI work, which is not necessarily the same as organizational change management work, and onto organizational processes and structures that can distribute that labor in an appropriate manner.

Mapping the aspirational future

In the future, interviewees envisioned internal organizational structures that would enable responsible AI responsibilities to be integrated throughout all business processes related to the work of product teams. One practitioner suggested that while a product is being developed, there could be a parallel development of product-specific artefacts that assess and mitigate potential responsible AI issues. The majority of interviewees imagined that responsible AI reviews and reports would be required prior to release of new features. New ML operations roles would be created as part of responsible AI audit teams. Currently, this work falls within ML engineering, but respondents identified the need for new organizational structures that would ensure that responsible AI concerns are being addressed while allowing ML engineers to be creative and experiment. For example, one practitioner suggested that a responsible AI operations role could act as a safeguard and ensure that continuous responsible AI assessments are being executed once a system is deployed. Some interviewees described the need for organizational structures that enable external critical scrutiny. Scale could be achieved through partnership-based and multistakeholder frameworks. In the future, public shaming of high-stakes AI failures would provide motivation towards building shared industry benchmarks, and structures would exist to allow organizations to share benchmark data with each other. External or internal stakeholders would need to call out high impact failure use cases to enable industry-wide learning from individual mistakes. Industry-wide standards could be employed to facilitate distributed accountability and sharing of data, guidelines, and best practices. Of note is that in the aspirational future, organizational structures and processes incorporate external parties and perspectives, providing organizations better channels to understand their societal impact.

How do we resolve tensions?

Lastly, responsible AI work brings new types of tensions that organizations may not yet have processes to resolve, especially related to the questions of ethics and unintended consequences of socio-technical systems like AI. This requires organizations to update their prevalent practices in their transitions to better enable responsible AI work. Resolving tensions requires organizations to choose what to prioritize in a situation where there’s a need for trade-offs. The practices described below show the different approaches that organizations are taking in prevalent practices, emerging practices, and in the aspirational future.

Prevalent work practices

The majority of respondents reported that they see misalignment between individual, team, and organizational level incentives and mission statements within their organization. Often, individuals reported doing ad hoc work based on their own values and personal assessment of relative importance. Similarly, the spread of information relies on individual relationships. Practitioners reported relying on their personal relationships and ability to navigate multiple levels of obscured organizational structures to drive responsible AI work. Related to the question about “What are the ethical tensions that you/your team faces?”, one of the interviewees shared, “We often work on prototypes for specific geographic units which are not meant to be scaled, it’s really meant not to be scaled. We need to step in and make that clear. Also sometimes people state the model is complete, we need a disclaimer that we’re still updating and validating it, it is work in progress.” Many of the interviewees had to navigate tensions related to scale and expectations on a daily basis. Like in the other transitions, this highlights a prevalent practice of relying on individuals to decide how to resolve tensions rather than organizational processes that would support individuals in evaluating tensions in alignment with the organization’s mission or values. This creates additional labor and uncertainty for individuals doing responsible AI work in organizations exhibiting prevalent practices.

Emerging work practices

One of the biggest challenges practitioners reported was that as responsible AI ethical tensions are identified, overly rigid organizational incentives may demotivate addressing them, compounded by organizational inertia which sustains those rigid incentives. In this case, although the organizational structures in the emerging work practice shift labor away from individuals onto organization-wide processes, the processes themselves are not sufficiently aligned with the ultimate goals of responsible AI. Therefore, this makes the transition from prevalent to emerging practice one that steers the organization away from, rather than towards, the aspirational future where organizations resolve tensions in a way that encourages responsible AI work. Respondents described that in this situation, research and product teams struggle to justify research agendas related to responsible AI. This was caused by competing priorities that may align better with existing incentives and metrics for success, which as reported in Section measuresuccess: How do we measure success? , do not adequately account for the impact of responsible AI initiatives.

Interviewees identified several factors that limit an organization’s ability to resolve tensions in a manner that enables, instead of hinders, responsible AI work: (1) incentives that reward complexity whether or not it is needed - individuals are rewarded for complex technical solutions; (2) lack of clarity around expectations and internal or external consequences; (3) impact of responsible AI work being perceived as diffuse and hard to identify; (4) lack of adequate support and communication structures - whether interviewees were able to address responsible AI tensions often depended on their network of high trust relationships within the organization; (5) lack of data for sensitive attributes, which can make it impossible to evaluate certain responsible AI concerns.

Mapping the aspirational future

When asked about their vision for the future of their responsible AI initiatives, several respondents wanted responsible AI tensions to be addressed in better alignment with organization-level values and mission statements. They imagined that organizational leadership would understand, support, and engage deeply with responsible AI concerns, which would be contextualized within their organizational context. Responsible AI would be prioritized as part of the high-level organizational mission and then translated into actionable goals down at the individual levels through established processes. Respondents wanted the spread of information to go through well-established channels so that people know where to look and how to share information. With communication and prioritization processes in place, finding a solution or best practice in one team or department would lead to rapid scaling via existing organizational protocols and internal infrastructure for communications, training, and compliance, in contrast to the current prevalent situation that respondents described. Respondents wanted organizational culture to be transformed to enable (1) releasing the fear of being scrutinized as a roadblock for allowing external critical review and (2) distributing accountability for responsible AI concerns across different organizational functions. In the future state, every single person in the organization would understand risk, teams would have a collective understanding of risk, while organizational leadership would talk about risk publicly, admit when failures happen, and take responsibility for broader socioeconomic and socio-cultural implications.

Results: Interdisciplinary Workshop

As described in the Section studyandmethods: Study and Methods , after the interviews with practitioners were completed, a workshop was held at a responsible AI oriented venue [anonymized for review]. Each of the four key organizational questions we identified in Section results: Results: Interviews needs to be considered within the unique socio-technical context of specific teams and organizations - (1) When and how do we act? (2) How do we measure success? (3) What are the internal structures we rely on? and (4) How do we resolve tensions? However, the literature and interview findings suggest that there are likely similar steps or tactics that could lead to positive outcomes. The workshop activity allowed groups to create landscapes of practices based on their own experiences and then illuminate connections and feedback loops between different practices. Participants were given a simple scenario describing the prevalent work practices and organizational structure of an AI product company in industry as described in Section studyandmethods: Study and Methods . They then engaged in identifying enablers and tensions elucidating current barriers and pointing the way towards possible solutions. The following themes emerged in the insights participants shared during the activity:

The importance of being able to veto an AI system

Multiple groups mentioned that before considering how the fairness or societal implications of an AI system can be addressed, it is crucial to ask whether an AI system is appropriate in the first place. It may not be due to risks of harm, or the problem may not need an AI solution. Crucially, if the answer is negative, then work must stop. They recommended designing a veto power that is available to people and committees across many different levels, from individual employees via whistleblower protections, to internal multidisciplinary oversight committees to external investors and board members. The most important design feature is that the decision to cease further development is respected and cannot be overruled by other considerations.

The role and balance of internal and external pressure to motivate corporate change

The different and synergistic roles of internal and external pressure was another theme across multiple groups’ discussions. Internal evaluation processes have more access to information and may provide higher levels of transparency, while external processes can leverage more stakeholders and increase momentum by building coalitions. External groups may be able to apply pressure more freely than internal employees that may worry about repercussions for speaking up.

Building channels for communication between people (employees and leadership, leadership and board, users and companies, impacted users and companies)

Fundamentally organizations are groups of people, and creating opportunities for different sets of people to exchange perspectives was another key enabler identified by multiple groups. One group recommended a regular town hall for employees to be able to provide input into organization-wide values in a semi-public forum.

Sequencing these actions will not be easy because they are highly interdependent

Many of the groups identified latent implementation challenges because the discussed organizational enablers work best in tandem. For example, whistleblower protections for employees and a culture that supports their creation would be crucial to ensure that people feel safe speaking candidly at town halls.

It is interesting to observe that workshop discussion groups identified organization-level structures and processes that support and amplify individual efforts as one of the key enablers for responsible AI work. Additionally, these themes are shared as a starting point to spark experimentation. Further pooling of results from trying these recommendations would accelerate learning and progress for all towards achieving positive societal outcomes through scaling responsible AI practices.

Discussion and Conclusion

As ML systems become more pervasive, there is growing interest and attention in protecting people from harms while also equitably distributing the benefits from these systems. This has led researchers to focus on algorithmic accountability and transparency as intermediary goals on the path to better outcomes. However, corporate responsibility and organizational change are not new themes. The processes we see elucidated by Orlikowskiand Mysersonalso apply to responsible AI. Myserson described how tempered radicals forge collective action through clarifying issues and creating movements, with a focus on internal culture and actively soliciting support using small but persistent steps. Our interviews and workshop discussions echo similar processes. The results suggest that what individuals working on responsible AI need is for the organizational structures around them to adapt in order to support rather than hinder their work. This can happen as a product of their own advocacy, demonstrated early successes, and/or from leadership proactively steering into these transitions. As Myerson points out, tempered radicals should be aware of new opportunities or threats during their work to elevate social responsibility to an internal corporate priority, and frame their work so it appeals to organizational interests. The resulting tensions in how labor and responsibility are distributed between individuals compared to supporting processes or structures were also prevalent in our findings.

In order to succeed, practitioners have to map out a route from prevalent work practices to their aspirational future state goals. Along the way, they need to leverage existing practices to build momentum for emerging work practices that can lead them there. Similarly, it is essential that practitioners are able to identify and avoid creating emerging work practices that work against their desired long-term outcomes. They need to have a clear enough view of what the aspirational future should be, while adjusting to changing circumstances. This means both maintaining alignment with the existing organizational state, while maintaining a long-term goal orientation. Our interviews and workshop discussion identified the resulting tensions in getting to that aspirational state.

Throughout the four key organizational questions in which transitions are necessary to accommodate responsible AI work, we saw that prevalent practices can place the burden of responsibility and labor squarely on individuals to identify issues and try to change outcomes within existing structures.

This means pushing for changes in those structures and processes, as their goals may be antithetical to what is currently supported by the organizational structure. Thus, individuals who want to bring responsible AI issues into their work must do their own jobs, do the responsible AI work if that is not their official job, do the difficult work of redesigning organizational structures around them to accommodate the responsible AI work, and on top of it all, do the change management to get those new organizational practices to be adopted. As a result, incentives may appear misaligned between individuals and their organizational context. This can make it challenging to create adequate support, which should come from communicating the (sometimes small) steps that together make the larger organizational successes. This can invoke feelings of a lack of clarity on expectations and impact. As summarized in Table table-1-overview , our participants had to decide when and how to act, how to reframe success, orient themselves within internal structures, and resolve tensions between incentives.

Navigating these questions in organizations exhibiting prevalent practices requires skills that are not necessarily part of the regular conversation at academic venues. Perhaps then, rather than focusing on technical complexity, or on calls to (ideal) action alone, we should as a research community also prioritize providing researchers and practitioners with the tools and organizational insight to ensure that they have clear strategies to face this challenge. Researchers who make transitions to industry need to communicate the impact of their work in ways that builds them support within organizations, and legitimacy along the way. They need to have the skills and tools to navigate internal structures and tensions. This requires training, mentorship and sponsorship much beyond technical or research skills. The most effective ways for a particular organization may not always be perfectly aligned with research community norms; perhaps there lies another tension. Rather than having individuals find out the organizational work required on their own, and encounter pitfalls anew as individuals, we can provide support as an insights community, but only when we take this less public work very seriously as a core field of inquiry, as well as education. Perhaps then, this could help us as a wider community to move towards an “open” system, as described in other organizational settings by Scott, linking different actors, resources and institutions, and solving complex problems, in similarly complex environments.

We observed that organizations exhibiting the emerging practices were beginning to implement new structures and processes or adapt existing ones, although some emerging structures like rigid organizational incentives within high-inertia contexts can hinder rather than support responsible AI work. The remaining emerging work practices did better enable responsible AI work, often by reducing the labor burden on individuals especially to identify what organizational processes and policies are necessary to support their responsible AI work and to change manage the transitions of adopting those new work practices. This frees up time that individuals can reclaim to focus on the responsible AI work itself.

In the aspirational future, organizational structures and processes would fully provide mechanisms for monitoring and adapting system-level practices to incorporate and address emergent ethical concerns, so individuals who care about algorithmic responsibility issues can easily devote their time and labor to making progress on the specific issues within their functions. The internal advocacy and change management work would be full-time roles given to people with the skills, training, and desire to focus on that work, who could also offer expertise and mentorship to other individuals as they band together to create system-level change inside and beyond their organizations. Individuals working on responsible AI would then be free to focus on their specific job, rather than on changing the job environment to make it possible to do their job.

The impact of ML systems on people cannot be changed without considering the people who build them and the organizational structure and culture of the human systems within which they operate. A qualitative methodological approach has allowed us to build rich context around the people and organizations building and deploying ML technology in industry. We have utilized this qualitative approach here in order to investigate the organizational tensions that practitioners need to navigate in practice. We describe existing enablers and barriers for the uptake of responsible AI practices and map a transition towards an aspirational future that practitioners describe for their work. In line with earlier organizationalresearch, we emphasize such transitions are not to be seen as linear movements from one fixed state to another, rather they represent persistent steps and coalition building within the ever-changing nature of organizational contexts themselves.

June 2020 [revised]October 2020 [accepted]December 2020

Questionnaire

Describing current work practices related to fairness, accountability, and transparency of ML products and services

Describe your role

• What is your formal role by title?

• How would you describe your role?

• Is your formal role matched to your actual role?

• If not, how is it not?

• Is your organization flexible in the way it sees roles?

• If not, what is it like?

• How did you assume your role?

• If hired in, were you hired externally or transitioned?
• If you transitioned, where did you transition from?
• Does your company generally move people around fluidly?
• Does your company reward broad knowledge/skills across different industries or specializations?

• How did your role change over time?

• From a responsibility perspective?
• From a people perspective?

• Is role scope change typical at your company?

• If yes, what does it typically look like - is it…

• Scope creep?
• Is it explicitly within your job description?
• Planned role expansion?
• Stretch assignments?

• Do you have autonomy to make impactful decisions?

• If yes, how?
• If no, what is the case instead?

How did your fairML effort start?

• Was it officially sponsored?

• If yes, by whom - what level of leader?
• If no, who launched the effort - was it a team? A motivated individual? What level of leadership?

• Why did the effort start?

• Was the effort communicated to employees?

• Who was it communicated to?
• How was it communicated?

• Is the effort part of a program or stand-alone?

• Is it tied to a specific product’s development or launch?

• What is the product?
• What is its primary use case?
• Who is the primary end user?
• When is it slated to launch?

• Are you part of a team or doing this kind of work by yourself?

• Is it a volunteering effort?

• If so, are you getting rewarded or recognized for your time? How?

• What types of activities have been done or are planned?

• Are you actively collaborating with external groups? What groups and why?

Responsibility and accountability

• Who is accountable for aspects around risk or unintended consequence…

• Identifying risk?
• Solutioning against risk?
• Fixing mistakes?

• Avoiding negative impact, including press? Is your sponsor connected to risk management?

• How so?
• What is their level of accountability relative to risk?
• What are they responsible for doing?

• Who are your main stakeholders?

• What are the other departments you work with?

• How has that changed since the effort launched?
• What was the business case for the fairML work/team?
• What teams are adjacent to this effort (ie, not directly involved but “friends and family”) - is there an active compliance function, eg?
• (if no answer, probe for e.g. product teams, compliance, trust and safety type teams, or value-based design efforts, ethics grassroots etc)
• What are other efforts in your organization that are similar to Accountability work for instance Diversity & Inclusion and what does that look like? Is there general support for this type of effort?

• Do you feel there is support for this effort?

• Why or why not?
• Who supports it (what company career level, function, role and/or geography)?
• Who doesn’t support it?

• Would you say this effort aligns to company culture? How or how not?

• Is scaling possible?

• If so, do you intend to scale?
• If not, why not?

Performance, rewards, and incentives

• How is performance for your Algorithmic Accountability effort defined at your company?

• What are you evaluated on in your role?

• What works about the way performance is measured? What are some flaws?

• What does your performance management system/compensation structure seek to incentivize people to do (what is the logic behind the approach)?

• What does your performance management system/compensation structure actually incentivize people to do?

• What kind of person gets consistently rewarded and incentivized?

Risk culture

• How do you work with external communication teams - PR, Policy?

• Who owns that relationship - is it a centralized team?
• What is that comms teams primary accountability (eg, press releases, think pieces, etc)?
• Has the team managed risk before?
• Is the team mobilized to manage risk?

• How do you work with Legal?

• Is it a visible function in the organization?
• Does it have authority to make decisions and company policy, from your PoV?
• How do you engage with communities?
• What types of communities?
• What does this look like?
• What types of communication have you set up?

• What are the ethical tensions that you/your team faces?

• On a scale of 1-5, what is your level of perception of your company’s risk tolerance?

Future dream state - a structured way of getting a mapping of future dream state

• What is your company’s current state for fairML practice? (people, process, technology)

• What is your vision for the future state of the fairML practices?

• What do you need to change to get to the future state?

• What do you need to retire to get to the future state?

• What can be salvaged/repurposed?

Ending notes

• What is the best about your current set up?

• How would you summarize the largest challenges? Aka what do you like least?

• Is there anything that I should have asked about?

Bibliography

  1@article{mittelstadt2019ai,
  2  year = {2019},
  3  journal = {arXiv preprint arXiv:1906.06668},
  4  author = {Mittelstadt, Brent},
  5  title = {AI Ethics--Too Principled to Fail?},
  6}
  7
  8@book{dawson2019reshaping,
  9  publisher = {Routledge},
 10  year = {2019},
 11  author = {Dawson, Patrick},
 12  title = {Reshaping change: A processual perspective. 2nd edition},
 13}
 14
 15@article{meyerson2004tempered,
 16  year = {2004},
 17  pages = {1--23},
 18  journal = {Stanford Social Innovation Review},
 19  author = {Meyerson, Debra},
 20  title = {The tempered radicals: How employees push their companies--little by little--to be more socially responsible},
 21}
 22
 23@article{krawiec2003cosmetic,
 24  publisher = {HeinOnline},
 25  year = {2003},
 26  pages = {487},
 27  volume = {81},
 28  journal = {Wash. ULQ},
 29  author = {Krawiec, Kimberly D},
 30  title = {Cosmetic compliance and the failure of negotiated governance},
 31}
 32
 33@article{kalev2006best,
 34  publisher = {Sage Publications Sage CA: Los Angeles, CA},
 35  year = {2006},
 36  pages = {589--617},
 37  number = {4},
 38  volume = {71},
 39  journal = {American sociological review},
 40  author = {Kalev, Alexandra and Dobbin, Frank and Kelly, Erin},
 41  title = {Best practices or best guesses? Assessing the efficacy of corporate affirmative action and diversity policies},
 42}
 43
 44@book{scott2015organizations,
 45  publisher = {Routledge},
 46  year = {2015},
 47  author = {Scott, W Richard and Davis, Gerald F},
 48  title = {Organizations and organizing: Rational, natural and open systems perspectives},
 49}
 50
 51@book{barak2016managing,
 52  publisher = {Sage Publications},
 53  year = {2016},
 54  author = {Barak, Michalle E Mor},
 55  title = {Managing diversity: Toward a globally inclusive workplace},
 56}
 57
 58@article{orlikowski1992duality,
 59  publisher = {INFORMS},
 60  year = {1992},
 61  pages = {398--427},
 62  number = {3},
 63  volume = {3},
 64  journal = {Organization science},
 65  author = {Orlikowski, Wanda J},
 66  title = {The duality of technology: Rethinking the concept of technology in organizations},
 67}
 68
 69@article{weaver1999corporate,
 70  publisher = {Springer},
 71  year = {1999},
 72  pages = {283--294},
 73  number = {3},
 74  volume = {18},
 75  journal = {Journal of Business Ethics},
 76  author = {Weaver, Gary R and Trevi{\~n}o, Linda Klebe and Cochran, Philip L},
 77  title = {Corporate ethics practices in the mid-1990's: An empirical study of the Fortune 1000},
 78}
 79
 80@article{trevino1999managing,
 81  publisher = {SAGE Publications Sage CA: Los Angeles, CA},
 82  year = {1999},
 83  pages = {131--151},
 84  number = {2},
 85  volume = {41},
 86  journal = {California management review},
 87  author = {Trevino, Linda Klebe and Weaver, Gary R and Gibson, David G and Toffler, Barbara Ley},
 88  title = {Managing ethics and legal compliance: What works and what hurts},
 89}
 90
 91@inproceedings{kaminski2020multi,
 92  year = {2020},
 93  pages = {68--79},
 94  booktitle = {Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency},
 95  author = {Kaminski, Margot E and Malgieri, Gianclaudio},
 96  title = {Multi-layered explanations from algorithmic impact assessments in the GDPR},
 97}
 98
 99@inproceedings{malgieri2020concept,
100  year = {2020},
101  pages = {154--166},
102  booktitle = {Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency},
103  author = {Malgieri, Gianclaudio},
104  title = {The concept of fairness in the GDPR: a linguistic and contextual interpretation},
105}
106
107@article{metcalfowners,
108  pages = {449--476},
109  issue = {2},
110  volume = {86},
111  journal = {Social Research: An International Quarterly},
112  year = {2019},
113  title = {Owning Ethics: Corporate Logics, Silicon Valley, and the Institutionalization of Ethics},
114  author = {Metcalf, Jacob and Moss, Emanuel and boyd, danah},
115}
116
117@inproceedings{tutorialfat,
118  booktitle = {Conference on Fairness, Accountability, and Transparency, FAT* 2019},
119  title = {Industry Translation Tutorial: Algorithmic fairness in practice},
120  author = {Henriette Cramer and Jenn Wortman-Vaughan and Kenneth Holstein and Hanna Wallach and Hal Daum{\'e} III and Miroslav Dudík and Sravana Reddy and Jean Garcia-Gathright},
121}
122
123@article{jobin2019global,
124  publisher = {Nature Publishing Group},
125  year = {2019},
126  pages = {389--399},
127  number = {9},
128  volume = {1},
129  journal = {Nature Machine Intelligence},
130  author = {Jobin, Anna and Ienca, Marcello and Vayena, Effy},
131  title = {The global landscape of AI ethics guidelines},
132}
133
134@inproceedings{buolamwini2018gender,
135  year = {2018},
136  pages = {77--91},
137  booktitle = {Conference on fairness, accountability and transparency},
138  author = {Buolamwini, Joy and Gebru, Timnit},
139  title = {Gender shades: Intersectional accuracy disparities in commercial gender classification},
140}
141
142@inproceedings{modelcards,
143  bibsource = {dblp computer science bibliography, https://dblp.org},
144  biburl = {https://dblp.org/rec/conf/fat/MitchellWZBVHSR19.bib},
145  timestamp = {Wed, 27 Feb 2019 10:53:44 +0100},
146  doi = {10.1145/3287560.3287596},
147  url = {https://doi.org/10.1145/3287560.3287596},
148  year = {2019},
149  publisher = {{ACM}},
150  pages = {220--229},
151  booktitle = {Proceedings of the Conference on Fairness, Accountability, and Transparency,
152FAT* 2019, Atlanta, GA, USA, January 29-31, 2019},
153  title = {Model Cards for Model Reporting},
154  author = {Margaret Mitchell and
155Simone Wu and
156Andrew Zaldivar and
157Parker Barnes and
158Lucy Vasserman and
159Ben Hutchinson and
160Elena Spitzer and
161Inioluwa Deborah Raji and
162Timnit Gebru},
163}
164
165@article{cramer2018assessing,
166  publisher = {ACM New York, NY, USA},
167  year = {2018},
168  pages = {58--63},
169  number = {6},
170  volume = {25},
171  journal = {Interactions},
172  author = {Cramer, Henriette and Garcia-Gathright, Jean and Springer, Aaron and Reddy, Sravana},
173  title = {Assessing and addressing algorithmic bias in practice},
174}
175
176@book{privacybook,
177  publisher = {MIT Press},
178  year = {2015},
179  title = {Driving Corporate Behavior in the United States and Europe},
180  author = {Kenneth A. Bamberger and Deirdre K. Mulligan},
181}
182
183@article{Orlikowski2000,
184  keywords = {Structuration theory, Work practices, Information technology, Organization},
185  numpages = {25},
186  pages = {404–428},
187  month = {July},
188  journal = {Organization Science},
189  doi = {10.1287/orsc.11.4.404.14600},
190  url = {https://doi.org/10.1287/orsc.11.4.404.14600},
191  issn = {1526-5455},
192  number = {4},
193  volume = {11},
194  address = {Linthicum, MD, USA},
195  publisher = {INFORMS},
196  issue_date = {July 2000},
197  year = {2000},
198  title = {Using Technology and Constituting Structures: A Practice Lens for Studying Technology in Organizations},
199  author = {Orlikowski, Wanda J.},
200}
201
202@inproceedings{selbst2019fairness,
203  year = {2019},
204  pages = {59--68},
205  booktitle = {Proceedings of the Conference on Fairness, Accountability, and Transparency},
206  author = {Selbst, Andrew D and Boyd, Danah and Friedler, Sorelle A and Venkatasubramanian, Suresh and Vertesi, Janet},
207  title = {Fairness and abstraction in sociotechnical systems},
208}
209
210@inproceedings{Mitchell20metrics,
211  series = {AIES ’20},
212  location = {New York, NY, USA},
213  keywords = {machine learning fairness, subset selection, diversity and inclusion},
214  numpages = {7},
215  pages = {117–123},
216  booktitle = {Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society},
217  doi = {10.1145/3375627.3375832},
218  url = {https://doi.org/10.1145/3375627.3375832},
219  address = {New York, NY, USA},
220  publisher = {Association for Computing Machinery},
221  isbn = {9781450371100},
222  year = {2020},
223  title = {Diversity and Inclusion Metrics in Subset Selection},
224  author = {Mitchell, Margaret and Baker, Dylan and Moorosi, Nyalleng and Denton, Emily and Hutchinson, Ben and Hanna, Alex and Gebru, Timnit and Morgenstern, Jamie},
225}
226
227@inproceedings{Krafft20,
228  series = {AIES ’20},
229  location = {New York, NY, USA},
230  keywords = {artificial intelligence, policy, definitions, sociotechnical imaginaries},
231  numpages = {7},
232  pages = {72–78},
233  booktitle = {Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society},
234  doi = {10.1145/3375627.3375835},
235  url = {https://doi.org/10.1145/3375627.3375835},
236  address = {New York, NY, USA},
237  publisher = {Association for Computing Machinery},
238  isbn = {9781450371100},
239  year = {2020},
240  title = {Defining AI in Policy versus Practice},
241  author = {Krafft, P. M. and Young, Meg and Katell, Michael and Huang, Karen and Bugingo, Ghislain},
242}
243
244@inproceedings{holstein2019improving,
245  year = {2019},
246  pages = {1--16},
247  booktitle = {Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems},
248  author = {Holstein, Kenneth and Wortman Vaughan, Jennifer and Daum{\'e} III, Hal and Dudik, Miro and Wallach, Hanna},
249  title = {Improving fairness in machine learning systems: What do industry practitioners need?},
250}
251
252@article{friedman1996bias,
253  publisher = {ACM New York, NY, USA},
254  year = {1996},
255  pages = {330--347},
256  number = {3},
257  volume = {14},
258  journal = {ACM Transactions on Information Systems (TOIS)},
259  author = {Friedman, Batya and Nissenbaum, Helen},
260  title = {Bias in computer systems},
261}
262
263@article{Shneiderman13538,
264  journal = {Proceedings of the National Academy of Sciences},
265  eprint = {https://www.pnas.org/content/113/48/13538.full.pdf},
266  url = {https://www.pnas.org/content/113/48/13538},
267  issn = {0027-8424},
268  publisher = {National Academy of Sciences},
269  doi = {10.1073/pnas.1618211113},
270  year = {2016},
271  pages = {13538--13540},
272  number = {48},
273  volume = {113},
274  title = {Opinion: The dangers of faulty, biased, or malicious algorithms requires independent oversight},
275  author = {Shneiderman, Ben},
276}
277
278@inproceedings{Fazelpour20,
279  series = {AIES ’20},
280  location = {New York, NY, USA},
281  keywords = {justice, algorithmic decision-making, political philosophy, fairness in machine learning},
282  numpages = {7},
283  pages = {57–63},
284  booktitle = {Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society},
285  doi = {10.1145/3375627.3375828},
286  url = {https://doi.org/10.1145/3375627.3375828},
287  address = {New York, NY, USA},
288  publisher = {Association for Computing Machinery},
289  isbn = {9781450371100},
290  year = {2020},
291  title = {Algorithmic Fairness from a Non-Ideal Perspective},
292  author = {Fazelpour, Sina and Lipton, Zachary C.},
293}
294
295@inproceedings{haydn2020,
296  series = {AIES ’20},
297  location = {New York, NY, USA},
298  keywords = {ethics, lethal autonomous weapons, organising, talent supply., bargaining, activism, epistemic community},
299  numpages = {7},
300  pages = {15–21},
301  booktitle = {Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society},
302  doi = {10.1145/3375627.3375814},
303  url = {https://doi.org/10.1145/3375627.3375814},
304  address = {New York, NY, USA},
305  publisher = {Association for Computing Machinery},
306  isbn = {9781450371100},
307  year = {2020},
308  title = {Activism by the AI Community: Analysing Recent Achievements and Future Prospects},
309  author = {Belfield, Haydn},
310}
311
312@inproceedings{madaio20,
313  series = {CHI ’20},
314  year = {2020},
315  author = {Madaio, Michael A and Stark, Luke and Vaughan, Jennifer Wortman and Wallach, Hanna},
316  title = {Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI},
317}
318
319@article{BERKOWITZ2018,
320  author = {Heloise Berkowitz},
321  url = {http://www.sciencedirect.com/science/article/pii/S095965261732958X},
322  doi = {https://doi.org/10.1016/j.jclepro.2017.12.028},
323  issn = {0959-6526},
324  year = {2018},
325  pages = {420 - 430},
326  volume = {175},
327  journal = {Journal of Cleaner Production},
328  title = {Meta-organizing firms' capabilities for sustainable innovation: A conceptual framework},
329}
330
331@misc{brundage2020trustworthy,
332  primaryclass = {cs.CY},
333  archiveprefix = {arXiv},
334  eprint = {2004.07213},
335  year = {2020},
336  author = {Miles Brundage et al},
337  title = {Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims},
338}
339
340@article{berkowitz2017collectively,
341  publisher = {Springer},
342  year = {2017},
343  pages = {753--769},
344  number = {4},
345  volume = {143},
346  journal = {Journal of Business Ethics},
347  author = {Berkowitz, Helo{\"\i}se and Bucheli, Marcelo and Dumez, Herv{\'e}},
348  title = {Collectively designing CSR through meta-organizations: A case study of the oil and gas industry},
349}
350
351@inproceedings{googleAudit2020,
352  series = {FAT* ’20},
353  location = {Barcelona, Spain},
354  keywords = {responsible innovation, accountability, algorithmic audits, machine learning},
355  numpages = {12},
356  pages = {33–44},
357  booktitle = {Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency},
358  doi = {10.1145/3351095.3372873},
359  url = {https://doi.org/10.1145/3351095.3372873},
360  address = {New York, NY, USA},
361  publisher = {Association for Computing Machinery},
362  isbn = {9781450369367},
363  year = {2020},
364  title = {Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing},
365  author = {Raji, Inioluwa Deborah and Smart, Andrew and White, Rebecca N. and Mitchell, Margaret and Gebru, Timnit and Hutchinson, Ben and Smith-Loud, Jamila and Theron, Daniel and Barnes, Parker},
366}
367
368@inproceedings{bietti2020ethics,
369  year = {2020},
370  pages = {210--219},
371  booktitle = {Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency},
372  author = {Bietti, Elettra},
373  title = {From ethics washing to ethics bashing: a view on tech ethics from within moral philosophy},
374}
375
376@misc{zeng2018linking,
377  primaryclass = {cs.AI},
378  archiveprefix = {arXiv},
379  eprint = {1812.04814},
380  year = {2018},
381  author = {Yi Zeng and Enmeng Lu and Cunqing Huangfu},
382  title = {Linking Artificial Intelligence Principles},
383}
384
385@book{holtzblatt1997contextual,
386  publisher = {Elsevier},
387  year = {1997},
388  author = {Holtzblatt, Karen and Beyer, Hugh},
389  title = {Contextual design: defining customer-centered systems},
390}

Attribution

arXiv:2006.12358v4 [cs.CY]
License: cc-by-4.0