Join Our Discord (940+ Members)

Requirements Engineering Framework for Human-Centered Artificial Intelligence Software Systems

Content License: cc-by

Requirements Engineering Framework for Human-centered Artificial Intelligence Software Systems

Papers is Alpha. This content is part of an effort to make research more accessible, and (most likely) has lost some details from the original. You can find the original paper here.

Introduction

 

AI-based software systems are rapidly becoming essential in many organizations. However, the focus on the technical side of building artificial intelligence (AI)-based systems are most common, and many projects, more often than not, fail to address critical human aspects during the development phases. These include but are not limited to age, gender, ethnicity, socio-economic status, education, language, culture, emotions, personality, and many others. Ignoring human-centered aspects in AI-based software tends to produce biased and non-inclusive outcomes. Shneidermanemphasizes the dangers of autonomy-first design in AI and the hidden biases that follow. Misrepresenting human aspects in requirements for model selection and data used in training AI algorithms can lead to discriminatory decision procedures even if the underlying computational processes were unbiased. For example, a study by Carnegie Mellon revealed that women were far less likely to receive high-paying job ads from Google than mendue to the under-representation of people of color and women in high paying IT jobs.

Studies on human-centered design aim to develop systems that put human needs and values at the center of software development and clearly understand the context of the software system’s usage. These human values include security, tradition, achievement, power, etc.. An increasing number of research resources are being invested in developing human-centered AI solutions. Large organizations such as Google, Microsoft, and Apple are moving towards including human-centered values when building AI-based software. These large organizations have now devised explicit guidelines for building human-centered AI systems. While these guidelines provide an excellent platform, they mainly focus on the design phase of software development and not on requirements engineering (RE).

Recent research efforts in RE have focused on including human-centered requirements in building software solutions, such as emotions,gender, power and politics, personality traits, age, and mental and physical challenges. For example, Miller et al.argued that “emotions should be considered as first-class citizens in software engineering methodology” and acknowledged the fact that software engineers usually overlook emotion when building software systems. Other studies have stressed that emotions are usually neglected when it comes to RE. Perera et al.emphasized the importance of including human values early on in RE.

Although this is a growing research area in RE, limited studies focus on including requirements for human-centered AI-based systems. We examined the studies presented in our systematic literature review (SLR)and a mapping studyon RE for AI (RE4AI) and found that most AI-based software lacks human-centered approaches when writing and modeling requirements, and existing research mainly focused on explainability, trust, and ethics with limited empirical evaluations.

We propose a new framework, the RE for human-centered AI (RE4HCAI), to specify and model requirements for human-centered AI-based software. The RE4HCAI framework is based on industrial human-centered guidelines, an analysis of studies obtained from the literature, and the results from an experts’ survey. The survey identified the guidelines and human-centered aspects that should be addressed and prioritized during RE. The framework is then evaluated in a case study, and requirements are elicited and modeled for a system that uses AI to enhance the quality of 360° videos for virtual reality (VR) users. The purpose of the AI-based software was to improve the quality of experience (QoE) and quality of service (QoS) for systems streaming and rendering 360° video contents. We discovered that some requirements could not be identified during the early stages of the project due to the black-box nature of the AI. The difficulty in explaining how it would respond to the available data also contributed to this issue. Therefore, a more iterative RE approach should be used to write requirements for AI-based software. The key contributions of this research can be described as follows:

  • We present a new framework to help elicit and specify requirements for AI-based software. The framework is based on the industry’s human-centered AI development, an SLR, and an expert survey. As part of our framework, we provide a catalog to aid the elicitation of requirements for AI-based software. The framework further provides a conceptual model to present the requirements visually.

  • We apply and evaluate the framework on a case study that uses AI to enhance the spatial quality of 360° VR videos and report on our findings.

The rest of the paper is structured as follows:Section sec-background provides a brief background related work. Section sec-framework presents details of our framework. Section sec-casestudy reports on implementing the framework and modeling language on a software system to enhance 360° videos for VR users. Section sec-discussion discusses key results and summarizes emerging theories. Section sec-threats addresses threats to validity. Section sec-relatedwork discusses related work, and section sec-conclusion concludes.

Background and Motivation

 

As mentioned in section sec-introduction, our framework is aimed at helping the elicitation and specification of requirements for human-centered AI-based software. We conducted an SLRand selected any existing study that focused on human-centered approaches in RE4AI. These papers include studies on emotion for a human-centered social robot, understating expectations and limitations of AI-based software, identifying and mitigating human-centered issues related to fairness and biases, ethics, explainability, and trust.Next, the industrial guidelines on human-centered AI development are analyzed, specifically Google’s PAIR guidebook,Apple’s human interface guidelines for building ML applications, Microsoft’s eighteen guidelines for human-centered AI interactionand the machine learning (ML) canvas. We note that ML canvas does not directly address the human-centered aspect. Nevertheless, we chose to include ML canvas for its relevance to our work as the ML canvas complemented the industrial guidelines by providing means for capturing relevant information for building ML models, e.g., ML business needs, decisions, data sources, and evaluation, and it facilitates collaboration between different stakeholders.

The gathered requirements from the SLR and the industrial guidelines are summarized, mapped, and categorized into six areas to build human-centered AI-based software. The categorization of six areas aligns with the five categories proposed in Google PAIR. The sixth area, i.e., model needs, identifies the human-centered approaches used when selecting and training an appropriate AI-based model. Each area is explained in detail below as follows:

Area#1 Requirements for User Needs

The first area focuses on ensuring that the user needs are captured first when building AI-based software, documenting the limitations and capabilities, what the system can do, how well it can do its assigned tasks, and identifying the users and stakeholders. Other factors include how the user will interact with the system and what approaches should be selected to match the user’s needs. Is the system going to be proactive or reactive? For example, in a proactive system, Google search provides a list of results based on the users’ entered keywords. The next step is to determine if the user is aware of the AI feature. Visible features are when the user is aware of the AI component, such as having auto-complete trying to guess what the user wants to input next. Invisible features are when the user is not aware of the AI feature. For example, a map would gather real-time data on traffic to provide the best route to users.

Next, we need to determine if the system is going to automate or augment the user’s needs. For instance, AI-based software can automate any task that does not require human oversight. Tasks that a user enjoys or requires a human-in-the-loop can be augmented by AI to improve the user’s experience and efficiency.

Finally, the evaluation behind the choice of reward function should be specified. How will the AI choose between right and wrong predictions of a learned model (LM)? An LM, as defined by Berry, “is the result of an instance of ML or deep learning (DL), whether the LM is taught, self-taught, or both with relevant real-world (RW) data”. In other words, an LM is a computational representation that is generated by an ML or DL algorithm. The model is learned by training on relevant RW data, which allows it to make predictions or decisions based on new, unseen data. The learning process typically involves feeding the algorithm a large dataset containing input-output pairs, from which the algorithm can discover patterns, relationships, or underlying structures in the data. During the training process, LM adjusts its internal parameters to minimize the error between its predictions and the actual outcomes. Once the model is trained and achieves a satisfactory level of performance, it can be applied to new, unseen data to make predictions, classify data points, or recommend actions, depending on the specific task it was designed to perform.

In the case of binary classification, a prediction by an LM can result in four categories, i.e., true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). The two reward functions available are precision and recall. Precision is the proportion of correctly picked TP out of the total positive predictions, i.e., precision = TP / (TP + FP). TP refers to the correctly predicted positive instances, while FP refers to instances where the LM predicts an incorrect positive outcome. A flight booking app with high precision would predict the cheapest flights for a specific date but might miss out on some flights. Recall measures the proportion of TP instances the LM can correctly identify. Thus, recall = TP / (TP + FN). Higher recall provides the confidence that the system has included all relevant results but might also include some irrelevant results. Evaluating the reward function will depend on the specific task and domain of the AI application used. The decision to use either would be based on the trade-off between selecting precision and recall. Therefore, when evaluating the reward function, a list of trade-offs should be documented to justify the selected function.

Area#2 Requirements for Model Needs

The second area includes documenting requirements for model needs. How do we choose an algorithm that optimizes stakeholder satisfaction? Do we need a system that is explainable or accurate? For example, Shin et al.look for ways to reduce cost by experimenting with different algorithms and found that some algorithms could produce better outcomes with lower costs. However, these algorithms might lack in other aspects, such as explainability. In contrast, different algorithms might provide a better explanation but with predictions with lower confidence. The choice of algorithm will also depend on the user’s needs for the system. Also, as part of requirement specifications for AI, there is the need to specify the possible settings of each variable for a given AI algorithm when applied to a particular task in a given context.

Part of the requirements specifications should consider how the incoming data affects model training. Thus, deciding how LM will improve on new data should be specified. Does the system need a dynamic LM that updates and trains online or a static system that improves only with updates? A dynamic system improves as the user interacts with the system and often involves using feedback in model training. A static system will improve with system updates and is trained offline. For example, an image recognition LM that depends on data that does not change over time will need updates with new releases, or else the LM might become biased or useless after a while.

Model selection depends on the choice of algorithm and can include supervised, unsupervised, and reinforcement learning. When training the system, we need to specify a threshold to avoid overfitting or underfitting the data and select tools that will be used to evaluate the LM. When tuning the LM, the following should be considered: types of feedback, user behavior, training data used in model tuning, and adjusting the parameters accordingly.

Area#3 Requirements for Data Needs

The third area focuses on data needs and data collection methods. Data collection includes the type and amount of data needed. Once data collection methods are specified, data requirements should be considered, including quality of data used, security, privacy settings, and fairness. Data quality includes five components: accuracy, completeness, consistency, credibility, and currentness. 1) Having correct data will ensure accuracy. 2) Completeness refers to the availability of all attributes and events associated with the data. 3) Consistency has no contradictions in the data used. 4) Ensuring credibility by having truthful data. 5) Finally, currentness means data must be collected within the correct time frame.

Data quantity focuses on data being diverse. And data quality addresses the completeness, consistency, and correctness of data being used. Selecting data should include identifying features, labels, and sampling rates. Labels identify the features needed to train the ML model, such as labeling a scanned image of a tumor as malignant. Explicit labeling is done manually, and implicit labeling is when the model learns the pattern independently. Examples represent a row of data and contain features, and labels represent descriptions given to data. More samples in the dataset ensure diversity but also increase costs. In this case, a threshold should be identified to set the amount of data needed within the given budget.

Identifying and reporting biases in data must be addressed. Such biases can include automation, selection, group attribution, etc. Automation biases are when preferences are selected based on automated suggestions from the system. Selection bias usually happens when data is not collected randomly from the target population but rather selected based on the stakeholder’s requests. Group attribution assumes that an output suitable for an individual will have the same impact on everyone in the group. Identifying key data characteristics should be set early to avoid discrimination and biases.

Area#4 Requirements for Feedback and User Control

This area deals with identifying and finding which kinds of user feedback need to be established in RE when building AI-based systems. Different types of feedback include implicit, explicit, and calibration. Implicit feedback provides information about the user’s interaction, preferences, and system behavior. Examples of implicit feedback include accepting or rejecting a recommendation, times of use, number of hours, when or how many times the user logged on, etc. The user provides explicit feedback when requested by the system, including surveys, forms, ratings, written feedback, likes, or dislikes. Calibration is the initial information the system might need from the user to function, such as scanning your fingerprint for the first time to activate touch ID.

It is essential to identify how and when the feedback will be used in model tuning and what changes it will have on the AI-based system. When asking for feedback, privacy measures should be considered to secure it. Also, give the user a choice to dismiss the feedback. Allowing users to feel that they control the system is essential to human-centered AI and can depend on many factors. Providing the users with control can be achieved by giving them complete control of the system in case of a failure or providing them with multiple options from which they can make the appropriate choice.

Area#5 Requirements for Explainability and Trust

AI-based systems reasoning can include explaining the results or predictions to the end user and stakeholders. From the stakeholders’ point of view, explainability requirements should consist of setting expectations of what the system can do and how it can do it and explaining its limitations and capabilities. On the other hand, explaining to the user will involve the data used and informing them about changes to the system that might happen with updates or LM improvements. Explaining predictions can be presented by either providing an example or displaying confidence. Confidence could be displayed in many ways depending on the situation and the AI-based system. Each situation should be evaluated to find an appropriate method to show confidence.

Explainable requirements can often conflict with others, such as performance and cost. It might be cheaper to build systems that are not very explainable with better performance. Therefore, it is vital to calculate the trade-off when favoring an explainable system and identify how it might conflict with other requirements. Schoonderwoerd et al.,indicate that explanations should be provided based on a specific context, with the need to identify which explanations should be provided and when. However, explainability might be necessary for some situations to ensure compliance with the European Union’s General Data Protection Regulation (GDPR). In this case, other measures need to be considered.

Providing realistic expectations helps users and stakeholders avoid over-trusting AI-based software. Explainability requirements might occur during the building process or after the LM is deployed. Explainability requirements can be divided into four components to include: Who the explanation is addressed to, what needs to be explained, when should the explanation happen, and who explains. Also, explanations should consist of consequences that might occur due to an action performed by the user. Showing the confidence can be a way to provide an explanation to users. However, it is essential to determine when and how to display predictions, as sometimes showing confidence could lead to mistrust.

Area#6 Requirements for Errors and Failure

None of the studies in the RE4AI literature mention the need to address errors in RE when building AI-based systems. The focus of most industrial guidelines is on designing AI-based software with a human-centered approach and does not focus on RE. Our survey resultsshow that practitioners working on AI-based systems wanted to know how to deal with errors and specify error sources during RE. The different types of errors include background, context, and system limitations. Some errors, such as background and context, are more challenging to identify and are invisible to the end user. Context errors happen for several reasons and can be avoided by ensuring the user understands how the system works and ensuring the system is aligned with their needs. They are usually an outcome that is a true positive yet does not provide a prediction that aligns with the user’s needs.

Error sources include system, incorrect predictions, data, and input and output errors. Data errors happen with mislabeled data and can be due to poor training or inaccurate labeling. Prediction errors can occur when an incorrect model is used, the data is not comprehensive, or missing some critical elements. Input errors happen when users input unexpected data and can be due to users’ old habits or having an abusive user. Output errors are when the system provides a prediction that is low in confidence or an irrelevant output but high in confidence. Moreover, finally, system errors would happen when multiple systems using AIintegrate or depend on each other. We need to identify error types and sources in RE and mitigate them by providing an action plan for how they should be addressed and fixed.

A Framework to Manage RE for Human-centered AI-based systems (RE4HCAI)

 

This section presents the RE4HCAI framework, illustrated in figure fig-re4hciframework, for eliciting and modeling requirements for human-centered AI-based systems. The framework consists of three layers. The first layer in section sec-referencemap presents the human-centered guidelines that should be included in RE. The guidelines are combined and mapped into a reference model. The second layer presents a catalog (provided in the appendix) to help elicit requirements for building human-centered AI-based software and is discussed in section sec-catalog. The last layer proposes a modeling language to present the requirements visually, as explained in section sec-modelinglanguage.

Our proposed framework to elicit and model requirements for human-centered AI.fig-re4hciframework

Our proposed framework to elicit and model requirements for human-centered AI.

Human-centered AI Guidelines for RE

 

< g r a p h i c s >

Human-centered needs for AI and the frequency of survey participants who selected them.

The coverage of these six areas in RE was investigated by conducting a user survey with 29 practitioners and researchers working on AI-related projects. The participants included data scientists, ML specialists, and software engineers working with AI-based software. We asked each participant to identify which of the mapped human-centered guidelines are or should be, included in RE based on their experience. The survey results, shown in figure fig-needs, confirmed that all six human-centered areas need to be specified in RE4HCAI. The survey participants responded more favorably towards including the first three areas of user, model, and data needs than the remaining three areas for RE4HCAI. Thus, our evaluation focuses on these three areas in section sec-casestudy.

The six areas of the human-centered AI components that are needed to be considered in RE are mapped to present a reference model that showcases the overall layout of the framework, as shown in figure fig-framework. Each area presented in the reference model is explained further in section sec-background.

Diagram showcasing the reference model for the RE4HCAI framework.fig-framework

Diagram showcasing the reference model for the RE4HCAI framework.

Catalog for Collecting Requirements

 

This section presents a catalog of human-centered AI requirements along the six areas in our framework. The gathered human-centered requirements and the mapped requirements from the SLR are listed in a tabular format to form our catalog. The catalog can be used as a checklist to elicit requirements for AI-based software and has six sections. Each section is dedicated to eliciting detailed requirements for each of the six areas explained in the background section sec-background. The catalog is provided in the appendix.

Modeling Requirements for Human-centered AI

 

The different roles of participants in the user survey.fig-roles

The different roles of participants in the user survey.

First level of the RE4HCAI framework showing the six different areas.fig-metamodel

First level of the RE4HCAI framework showing the six different areas.

In our SLR, we sought any existing modeling language or requirements notations used in RE4AI. UML was the most popular method, making it easier for non-software engineers to work with and learn. However, UML has limitations as it does not support modeling non-functional requirements (NFR) and business rules. Goal-oriented requirements engineering (GORE) had better support for NFR and business rules but was more challenging to learn and was mainly used by requirements engineers and software engineers. Also, GORE could model requirements at lower levels of abstraction than UML. Silva et al.explained that using GORE to model a requirement or concept could be presented with fewer structural diagrams than UML. However, UML is more widely known and used than GORE. Also, GORE was reported to be more challenging to learn among non-software engineers.

Looking at the different roles from the survey, we found that around 25% of the people involved in building AI-based software were software engineers and requirements engineers. Data scientists and ML specialists contributed around 27% of the team building AI-based software. Other roles included system and business analysts, developers, and researchers, as shown in figure fig-roles. Due to the diverse nature of team structure in building AI-based software and the need to use more specific modeling concepts to reflect on the process provided in the framework, we decided to create a conceptual model for the RE4HCAI framework, as conceptual modeling allows for precise concepts to be modeled as well as presenting a holistic view of the application.

Legend

Table Label: table-legend

Download PDF to view table

Our conceptual model consists of two levels. The first level presented in figure fig-metamodel provides a holistic view of the system requirements. We use an oval shape notation to list the main goal that needs to be addressed. For example, in the case study we conduct in section sec-casestudy, the goal would be enhancing the quality of 360°videos by four times. This goal would connect to the six sub-models to include the six areas of our framework, areas#1-6 covered in section sec-background. For each area, we use a UML class diagram to present it. Each area contains the main attributes in the reference model in figure fig-framework. The second level consists of a sub-model to show each area in further detail. We provide the notations to model these requirements in table table-legend. When building the visual notations for our language, we attempt to comply with the nine principles of notationsto reduce the complexity and cognitive load for visual notations in SE modeling. We present our notations using different shapes, textures, icons, and colors.

Goals are presented using an oval shape; each need includes the components modeled in the sub-models. Since our framework focuses mostly on needs, we use a square with dash lines to visually emphasize the concept of needs. Elongated hexagons are used to present processes or tasks. Trade-offs are displayed using an equilateral octagon. Limitations and capabilities are modeled using rectangles to show what the AI component can and cannot do, with limitations shaded in yellow to differentiate between them visually. Icons were used to present some notations.The folder icon is used for attributes, the stick icon for users, and the data icon for data sources.Moreover, finally, the components are joined using connectors to either show the flow of data or evaluate the data.

When presenting the models, we use blue tick icons on notations to represent the parts that belong to the framework. The notations that do not include a blue tick icon are system specific, as shown in figure fig-sysvsarch. System requirements will change when modeling a different AI-based system.

The blue icon used with modeling notations shows the difference between architectural and system requirements.fig-sysvsarch

The blue icon used with modeling notations shows the difference between architectural and system requirements.

Case Study Application and Evaluation

 

This section describes the process we used to design, select, and conduct the case study to implement our proposed framework. The case study answers the following research questions (RQs):

  • How does our RE4HCAI framework benefit the process of eliciting and modeling requirements for human-centered AI software? RQ1 aims to assess the usefulness of our framework in eliciting and modeling requirements. It further contrasts the RE process before and after implementing our framework in the case study.

  • How can the RE practices be aligned in the life cycle of AI projects? RQ2 aims to provide an understanding and alignment of the RE practices in AI-based software development projects.

The case study follows the guidelines for conducting case study researchand empirical evaluations in software engineering. The main reason for conducting this study was to find how to extract requirements for building human-centered AI-based software and if implementing the framework benefited the process. The catalog is used to elicit requirements for the AI system, to identify user needs, model needs, and data needs as presented in tables table-userneedstable-modelneeds, and table-dataneeds. These requirements are then modeled, as shown in figures fig-userneeds, fig-modelneeds, and fig-dataneeds.

Study selection

When selecting the case studies, we had the following selection criteria in mind:

  • Software project with a significant AI component: Since our framework focuses on RE4AI, we required a project with an AI component to answer our RQs, and, ideally, a project that involved interaction with users.

  • Project maturity: The project should be at a reasonably mature stage of development but ideally still under development. Our previous experience motivated this criterion, as we found that experts tend to miss out on details in already completed projects. Furthermore, to address RQ2, we required a project in which the requirements had been established using traditional methods.

  • Availability and background of the expert: We needed to select a project with experts that had the requisite knowledge of the AI component of the system so we could elicit and model the requirements for model and data needs. Also, the case study required substantial time commitment from the expert(s), and hence we sought projects in which at least one expert agreed to the time commitment.

Our case study on VR-360° video enhancer matched the abovementioned criteria. The VR project builds on a DL model to enhance the quality of 360° videos for VR platforms. Hence, it has a major AI component that satisfies the first criteria. Also, the project meets the second selection criteria as it was at the later stages of development. The project’s requirements were finalized as a software feature plan, and most of the features were implemented when we conducted the case study. Furthermore, we had access to the primary ML expert of the project, who developed the AI component and agreed on the time commitment, which satisfies the third selection criteria.

Data Collection

We collected data over four sessions with the ML expert. Of these four sessions, three were dedicated to extracting and modeling the requirements for the project, and a fourth session evaluated the modeled human-centered requirements. The ML expert who built the AI project, and ended up being the fourth author, had no prior background knowledge of how our framework or the modeling language worked. They were given a brief overview of our framework in the first session. During the fourth session, we identified which requirements could be established at the start of the project. In other words, these were the requirements that the ML expert found important for them to know at the start of the project’s lifecycle. However, some of these requirements were not necessarily known originally by the experts and were explored through our framework.

We note that we could elicit and model the requirements only for the first three areas of the framework collaboratively. This was the case due to the limited availability of the ML expert and the fact that the elicitation and modeling for each area required significant time commitment. While we ideally wanted to cover all six areas, we had a choice of covering a subset thoroughly or covering all areas superficially. We made the former choice, which aligns with our survey findings that practitioners’ first three areas are the most focused areas when eliciting requirements.

In the first session, the ML expert gave an overview of the project, and the expert was given a brief overview of the motivation of the study. The expert was further given a brief overview of the framework. The first session was dedicated to collaboratively eliciting and then modeling the first area for user needs. The first session was the longest and lasted $\approx$ 4 hours and was divided into two periods. In the first half, the requirements were elicited, and the second half consisted of modeling the requirements. The second session was dedicated to the second area model needs. The session took $\approx$ 2.5 hours and was done over two periods. Similar to the first session, the requirements were elicited in the first half, followed by modeling the requirements in the second half. In line with the first two sessions, the third session was dedicated to eliciting and modeling the third area data needs. The session took $\approx$ 3 hours over two periods. The fourth session took two hours, and we evaluated all three models. We instructed the ML expert that they were free to suggest any changes to the elicited and modeled requirements from the three sessions. Also, the ML expert reflected on the comparison between the RE process followed originally in the project and the requirements collected from the sessions after applying our framework. The holistic view of the system is presented in figure fig-level1.

Model presenting the holistic view of the requirements elicited for enhancing the quality of 360°video.fig-level1

Model presenting the holistic view of the requirements elicited for enhancing the quality of 360°video.

Improving the Quality of 360° Videos

The case study was conducted on a project that uses DL to enhance the quality of 360° videos for VR platforms. Current approaches to building 360° videos can capture and facilitate the immersive and interactive viewing experience. However, the quality of the final product can degrade due to the limitations of consumer-grade hardware, bandwidth streaming constraints, and processing that requires stitching videos from multiple cameras. The proposed solution was to enhance the quality of the final product using AI-based software.

Requirements for User Needs

The first session was held to elicit requirements for identifying the user’s needs. The session started by identifying the need for the proposed AI-based software using the catalog in the appendix. The need for the system was determined by the quality of the existing 360° videos. Most 360° videos had low resolution and contained artifacts and noise, thus affecting the quality of the final product. The process of removing unwanted noise and artifacts was not achievable without the AI component, therefore, establishing the need for an AI-based software.

Identified user needs requirements for the 360video enhancer.* requirements that could be specified before testing the data on the selected model

Table Label: table-userneeds

Download PDF to view table
Model presenting the requirements for User Needs.fig-userneeds

Model presenting the requirements for User Needs.

To identify the system’s capabilities, we needed to understand what the expected system could and could not do. System capabilities covered both the end-user and stakeholders. From the user’s perspective, the end product would improve the resolution four times while using the same hardware equipment. Therefore, improving the QoE in either entertainment or educational settings. The limitations affected both the end-user and the stakeholders. Listing these limitations to avoid high expectations from either side was important. The limitations for the end-user were that users had to own VR equipment capable of rendering high-quality 360° videos. The second limitation to the user was that they could watch these videos only offline. Therefore, online streaming would not be possible. Furthermore, the limitation for the stakeholder was towards building the system, which included hardware resources and time needed to process the videos. The stakeholders needed to establish ways to improve quality without increasing expenses, which will later link to model selection and collected data.

When modeling limitations and challenges, we found that using color helped distinguish between them visually, as shown in figure fig-userneeds. We modeled the human-centered aspects as goals, and each goal would be mapped to either a need, process, capability, or limitation. For example, the system needed to be automated because it was not possible for a human to do the system’s task. Processes involved actions that needed to be implemented, such as building a reactive system that doesn’t require user interaction. Also, having different textures for the needs made it easier to identify them visually. The advantage of modeling the first area was that it was easy to recognize limitations, capabilities, and needs when building human-centered AI-based software.

Requirements for Model Needs

The second session involved collecting requirements for model needs from our catalog, which helped us understand the rationale behind selecting the type of algorithm used to train the LM and other aspects considered during optimization. The LM was built using deep learning techniques and as a regression problem and optimized for accuracy in order to produce better-quality frames. Model tuning included adjusting the LM to user feedback, user behavior, and output errors. For this particular case, user feedback would be done with a ranking scale that is given after the user watches the enhanced video. The other form of feedback would be to perform a user study to evaluate the quality of the generated outcome from the model. The study would involve human subjects ranking the quality on a scale of 1 to 100. This feedback will indicate if the human-centered perceptual quality of the predicted outcome is acceptable. If not, the plan would be to change the model training and optimize it to improvise perceptual quality. We note that we modeled as limitations the cases where user or system goals could not be achieved, e.g., tracing LM output errors back to data.

Identified model needs requirements for the 360 video enhancer.* Requirements that could be specified before testing the data on the selected model

Table Label: table-modelneeds

Download PDF to view table
Model presenting the requirements for Model Needs.fig-modelneeds

Model presenting the requirements for Model Needs.

Trade-offs that needed to be calculated and modeled were connected to scaling the LM. Scaling was achieved by changing the width or depth in the deep learning model. Increasing the depth meant adding more convolution layers and allowing the model to learn more complex patterns. The depth would have to increase if the performance were not good enough. However, the trade-off to increasing the depth is that more computational resources are required, resulting in delays in the output generation. On the contrary, changing the width would require that features extracted from the input become wider. This would require more memory allocation for data, as it would store more features for the given input at multiple layers. For the model used, an experiment would be set to identify when to modify width or depth and at what intervals while evaluating the trade-offs for time vs. costs to achieve the most effective results. Table table-modelneeds shows the collected requirements for model needs.

Identified data needs requirements for the 360 video enhancer.* Requirements that could be specified before testing the data on the selected model

Table Label: table-dataneeds

Download PDF to view table

Requirements for Data Needs

 

Model presenting the requirements for Data Needs.fig-dataneeds

Model presenting the requirements for Data Needs.

The third session involved extracting requirements for data needs. While conducting this session, it was found that the data collected originally was unsuitable for the model’s training. Therefore, the data had to be cleaned and adjusted to become accurate, correct, consistent, and current. Sudden changes in scene, motion, or light and removing rolling and credit information were removed to make the data accurate. Next, to ensure data completeness, a list of all attributes and events needed for model training was listed, and the expert ensured each attribute had data to represent. Completeness was established technically to show the diversity of the presented data to ensure that the dataset had diversity in terms of 1) motion,2) content, which included objects and scenes, and 3) light. The initially collected data was not consistent in quality. Thus, videos were processed to the same length and resolution to ensure consistency. Finally, currentness was provided by creating video shots representing a unique single scene, thus ensuring that each clip/shot had current data for a continuous period.

Data requirements presented a higher number of needs. The need to specify how the data will be selected and collected for use in model training and testing. For data selection, the needs consisted of setting features, labels, examples, and sampling rates. Features include, among others, color, size, and location. An essential aspect of data needs was to identify any biases that might happen due to the data used. The proposed system was designed for videos with insignificant motion, content, and light changes across time. However, the model might perform differently if the videos had larger motion or luminance changes. Moreover, the diversity of the scenes and objects found was limited to what was available in the open-sourced platforms creating the possibility of the same scene/object represented across multiple videos. Thus, more diverse scenes would be needed to avoid under or over-representing data.

Data collection involved gathering diverse content, including different objects such as trees, buildings, humans, and animals, and various motions such as how objects move across the video and lighting situations such as day, night, bright light, and dim. These measures were identified before data collection and obtained from literature based on the International Telecommunication Union’s standard guidelines for spatio-temporal complexity measures. Although measures were taken into account, some constraints were found after training the model with the collected dataset. The constraints were related to image quality and size due to hardware and model memory-related limitations. These constraints were modeled as limitations to show that data was limited to the specified size and frame rate.

Discussion

 

This section reflects on how applying our framework helped write and model requirements for AI-based systems.

RQ1. How does our RE4HCAI framework benefit the process of eliciting and modeling requirements for human-centered AI software?

Most traditional RE methods are not equipped to manage AI-based software, and requirements are sometimes difficult to write, especially during the early stages of systems development. The problem with specifying requirements for an AI-based software project is that it is difficult to explain the black-box nature of AI and how the system would work or what outcome it will predict. In our case study, we found that the process of eliciting and specifying requirements for AI-based software is vastly different from traditional approaches, as some aspects of the system are unpredictable and difficult to explain.

To evaluate the framework and modeling tool, we compared the building process of the 360° VR model before and after the implementation of our framework. Before implementing the framework, many human-centered aspects were missing from the final product. Most of the development was based on the technical aspects of building the AI-based system. For example, classifying what data was needed was based on technical aspects (e.g., model and hardware needs) rather than focusing on the human perspective. After applying the framework, we identified possible biases that might lead to issues, e.g., biases related to the lack of diversity of data sources and the biases related to the lack of diversity of the type of videos.

The catalog helped elicit most of the requirements to build the AI-based software project from a human-centered perspective. The models presented in figures fig-userneeds, fig-modelneeds, and fig-dataneeds helped the ML expert visualize what needed to be considered when implementing the AI-based software. The conceptual models for our RE4HCAI framework provided a visual view of all the components in the system. Our participant said “It helped me visualize all the components needed to be done in my project”. Also, we found that limitations and system needs were easier to spot with the visual representation via the conceptual model. Another observation we found missing was using feedback in the LM. The initial configuration of the LM did not consider the users’ view of how good the perceptual quality of the improved 360° videos was. However, after applying the framework, the decision was made to include user feedback to account for model tuning in future work of the 360° VR project.

We also noticed that some framework aspects would change depending on the application domain and the AI type used. For example, in this case study, regression was used, and the optimized reward function would be the loss function. Therefore, we need to consider other requirements that might need to be elicited and specified in our framework. This relates to Berry’s findingthat different AI algorithms and their individual settings in different application contexts would require different requirements specifications. However, if the system were designed by applying a classification algorithm and we had to choose between precision or recall, one would need to calculate the trade-offs. When calculating the trade-off between recall vs. precision, we need to list the implications of having an FP vs. FN. In this case, if precision is favored, i.e., we would opt to reduce the number of FPs at the cost of tolerating more FNs, then the 360° VR system will use the correctly selected TP frames but remove some frames from the outcome FN. Thus, the final outcome will have visual gaps, as some relevant frames will be missing. On the other hand, favoring recall meant that most frames would be included, i.e., more FPs. Thus, the final output will also include frames that are not enhanced FP, compromising the quality of experience as the visual quality will not always be the same in the final product, and some of the frames will still have the same quality as the original video.

The downside to favoring precision would be the possibility of having lags when rendering frames, as the system might choose to exclude frames that should have been enhanced from the frame sequence. These lags would increase the chance of latency which is a potential cause of VR sickness. Decreasing this delay or latency would result in reduced sickness. Recall would still include these frames when rendering, even if these frames might not have the expected quality. In this case, the trade-off is to compromise the visual quality instead of causing lags or delays in the visual display.

RQ2. How can the RE practices be aligned in the life cycle of AI projects?

When applying the framework to the case study, not all requirements could be elicited at the start of the project. We found that it was important to identify some requirements at the start. For example, when identifying user needs, we found that it was necessary to have initial specifications before building AI-based software. Also, some of the capabilities could be identified only after getting initial results from the model training, such as how much it would improve the quality of experience and if it would affect the quality of service. Furthermore, limitations such as hardware resources and processing time could not be identified at the start of the project. The extent of the limitation was found after training the model with the existing dataset. The reason was that the ML expert could not know how much scaling the model needed. Adding more features and layers resulted in higher processing time and required more resources to accommodate the change. They could establish how many features or layers were needed to improve model performance only after several training iterations.

Similar patterns were found in both requirements for model needs and data needs. We observed that these requirements would change over time as they learned how the model would interact with the available data. For example, information regarding the needed frames could be specified only after initial testing. The datasets were limited to using up to 20 frames per clip, which was unknown at the beginning of the project. However, after initial testing, it was found that this was a constraint due to the model and hardware limitations. The same applied to identifying the quantity and diversity of the data. After experimenting on the first round of training and testing, they found that the existing 360 dataset was inadequate. Thus, the dataset had to be modified accordingly.

We established that it was important to specify requirements for data needs, such as data sources, charges, diversity of collected data, data quality, if data was up to date, and other requirements involving cleaning the data to provide accurate and current results. Some of these data requirements were obtained from literature and standard guidelines. Therefore, not all requirements could be specified at the start of the project, and some might change or appear while model testing is in progress. We highlight the requirements that we could identify prior to starting the project in tables table-userneeds, table-modelneeds, and table-dataneeds by adding a * character before the requirement.

Threats to Validity

 

In all the phases of our research method and case study design, we attempted to mitigate and reduce any threat to validity as follows :

Internal Validity:

Case studies are easier to perform than experiments., but more challenging to interpret. The main disadvantage to conducting case studies is that the outcomes are more susceptible to researcher bias, and evaluation usually depends on how the researchers interpret the results. To reduce potential threats and biases, we ensured when selecting the case study that the person leading the AI project had no prior knowledge of our framework. Also, we collected requirements based on our framework and compared the results to how it was built without applying the proposed framework.

Construct Validity:

Another threat was the selection of studies to build the catalog. The risks of building a catalog for requirements are that it might miss out on some requirements and not be comprehensive. We established our requirements for the framework based on the guidelines and the literature presented on human-centered AI. Although we might have missed out on some requirements, we did base our findings on our SLR, which covered a comprehensive list of studies on RE4AI research. Also, we used industrial guidelines such as Google and Microsoft, which have already done extensive research to provide their guidelines.

External Validity:

The framework has been applied to only one case study, which threatens external validity. We will address this threat by conducting more case studies in future work and investigating how the framework applies to projects from different application domains and multiple stages in the software development lifecycle. Also, due to time limitations, we could only assess the first three areas of the framework and plan to conduct a further evaluation to assess the last three areas in future work.

 

AI-based software should be carefully assessed not to replace people’s abilities but augment their capabilities and allow people to make the final choice. Shneidermanexplained the importance of enabling humans to control the AI-based system. Human needs and values in building AI-based software need to be researched and examined carefully. For example, AI bots such as Alexia and Siri require extensive training to obtain the right personality, and some companies are investing in creating algorithms for chatbots that can detect sarcasm or respond with empathy.

Recent studies have shown that many AI-based systems lack requirements specifications, which is mainly due to the difference in the building process between traditional systems vs. AI-based software. We conducted an SLRto identify literature on RE4AI. From the results, we identified existing frameworks in RE4AI to include. Nalchigar et al.offers a framework to manage RE in building ML systems. The presented GR4ML framework covered three views: the business view, the analytic design view, and the data preparation view. A conceptual modeling language was used to present each view visually. The framework was applied to a case study that investigated the use of ML in the medical domain.

Most AI-based systems are guided by large amounts of data and limited given resources. Bosch et al.proposed the Holistic DevOps framework for building AI-based software. The framework combined three practices: requirements-driven, data-driven, and AI-driven software systems. Requirements-driven approaches are used for systems that don’t require frequent changes. This method was found to use fewer resources when testing the system. On the other hand, data-driven approaches are used for systems that are constantly changing and updatedand are designed based on the analysis of available data. Companies that usually use automation, such as speech and image recognition, tend to use AI-driven approaches.

Aydemir and Dalpiazproposed a framework to aid requirements engineers and stakeholders in analyzing ethical requirements. The framework assisted in extracting, managing, and evaluating ethical requirements. However, the focus was on ethical requirements only and did not include other aspects of human-centered AI requirements. The last frameworkprovides a virtual framework to test specified requirements for a self-driving car. Also, frameworks are built to manage aspects of AI-bases systems, such as Khalajzadeh et al.who created a domain-specific modeling tool to support data analytic solutions for ML systems (BiDaML). The toolset provides five diagrams to support a different aspect of big data analytics.

Shneidermanpresented a framework that proposes 15 recommendations for engineering trustworthy, reliable, and safe human-centered AI-based systems. To ensure reliable human-centered AI, software engineering teams need to apply technically sound practices such as using appropriate analysis tools, updating workflows for each task and domain, applying new forms of validation and verification, testing for bias detection, and providing explainable user interfaces. For reliable human-centered AI, there should be more commitment and training to ensure safety measures, reporting, and addressing errors and failures. And lastly, promoting trustworthy systems by providing independent oversight reviews to support legal and ethical codes of conduct. Another method used to aid in creating awareness of ethics when building AI software is the ECCOLAmethod, which is based on the ethical AI guidelines and involves using a deck of 21 cards, with each card targeting an aspect of ethics.

Although some frameworks are proposed to provide methods for managing RE4AI, none have focused on delivering an overall solution to building human-centered AI in RE. An exception is Shneiderman’s frameworkthat relies on promoting human-centered AI-based software. However, it focuses on the entire Software Engineering process, not RE. This calls for researching and studying appropriate human-centered AI methods before including them in software systems. We argue that the proposed solutions in RE should address human needs, as having a very well-engineered product would not be useful if it does not satisfy the user’s needs.

Conclusion & Future Work

 

This paper offers a framework to extract and model requirements for human-centered AI. The proposed framework consists of three layers. The first layer provides a reference map to six areas from the human-centered AI guidelines. The second layer presents a catalog to elicit these requirements. The third layer provides a modeling tool to show the second layer’s elicited requirements visually. The framework was applied to a case study, and we extracted requirements for the first three layers of user, model, and data needs. The case included specifying and modeling requirements for an AI system that enhanced the quality of 360° VR videos. We found that some requirements were more difficult to specify initially and could be identified only after testing the model with the existing data.

We plan further to evaluate our framework in the future in several workshops and compare it with existing frameworks and modeling platforms. Also, we do not provide details of how attributes are connected in sub-level models. For example, in model needs, when modeling the goal for balancing overfitting and underfitting, we did not model the actual process. This would have to be presented in a separate sub-model. We plan to extend our model to present this level of detail in future work.

Appendix

1.2 p3cm p6cm p6cm

Catalog presenting requirements needed for Human-centered AI

3lc]@l@User Needs
29emCan AI add value or solve the problem to user’s needs? Do you have the expertise to manage AI? (Data scientist, ML specialist, SE) IF no – use a non-AI solution

Is data available?     (format – amount – diverse) 

3*Identify need for AI? Who are the users

Why do we need the system    

What is the system used for    

49emWhat are the system capabilities?
2*Limitations Limitations to the user:

     Limitations to stakeholders: 

 Capabilities (What can the system do) 
 How well can the system do what it does?

Interaction with user action Proactive vs. Reactive (Provides results when people request them) or Reactive (Interacts with the user without requesting)

Is the user aware of AI feature? Visible or Invisible features

Evaluate approach Augmentation vs Automation (user enjoyment, control, high stakes) vs (boring, dangerous, cannot be performed by humans)

89emReward function or evaluation matrix Impact of having a false positive

 Impact of having a false negative    

 Trade off - e.g precision vs recall    Precision (excludes relevant results, removes all false positives, and misses some of the true positive predictions) vs recall (includes irrelevant results as it includes all true positive but might also include some false positive.) 

 Choice of reward function     e.g precision or recall 

 Impact on user    

 List potential pitfalls    

 How do you provide inclusion     

 Predictions    (How are predictions used in making decisions)

3*Predictions How are predictions used in making decisions

 Impact of new input on predictions    

 Impact of feedback on predictions    

3lc]@l@Model Needs
Optimize algorithm for? Optimize for accuracy, explainability, robust, etc.
Choose the ML type Supervised , Unsupervised, Reinforcement learning
Model training Static vs Dynamic (Offline evaluation / Training improves with updates) vs (Learns from user behavior, Improves from feedback) 2lc]@l@Balance between overfitting & underfitting
510emModel Tuning (Include parameter tuning and architecture changes) User feedback

 Adjusting to user behavior     

 Trace output errors to data     

Parameter tuning    

Quantitative feedback    

2lc]@l@Specify scalability issues
2lc]@l@Choose tools to use to evaluate the model
2lc]@l@ Evaluate the quality of the model

3lc]@l@Data Needs

4*Data selection Features (a characteristics of an input variable.E.g. Feature would include color, size, location, etc)

 Labels     descriptions given to data (Explicit -manually vs Implicit – model learns) 

 Example     (a row of data and contains features and labels)

 Sampling rate     amount of data in a given dataset 

39emData collection (Match data to user needs) Identify type of data needed

 Amount of data needed    

 Diversity of data     

Constraints cost, accuracy, quality, time?
3*Data source Type of data source (public, private, mixed)

 Is data responsibly sourced?    

 Using feedback as data    

 Charges in obtaining data     

 Measures taken to ensure data is up to date?    

2*Split Data Training data (Model tuning)

 Testing data     

5*Data quality Accuracy (The correctness of the data collected)

 Completeness     (all attributes and events should have data to associate them).

-List all attributes and events needed for the AI product -Make sure each attributes has data to represent

 Consistency     (Collected data has to be free from contradictions)

 Credibility     (Credible data had to be authentic and truthful) 

 Currentness    (corresponded to having data collected within the correct time frame; for example, when collecting images of people for a facial recognition application, a picture of an adult person when they were a baby will not be the correct data) 

2*Data requirements Protect personal information

 Does data comply with privacy and law    

4*Fairness Identify biases

 Missing features    

 Under or over representative data     

Other     

5*Reporting biases Automation bias Automation biases are when preferences are selected based on automated suggestions from the system

 Selection bias     Selection bias usually happens when data is not collected randomly from the target population but rather selected based on the stakeholder's requests 

 Impact bias     Making an assumption based on the persons judgment (e.g. all wedding dresses are white)
 Group attribution     Group attribution assumes that an output suitable for an individual will have the same impact on everyone in the group 

Other     

Address biases What methods will be used?

3lc]@l@Feedback & Control
69emExplicit feedback (Only ask when needed) What reward will be provided to the end-user or raters to provide explicit feedback (Material / symbolic)

 If the reward is symbolic     What impact will it have on the user? What benefit will the user gain from it? 

 How is it going to be reviewed and evaluated?    

 What changes will it make to the AI model?     

 How will it be used in model tuning?     

 How is feedback going to be conducted     (surveys, notifications, ratings, etc.) 

3*Implicit feedback User interactions / behaviours to use: Frequency of use, time of use, amount of time spent interacting with the system, accept / reject recommendations, etc.

Review feedback from user interaction to how it will make changes to AI    

 How will it be used in model tuning     

3*Calibration Reason for using calibration

When should the user use calibration    

 How is it going to affect the AI    

49emWhen asking for feedback Measures taken to secure user privacy

Provide multiple options to feedback     (what options will you provide?) 

Is feedback in line with user mental map or users understanding of the system

 Allow the user to dismiss feedback or opt-out     (How are you going to provide this option?) 

4*User control When to give the user control (High stakes, legal, safety )

Allow user to adjust preferences    

Allow user dismissal     

 Level of control     (high, medium, low or no control) 

3lc]@l@Explainability & Trust
2*Explain to stakeholders Limitations of AI

Capabilities of AI    

2*Explain to user Inform user when changes happen

Explain consequences to users action    

3*Explain data: Who can access or use the user’s personal data

How data is shared between apps    

Explain how predictions are based on data     

3*Explain predictions Explain with examples

Explain with confidence:     Data visualization (Expert users).Numerical (Might cause confusion).Categorial (e.g., Low - Med - High).N-best Alternatives (Low confidence) 

 Do not display Confidence:    Risks (misleading, no impact to user decisions when explaining output, distracting).Low confidence (Can provide partial explanation

Calculate Trade-off Identify conflicts with other requirements
3lc]@l@Calibrate user trust
2*Explain special cases Law & rules

Third party involvement    

2*Explain feedback Impact of feedback on AI

How feedback is used to improve model
3*Avoid over trusting Only show relevant info

Account for different situations    -Low stakes.High stakes

Explain errors     

3lc]@l@Errors & Failures

39emUser perceived errors (might change over time) Failstates True negative (Not included in system training).System limitation

Context errors (these are true positive but)    AI making incorrect assumptions.User has poor mental model.System not aligned with users needs

Background errors Invisible errors the user cannot perceive
139emList possible errors source Prediction & data errors

Mislabeled or misclassified results    Poor training.Incorrect labeling 

Incorrect Model    

Incomplete data    

**Input Errors**      

Unexpected input     

Old habits    

Mis-calibrated input    

**Output Quality**      

Low confidence    Lack of data. Uncertain prediction accuracy. Unstable information 

**systems hierarchy error**      

Multiple systems    (Allow user to give one of the system priorities) 

Crashing signals    (Allow signals from the primary system only) 

2*Error risks High stakes Health, safety and financial decisions -Sensitive / private data

Abusive users    

3*Action Mitigate Errors

Allow users to fix mistakes    

 Provide suggestions when in doubt    

Bibliography

   1@article{milne2012power,
   2  publisher = {Springer},
   3  year = {2012},
   4  pages = {83--98},
   5  volume = {17},
   6  journal = {Requirements Engineering},
   7  author = {Milne, Alastair and Maiden, Neil},
   8  title = {Power and politics in requirements engineering: embracing the dark side?},
   9}
  10
  11@article{callele2008emotional,
  12  publisher = {IEEE},
  13  year = {2008},
  14  pages = {43--45},
  15  number = {1},
  16  volume = {25},
  17  journal = {IEEE software},
  18  author = {Callele, David and Neufeld, Eric and Schneider, Kevin},
  19  title = {Emotional requirements},
  20}
  21
  22@inproceedings{thompson2013evoking,
  23  year = {2013},
  24  booktitle = {Looking into the Future of Creativity and Decision Support Systems: Proceedings of the 8th International Conference on Knowledge, Information and Creativity Support Systems},
  25  author = {Thompson, Clare and Maiden, Neil and Nouri, Mobina and Zachos, Konstantinos},
  26  title = {Evoking emotion through stories in creative dementia care},
  27}
  28
  29@article{maiden2012spocks,
  30  publisher = {IEEE Computer Society},
  31  year = {2012},
  32  pages = {84--85},
  33  number = {04},
  34  volume = {29},
  35  journal = {IEEE software},
  36  author = {Maiden, Neil},
  37  title = {Spocks and kirks in the requirements universe},
  38}
  39
  40@article{ramos2005emotion,
  41  publisher = {Springer},
  42  year = {2005},
  43  pages = {238--242},
  44  volume = {10},
  45  journal = {Requirements Engineering},
  46  author = {Ramos, Isabel and Berry, Daniel M},
  47  title = {Is emotion relevant to requirements engineering?},
  48}
  49
  50@article{agrahari2022omnidirectional,
  51  publisher = {TechRxiv},
  52  year = {2022},
  53  author = {Agrahari Baniya, Arbind and Lee, Glory and Eklund, Peter and Aryal, Sunil},
  54  title = {Omnidirectional Video Super-Resolution using Deep Learning},
  55}
  56
  57@article{vakkuri2021eccola,
  58  publisher = {Elsevier},
  59  year = {2021},
  60  pages = {111067},
  61  volume = {182},
  62  journal = {Journal of Systems and Software},
  63  author = {Vakkuri, Ville and Kemell, Kai-Kristian and Jantunen, Marianna and Halme, Erika and Abrahamsson, Pekka},
  64  title = {ECCOLA—A method for implementing ethically aligned AI systems},
  65}
  66
  67@inproceedings{Zamani:21,
  68  doi = {10.1109/REW53955.2021.00023},
  69  pages = {116-125},
  70  number = {},
  71  volume = {},
  72  year = {2021},
  73  title = {Machine Learning in Requirements Engineering: A Mapping Study},
  74  booktitle = {2021 IEEE 29th International Requirements Engineering Conference Workshops (REW)},
  75  author = {Zamani, Kareshna and Zowghi, Didar and Arora, Chetan},
  76}
  77
  78@inproceedings{khalajzadeh2020visual,
  79  year = {2020},
  80  pages = {15--26},
  81  booktitle = {ENASE},
  82  author = {Khalajzadeh, Hourieh and Simmons, Andrew J and Abdelrazek, Mohamed and Grundy, John and Hosking, John G and He, Qiang},
  83  title = {Visual Languages for Supporting Big Data Analytics Development.},
  84}
  85
  86@inproceedings{khalajzadeh2019bidaml,
  87  organization = {IEEE},
  88  year = {2019},
  89  pages = {93--97},
  90  booktitle = {2019 IEEE International Congress on Big Data (BigDataCongress)},
  91  author = {Khalajzadeh, Hourieh and Abdelrazek, Mohamed and Grundy, John and Hosking, John and He, Qiang},
  92  title = {Bidaml: A suite of visual languages for supporting end-user data analytics},
  93}
  94
  95@article{wachter2017counterfactual,
  96  publisher = {HeinOnline},
  97  year = {2017},
  98  pages = {841},
  99  volume = {31},
 100  journal = {Harv. JL \& Tech.},
 101  author = {Wachter, Sandra and Mittelstadt, Brent and Russell, Chris},
 102  title = {Counterfactual explanations without opening the black box: Automated decisions and the GDPR},
 103}
 104
 105@article{goodman2017european,
 106  year = {2017},
 107  pages = {50--57},
 108  number = {3},
 109  volume = {38},
 110  journal = {AI magazine},
 111  author = {Goodman, Bryce and Flaxman, Seth},
 112  title = {European Union regulations on algorithmic decision-making and a “right to explanation”},
 113}
 114
 115@article{shneiderman2020bridging,
 116  publisher = {ACM New York, NY, USA},
 117  year = {2020},
 118  pages = {1--31},
 119  number = {4},
 120  volume = {10},
 121  journal = {ACM Transactions on Interactive Intelligent Systems (TiiS)},
 122  author = {Shneiderman, Ben},
 123  title = {Bridging the gap between ethics and practice: guidelines for reliable, safe, and trustworthy human-centered AI systems},
 124}
 125
 126@article{lloyd2018bias,
 127  year = {2018},
 128  journal = {arXiv preprint arXiv:1809.07842},
 129  author = {Lloyd, Kirsten},
 130  title = {Bias amplification in artificial intelligence systems},
 131}
 132
 133@article{itu1999subjective,
 134  year = {1999},
 135  author = {ITU-T RECOMMENDATION, P},
 136  title = {Subjective video quality assessment methods for multimedia applications},
 137}
 138
 139@incollection{calders2013unbiased,
 140  publisher = {Springer},
 141  year = {2013},
 142  pages = {43--57},
 143  booktitle = {Discrimination and privacy in the information society},
 144  author = {Calders, Toon and {\v{Z}}liobait{\.e}, Indr{\.e}},
 145  title = {Why unbiased computational processes can lead to discriminative decision procedures},
 146}
 147
 148@article{shneiderman2020Three,
 149  year = {2020},
 150  pages = {109--124},
 151  number = {3},
 152  volume = {12},
 153  journal = {AIS Transactions on Human-Computer Interaction},
 154  author = {Shneiderman, Ben},
 155  title = {Human-centered artificial intelligence: three fresh ideas},
 156}
 157
 158@inproceedings{grundy2021impact,
 159  year = {2021},
 160  pages = {9--20},
 161  booktitle = {ENASE},
 162  author = {Grundy, John C},
 163  title = {Impact of End User Human Aspects on Software Engineering.},
 164}
 165
 166@inproceedings{mcintosh2021evaluating,
 167  organization = {IEEE},
 168  year = {2021},
 169  pages = {31--40},
 170  booktitle = {2021 IEEE/ACM 13th International Conference on Cooperative and Human Aspects of Software Engineering (CHASE)},
 171  author = {McIntosh, Jennifer and Du, Xiaojiao and Wu, Zexian and Truong, Giahuy and Ly, Quang and How, Richard and Viswanathan, Sriram and Kanij, Tanjila},
 172  title = {Evaluating Age Bias In E-commerce},
 173}
 174
 175@book{shneiderman2022human,
 176  publisher = {Oxford University Press},
 177  year = {2022},
 178  author = {Shneiderman, Ben},
 179  title = {Human-Centered AI},
 180}
 181
 182@article{schwartz2012overview,
 183  year = {2012},
 184  pages = {2307--0919},
 185  number = {1},
 186  volume = {2},
 187  journal = {Online readings in Psychology and Culture},
 188  author = {Schwartz, Shalom H},
 189  title = {An overview of the Schwartz theory of basic values},
 190}
 191
 192@inproceedings{riedl2016using,
 193  year = {2016},
 194  booktitle = {Workshops at the Thirtieth AAAI Conference on Artificial Intelligence},
 195  author = {Riedl, Mark O and Harrison, Brent},
 196  title = {Using stories to teach human values to artificial agents},
 197}
 198
 199@article{whittle2019your,
 200  publisher = {IEEE},
 201  year = {2019},
 202  pages = {112--115},
 203  number = {3},
 204  volume = {36},
 205  journal = {IEEE Software},
 206  author = {Whittle, Jon},
 207  title = {Is your software valueless?},
 208}
 209
 210@article{maguire2001methods,
 211  publisher = {Elsevier},
 212  year = {2001},
 213  pages = {587--634},
 214  number = {4},
 215  volume = {55},
 216  journal = {International journal of human-computer studies},
 217  author = {Maguire, Martin},
 218  title = {Methods to support human-centred design},
 219}
 220
 221@inproceedings{schmidt2020interactive,
 222  year = {2020},
 223  pages = {1--4},
 224  booktitle = {Proceedings of the International Conference on Advanced Visual Interfaces},
 225  author = {Schmidt, Albrecht},
 226  title = {Interactive human centered artificial intelligence: a definition and research challenges},
 227}
 228
 229@article{bellamy2018ai,
 230  year = {2018},
 231  journal = {arXiv preprint arXiv:1810.01943},
 232  author = {Bellamy, Rachel KE and Dey, Kuntal and Hind, Michael and Hoffman, Samuel C and Houde, Stephanie and Kannan, Kalapriya and Lohia, Pranay and Martino, Jacquelyn and Mehta, Sameep and Mojsilovic, Aleksandra and others},
 233  title = {AI Fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias},
 234}
 235
 236@inproceedings{roselli2019managing,
 237  year = {2019},
 238  pages = {539--544},
 239  booktitle = {Companion Proceedings of The 2019 World Wide Web Conference},
 240  author = {Roselli, Drew and Matthews, Jeanna and Talagala, Nisha},
 241  title = {Managing bias in AI},
 242}
 243
 244@article{whittaker2019disability,
 245  year = {2019},
 246  journal = {AI Now Institute},
 247  author = {Whittaker, Meredith and Alper, Meryl and Bennett, Cynthia L and Hendren, Sara and Kaziunas, Liz and Mills, Mara and Morris, Meredith Ringel and Rankin, Joy and Rogers, Emily and Salas, Marcel and others},
 248  title = {Disability, bias, and AI},
 249}
 250
 251@article{dignum2017responsible,
 252  publisher = {Daffodil International University},
 253  year = {2017},
 254  author = {Dignum, Virginia},
 255  title = {Responsible artificial intelligence: designing AI for human values},
 256}
 257
 258@article{hussain2020human,
 259  publisher = {IEEE},
 260  year = {2020},
 261  journal = {IEEE Transactions on Software Engineering},
 262  author = {Hussain, Waqar and Perera, Harsha and Whittle, Jon and Nurwidyantoro, Arif and Hoda, Rashina and Shams, Rifat Ara and Oliver, Gillian},
 263  title = {Human values in software engineering: Contrasting case studies of practice},
 264}
 265
 266@inproceedings{perera2020study,
 267  organization = {IEEE},
 268  year = {2020},
 269  pages = {409--420},
 270  booktitle = {2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE)},
 271  author = {Perera, Harsha and Hussain, Waqar and Whittle, Jon and Nurwidyantoro, Arif and Mougouei, Davoud and Shams, Rifat Ara and Oliver, Gillian},
 272  title = {A study on the prevalence of human values in software engineering publications, 2015-2018},
 273}
 274
 275@inproceedings{viyovic2014sirius,
 276  organization = {IEEE},
 277  year = {2014},
 278  pages = {233--238},
 279  booktitle = {IEEE 18th International Conference on Intelligent Engineering Systems INES 2014},
 280  author = {Viyovi{\'c}, Vladimir and Maksimovi{\'c}, Mirjam and Perisi{\'c}, Branko},
 281  title = {Sirius: A rapid development of DSM graphical editor},
 282}
 283
 284@article{moody2009physics,
 285  publisher = {IEEE},
 286  year = {2009},
 287  pages = {756--779},
 288  number = {6},
 289  volume = {35},
 290  journal = {IEEE Transactions on software engineering},
 291  author = {Moody, Daniel},
 292  title = {The “physics” of notations: toward a scientific basis for constructing visual notations in software engineering},
 293}
 294
 295@article{kitchenham1995case,
 296  publisher = {IEEE},
 297  year = {1995},
 298  pages = {52--62},
 299  number = {4},
 300  volume = {12},
 301  journal = {IEEE software},
 302  author = {Kitchenham, Barbara and Pickard, Lesley and Pfleeger, Shari Lawrence},
 303  title = {Case studies for method and tool evaluation},
 304}
 305
 306@incollection{bosch2021engineering,
 307  publisher = {IGI global},
 308  year = {2021},
 309  pages = {1--19},
 310  booktitle = {Artificial Intelligence Paradigms for Smart Cyber-Physical Systems},
 311  author = {Bosch, Jan and Olsson, Helena Holmstr{\"o}m and Crnkovic, Ivica},
 312  title = {Engineering ai systems: A research agenda},
 313}
 314
 315@article{glymour2008causal,
 316  publisher = {Lippincott Williams \& Wilkins Philadelphia, PA},
 317  year = {2008},
 318  pages = {183--209},
 319  volume = {3},
 320  journal = {Modern epidemiology},
 321  author = {Glymour, M Maria and Greenland, Sander},
 322  title = {Causal diagrams},
 323}
 324
 325@article{pearl1995causal,
 326  publisher = {Oxford University Press},
 327  year = {1995},
 328  pages = {669--688},
 329  number = {4},
 330  volume = {82},
 331  journal = {Biometrika},
 332  author = {Pearl, Judea},
 333  title = {Causal diagrams for empirical research},
 334}
 335
 336@article{greenland1999causal,
 337  publisher = {JSTOR},
 338  year = {1999},
 339  pages = {37--48},
 340  journal = {Epidemiology},
 341  author = {Greenland, Sander and Pearl, Judea and Robins, James M},
 342  title = {Causal diagrams for epidemiologic research},
 343}
 344
 345@inproceedings{blumreiter2019towards,
 346  organization = {IEEE},
 347  year = {2019},
 348  pages = {543--548},
 349  booktitle = {2019 ACM/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C)},
 350  author = {Blumreiter, Mathias and Greenyer, Joel and Garcia, Francisco Javier Chiyah and Kl{\"o}s, Verena and Schwammberger, Maike and Sommer, Christoph and Vogelsang, Andreas and Wortmann, Andreas},
 351  title = {Towards self-explainable cyber-physical systems},
 352}
 353
 354@book{embley2012handbook,
 355  publisher = {Springer},
 356  year = {2012},
 357  author = {Embley, David W and Thalheim, Bernhard},
 358  title = {Handbook of conceptual modeling: theory, practice, and research challenges},
 359}
 360
 361@article{martinez2021software,
 362  year = {2021},
 363  journal = {arXiv preprint arXiv:2105.01984},
 364  author = {Mart{\'\i}nez-Fern{\'a}ndez, Silverio and Bogner, Justus and Franch, Xavier and Oriol, Marc and Siebert, Julien and Trendowicz, Adam and Vollmer, Anna Maria and Wagner, Stefan},
 365  title = {Software Engineering for AI-Based Systems: A Survey},
 366}
 367
 368@inproceedings{zhou2016map,
 369  organization = {IEEE},
 370  year = {2016},
 371  pages = {153--160},
 372  booktitle = {2016 23rd Asia-Pacific Software Engineering Conference (APSEC)},
 373  author = {Zhou, Xin and Jin, Yuqin and Zhang, He and Li, Shanshan and Huang, Xin},
 374  title = {A map of threats to validity of systematic literature reviews in software engineering},
 375}
 376
 377@article{holmquist2017intelligence,
 378  publisher = {ACM},
 379  year = {2017},
 380  pages = {28--33},
 381  number = {4},
 382  volume = {24},
 383  journal = {interactions},
 384  author = {Holmquist, Lars Erik},
 385  title = {Intelligence on tap: artificial intelligence as a new design material},
 386}
 387
 388@article{price2016microsoft,
 389  year = {2016},
 390  journal = {Business Insider},
 391  author = {Price, Rob},
 392  title = {Microsoft is deleting its \uppercase{AI} chatbot’s incredibly racist tweets},
 393}
 394
 395@article{khomh2018software,
 396  publisher = {IEEE},
 397  year = {2018},
 398  pages = {81--84},
 399  number = {5},
 400  volume = {35},
 401  journal = {IEEE Software},
 402  author = {Khomh, Foutse and Adams, Bram and Cheng, Jinghui and Fokaefs, Marios and Antoniol, Giuliano},
 403  title = {Software Engineering for Machine-Learning Applications: The Road Ahead},
 404}
 405
 406@inproceedings{houdek2017automotive,
 407  year = {2017},
 408  booktitle = {REFSQ Workshops},
 409  author = {Houdek, Frank and Schmerler, Stefan},
 410  title = {Automotive Future and its Impact on Requirements Engineering.},
 411}
 412
 413@incollection{boy2014unifying,
 414  publisher = {Springer},
 415  year = {2014},
 416  pages = {151--162},
 417  booktitle = {Complex Systems Design \& Management},
 418  author = {Boy, Guy A and Narkevicius, Jennifer McGovern},
 419  title = {Unifying human centered design and systems engineering for human systems integration},
 420}
 421
 422@book{li2017artificial,
 423  publisher = {CRC press},
 424  year = {2017},
 425  author = {Li, Deyi and Du, Yi},
 426  title = {Artificial intelligence with uncertainty},
 427}
 428
 429@inproceedings{zowghi2002three,
 430  year = {2002},
 431  pages = {155--164},
 432  booktitle = {International Workshop on Requirements Engineering: Foundations for Software Quality, Essen, Germany: Essener Informatik Beitiage},
 433  author = {Zowghi, Didar and Gervasi, Vincenzo},
 434  title = {The Three {C}s of requirements: consistency, completeness, and correctness},
 435}
 436
 437@article{jiang2017artificial,
 438  publisher = {BMJ Specialist Journals},
 439  year = {2017},
 440  pages = {230--243},
 441  number = {4},
 442  volume = {2},
 443  journal = {Stroke and vascular neurology},
 444  author = {Jiang, Fei and Jiang, Yong and Zhi, Hui and Dong, Yi and Li, Hao and Ma, Sufeng and Wang, Yilong and Dong, Qiang and Shen, Haipeng and Wang, Yongjun},
 445  title = {Artificial intelligence in healthcare: past, present and future},
 446}
 447
 448@article{washizaki2019studying,
 449  year = {2019},
 450  journal = {arXiv preprint arXiv:1910.04736},
 451  author = {Washizaki, Hironori and Uchida, Hiromu and Khomh, Foutse and Gueheneuc, Yann-Gael},
 452  title = {Studying Software Engineering Patterns for Designing Machine Learning Systems},
 453}
 454
 455@inproceedings{kanij2015empirical,
 456  organization = {IEEE Press},
 457  year = {2015},
 458  pages = {1--7},
 459  booktitle = {Proceedings of the Eighth International Workshop on Cooperative and Human Aspects of Software Engineering},
 460  author = {Kanij, Tanjila and Merkel, Robert and Grundy, John},
 461  title = {An empirical investigation of personality traits of software testers},
 462}
 463
 464@article{soomro2016effect,
 465  publisher = {Elsevier},
 466  year = {2016},
 467  pages = {52--65},
 468  volume = {73},
 469  journal = {Information and Software Technology},
 470  author = {Soomro, Arjumand Bano and Salleh, Norsaremah and Mendes, Emilia and Grundy, John and Burch, Giles and Nordin, Azlin},
 471  title = {The effect of software engineers’ personality traits on team climate and performance: A Systematic Literature Review},
 472}
 473
 474@incollection{kamalrudin2014pair,
 475  publisher = {Springer},
 476  year = {2014},
 477  pages = {150--164},
 478  booktitle = {Requirements Engineering},
 479  author = {Kamalrudin, Massila and Sidek, Safiah and Salleh, Norsaremah and Hosking, John and Grundy, John},
 480  title = {A Pair-Oriented Requirements Engineering Approach for Analysing Multi-lingual Requirements},
 481}
 482
 483@article{miller2015emotion,
 484  publisher = {Elsevier},
 485  year = {2015},
 486  pages = {54--71},
 487  volume = {105},
 488  journal = {Journal of Systems and Software},
 489  author = {Miller, Tim and Pedell, Sonja and Lopez-Lorca, Antonio A and Mendoza, Antonette and Sterling, Leon and Keirnan, Alen},
 490  title = {Emotion-led modelling for people-oriented requirements engineering: The case study of emergency systems},
 491}
 492
 493@article{miller2012understanding,
 494  publisher = {Elsevier},
 495  year = {2012},
 496  pages = {2160--2170},
 497  number = {9},
 498  volume = {85},
 499  journal = {Journal of Systems and Software},
 500  author = {Miller, Tim and Pedell, Sonja and Sterling, Leon and Vetere, Frank and Howard, Steve},
 501  title = {Understanding socially oriented roles and goals through motivational modelling},
 502}
 503
 504@inproceedings{burnett2016finding,
 505  organization = {ACM},
 506  year = {2016},
 507  pages = {2586--2598},
 508  booktitle = {Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems},
 509  author = {Burnett, Margaret and Peters, Anicia and Hill, Charles and Elarief, Noha},
 510  title = {Finding gender-inclusiveness software issues with {G}ender{M}ag: a field investigation},
 511}
 512
 513@article{kissoon2019emotion,
 514  publisher = {Elsevier},
 515  year = {2019},
 516  pages = {215--229},
 517  volume = {147},
 518  journal = {Journal of systems and software},
 519  author = {Kissoon Curumsing, Maheswaree and Fernando, Niroshinie and Abdelrazek, Mohamed and Vasa, Rajesh and Mouzakis, Kon and Grundy, John},
 520  title = {Emotion-oriented requirements engineering: a case study in developing a smart home system for the elderly},
 521}
 522
 523@book{dick2017requirements,
 524  publisher = {Springer},
 525  year = {2017},
 526  author = {Dick, Jeremy and Hull, Elizabeth and Jackson, Ken},
 527  title = {Requirements engineering},
 528}
 529
 530@inproceedings{vorvoreanu2019From,
 531  year = {2019},
 532  booktitle = {Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI’19). ACM, New York, NY, USA. https://doi. org/10.1145/3290605.3300283},
 533  author = {Vorvoreanu, Mihaela and Zhang, Lingyi and Huang, Yun-Han and Hilderbrand, Claudia and Steine-Hanson, Zoe and Burnett, Margaret},
 534  title = {From Gender Biases to Gender-Inclusive Design: An Empirical Investigation},
 535}
 536
 537@article{grundy2018supporting,
 538  year = {2018},
 539  pages = {75--90},
 540  volume = {246},
 541  journal = {Studies in health technology and informatics},
 542  author = {Grundy, John and Mouzakis, Kon and Vasa, Rajesh and Cain, Andrew and Curumsing, Maheswaree and Abdelrazek, Mohamed and Fernando, Niroshine},
 543  title = {Supporting Diverse Challenges of Ageing with Digital Enhanced Living Solutions.},
 544}
 545
 546@inproceedings{sneed2007testing,
 547  organization = {IEEE},
 548  year = {2007},
 549  pages = {380--387},
 550  booktitle = {Seventh International Conference on Quality Software (QSIC 2007)},
 551  author = {Sneed, Harry M},
 552  title = {Testing against natural language requirements},
 553}
 554
 555@article{graham2002requirements,
 556  publisher = {IEEE},
 557  year = {2002},
 558  pages = {15--17},
 559  number = {5},
 560  volume = {19},
 561  journal = {IEEE Software},
 562  author = {Graham, Dorothy},
 563  title = {Requirements and testing: Seven missing-link myths},
 564}
 565
 566@inproceedings{sneed2018requirement,
 567  organization = {Springer},
 568  year = {2018},
 569  pages = {60--79},
 570  booktitle = {International Conference on Software Quality},
 571  author = {Sneed, Harry M},
 572  title = {Requirement-based testing-extracting logical test cases from requirement documents},
 573}
 574
 575@inproceedings{vorvoreanu2019rom,
 576  year = {2019},
 577  booktitle = {Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI’19). ACM, New York, NY, USA. https://doi. org/10.1145/3290605.3300283},
 578  author = {Vorvoreanu, Mihaela and Zhang, Lingyi and Huang, Yun-Han and Hilderbrand, Claudia and Steine-Hanson, Zoe and Burnett, Margaret},
 579  title = {from Gender Biases to Gender-Inclusive Design: An Empirical Investigation},
 580}
 581
 582@inproceedings{amershi2019software,
 583  organization = {IEEE Press},
 584  year = {2019},
 585  pages = {291--300},
 586  booktitle = {Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice},
 587  author = {Amershi, Saleema and Begel, Andrew and Bird, Christian and DeLine, Robert and Gall, Harald and Kamar, Ece and Nagappan, Nachiappan and Nushi, Besmira and Zimmermann, Thomas},
 588  title = {Software engineering for machine learning: a case study},
 589}
 590
 591@article{inayat2015systematic,
 592  publisher = {Elsevier},
 593  year = {2015},
 594  pages = {915--929},
 595  volume = {51},
 596  journal = {Computers in human behavior},
 597  author = {Inayat, Irum and Salim, Siti Salwah and Marczak, Sabrina and Daneva, Maya and Shamshirband, Shahaboddin},
 598  title = {A systematic literature review on agile requirements engineering practices and challenges},
 599}
 600
 601@article{wilson2018collaborative,
 602  year = {2018},
 603  pages = {114--123},
 604  number = {4},
 605  volume = {96},
 606  journal = {Harvard Business Review},
 607  author = {Wilson, H James and Daugherty, Paul R},
 608  title = {Collaborative intelligence: humans and {AI} are joining forces},
 609}
 610
 611@article{bellamy2019think,
 612  publisher = {IEEE},
 613  year = {2019},
 614  pages = {76--80},
 615  number = {4},
 616  volume = {36},
 617  journal = {IEEE Software},
 618  author = {Bellamy, Rachel KE and Dey, Kuntal and Hind, Michael and Hoffman, Samuel C and Houde, Stephanie and Kannan, Kalapriya and Lohia, Pranay and Mehta, Sameep and Mojsilovic, Aleksandra and Nagar, Seema and others},
 619  title = {Think your artificial intelligence software is fair? Think again},
 620}
 621
 622@inproceedings{sculley2015hidden,
 623  year = {2015},
 624  pages = {2503--2511},
 625  booktitle = {Advances in neural information processing systems},
 626  author = {Sculley, David and Holt, Gary and Golovin, Daniel and Davydov, Eugene and Phillips, Todd and Ebner, Dietmar and Chaudhary, Vinay and Young, Michael and Crespo, Jean-Francois and Dennison, Dan},
 627  title = {Hidden technical debt in machine learning systems},
 628}
 629
 630@incollection{maalej2013introduction,
 631  publisher = {Springer},
 632  year = {2013},
 633  pages = {1--20},
 634  booktitle = {Managing Requirements Knowledge},
 635  author = {Maalej, Walid and Thurimella, Anil Kumar},
 636  title = {An introduction to requirements knowledge},
 637}
 638
 639@incollection{fricker2015requirements,
 640  publisher = {Springer},
 641  year = {2015},
 642  pages = {25--46},
 643  booktitle = {Requirements Engineering for Digital Health},
 644  author = {Fricker, Samuel A and Grau, Rainer and Zwingli, Adrian},
 645  title = {Requirements engineering: best practice},
 646}
 647
 648@book{koelsch2016requirements,
 649  publisher = {Springer},
 650  year = {2016},
 651  author = {Koelsch, George},
 652  title = {Requirements writing for system engineering},
 653}
 654
 655@book{dean2003managing,
 656  publisher = {Addison-Wesley Professional},
 657  year = {2003},
 658  author = {Dean, Leffingwell and Don, Widrig},
 659  title = {Managing software requirements: A use case approach},
 660}
 661
 662@book{marshall2009solid,
 663  publisher = {Microsoft Press},
 664  year = {2009},
 665  author = {Marshall, Donis and Bruno, John},
 666  title = {Solid Code: Optimizing the Software Development Life Cycle},
 667}
 668
 669@book{mcconnell1998software,
 670  publisher = {Pearson Education},
 671  year = {1998},
 672  author = {McConnell, Steve},
 673  title = {Software project survival guide},
 674}
 675
 676@article{horkoff2019goal,
 677  publisher = {Springer},
 678  year = {2019},
 679  pages = {133--160},
 680  number = {2},
 681  volume = {24},
 682  journal = {Requirements Engineering},
 683  author = {Horkoff, Jennifer and Aydemir, Fatma Ba{\c{s}}ak and Cardoso, Evellin and Li, Tong and Mat{\'e}, Alejandro and Paja, Elda and Salnitri, Mattia and Piras, Luca and Mylopoulos, John and Giorgini, Paolo},
 684  title = {Goal-oriented requirements engineering: an extended systematic mapping study},
 685}
 686
 687@inproceedings{bano2014systematic,
 688  organization = {IEEE},
 689  year = {2014},
 690  pages = {9--16},
 691  booktitle = {2014 IEEE 4th International Workshop on Empirical Requirements Engineering (EmpiRE)},
 692  author = {Bano, Muneera and Zowghi, Didar and Ikram, Naveed},
 693  title = {Systematic reviews in requirements engineering: A tertiary study},
 694}
 695
 696@article{amyot2010evaluating,
 697  publisher = {Wiley Online Library},
 698  year = {2010},
 699  pages = {841--877},
 700  number = {8},
 701  volume = {25},
 702  journal = {International Journal of Intelligent Systems},
 703  author = {Amyot, Daniel and Ghanavati, Sepideh and Horkoff, Jennifer and Mussbacher, Gunter and Peyton, Liam and Yu, Eric},
 704  title = {Evaluating goal models within the goal-oriented requirement language},
 705}
 706
 707@article{daneva2014empirical,
 708  publisher = {Elsevier},
 709  year = {2014},
 710  pages = {1--9},
 711  volume = {95},
 712  journal = {Journal of systems and software},
 713  author = {Daneva, Maya and Damian, Daniela and Marchetto, Alessandro and Pastor, Oscar},
 714  title = {Empirical research methodologies and studies in Requirements Engineering: How far did we come?},
 715}
 716
 717@inproceedings{lwakatare2019taxonomy,
 718  organization = {Springer},
 719  year = {2019},
 720  pages = {227--243},
 721  booktitle = {International Conference on Agile Software Development},
 722  author = {Lwakatare, Lucy Ellen and Raj, Aiswarya and Bosch, Jan and Olsson, Helena Holmstr{\"o}m and Crnkovic, Ivica},
 723  title = {A taxonomy of software engineering challenges for machine learning systems: An empirical investigation},
 724}
 725
 726@article{ur2013analysis,
 727  year = {2013},
 728  pages = {40},
 729  number = {3},
 730  volume = {5},
 731  journal = {International Journal of Information Technology and Computer Science (IJITCS)},
 732  author = {ur Rehman, Tousif and Khan, Muhammad Naeem Ahmed and Riaz, Naveed},
 733  title = {Analysis of requirement engineering processes, tools/techniques and methodologies},
 734}
 735
 736@inproceedings{van2004goal,
 737  organization = {IEEE},
 738  year = {2004},
 739  pages = {4--7},
 740  booktitle = {Proceedings. 12th IEEE International Requirements Engineering Conference, 2004.},
 741  author = {Van Lamsweerde, Axel},
 742  title = {Goal-oriented requirements enginering: a roundtrip from research to practice [enginering read engineering]},
 743}
 744
 745@article{amyot2011user,
 746  year = {2011},
 747  pages = {747--768},
 748  number = {5},
 749  volume = {6},
 750  journal = {JSW},
 751  author = {Amyot, Daniel and Mussbacher, Gunter},
 752  title = {User requirements notation: the first ten years, the next ten years},
 753}
 754
 755@inproceedings{belani2019requirements,
 756  organization = {IEEE},
 757  year = {2019},
 758  pages = {252--255},
 759  booktitle = {2019 IEEE 27th International Requirements Engineering Conference Workshops (REW)},
 760  author = {Belani, Hrvoje and Vukovic, Marin and Car, {\v{Z}}eljka},
 761  title = {Requirements Engineering Challenges in Building {AI}-Based Complex Systems},
 762}
 763
 764@inproceedings{arpteg2018software,
 765  organization = {IEEE},
 766  year = {2018},
 767  pages = {50--59},
 768  booktitle = {2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)},
 769  author = {Arpteg, Anders and Brinne, Bj{\"o}rn and Crnkovic-Friis, Luka and Bosch, Jan},
 770  title = {Software engineering challenges of deep learning},
 771}
 772
 773@article{lovejoy2018human,
 774  year = {2020. https://medium.com/google-design/human-centered-machine-learning-a770d10562cd},
 775  journal = {Medium. Retrieved March},
 776  author = {Lovejoy, Josh and Holbrook, Jess},
 777  title = {Human-Centered Machine Learning. 7 steps to stay focused on the user when designing with {ML}},
 778}
 779
 780@inproceedings{amershi2019guidelines,
 781  year = {2019},
 782  pages = {1--13},
 783  booktitle = {Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems},
 784  author = {Amershi, Saleema and Weld, Dan and Vorvoreanu, Mihaela and Fourney, Adam and Nushi, Besmira and Collisson, Penny and Suh, Jina and Iqbal, Shamsi and Bennett, Paul N and Inkpen, Kori and others},
 785  title = {Guidelines for human-{AI} interaction},
 786}
 787
 788@article{kitchenham2007guidelines,
 789  publisher = {Citeseer},
 790  year = {2007},
 791  author = {Kitchenham, Barbara and Charters, Stuart},
 792  title = {Guidelines for performing systematic literature reviews in software engineering},
 793}
 794
 795@article{miller2017explainable,
 796  year = {2017},
 797  journal = {IJCAI 2017 Workshop on Explainable Artificial Intelligence (XAI), 36–42, URL http://people. eng.unimelb.edu.au/tmiller/pubs/explanation-inmates.pdf},
 798  author = {Miller, Tim and Howe, Piers and Sonenberg, Liz},
 799  title = {Explainable {AI}: Beware of inmates running the asylum or: How I learnt to stop worrying and love the social and behavioural sciences},
 800}
 801
 802@inproceedings{sakamoto2013goal,
 803  organization = {IEEE},
 804  year = {2013},
 805  pages = {32--35},
 806  booktitle = {2013 3rd International Workshop on Games and Software Engineering: Engineering Computer Games to Enable Positive, Progressive Change (GAS)},
 807  author = {Sakamoto, Kazunori and Hosono, Hiroaki and Sato, Seiji and Washizaki, Hironori and Fukazawa, Yoshiaki},
 808  title = {Goal-oriented requirements analysis and an extended design pattern using scala for artificial intelligence programming contests},
 809}
 810
 811@incollection{vassev2014autonomy,
 812  publisher = {Springer},
 813  year = {2014},
 814  pages = {105--172},
 815  booktitle = {Autonomy Requirements Engineering for Space Missions},
 816  author = {Vassev, Emil and Hinchey, Mike},
 817  title = {Autonomy requirements engineering},
 818}
 819
 820@inproceedings{kondermann2013ground,
 821  year = {2013},
 822  pages = {1--4},
 823  booktitle = {Proceedings of the International Workshop on Video and Image Ground Truth in Computer Vision Applications},
 824  author = {Kondermann, Daniel},
 825  title = {Ground truth design principles: an overview},
 826}
 827
 828@inproceedings{hajian2016algorithmic,
 829  year = {2016},
 830  pages = {2125--2126},
 831  booktitle = {Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining},
 832  author = {Hajian, Sara and Bonchi, Francesco and Castillo, Carlos},
 833  title = {Algorithmic bias: From discrimination discovery to fairness-aware data mining},
 834}
 835
 836@article{morandini2017engineering,
 837  publisher = {Springer},
 838  year = {2017},
 839  pages = {77--103},
 840  number = {1},
 841  volume = {22},
 842  journal = {Requirements Engineering},
 843  author = {Morandini, Mirko and Penserini, Loris and Perini, Anna and Marchetto, Alessandro},
 844  title = {Engineering requirements for adaptive systems},
 845}
 846
 847@inproceedings{dimatteo2020requirements,
 848  year = {2020},
 849  booktitle = {REFSQ Workshops},
 850  author = {DiMatteo, Johnathan and Berry, Daniel M and Czarnecki, Krzysztof},
 851  title = {Requirements for Monitoring Inattention of the Responsible Human in an Autonomous Vehicle: The Recall and Precision Tradeoff.},
 852}
 853
 854@inproceedings{wickramasinghe2020trustworthy,
 855  organization = {IEEE},
 856  year = {2020},
 857  pages = {130--136},
 858  booktitle = {2020 13th International Conference on Human System Interaction (HSI)},
 859  author = {Wickramasinghe, Chathurika S and Marino, Daniel L and Grandio, Javier and Manic, Milos},
 860  title = {Trustworthy {AI} Development Guidelines for Human System Interaction},
 861}
 862
 863@inproceedings{krause2016interacting,
 864  year = {2016},
 865  pages = {5686--5697},
 866  booktitle = {Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems},
 867  author = {Krause, Josua and Perer, Adam and Ng, Kenney},
 868  title = {Interacting with predictions: Visual inspection of black-box machine learning models},
 869}
 870
 871@article{miller2019explanation,
 872  publisher = {Elsevier},
 873  year = {2019},
 874  pages = {1--38},
 875  volume = {267},
 876  journal = {Artificial Intelligence},
 877  author = {Miller, Tim},
 878  title = {Explanation in artificial intelligence: Insights from the social sciences},
 879}
 880
 881@inproceedings{de2017people,
 882  year = {2017},
 883  booktitle = {2017 AAAI Fall Symposium Series},
 884  author = {De Graaf, Maartje MA and Malle, Bertram F},
 885  title = {How people explain action (and autonomous intelligent systems should too)},
 886}
 887
 888@article{jobin2019global,
 889  publisher = {Nature Publishing Group},
 890  year = {2019},
 891  pages = {389--399},
 892  number = {9},
 893  volume = {1},
 894  journal = {Nature Machine Intelligence},
 895  author = {Jobin, Anna and Ienca, Marcello and Vayena, Effy},
 896  title = {The global landscape of {AI} ethics guidelines},
 897}
 898
 899@inproceedings{pedreschi2019meaningful,
 900  year = {2019},
 901  pages = {9780--9784},
 902  volume = {33},
 903  booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
 904  author = {Pedreschi, Dino and Giannotti, Fosca and Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Turini, Franco},
 905  title = {Meaningful explanations of Black Box {AI} decision systems},
 906}
 907
 908@article{guidotti2018survey,
 909  publisher = {ACM New York, NY, USA},
 910  year = {2018},
 911  pages = {1--42},
 912  number = {5},
 913  volume = {51},
 914  journal = {ACM computing surveys (CSUR)},
 915  author = {Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Turini, Franco and Giannotti, Fosca and Pedreschi, Dino},
 916  title = {A survey of methods for explaining black box models},
 917}
 918
 919@book{IOS2014SQuaRE,
 920  publisher = {ISO/IEC},
 921  year = {2014},
 922  author = {International Organization for Standardization},
 923  title = {Systems and Software Engineering: Systems and Software Quality Requirements and Evaluation ({SQuaRE}): Guide to {SQuaRE}},
 924}
 925
 926@article{HLEG2019ethics,
 927  note = {Document can be accessed at:  https://ec.europa.eu/digital-single-market/en/news/ethics-guidelines-trustworthy-ai},
 928  publisher = { European Commission },
 929  year = {8 April 2019},
 930  author = {High-Level Expert Group on Artificial Intelligence},
 931  title = {Ethics guidelines for trustworthy {AI}},
 932}
 933
 934@inproceedings{brun2018software,
 935  year = {2018},
 936  pages = {754--759},
 937  booktitle = {Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering},
 938  author = {Brun, Yuriy and Meliou, Alexandra},
 939  title = {Software fairness},
 940}
 941
 942@inproceedings{galhotra2017fairness,
 943  year = {2017},
 944  pages = {498--510},
 945  booktitle = {Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering},
 946  author = {Galhotra, Sainyam and Brun, Yuriy and Meliou, Alexandra},
 947  title = {Fairness testing: testing software for discrimination},
 948}
 949
 950@article{garvie2016facial,
 951  year = {2016},
 952  volume = {7},
 953  journal = {The Atlantic},
 954  author = {Garvie, Clare and Frankle, Jonathan},
 955  title = {Facial-recognition software might have a racial bias problem},
 956}
 957
 958@article{mendoza2013role,
 959  year = {2013},
 960  author = {Mendoza, Antonette and Miller, Tim and Pedell, Sonja and Sterling and others},
 961  title = {The role of users’ emotions and associated quality goals on appropriation of systems: two case studies},
 962}
 963
 964@inproceedings{schroeder2015design,
 965  organization = {IEEE},
 966  year = {2015},
 967  pages = {189--198},
 968  volume = {2},
 969  booktitle = {2015 IEEE/ACM 37th IEEE International Conference on Software Engineering},
 970  author = {Schroeder, Jan and Holzner, Daniela and Berger, Christian and Hoel, Carl-Johan and Laine, Leo and Magnusson, Anders},
 971  title = {Design and evaluation of a customizable multi-domain reference architecture on top of product lines of self-driving heavy vehicles-an industrial case study},
 972}
 973
 974@inproceedings{mekni2020smart,
 975  year = {2020},
 976  pages = {1--6},
 977  booktitle = {Proceedings of the 3rd International Conference on Applications of Intelligent Systems},
 978  author = {Mekni, Mehdi and Baani, Zakaria and Sulieman, Dalia},
 979  title = {A Smart Virtual Assistant for Students},
 980}
 981
 982@article{lee2019confident,
 983  publisher = {ACM New York, NY, USA},
 984  year = {2019},
 985  pages = {1--39},
 986  number = {1},
 987  volume = {27},
 988  journal = {ACM Transactions on Computer-Human Interaction (TOCHI)},
 989  author = {Lee, Hosub and Kobsa, Alfred},
 990  title = {Confident Privacy Decision-Making in {IoT} Environments},
 991}
 992
 993@inproceedings{theosaksomo2019conversational,
 994  organization = {IEEE},
 995  year = {2019},
 996  pages = {154--159},
 997  booktitle = {2019 IEEE 13th International Conference on Telecommunication Systems, Services, and Applications (TSSA)},
 998  author = {Theosaksomo, David and Widyantoro, Dwi H},
 999  title = {Conversational Recommender System Chatbot Based on Functional Requirement},
1000}
1001
1002@inproceedings{agarwal2014expert,
1003  organization = {IEEE},
1004  year = {2014},
1005  pages = {1--4},
1006  booktitle = {International Conference on Recent Advances and Innovations in Engineering (ICRAIE-2014)},
1007  author = {Agarwal, Mahak and Goel, Shivani},
1008  title = {Expert system and it's requirement engineering process},
1009}
1010
1011@inproceedings{singh2017reconcile,
1012  organization = {IEEE},
1013  year = {2017},
1014  pages = {1646--1651},
1015  booktitle = {2017 17th International Conference on Control, Automation and Systems (ICCAS)},
1016  author = {Singh, Madhusudan and Kim, Shiho},
1017  title = {Reconcile security requirements for intelligent vehicles},
1018}
1019
1020@inproceedings{cysneiros2018software,
1021  organization = {IEEE},
1022  year = {2018},
1023  pages = {382--387},
1024  booktitle = {2018 IEEE 26th International Requirements Engineering Conference (RE)},
1025  author = {Cysneiros, Luiz Marcio and Raffi, Majid and do Prado Leite, Julio Cesar Sampaio},
1026  title = {Software transparency as a key requirement for self-driving cars},
1027}
1028
1029@inproceedings{schrader2019approach,
1030  organization = {IEEE},
1031  year = {2019},
1032  pages = {1597--1603},
1033  booktitle = {2019 IEEE Intelligent Vehicles Symposium (IV)},
1034  author = {Schr{\"a}der, Tobias and Stolte, Torben and Jatzkowski, Inga and Graubohm, Robert and Maurer, Markus},
1035  title = {An Approach for a Requirement Analysis for an Autonomous Family Vehicle},
1036}
1037
1038@inproceedings{martin2019graded,
1039  organization = {IEEE},
1040  year = {2019},
1041  pages = {1--6},
1042  booktitle = {2019 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)},
1043  author = {Martin, Trevor P and Anggraini, Ratih NE},
1044  title = {A Graded Approach to Requirement Satisfaction for Evolving Systems},
1045}
1046
1047@inproceedings{bencomo2019ram,
1048  organization = {IEEE},
1049  year = {2019},
1050  pages = {216--226},
1051  booktitle = {2019 ACM/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems (MODELS)},
1052  author = {Bencomo, Nelly and Paucar, Luis H Garcia},
1053  title = {{RaM}: Causally-connected and Requirements-aware Runtime Models using Bayesian Learning},
1054}
1055
1056@inproceedings{chazette2019mitigating,
1057  organization = {IEEE},
1058  year = {2019},
1059  pages = {470--475},
1060  booktitle = {2019 IEEE 27th International Requirements Engineering Conference (RE)},
1061  author = {Chazette, Larissa},
1062  title = {Mitigating Challenges in the Elicitation and Analysis of Transparency Requirements},
1063}
1064
1065@inproceedings{fagbola2019towards,
1066  organization = {IEEE},
1067  year = {2019},
1068  pages = {200--204},
1069  booktitle = {2019 International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS)},
1070  author = {Fagbola, Temitayo M and Thakur, Surendra C},
1071  title = {Towards the Development of Artificial Intelligence-based Systems: Human-Centered Functional Requirements and Open Problems},
1072}
1073
1074@inproceedings{brugali2019non,
1075  organization = {IEEE},
1076  year = {2019},
1077  pages = {743--748},
1078  booktitle = {2019 IEEE International Conference on Real-time Computing and Robotics (RCAR)},
1079  author = {Brugali, Davide},
1080  title = {Non-Functional Requirements in Robotic Systems: Challenges and State of the Art},
1081}
1082
1083@article{lutje2020requirements,
1084  publisher = {Multidisciplinary Digital Publishing Institute},
1085  year = {2020},
1086  pages = {10},
1087  number = {1},
1088  volume = {10},
1089  journal = {Administrative Sciences},
1090  author = {L{\"u}tje, Anna and Wohlgemuth, Volker},
1091  title = {Requirements Engineering for an Industrial Symbiosis Tool for Industrial Parks Covering System Analysis, Transformation Simulation and Goal Setting},
1092}
1093
1094@inproceedings{woodman2019human,
1095  organization = {IEEE},
1096  year = {2019},
1097  pages = {2371--2376},
1098  booktitle = {2019 IEEE Intelligent Vehicles Symposium (IV)},
1099  author = {Woodman, Roger and Lu, Ke and Higgins, Matthew D and Brewerton, Simon and Jennings, Paul and Birrell, Stewart},
1100  title = {A human factors approach to defining requirements for low-speed autonomous vehicles to enable intelligent platooning},
1101}
1102
1103@incollection{cysneiros2020non,
1104  publisher = {Springer},
1105  year = {2020},
1106  pages = {335--342},
1107  booktitle = {Enterprise, Business-Process and Information Systems Modeling},
1108  author = {Cysneiros, Luiz Marcio and do Prado Leite, Julio Cesar Sampaio},
1109  title = {Non-Functional Requirements Orienting the Development of Socially Responsible Software},
1110}
1111
1112@inproceedings{guizzardi2020ethical,
1113  organization = {Springer},
1114  year = {2020},
1115  pages = {251--256},
1116  booktitle = {Canadian Conference on Artificial Intelligence},
1117  author = {Guizzardi, Renata and Amaral, Glenda and Guizzardi, Giancarlo and Mylopoulos, John},
1118  title = {Ethical Requirements for {AI} Systems},
1119}
1120
1121@article{meinert2020agile,
1122  publisher = {JMIR Publications Inc., Toronto, Canada},
1123  year = {2020},
1124  pages = {e19297},
1125  number = {2},
1126  volume = {6},
1127  journal = {JMIR Public Health and Surveillance},
1128  author = {Meinert, Edward and Milne-Ives, Madison and Surodina, Svitlana and Lam, Ching},
1129  title = {Agile requirements engineering and software planning for a digital health platform to engage the effects of isolation caused by social distancing: case study},
1130}
1131
1132@inproceedings{kaindl2020towards,
1133  year = {2020},
1134  booktitle = {REFSQ Workshops},
1135  author = {Kaindl, Hermann and Ferdigg, Jonas},
1136  title = {Towards Requirements Engineering for Superintelligence Safety},
1137}
1138
1139@inproceedings{kaindl2020towards,
1140  organization = {IEEE},
1141  year = {2020},
1142  pages = {33--38},
1143  booktitle = {2020 IEEE Seventh International Workshop on Artificial Intelligence for Requirements Engineering (AIRE)},
1144  author = {Kaindl, Hermann and Ferdigg, Jonas},
1145  title = {Towards an Extended Requirements Problem Formulation for Superintelligence Safety},
1146}
1147
1148@inproceedings{vergne2020trustworthy,
1149  year = {2020},
1150  booktitle = {REFSQ Workshops},
1151  author = {Vergne, Matthieu},
1152  title = {Trustworthy {AI}: Towards the Golden Age of {RE}?},
1153}
1154
1155@inproceedings{kostova2020interplay,
1156  year = {2020},
1157  booktitle = {REFSQ Workshops},
1158  author = {Kostova, Blagovesta and Gurses, Seda and Wegmann, Alain},
1159  title = {On the Interplay between Requirements, Engineering, and Artificial Intelligence.},
1160}
1161
1162@inproceedings{schafer2013technical,
1163  organization = {IEEE},
1164  year = {2013},
1165  pages = {1--9},
1166  booktitle = {2013 IEEE 18th Conference on Emerging Technologies \& Factory Automation (ETFA)},
1167  author = {Sch{\"a}fer, Stephan and Sch{\"o}ttke, Dirk and K{\"a}mpfe, Thomas and Berger, Ulrich},
1168  title = {Technical conditions for the use of autonomous systems: A general approach on an example},
1169}
1170
1171@article{crnkovic2012robots,
1172  publisher = {Springer},
1173  year = {2012},
1174  pages = {61--71},
1175  number = {1},
1176  volume = {14},
1177  journal = {Ethics and Information Technology},
1178  author = {Crnkovic, Gordana Dodig and {\c{C}}{\"u}r{\"u}kl{\"u}, Baran},
1179  title = {Robots: ethical by design},
1180}
1181
1182@article{kuwajima2020engineering,
1183  publisher = {Springer},
1184  year = {2020},
1185  pages = {1--24},
1186  journal = {Machine Learning},
1187  author = {Kuwajima, Hiroshi and Yasuoka, Hirotoshi and Nakae, Toshihiro},
1188  title = {Engineering problems in machine learning systems},
1189}
1190
1191@inproceedings{arruda2017towards,
1192  organization = {IEEE},
1193  year = {2017},
1194  pages = {2314--2319},
1195  booktitle = {2017 IEEE international conference on big data (big data)},
1196  author = {Arruda, Darlan and Madhavji, Nazim H},
1197  title = {Towards a requirements engineering artefact model in the context of big data software development projects: Research in progress},
1198}
1199
1200@article{felzmann2019transparency,
1201  publisher = {SAGE Publications Sage UK: London, England},
1202  year = {2019},
1203  pages = {2053951719860542},
1204  number = {1},
1205  volume = {6},
1206  journal = {Big Data \& Society},
1207  author = {Felzmann, Heike and Villaronga, Eduard Fosch and Lutz, Christoph and Tam{\`o}-Larrieux, Aurelia},
1208  title = {Transparency you can trust: Transparency requirements for artificial intelligence between legal norms and contextual concerns},
1209}
1210
1211@article{cuer2018formal,
1212  publisher = {Elsevier},
1213  year = {2018},
1214  pages = {29--40},
1215  volume = {174},
1216  journal = {Reliability Engineering \& System Safety},
1217  author = {Cuer, Romain and Pi{\'e}trac, Laurent and Niel, Eric and Diallo, Saidou and Minoiu-Enache, Nicoleta and Dang-Van-Nhan, Christophe},
1218  title = {A formal framework for the safe design of the Autonomous Driving supervision},
1219}
1220
1221@article{chazette2020explainability,
1222  publisher = {Springer},
1223  year = {2020},
1224  pages = {1--22},
1225  journal = {Requirements Engineering},
1226  author = {Chazette, Larissa and Schneider, Kurt},
1227  title = {Explainability as a non-functional requirement: challenges and recommendations},
1228}
1229
1230@book{murphy2012machine,
1231  publisher = {MIT press},
1232  year = {2012},
1233  author = {Murphy, Kevin P},
1234  title = {Machine learning: a probabilistic perspective},
1235}
1236
1237@inproceedings{wheatcraft2018communicating,
1238  organization = {Wiley Online Library},
1239  year = {2018},
1240  pages = {716--732},
1241  number = {1},
1242  volume = {28},
1243  booktitle = {INCOSE International Symposium},
1244  author = {Wheatcraft, Louis S and Ryan, Michael J},
1245  title = {Communicating Requirements--Effectively!},
1246}
1247
1248@article{amershi2014power,
1249  year = {2014},
1250  pages = {105--120},
1251  number = {4},
1252  volume = {35},
1253  journal = {Ai Magazine},
1254  author = {Amershi, Saleema and Cakmak, Maya and Knox, William Bradley and Kulesza, Todd},
1255  title = {Power to the people: The role of humans in interactive machine learning},
1256}
1257
1258@article{how2020artificial,
1259  publisher = {Multidisciplinary Digital Publishing Institute},
1260  year = {2020},
1261  pages = {39},
1262  number = {1},
1263  volume = {11},
1264  journal = {Information},
1265  author = {How, Meng-Leong and Cheah, Sin-Mei and Chan, Yong-Jiet and Khor, Aik Cheow and Say, Eunice Mei Ping},
1266  title = {Artificial intelligence-enhanced decision support for informing global sustainable development: A human-centric {AI}-thinking approach},
1267}
1268
1269@inproceedings{wang2019designing,
1270  year = {2019},
1271  pages = {1--15},
1272  booktitle = {Proceedings of the 2019 CHI conference on human factors in computing systems},
1273  author = {Wang, Danding and Yang, Qian and Abdul, Ashraf and Lim, Brian Y},
1274  title = {Designing theory-driven user-centric explainable {AI}},
1275}
1276
1277@inproceedings{dodge2019explaining,
1278  year = {2019},
1279  pages = {275--285},
1280  booktitle = {Proceedings of the 24th International Conference on Intelligent User Interfaces},
1281  author = {Dodge, Jonathan and Liao, Q Vera and Zhang, Yunfeng and Bellamy, Rachel KE and Dugan, Casey},
1282  title = {Explaining models: an empirical study of how explanations impact fairness judgment},
1283}
1284
1285@inproceedings{alatawi2018psychologically,
1286  organization = {IEEE},
1287  year = {2018},
1288  pages = {41--50},
1289  booktitle = {2018 25th Australasian Software Engineering Conference (ASWEC)},
1290  author = {Alatawi, Eman and Mendoza, Antonette and Miller, Tim},
1291  title = {Psychologically-driven requirements engineering: A case study in depression care},
1292}
1293
1294@inproceedings{teruel2011comparing,
1295  organization = {Springer},
1296  year = {2011},
1297  pages = {169--184},
1298  booktitle = {International Conference on Evaluation of Novel Approaches to Software Engineering},
1299  author = {Teruel, Miguel A and Navarro, Elena and L{\'o}pez-Jaquero, V{\'\i}ctor and Montero, Francisco and Gonz{\'a}lez, Pascual},
1300  title = {Comparing goal-oriented approaches to model requirements for {CSCW}},
1301}
1302
1303@article{dalpiaz2016istar,
1304  year = {2016},
1305  journal = {arXiv preprint arXiv:1605.07767},
1306  author = {Dalpiaz, Fabiano and Franch, Xavier and Horkoff, Jennifer},
1307  title = {istar 2.0 language guide},
1308}
1309
1310@inproceedings{van2001goal,
1311  organization = {IEEE},
1312  year = {2001},
1313  pages = {249--262},
1314  booktitle = {Proceedings fifth ieee international symposium on requirements engineering},
1315  author = {Van Lamsweerde, Axel},
1316  title = {Goal-oriented requirements engineering: A guided tour},
1317}
1318
1319@article{lapouchnian2005goal,
1320  year = {2005},
1321  volume = {32},
1322  journal = {University of Toronto},
1323  author = {Lapouchnian, Alexei},
1324  title = {Goal-oriented requirements engineering: An overview of the current research},
1325}
1326
1327@article{horkoff2013comparison,
1328  publisher = {Springer},
1329  year = {2013},
1330  pages = {199--222},
1331  number = {3},
1332  volume = {18},
1333  journal = {Requirements Engineering},
1334  author = {Horkoff, Jennifer and Yu, Eric},
1335  title = {Comparison and evaluation of goal-oriented satisfaction analysis techniques},
1336}
1337
1338@inproceedings{colomo2010study,
1339  organization = {Springer},
1340  year = {2010},
1341  pages = {1--7},
1342  booktitle = {World Summit on Knowledge Society},
1343  author = {Colomo-Palacios, Ricardo and Hern{\'a}ndez-L{\'o}pez, Adri{\'a}n and Garc{\'\i}a-Crespo, {\'A}ngel and Soto-Acosta, Pedro},
1344  title = {A study of emotions in requirements engineering},
1345}
1346
1347@book{crotty1998foundations,
1348  publisher = {Sage},
1349  year = {1998},
1350  author = {Crotty, Michael},
1351  title = {The foundations of social research: Meaning and perspective in the research process},
1352}
1353
1354@book{holstein2013handbook,
1355  publisher = {Guilford Publications},
1356  year = {2013},
1357  author = {Holstein, James A and Gubrium, Jaber F},
1358  title = {Handbook of constructionist research},
1359}
1360
1361@article{berland2014educational,
1362  publisher = {Springer},
1363  year = {2014},
1364  pages = {205--220},
1365  number = {1-2},
1366  volume = {19},
1367  journal = {Technology, Knowledge and Learning},
1368  author = {Berland, Matthew and Baker, Ryan S and Blikstein, Paulo},
1369  title = {Educational data mining and learning analytics: Applications to constructionist research},
1370}
1371
1372@article{gonccalves2019understanding,
1373  publisher = {Springer},
1374  year = {2019},
1375  pages = {55--84},
1376  number = {1},
1377  volume = {24},
1378  journal = {Requirements Engineering},
1379  author = {Gon{\c{c}}alves, Enyo and de Oliveira, Marcos Ant{\^o}nio and Monteiro, Ingrid and Castro, Jaelson and Ara{\'u}jo, Jo{\~a}o},
1380  title = {Understanding what is important in i{S}tar extension proposals: the viewpoint of researchers},
1381}
1382
1383@book{friedenthal2014practical,
1384  publisher = {Morgan Kaufmann},
1385  year = {2014},
1386  author = {Friedenthal, Sanford and Moore, Alan and Steiner, Rick},
1387  title = {A practical guide to {SysML}: the systems modeling language},
1388}
1389
1390@inproceedings{palomares2018personal,
1391  organization = {Springer},
1392  year = {2018},
1393  pages = {297--304},
1394  booktitle = {International working conference on requirements engineering: foundation for software quality},
1395  author = {Palomares, Cristina and Franch, Xavier and Fucci, Davide},
1396  title = {Personal recommendations in requirements engineering: the {OpenReq} approach},
1397}
1398
1399@book{daugherty2018human,
1400  publisher = {Harvard Business Press},
1401  year = {2018},
1402  author = {Daugherty, Paul R and Wilson, H James},
1403  title = {Human+ machine: Reimagining work in the age of {AI}},
1404}
1405
1406@inproceedings{bach2017data,
1407  organization = {IEEE},
1408  year = {2017},
1409  pages = {1--6},
1410  booktitle = {2017 IEEE International Systems Engineering Symposium (ISSE)},
1411  author = {Bach, Johannes and Langner, Jacob and Otten, Stefan and Holz{\"a}pfel, Marc and Sax, Eric},
1412  title = {Data-driven development, a complementing approach for automotive systems engineering},
1413}
1414
1415@incollection{bennaceur2019requirements,
1416  publisher = {Springer},
1417  year = {2019},
1418  pages = {51--92},
1419  booktitle = {Handbook of Software Engineering},
1420  author = {Bennaceur, Amel and Tun, Thein Than and Yu, Yijun and Nuseibeh, Bashar},
1421  title = {Requirements Engineering},
1422}
1423
1424@book{cooper2004inmates,
1425  publisher = {Sams Indianapolis},
1426  year = {2004},
1427  volume = {2},
1428  author = {Cooper, Alan and others},
1429  title = {The inmates are running the asylum: Why high-tech products drive us crazy and how to restore the sanity},
1430}
1431
1432@inproceedings{kaiya2002agora,
1433  organization = {Ieee},
1434  year = {2002},
1435  pages = {13--22},
1436  booktitle = {Proceedings IEEE joint international conference on requirements engineering},
1437  author = {Kaiya, Haruhiko and Horai, Hisayuki and Saeki, Motoshi},
1438  title = {{AGORA}: Attributed goal-oriented requirements analysis method},
1439}
1440
1441@inproceedings{anton1998GBRAM,
1442  organization = {IEEE},
1443  year = {1998},
1444  pages = {157--166},
1445  booktitle = {Proceedings of the 20th international Conference on Software Engineering},
1446  author = {Ant{\'o}n, Annie I and Potts, Colin},
1447  title = {The use of goals to surface requirements for evolving systems},
1448}
1449
1450@book{chung2012NRL,
1451  publisher = {Springer Science \& Business Media},
1452  year = {2012},
1453  volume = {5},
1454  author = {Chung, Lawrence and Nixon, Brian A and Yu, Eric and Mylopoulos, John},
1455  title = {Non-functional requirements in software engineering},
1456}
1457
1458@article{bresciani2004tropos,
1459  publisher = {Springer},
1460  year = {2004},
1461  pages = {203--236},
1462  number = {3},
1463  volume = {8},
1464  journal = {Autonomous Agents and Multi-Agent Systems},
1465  author = {Bresciani, Paolo and Perini, Anna and Giorgini, Paolo and Giunchiglia, Fausto and Mylopoulos, John},
1466  title = {Tropos: An agent-oriented software development methodology},
1467}
1468
1469@article{liu2004designing,
1470  publisher = {Elsevier},
1471  year = {2004},
1472  pages = {187--203},
1473  number = {2},
1474  volume = {29},
1475  journal = {Information systems},
1476  author = {Liu, Lin and Yu, Eric},
1477  title = {Designing information systems in social context: a goal and scenario modelling approach},
1478}
1479
1480@article{agrawal2019cloudy,
1481  year = {2019},
1482  journal = {10th Annual Conference on Innovative Data Systems Research (CIDR ‘20)},
1483  author = {Agrawal, Ashvin and Chatterjee, Rony and Curino, Carlo and Floratou, Avrilia and Gowdal, Neha and Interlandi, Matteo and Jindal, Alekh and Karanasos, Kostantinos and Krishnan, Subru and Kroth, Brian and others},
1484  title = {Cloudy with high chance of {DBMS}: A 10-year prediction for Enterprise-Grade {ML}},
1485}
1486
1487@article{amyot2011grl,
1488  year = {2011},
1489  pages = {160--162},
1490  volume = {766},
1491  journal = {iStar},
1492  author = {Amyot, Daniel and Mussbacher, Gunter and Ghanavati, Sepideh and Kealey, Jason},
1493  title = {{GRL} Modeling and Analysis with {jUCMNav}},
1494}
1495
1496@inproceedings{baresi2010fuzzy,
1497  organization = {IEEE},
1498  year = {2010},
1499  pages = {125--134},
1500  booktitle = {2010 18th IEEE international requirements engineering conference},
1501  author = {Baresi, Luciano and Pasquale, Liliana and Spoletini, Paola},
1502  title = {Fuzzy goals for requirements-driven adaptation},
1503}
1504
1505@incollection{bartocci2018STL,
1506  publisher = {Springer},
1507  year = {2018},
1508  pages = {135--175},
1509  booktitle = {Lectures on Runtime Verification},
1510  author = {Bartocci, Ezio and Deshmukh, Jyotirmoy and Donz{\'e}, Alexandre and Fainekos, Georgios and Maler, Oded and Ni{\v{c}}kovi{\'c}, Dejan and Sankaranarayanan, Sriram},
1511  title = {Specification-based monitoring of cyber-physical systems: a survey on theory, tools and applications},
1512}
1513
1514@article{bellamy2019Fairness360,
1515  publisher = {IBM},
1516  year = {2019},
1517  pages = {4--1},
1518  number = {4/5},
1519  volume = {63},
1520  journal = {IBM Journal of Research and Development},
1521  author = {Bellamy, Rachel KE and Dey, Kuntal and Hind, Michael and Hoffman, Samuel C and Houde, Stephanie and Kannan, Kalapriya and Lohia, Pranay and Martino, Jacquelyn and Mehta, Sameep and Mojsilovi{\'c}, Aleksandra and others},
1522  title = {A{I} Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias},
1523}
1524
1525@inproceedings{tramer2017fairtest,
1526  organization = {IEEE},
1527  year = {2017},
1528  pages = {401--416},
1529  booktitle = {2017 IEEE European Symposium on Security and Privacy (EuroS\&P)},
1530  author = {Tramer, Florian and Atlidakis, Vaggelis and Geambasu, Roxana and Hsu, Daniel and Hubaux, Jean-Pierre and Humbert, Mathias and Juels, Ari and Lin, Huang},
1531  title = {Fairtest: Discovering unwarranted associations in data-driven applications},
1532}
1533
1534@book{urquhart2012grounded,
1535  publisher = {Sage},
1536  year = {2012},
1537  author = {Urquhart, Cathy},
1538  title = {Grounded theory for qualitative research: A practical guide},
1539}
1540
1541@inproceedings{wohlin2014guidelines,
1542  year = {2014},
1543  pages = {1--10},
1544  booktitle = {Proceedings of the 18th international conference on evaluation and assessment in software engineering},
1545  author = {Wohlin, Claes},
1546  title = {Guidelines for snowballing in systematic literature studies and a replication in software engineering},
1547}
1548
1549@incollection{easterbrook2008selecting,
1550  publisher = {Springer},
1551  year = {2008},
1552  pages = {285--311},
1553  booktitle = {Guide to advanced empirical software engineering},
1554  author = {Easterbrook, Steve and Singer, Janice and Storey, Margaret-Anne and Damian, Daniela},
1555  title = {Selecting empirical methods for software engineering research},
1556}
1557
1558@inproceedings{vassev2013autonomy,
1559  organization = {IEEE},
1560  year = {2013},
1561  pages = {1--10},
1562  booktitle = {16th IEEE International Symposium on Object/component/service-oriented Real-time distributed Computing (ISORC 2013)},
1563  author = {Vassev, Emil and Hinchey, Mike},
1564  title = {On the autonomy requirements for space missions},
1565}
1566
1567@inproceedings{nuseibeh2000requirements,
1568  year = {2000},
1569  pages = {35--46},
1570  booktitle = {Proceedings of the Conference on the Future of Software Engineering},
1571  author = {Nuseibeh, Bashar and Easterbrook, Steve},
1572  title = {Requirements engineering: a roadmap},
1573}
1574
1575@article{tellis1997application,
1576  year = {1997},
1577  pages = {1--19},
1578  number = {3},
1579  volume = {3},
1580  journal = {The qualitative report},
1581  author = {Tellis, Winston},
1582  title = {Application of a case study methodology},
1583}
1584
1585@article{meyer2001case,
1586  publisher = {Sage Publications Sage CA: Thousand Oaks, CA},
1587  year = {2001},
1588  pages = {329--352},
1589  number = {4},
1590  volume = {13},
1591  journal = {Field methods},
1592  author = {Meyer, Christine Benedichte},
1593  title = {A case in case study methodology},
1594}
1595
1596@article{glasow2005fundamentals,
1597  year = {2005},
1598  pages = {2013},
1599  volume = {18},
1600  journal = {Retrieved January},
1601  author = {Glasow, Priscilla A},
1602  title = {Fundamentals of survey research methodology},
1603}
1604
1605@article{gable1994integrating,
1606  publisher = {Taylor \& Francis},
1607  year = {1994},
1608  pages = {112--126},
1609  number = {2},
1610  volume = {3},
1611  journal = {European journal of information systems},
1612  author = {Gable, Guy G},
1613  title = {Integrating case study and survey research methods: an example in information systems},
1614}
1615
1616@inproceedings{yu2020bdd100k,
1617  year = {2020},
1618  pages = {2636--2645},
1619  booktitle = {Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
1620  author = {Yu, Fisher and Chen, Haofeng and Wang, Xin and Xian, Wenqi and Chen, Yingying and Liu, Fangchen and Madhavan, Vashisht and Darrell, Trevor},
1621  title = {Bdd100k: A diverse driving dataset for heterogeneous multitask learning},
1622}
1623
1624@inproceedings{viyovic2014sirius,
1625  organization = {IEEE},
1626  year = {2014},
1627  pages = {233--238},
1628  booktitle = {IEEE 18th International Conference on Intelligent Engineering Systems INES 2014},
1629  author = {Viyovi{\'c}, Vladimir and Maksimovi{\'c}, Mirjam and Perisi{\'c}, Branko},
1630  title = {Sirius: A rapid development of DSM graphical editor},
1631}
1632
1633@inproceedings{bosch2018takes,
1634  year = {2018},
1635  pages = {177--192},
1636  booktitle = {SiBW},
1637  author = {Bosch, Jan and Olsson, Helena Holmstr{\"o}m and Crnkovic, Ivica},
1638  title = {It takes three to tango: Requirement, outcome/data, and {AI} driven development.},
1639}
1640
1641@inproceedings{bonfe2012towards,
1642  organization = {IEEE},
1643  year = {2012},
1644  pages = {56--61},
1645  booktitle = {2012 4th IEEE RAS \& EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob)},
1646  author = {Bonfe, Marcello and Boriero, Fabrizio and Dodi, Riccardo and Fiorini, Paolo and Morandi, Angelica and Muradore, Riccardo and Pasquale, Liliana and Sanna, Alberto and Secchi, Cristian},
1647  title = {Towards automated surgical robotics: A requirements engineering approach},
1648}
1649
1650@inproceedings{altarturi2017requirement,
1651  organization = {IEEE},
1652  year = {2017},
1653  pages = {111--117},
1654  booktitle = {2017 IEEE Conference on Big Data and Analytics (ICBDA)},
1655  author = {Altarturi, Hamza Hussein and Ng, Keng-Yap and Ninggal, Mohd Izuan Hafez and Nazri, Azree Shahrel Ahmad and Abd Ghani, Abdul Azim},
1656  title = {A requirement engineering model for big data software},
1657}
1658
1659@inproceedings{hu2020towards,
1660  organization = {IEEE},
1661  year = {2020},
1662  pages = {},
1663  booktitle = {Workshop on Artificial Intelligence for Requirements Engineering (AIRE)},
1664  author = {Hu, Boyue Caroline and Salay, Rick and Czarnecki, Krzysztof and Rahimi, Mona and Selim, Gehan and Chechik, Marsha},
1665  title = {Towards Requirements Specification for Machine-learned Perception Based on Human Performance},
1666}
1667
1668@inproceedings{gruber2017integrated,
1669  organization = {IEEE},
1670  year = {2017},
1671  pages = {27--31},
1672  booktitle = {2017 7th IEEE International Conference on System Engineering and Technology (ICSET)},
1673  author = {Gruber, Kristina and Huemer, Jakob and Zimmermann, Armin and Maschotta, Ralph},
1674  title = {Integrated description of functional and non-functional requirements for automotive systems design using {SysML}},
1675}
1676
1677@inproceedings{sandkuhl2019putting,
1678  organization = {IEEE},
1679  year = {2019},
1680  pages = {157--164},
1681  volume = {1},
1682  booktitle = {2019 IEEE 21st Conference on Business Informatics (CBI)},
1683  author = {Sandkuhl, Kurt},
1684  title = {Putting {AI} into Context-Method Support for the Introduction of Artificial Intelligence into Organizations},
1685}
1686
1687@inproceedings{rahimi2019toward,
1688  organization = {IEEE},
1689  year = {2019},
1690  pages = {241--244},
1691  booktitle = {2019 IEEE 27th International Requirements Engineering Conference Workshops (REW)},
1692  author = {Rahimi, Mona and Guo, Jin LC and Kokaly, Sahar and Chechik, Marsha},
1693  title = {Toward Requirements Specification for Machine-Learned Components},
1694}
1695
1696@inproceedings{jakob2017defining,
1697  organization = {IEEE},
1698  year = {2017},
1699  pages = {128--133},
1700  booktitle = {2017 IEEE 14th International Scientific Conference on Informatics},
1701  author = {Jakob, Judith and Kugele, Kordula and Tick, J{\'o}zsef},
1702  title = {Defining camera-based traffic scenarios and use cases for the visually impaired by means of expert interviews},
1703}
1704
1705@article{vogelsang2019requirements,
1706  year = {2019},
1707  journal = {IEEE International Requirements Engineering Conference Workshops},
1708  author = {Vogelsang, Andreas and Borg, Markus},
1709  title = {Requirements Engineering for Machine Learning: Perspectives from Data Scientists},
1710}
1711
1712@article{tuncali2019requirements,
1713  publisher = {IEEE},
1714  year = {2019},
1715  pages = {265--280},
1716  number = {2},
1717  volume = {5},
1718  journal = {IEEE Transactions on Intelligent Vehicles},
1719  author = {Tuncali, Cumhur Erkan and Fainekos, Georgios and Prokhorov, Danil and Ito, Hisahiro and Kapinski, James},
1720  title = {Requirements-driven test generation for autonomous vehicles with machine learning components},
1721}
1722
1723@inproceedings{becker2020partial,
1724  year = {2020},
1725  booktitle = {Software Engineering (Workshops)},
1726  author = {Becker, Jan Steffen},
1727  title = {Partial Consistency for Requirement Engineering with Traffic Sequence Charts.},
1728}
1729
1730@article{shin2019data,
1731  publisher = {Multidisciplinary Digital Publishing Institute},
1732  year = {2019},
1733  pages = {1696},
1734  number = {9},
1735  volume = {12},
1736  journal = {Energies},
1737  author = {Shin, Changho and Rho, Seungeun and Lee, Hyoseop and Rhee, Wonjong},
1738  title = {Data requirements for applying machine learning to energy disaggregation},
1739}
1740
1741@article{dimitrakopoulos2019alpha,
1742  publisher = {Elsevier},
1743  year = {2019},
1744  pages = {28--47},
1745  volume = {91},
1746  journal = {Simulation Modelling Practice and Theory},
1747  author = {Dimitrakopoulos, George and Kavakli, Evangelia and Loucopoulos, Peri and Anagnostopoulos, Dimosthenis and Zographos, Theodoros},
1748  title = {A capability-oriented modelling and simulation approach for autonomous vehicle management},
1749}
1750
1751@article{weihrauch2018conceptual,
1752  publisher = {Elsevier},
1753  year = {2018},
1754  pages = {386--391},
1755  volume = {67},
1756  journal = {Procedia CIRP},
1757  author = {Weihrauch, Dennis and Schindler, Paul Anton and Sihn, Wilfried},
1758  title = {A conceptual model for developing a smart process control system},
1759}
1760
1761@article{fenn2016addressing,
1762  publisher = {Springer},
1763  year = {2016},
1764  pages = {77--86},
1765  number = {1},
1766  volume = {27},
1767  journal = {Machine Vision and Applications},
1768  author = {Fenn, Shannon and Mendes, Alexandre and Budden, David M},
1769  title = {Addressing the non-functional requirements of computer vision systems: a case study},
1770}
1771
1772@article{neace2018goal,
1773  publisher = {Springer},
1774  year = {2018},
1775  pages = {509--555},
1776  number = {4},
1777  volume = {23},
1778  journal = {Requirements Engineering},
1779  author = {Neace, Kerry and Roncace, Robert and Fomin, Pavel},
1780  title = {Goal model analysis of autonomy requirements for Unmanned Aircraft Systems},
1781}
1782
1783@inproceedings{lockerbie2020using,
1784  year = {2020},
1785  booktitle = {REFSQ Workshops},
1786  author = {Lockerbie, James and Maiden, Neil AM},
1787  title = {Using a Requirements Modelling Language to Co-Design Intelligent Support for People Living with Dementia.},
1788}
1789
1790@inproceedings{nakamichi2020requirements,
1791  organization = {IEEE},
1792  year = {2020},
1793  pages = {260--270},
1794  booktitle = {2020 IEEE 28th International Requirements Engineering Conference (RE)},
1795  author = {Nakamichi, Koji and Ohashi, Kyoko and Namba, Isao and Yamamoto, Rieko and Aoyama, Mikio and Joeckel, Lisa and Siebert, Julien and Heidrich, Jens},
1796  title = {Requirements-driven method to determine quality characteristics and measurements for machine learning software and its evaluation},
1797}
1798
1799@inproceedings{challa2020faulty,
1800  organization = {IEEE},
1801  year = {2020},
1802  pages = {61--69},
1803  booktitle = {2020 IEEE Seventh International Workshop on Artificial Intelligence for Requirements Engineering (AIRE)},
1804  author = {Challa, Harshitha and Niu, Nan and Johnson, Reese},
1805  title = {Faulty Requirements Made Valuable: On the Role of Data Quality in Deep Learning},
1806}
1807
1808@inproceedings{aydemir2018roadmap,
1809  organization = {IEEE},
1810  year = {2018},
1811  pages = {15--21},
1812  booktitle = {2018 IEEE/ACM International Workshop on Software Fairness (FairWare)},
1813  author = {Aydemir, Fatma Basak and Dalpiaz, Fabiano},
1814  title = {A roadmap for ethics-aware software engineering},
1815}
1816
1817@inproceedings{horkoff2019nonFunctional,
1818  organization = {IEEE},
1819  year = {2019},
1820  pages = {386--391},
1821  booktitle = {2019 IEEE 27th International Requirements Engineering Conference (RE)},
1822  author = {Horkoff, Jennifer},
1823  title = {Non-functional requirements for machine learning: Challenges and new directions},
1824}
1825
1826@inproceedings{kohl2019explainability,
1827  organization = {IEEE},
1828  year = {2019},
1829  pages = {363--368},
1830  booktitle = {2019 IEEE 27th International Requirements Engineering Conference (RE)},
1831  author = {K{\"o}hl, Maximilian A and Baum, Kevin and Langer, Markus and Oster, Daniel and Speith, Timo and Bohlender, Dimitri},
1832  title = {Explainability as a non-functional requirement},
1833}
1834
1835@inproceedings{ang2011requirement,
1836  organization = {IEEE},
1837  year = {2011},
1838  pages = {640--645},
1839  booktitle = {2011 IEEE Symposium on Computers \& Informatics},
1840  author = {Ang, Jac Ky and Leong, Sook Bing and Lee, Chin Fei and Yusof, Umi Kalsom},
1841  title = {Requirement engineering techniques in developing expert systems},
1842}
1843
1844@inproceedings{ishikawa2020evidence,
1845  organization = {IEEE},
1846  year = {2020},
1847  pages = {346--351},
1848  booktitle = {2020 IEEE 28th International Requirements Engineering Conference (RE)},
1849  author = {Ishikawa, Fuyuki and Matsuno, Yutaka},
1850  title = {Evidence-driven Requirements Engineering for Uncertainty of Machine Learning-based Systems},
1851}
1852
1853@inproceedings{kuwajima2019adapting,
1854  organization = {IEEE},
1855  year = {2019},
1856  pages = {13--18},
1857  booktitle = {2019 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)},
1858  author = {Kuwajima, Hiroshi and Ishikawa, Fuyuki},
1859  title = {Adapting {SQuaRE} for Quality Assessment of Artificial Intelligence Systems},
1860}
1861
1862@inproceedings{amaral2020ontology,
1863  organization = {Springer},
1864  year = {2020},
1865  pages = {342--352},
1866  booktitle = {International Conference on Conceptual Modeling},
1867  author = {Amaral, Glenda and Guizzardi, Renata and Guizzardi, Giancarlo and Mylopoulos, John},
1868  title = {Ontology-Based Modeling and Analysis of Trustworthiness Requirements: Preliminary Results},
1869}
1870
1871@inproceedings{bruno2013functional,
1872  organization = {IEEE},
1873  year = {2013},
1874  pages = {768--773},
1875  booktitle = {2013 IEEE RO-MAN},
1876  author = {Bruno, Barbara and Mastrogiovanni, Fulvio and Sgorbissa, Antonio},
1877  title = {Functional requirements and design issues for a socially assistive robot for elderly people with mild cognitive impairments},
1878}
1879
1880@inproceedings{ries2021mde,
1881  organization = {SCITEPRESS},
1882  year = {2021},
1883  pages = {41--52},
1884  booktitle = {Proceedings of the 9th International Conference on Model-Driven Engineering and Software Development},
1885  author = {Ries, Benoit and Guelfi, Nicolas and Jahic, Benjamin},
1886  title = {An {MDE} Method for Improving Deep Learning Dataset Requirements Engineering using Alloy and {UML}},
1887}
1888
1889@inproceedings{olmos2020helping,
1890  organization = {IEEE},
1891  year = {2020},
1892  pages = {1--10},
1893  booktitle = {2020 8th International Conference in Software Engineering Research and Innovation (CONISOFT)},
1894  author = {Olmos-S{\'a}nchez, Karla and Rodas-Osollo, Jorge},
1895  title = {Helping organizations manage the innovation process to join the Cognitive era},
1896}
1897
1898@inproceedings{agrawal2020model,
1899  organization = {IEEE},
1900  year = {2020},
1901  pages = {1--10},
1902  booktitle = {2020 IEEE Tenth International Model-Driven Requirements Engineering (MoDRE)},
1903  author = {Agrawal, Ankit and Cleland-Huang, Jane and Stegh{\"o}fer, Jan-Philipp},
1904  title = {Model-driven requirements for humans-on-the-loop multi-uav missions},
1905}
1906
1907@inproceedings{samin2021towards,
1908  year = {2021},
1909  pages = {1328--1337},
1910  booktitle = {Proceedings of the 36th Annual ACM Symposium on Applied Computing},
1911  author = {Samin, Huma and Paucar, Luis H Garcia and Bencomo, Nelly and Sawyer, Peter},
1912  title = {Towards priority-awareness in autonomous intelligent systems},
1913}
1914
1915@inproceedings{cirqueira2020scenario,
1916  organization = {Springer},
1917  year = {2020},
1918  pages = {321--341},
1919  booktitle = {International Cross-Domain Conference for Machine Learning and Knowledge Extraction},
1920  author = {Cirqueira, Douglas and Nedbal, Dietmar and Helfert, Markus and Bezbradica, Marija},
1921  title = {Scenario-based requirements elicitation for user-centric explainable ai},
1922}
1923
1924@article{ntakolia2020user,
1925  publisher = {Springer},
1926  year = {2020},
1927  pages = {1--26},
1928  journal = {Universal Access in the Information Society},
1929  author = {Ntakolia, Charis and Dimas, George and Iakovidis, Dimitris K},
1930  title = {User-centered system design for assisted navigation of visually impaired individuals in outdoor cultural environments},
1931}
1932
1933@article{nalchigar2021modeling,
1934  publisher = {Springer},
1935  year = {2021},
1936  pages = {237--254},
1937  number = {2},
1938  volume = {26},
1939  journal = {Requirements Engineering},
1940  author = {Nalchigar, Soroosh and Yu, Eric and Keshavjee, Karim},
1941  title = {Modeling machine learning requirements from three perspectives: a case report from the healthcare domain},
1942}
1943
1944@article{nalchigar2018business,
1945  publisher = {Elsevier},
1946  year = {2018},
1947  pages = {359--372},
1948  volume = {117},
1949  journal = {Data \& Knowledge Engineering},
1950  author = {Nalchigar, Soroosh and Yu, Eric},
1951  title = {Business-driven data analytics: a conceptual modeling framework},
1952}
1953
1954@article{schoonderwoerd2021human,
1955  publisher = {Elsevier},
1956  year = {2021},
1957  pages = {102684},
1958  journal = {International Journal of Human-Computer Studies},
1959  author = {Schoonderwoerd, Tjeerd AJ and Jorritsma, Wiard and Neerincx, Mark A and van den Bosch, Karel},
1960  title = {Human-Centered XAI: Developing Design Patterns for Explanations of Clinical Decision Support Systems},
1961}
1962
1963@inproceedings{islam2021mobile,
1964  organization = {Springer},
1965  year = {2021},
1966  pages = {33--42},
1967  booktitle = {International Conference on Computational Intelligence in Information System},
1968  author = {Islam, Muhammad Nazrul and Khan, Shahriar Rahman and Islam, Noor Nafiz and Rezwan-A-Rownok, Md and Zaman, Syed Rohit and Zaman, Samiha Raisa},
1969  title = {A Mobile Application for Mental Health Care During COVID-19 Pandemic: Development and Usability Evaluation with System Usability Scale},
1970}
1971
1972@inproceedings{cleland2020requirements,
1973  year = {2020},
1974  pages = {1--12},
1975  booktitle = {Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume A-Volume A},
1976  author = {Cleland-Huang, Jane and Agrawal, Ankit and Islam, Md Nafee Al and Tsai, Eric and Van Speybroeck, Maxime and Vierhauser, Michael},
1977  title = {Requirements-driven configuration of emergency response missions with small aerial vehicles},
1978}
1979
1980@article{clauer2021usage,
1981  publisher = {Elsevier},
1982  year = {2021},
1983  pages = {242--247},
1984  volume = {96},
1985  journal = {Procedia CIRP},
1986  author = {Clauer, Dana and Fottner, Johannes and Rauch, Erwin and Pr{\"u}glmeier, Marco},
1987  title = {Usage of Autonomous Mobile Robots Outdoors-an Axiomatic Design Approach},
1988}
1989
1990@inproceedings{hall2019systematic,
1991  year = {2019},
1992  volume = {11},
1993  booktitle = {Proceedings of the IJCAI Workshop on eXplainable Artificial Intelligence (XAI 2019), Macau, China},
1994  author = {Hall, Mark and Harborne, Daniel and Tomsett, Richard and Galetic, Vedran and Quintana-Amate, Santiago and Nottle, Alistair and Preece, Alun},
1995  title = {A systematic method to understand requirements for explainable AI (XAI) systems},
1996}
1997
1998@article{habibullah2021non,
1999  year = {2021},
2000  journal = {IEEE 29th International Requirements Engineering Conference (RE)},
2001  author = {Habibullah, Khan Mohammad and Horkoff, Jennifer},
2002  title = {Non-functional Requirements for Machine Learning: Understanding Current Use and Challenges in Industry},
2003}
2004
2005@inproceedings{rivero2021lessons,
2006  organization = {Springer},
2007  year = {2021},
2008  pages = {227--243},
2009  booktitle = {International Conference on Human-Computer Interaction},
2010  author = {Rivero, Luis and Portela, Carlos and Boaro, Jos{\'e} and Santos, Pedro and Rego, Venicius and Braz Junior, Geraldo and Paiva, Anselmo and Alves, Erika and Oliveira, Milton and Moraes, Renato and others},
2011  title = {Lessons Learned from Applying Requirements and Design Techniques in the Development of a Machine Learning System for Predicting Lawsuits Against Power Companies},
2012}
2013
2014@inproceedings{khatamino2021nlp,
2015  year = {2021},
2016  booktitle = {REFSQ Workshops},
2017  author = {Khatamino, Pedram and Camli, Mustafa Baris and {\"O}ztekin, Bilgehan and Gozumoglu, Umutcan and Tortumlu, Emre and Gezer, Huseyin Murat},
2018  title = {An NLP-based Chatbot to Facilitate RE Activities: An Experience Paper on Human Resources Application.},
2019}
2020
2021@inproceedings{schwammberger2021quest,
2022  organization = {IEEE},
2023  year = {2021},
2024  pages = {195--199},
2025  booktitle = {2021 IEEE 29th International Requirements Engineering Conference Workshops (REW)},
2026  author = {Schwammberger, Maike},
2027  title = {A Quest of Self-Explainability: When Causal Diagrams meet Autonomous Urban Traffic Manoeuvres},
2028}
2029
2030@inproceedings{ahmad2021s,
2031  organization = {IEEE},
2032  year = {2021},
2033  pages = {1--12},
2034  booktitle = {2021 IEEE 29th International Requirements Engineering Conference (RE)},
2035  author = {Ahmad, Khlood and Bano, Muneera and Abdelrazek, Mohamed and Arora, Chetan and Grundy, John},
2036  title = {What’s up with Requirements Engineering for Artificial Intelligence Systems?},
2037}
2038
2039@article{shneiderman2021human,
2040  publisher = {Issues in Science and Technology},
2041  year = {2021},
2042  pages = {56--61},
2043  number = {2},
2044  volume = {37},
2045  journal = {Issues in Science and Technology},
2046  author = {Shneiderman, Ben},
2047  title = {Human-centered AI},
2048}
2049
2050@incollection{frank2013domain,
2051  publisher = {Springer},
2052  year = {2013},
2053  pages = {133--157},
2054  booktitle = {Domain engineering},
2055  author = {Frank, Ulrich},
2056  title = {Domain-specific modeling languages: requirements analysis and design guidelines},
2057}
2058
2059@inproceedings{villamizar2021requirements,
2060  organization = {IEEE},
2061  year = {2021},
2062  pages = {29--36},
2063  booktitle = {2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)},
2064  author = {Villamizar, Hugo and Escovedo, Tatiana and Kalinowski, Marcos},
2065  title = {Requirements engineering for machine learning: A systematic mapping study},
2066}
2067
2068@article{sokol2020one,
2069  publisher = {Springer},
2070  year = {2020},
2071  pages = {235--250},
2072  number = {2},
2073  volume = {34},
2074  journal = {KI-K{\"u}nstliche Intelligenz},
2075  author = {Sokol, Kacper and Flach, Peter},
2076  title = {One explanation does not fit all},
2077}
2078
2079@article{felfernig2000uml,
2080  publisher = {World Scientific},
2081  year = {2000},
2082  pages = {449--469},
2083  number = {04},
2084  volume = {10},
2085  journal = {International Journal of Software Engineering and Knowledge Engineering},
2086  author = {Felfernig, Alexander and Friedrich, Gerhard E and Jannach, Dietmar},
2087  title = {UML as domain specific language for the construction of knowledge-based configuration systems},
2088}
2089
2090@inproceedings{silva2018requirements,
2091  year = {2018},
2092  booktitle = {proc. of the XXII Brazilian congress on automation. Brazilian Society of Automation},
2093  author = {Silva, J and Salmon, A and del Foyo, P and Silva, J and Mec{\^a}nica, Depto de Eng},
2094  title = {Requirements engineering at a glance: Comparing gore and uml methods in the design of automated systems},
2095}
2096
2097@book{kelly2008domain,
2098  publisher = {John Wiley \& Sons},
2099  year = {2008},
2100  author = {Kelly, Steven and Tolvanen, Juha-Pekka},
2101  title = {Domain-specific modeling: enabling full code generation},
2102}
2103
2104@book{fowler2004uml,
2105  publisher = {Addison-Wesley Professional},
2106  year = {2004},
2107  author = {Fowler, Martin},
2108  title = {UML distilled: a brief guide to the standard object modeling language},
2109}
2110
2111@book{fowler2010domain,
2112  publisher = {Pearson Education},
2113  year = {2010},
2114  author = {Fowler, Martin},
2115  title = {Domain-specific languages},
2116}
2117
2118@inproceedings{hause2006sysml,
2119  year = {2006},
2120  pages = {1--12},
2121  volume = {9},
2122  booktitle = {Fifteenth European Systems Engineering Conference},
2123  author = {Hause, Matthew and others},
2124  title = {The SysML modelling language},
2125}
2126
2127@inproceedings{berry2022requirements,
2128  organization = {Springer},
2129  year = {2022},
2130  pages = {19--25},
2131  booktitle = {International Working Conference on Requirements Engineering: Foundation for Software Quality},
2132  author = {Berry, Daniel M},
2133  title = {Requirements Engineering for Artificial Intelligence: What Is a Requirements Specification for an Artificial Intelligence?},
2134}
2135
2136@inproceedings{balasubramaniam2022transparency,
2137  organization = {Springer},
2138  year = {2022},
2139  pages = {3--18},
2140  booktitle = {International Working Conference on Requirements Engineering: Foundation for Software Quality},
2141  author = {Balasubramaniam, Nagadivya and Kauppinen, Marjo and Hiekkanen, Kari and Kujala, Sari},
2142  title = {Transparency and Explainability of AI Systems: Ethical Guidelines in Practice},
2143}
2144
2145@article{mihelj2014virtual,
2146  publisher = {Springer},
2147  year = {2014},
2148  author = {Mihelj, Matja{\v{z}} and Novak, Domen and Begu{\v{s}}, Samo},
2149  title = {Virtual reality technology and applications},
2150}
2151
2152@article{miller2020personal,
2153  publisher = {Nature Publishing Group},
2154  year = {2020},
2155  pages = {1--10},
2156  number = {1},
2157  volume = {10},
2158  journal = {Scientific Reports},
2159  author = {Miller, Mark Roman and Herrera, Fernanda and Jun, Hanseul and Landay, James A and Bailenson, Jeremy N},
2160  title = {Personal identifiability of user tracking data during observation of 360-degree VR video},
2161}
2162
2163@inproceedings{tran2017evaluation,
2164  organization = {IEEE},
2165  year = {2017},
2166  pages = {7--11},
2167  booktitle = {2017 ninth international conference on ubiquitous and future networks (ICUFN)},
2168  author = {Tran, Huyen TT and Ngoc, Nam Pham and Bui, Cuong Manh and Pham, Minh Hong and Thang, Truong Cong},
2169  title = {An evaluation of quality metrics for 360 videos},
2170}
2171
2172@article{kawaguchi2019effect,
2173  publisher = {MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info~…},
2174  year = {2019},
2175  pages = {1462--1498},
2176  number = {7},
2177  volume = {31},
2178  journal = {Neural computation},
2179  author = {Kawaguchi, Kenji and Huang, Jiaoyang and Kaelbling, Leslie Pack},
2180  title = {Effect of depth and width on local minima in deep learning},
2181}
2182
2183@inproceedings{safran2017depth,
2184  organization = {PMLR},
2185  year = {2017},
2186  pages = {2979--2987},
2187  booktitle = {International conference on machine learning},
2188  author = {Safran, Itay and Shamir, Ohad},
2189  title = {Depth-width tradeoffs in approximating natural functions with neural networks},
2190}
2191
2192@article{fan2019survey,
2193  publisher = {ACM New York, NY, USA},
2194  year = {2019},
2195  pages = {1--36},
2196  number = {4},
2197  volume = {52},
2198  journal = {ACM Computing Surveys (CSUR)},
2199  author = {Fan, Ching-Ling and Lo, Wen-Chih and Pai, Yu-Tung and Hsu, Cheng-Hsin},
2200  title = {A survey on 360 video streaming: Acquisition, transmission, and display},
2201}
2202
2203@article{anwar2020subjective,
2204  publisher = {IEEE},
2205  year = {2020},
2206  pages = {148084--148099},
2207  volume = {8},
2208  journal = {IEEE Access},
2209  author = {Anwar, Muhammad Shahid and Wang, Jing and Khan, Wahab and Ullah, Asad and Ahmad, Sadique and Fei, Zesong},
2210  title = {Subjective QoE of 360-degree virtual reality videos and machine learning predictions},
2211}
2212
2213@inproceedings{huang20176,
2214  organization = {IEEE},
2215  year = {2017},
2216  pages = {37--44},
2217  booktitle = {2017 IEEE Virtual Reality (VR)},
2218  author = {Huang, Jingwei and Chen, Zhili and Ceylan, Duygu and Jin, Hailin},
2219  title = {6-DOF VR videos with a single 360-camera},
2220}
2221
2222@article{roberto2019visual,
2223  publisher = {IEEE},
2224  year = {2019},
2225  pages = {2524--2537},
2226  number = {8},
2227  volume = {30},
2228  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
2229  author = {Roberto, G de A and Birkbeck, Neil and De Simone, Francesca and Janatra, Ivan and Adsumilli, Balu and Frossard, Pascal},
2230  title = {Visual distortions in 360° videos},
2231}
2232
2233@article{wan2019does,
2234  publisher = {IEEE},
2235  year = {2019},
2236  pages = {1857--1871},
2237  number = {9},
2238  volume = {47},
2239  journal = {IEEE Transactions on Software Engineering},
2240  author = {Wan, Zhiyuan and Xia, Xin and Lo, David and Murphy, Gail C},
2241  title = {How does machine learning change software development practices?},
2242}
2243
2244@article{ahmad2022mapping,
2245  year = {2022},
2246  journal = {arXiv preprint arXiv:2212.10693},
2247  author = {Ahmad, Khlood and Abdelrazek, Mohamed and Arora, Chetan and Bano, Muneera and Grundy, John},
2248  title = {Requirements Engineering for Artificial Intelligence Systems: A Systematic Mapping Study},
2249}
2250
2251@article{ahmad2023requirements,
2252  year = {2023},
2253  journal = {arXiv preprint arXiv:2301.10404 (accepted at: Applied Soft Computing Journal)},
2254  author = {Ahmad, Khlood and Abdelrazek, Mohamed and Arora, Chetan and Bano, Muneera and Grundy, John},
2255  title = {Requirements Practices and Gaps When Engineering Human-Centered Artificial Intelligence Systems},
2256}

Attribution

arXiv:2303.02920v2 [cs.SE]
License: cc-by-4.0

Related Posts

The Right Tool for the Job: Open-Source Auditing Tools in Machine Learning

The Right Tool for the Job: Open-Source Auditing Tools in Machine Learning

Introduction As a society, we are tossed around on a sea of competing ideas, backed by statistics, data, and algorithms.

The Forgotten Margins of AI Ethics

The Forgotten Margins of AI Ethics

Introduction The urgency and profound importance of fairness, justice, and ethics in Artificial Intelligence (AI) has been made evident by grave failures, limitations, harms, threats, and inaccuracies emanating from algorithmic systems deployed in various domains from criminal justice systems, and education, to medicine.

Fairness and Preventing Discrimination in AI Governance

Fairness and Preventing Discrimination in AI Governance

In the landscape of AI governance, ensuring fairness and preventing unintended discrimination are paramount.