Trust-Calibrated Human-AI Collaboration for Engineering Design Review and Program Governance: A Systematic Review
Article Sidebar
Main Article Content
Abstract
Artificial Intelligence (AI) is now viewed as an analyst and decision support in the field of engineering design review and program governance, but the benefit of using AI as such will be contingent upon the level of human dependency to gain from system reliability. This systematic review is a synthesis of 28 sources published from 1997–2023 about human-AI interaction, trust in automation, explainable AI, AI ethics, and algorithmic auditing, and maps these sources to the stages of systematic engineering design. Each source was assigned to one of the constructs of trust, reliability outcomes, types of interventions, and governance functions. From the synthesis, reliability, transparency, and feedback are the main drivers of trust in AI, having been identified as an appropriate design target by over 18 interaction guidelines distilled from well over 150 recommendations, a review of 127 trust in automation studies, and a review of approximately 150 empirical trust in AI studies. While explanations led to higher acceptance of correct and incorrect recommendations, cognitive forcing and confidence displays resulted in better calibration at the expense of usability. Transparency was mentioned in 73 of 84 ethics guidelines and a national risk management framework was recently established, in which governance has four functions. A trust-calibrated review architecture is suggested that combines the level of automation per phase, seven formalized metrics and auditable governance gates, providing measurable criteria for implementing AI in review processes with safety and cost implications.
Article Details
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
References
[1] S. Amershi et al., “Guidelines for Human-AI Interaction,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, in CHI ’19. New York, NY, USA: Association for Computing Machinery, 2019, pp. 1–13. doi: 10.1145/3290605.3300233.
[2] A. Barredo Arrieta et al., “Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI,” Inf. Fusion, vol. 58, pp. 82–115, 2020, doi: https://doi.org/10.1016/j.inffus.2019.12.012.
[3] G. Bansal et al., “Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, in CHI ’21. New York, NY, USA: Association for Computing Machinery, 2021, pp. 1–16. doi: 10.1145/3411764.3445717.
[4] Z. Buçinca, M. B. Malaya, and K. Z. Gajos, “To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making,” Proc. ACM Hum.-Comput. Interact., vol. 5, no. CSCW1, Apr. 2021, doi: 10.1145/3449287.
[5] B. J. Dietvorst, J. P. Simmons, and C. Massey, “Algorithm aversion: people erroneously avoid algorithms after seeing them err.,” J. Exp. Psychol. Gen., vol. 144, no. 1, pp. 114–126, 2015, doi: https://psycnet.apa.org/doi/10.1037/xge0000033.
[6] U. Ehsan, Q. V. Liao, M. Muller, M. O. Riedl, and J. D. Weisz, “Expanding Explainability: Towards Social Transparency in AI systems,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, in CHI ’21. New York, NY, USA: Association for Computing Machinery, 2021, pp. 1–19. doi: 10.1145/3411764.3445188.
[7] M. R. Endsley, “From Here to Autonomy: Lessons Learned From Human–Automation Research,” Hum. Factors, vol. 59, no. 1, pp. 5–27, Feb. 2017, doi: 10.1177/0018720816681350.
[8] L. Floridi et al., “AI4People—An Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations,” Minds Mach., vol. 28, no. 4, pp. 689–707, 2018, doi: 10.1007/s11023-018-9482-5.
[9] E. Glikson and A. W. Woolley, “Human trust in artificial intelligence: Review of empirical research,” Acad. Manag. Ann., vol. 14, no. 2, pp. 627–660, 2020, doi: https://doi.org/10.5465/annals.2018.0057.
[10] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, and D. Pedreschi, “A Survey of Methods for Explaining Black Box Models,” ACM Comput. Surv., vol. 51, no. 5, pp. 1–42, 2018, doi: 10.1145/3236009.
[11] K. A. Hoff and M. Bashir, “Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust,” Hum. Factors, vol. 57, no. 3, pp. 407–434, May 2015, doi: 10.1177/0018720814547570.
[12] A. Jacovi, A. Marasović, T. Miller, and Y. Goldberg, “Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI,” in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, in FAccT ’21. New York, NY, USA: Association for Computing Machinery, 2021, pp. 624–635. doi: 10.1145/3442188.3445923.
[13] A. Jobin, M. Ienca, and E. Vayena, “The global landscape of AI ethics guidelines,” Nat. Mach. Intell., vol. 1, no. 9, pp. 389–399, 2019, doi: 10.1038/s42256-019-0088-2.
[14] V. Lai, C. Chen, A. Smith-Renner, Q. V. Liao, and C. Tan, “Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies,” in Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, in FAccT ’23. New York, NY, USA: Association for Computing Machinery, 2023, pp. 1369–1385. doi: 10.1145/3593013.3594087.
[15] J. D. Lee and K. A. See, “Trust in Automation: Designing for Appropriate Reliance,” Hum. Factors, vol. 46, no. 1, pp. 50–80, Mar. 2004, doi: 10.1518/hfes.46.1.50_30392.
[16] J. M. Logg, J. A. Minson, and D. A. Moore, “Algorithm appreciation: People prefer algorithmic to human judgment,” Organ. Behav. Hum. Decis. Process., vol. 151, pp. 90–103, 2019, doi: https://doi.org/10.1016/j.obhdp.2018.12.005.
[17] T. Miller, “Explanation in artificial intelligence: Insights from the social sciences,” Artif. Intell., vol. 267, pp. 1–38, 2019, doi: https://doi.org/10.1016/j.artint.2018.07.007.
[18] J. Mökander, J. Morley, M. Taddeo, and L. Floridi, “Ethics-Based Auditing of Automated Decision-Making Systems: Nature, Scope, and Limitations: J. Mökander et al.,” Sci. Eng. Ethics, vol. 27, no. 4, p. 44, 2021, doi: https://doi.org/10.1007/s11948-021-00319-4.
[19] G. Pahl, W. Beitz, J. Feldhusen, and K.-H. Grote, Engineering Design: A Systematic Approach, 3rd ed. Springer, 2007. doi: 10.1007/978-1-84628-319-2.
[20] R. Parasuraman and V. Riley, “Humans and Automation: Use, Misuse, Disuse, Abuse,” Hum. Factors, vol. 39, no. 2, pp. 230–253, Jun. 1997, doi: 10.1518/001872097778543886.
[21] I. Rahwan et al., “Machine behaviour,” Nature, vol. 568, no. 7753, pp. 477–486, 2019, doi: 10.1038/s41586-019-1138-y.
[22] S. Raisch and S. Krakowski, “Artificial intelligence and management: The automation–augmentation paradox,” Acad. Manag. Rev., vol. 46, no. 1, pp. 192–210, 2021, doi: https://doi.org/10.5465/amr.2018.0072.
[23] I. D. Raji et al., “Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing,” in Proceedings of the 2020 conference on fairness, accountability, and transparency, 2020, pp. 33–44. doi: https://doi.org/10.1145/3351095.3372873.
[24] B. Shneiderman, “Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy,” Int. J. Human–Computer Interact., vol. 36, no. 6, pp. 495–504, Apr. 2020, doi: 10.1080/10447318.2020.1741118.
[25] E. Tabassi, “Artificial intelligence risk management framework (AI RMF 1.0),” National Institute of Standards and Technology, 2023. doi: https://doi.org/10.6028/NIST.AI.100-1.
[26] O. Vereschak, G. Bailly, and B. Caramiaux, “How to Evaluate Trust in AI-Assisted Decision Making? A Survey of Empirical Methodologies,” Proc. ACM Hum.-Comput. Interact., vol. 5, no. CSCW2, pp. 1–39, 2021, doi: 10.1145/3476068.
[27] M. Yin, J. Wortman Vaughan, and H. Wallach, “Understanding the Effect of Accuracy on Trust in Machine Learning Models,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, in CHI ’19. New York, NY, USA: Association for Computing Machinery, 2019, pp. 1–12. doi: 10.1145/3290605.3300509.
[28] Y. Zhang, Q. V. Liao, and R. K. E. Bellamy, “Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making,” in Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, in FAT* ’20. New York, NY, USA: Association for Computing Machinery, 2020, pp. 295–305. doi: 10.1145/3351095.3372852.