From Accuracy to Alignment: The Practical Logic of ‘Trustworthy AI’ among Chinese Radiologists
Abstract: The increasing use of machine learning algorithms to support human decision-making has brought about the popular notion of “trustworthy AI”. Accuracy and explainability, among other things, are considered two key elements in the trustworthiness of machine learning systems and are formulating ethical AI guidelines as well as major research efforts in computer science. The underlying assumption is that, if the output of AI systems is more “accurate” and “explainable,” then they become more trustworthy and trusted by users. Drawing on extensive participant observations and interviews with radiologists in China, this paper problematizes such universal assumptions and proposes an alternative, locally-rooted framework centered on “human-machine alignment” to understand AI trustworthiness. I argue that radiologists in China develop their trust based on the degree of alignment between their own judgment and the algorithmic output, including “direct alignment” and “adjusted alignment.” Regardless of the claimed performance indicated by statistical parameters, Chinese radiologists are still prompted to judge if the algorithmic decisions directly align with their own because of two factors. First, the probabilistic nature of evaluation metrics cannot guarantee algorithms’ correctness in individual cases in the clinical setting, for which typically no ready “ground truths” are available. Second, under current Chinese legal and regulatory regimes, radiologists are held accountable for medical reports and are therefore motivated to doublecheck AI’s recommendations. Yet, even if the direct alignment is low, radiologists may still trust and use the algorithmic output if they can observe certain patterns of, and thus explain away, the misaligned algorithmic output. This leads to an “adjusted alignment” based on the radiologist’s own interpretations. In conclusion, the paper suggests that universal notions of accuracy and explainability are misplaced in conceptualizing and regulating trustworthy AI in the real world; instead, trust in AI is a result of human-machine alignment that is subject to social and institutional shaping, and could not be reduced to some intrinsic technical features of the algorithms.
Panelist bios: Wanheng Hu is a Ph.D. candidate in Science and Technology Studies at Cornell University and a research fellow in the Program of Science, Technology and Society of the Harvard Kennedy School. At Cornell, he is also a member of the Artificial Intelligence, Policy, and Practice (AIPP) initiative and a graduate affiliate of the East Asia Program. His dissertation research examines the use of machine learning algorithms to cope with expert tasks, with an empirical focus on the development, application and regulation of AI systems for image-based medical diagnosis in China. The project has been supported by the National Science Foundation, China Times Cultural Foundation, and a Hu Shih Fellowship in Chinese Studies, among others. His research is broadly situated at the intersection of the sociology of expertise, medical sociology, critical data/algorithm studies, and development studies. Wanheng holds an M.Phil. in Philosophy of Science and Technology, a B.L. in Sociology, and a B.Sc. in Biomedical English, all from Peking University.
Recorded Presentation | 26 April 2023
#Trustworthiness #Healthcare #HealthcareWorkers #China
