Liu, J., Wu, D., Sun, G., & Deng, Y. (2024). Dynamic algorithmic awareness based on FAT evaluation: Heuristic intervention and multidimensional prediction. Journal of the Association for Information Science and Technology, 1–22. https://doi.org/10.1002/asi.24969

近日，中心2021级博士研究生刘静以第一作者身份在期刊Journal of the Association for Information Science and Technology发表论文“Dynamic algorithmic awareness based on FAT evaluation: Heuristic intervention and multidimensional prediction”，通讯作者为吴丹教授。

Journal of the Association for Information Science and Technology是图情领域顶级期刊，2023年影响因子为2.8，在SSCI索引INFORMATION SCIENCE & LIBRARY SCIENCE类别属于Q1分区。

Abstract

As the widespread use of algorithms and artificial intelligence (AI) technologies, understanding the interaction process of human–algorithm interaction becomes increasingly crucial. From the human perspective, algorithmic awareness is recognized as a significant factor influencing how users evaluate algorithms and engage with them. In this study, a formative study identified four dimensions of algorithmic awareness: conceptions awareness (AC), data awareness (AD), functions awareness (AF), and risks awareness (AR). Subsequently, we implemented a heuristic intervention and collected data on users’ algorithmic awareness and FAT (fairness, accountability, and transparency) evaluation in both pre-test and post-test stages (N = 622). We verified the dynamics of algorithmic awareness and FAT evaluation through fuzzy clustering and identified three patterns of FAT evaluation changes: “Stable high rating pattern,” “Variable medium rating pattern,” and “Unstable low rating pattern.” Using the clustering results and FAT evaluation scores, we trained classification models to predict different dimensions of algorithmic awareness by applying different machine learning techniques, namely Logistic Regression (LR), Random Forest (RF), Support Vector Machine (SVM), Linear Discriminant Analysis (LDA), and XGBoost (XGB). Comparatively, experimental results show that the SVM algorithm accomplishes the task of predicting the four dimensions of algorithmic awareness with better results and interpretability. Its F1 scores are 0.6377, 0.6780, 0.6747, and 0.75. These findings hold great potential for informing human-centered algorithmic practices and HCI design.

摘要

算法及人工智能技术的发展和广泛应用使得从用户视角探索用户与算法的交互过程显得更为重要。算法感知被认为是用户形成算法评估并产生算法交互行为的重要环节。本研究通过前序研究识别了算法感知的四个维度：概念感知、数据感知、风险感知和功能感知。随后我们设计了设计启发式介入措施，通过用户实验采集用户在措施介入前和介入后两个阶段算法感知和FAT评估的数据（N=622）。通过数据分析，我们首先验证了算法感知与FAT评估的动态性，并在此基础上通过模糊聚类识别了3种FAT评估变化模式：稳定高评价模式、多变中庸评价模式和不稳定低评价模式。随后利用聚类结果和 FAT 评价得分，我们训练了分类模型，分别使用逻辑回归（LR）、随机森林（RF）、支持向量机（SVM）、线性判别分析（LDA）和 XGBoost（XGB）来预测算法意识的不同维度。实验结果表明，SVM 算法在预测算法感知的四个维度上都取得了较好的效果和可解释性，其 F1 值分别为 0.6377、0.6780、0.6747 和 0.75。这些发现能够以人为本的算法实践和人机交互设计提供参考与借鉴。