Fact-check Your Information (FYI): A Design Probe to Understand How People Actually Fact-check Data-Driven Articles

TL;DR

A browser-extension design probe (N=22) reveals three human-AI workflow archetypes for fact-checking data-driven journalism.

Abstract

Data claims—statements grounded in numbers and statistics—are common in journalism and policy reports, yet verifying them requires significant analytical effort that most readers cannot undertake alone. Existing tools either fully automate verification, risking blind trust in AI, or relying entirely on manual data exploration with a high cognitive burden. We present FYI, a browser extension that bridges this gap through four complementary tools within a unified side panel, spanning the spectrum from automation to user-driven exploration. Using FYI as a design probe, we conducted an exploratory study (N = 22) in which participants verified claims in a data-driven article while thinking aloud. We find that participants adopted three distinct workflow archetypes—AI-first with manual confirmation, manual-first with AI supplement, and parallel co-review—with visualization serving as the primary mechanism for auditing AI conclusions. Trust in AI was not static but shifted dynamically. Specifically, it grew when multiple tools converged on the same answer and eroded when AI outputs were inconsistent. These findings suggest that fact-checking systems should treat AI as a starting point rather than a definitive authority, elevate visualization as a core verification capability, and support flexible, user-driven workflows. We contribute FYI as open-source at https://github.com/DataVisards/FYI.