Skip to content

Menu

Research Seminars

The Limits of AI-Mediated Decision Systems: Strategic Manipulation and Endogenous Reliance

Governing decision systems vulnerable to manipulation by the agents they evaluate is a longstanding challenge. Shared AI-mediated channels, however, introduce a dynamic feedback typically absent from standard incentive-design frameworks: downstream reliance raises the return to manipulation, while manipulation degrades channel performance and subsequently weakens reliance. We develop a sender-receiver population model to study how this feedback shapes system performance and governance. In the model, senders choose whether to manipulate inputs, and receivers whether to rely on channel outputs. We show how this interaction creates a self-limiting dynamic. By attracting additional reliance, a well-performing channel can generate incentives that prevent it from realizing its full technological potential. In high-stakes environments in particular, equilibrium leaves receivers indifferent between relying on the channel and taking their outside option. Expanding reach need not expand effective use and can instead lower the equilibrium reliance rate. Governance therefore exhibits a locally all-or-nothing logic: below a critical threshold, stronger detection raises the equilibrium share of manipulators without improving delivered value. We further show that an interior level of algorithmic noise uniquely maximizes both equilibrium system effectiveness and aggregate welfare. Noise can thus serve as a governance instrument, as noisy honesty may outperform manipulable precision. Overall, our work highlights that sustaining progress in AI-mediated decision systems requires accounting for the joint endogeneity of manipulation and downstream reliance throughout system design, deployment, and oversight.

Date
Time
Location

Room 928, Cheng Yu Tung Building, CUHK Business School

Speaker(s)

Prof Jussi Keppo
Professor,
Department of Analytics & Operations,
NUS Business School,
National University of Singapore,
Singapore

100%

of Undergraduates Have Global Experience