In our latest report, The Protocol Gap: Indonesia, published in collaboration with the Journalism Relay Project and co-authors AMSI, Monash University Indonesia, and PR2Media. We examined an initial sample of 1,862 URLs to explore how Indonesian news websites manage AI crawler access.
The study focuses on robots.txt, a file that website operators can use to signal which parts of their sites automated crawlers may or may not access. The findings examine whether Indonesian news publishers are using this mechanism to manage AI access to their content and reveal the gap between the ability to signal access preferences and the capacity to protect journalistic content in an AI-driven ecosystem.
Without effective ways to manage AI crawlers, publishers risk losing control over how their journalism is accessed and reused, while facing declining referral traffic, limited bargaining power, and uncertainty over compensation.
The Protocol Gap: Indonesia is part of a cross-border research collaboration by the CTRL+J Alliance and its partners in Brazil, South Africa, and Indonesia, examining how news publishers signal and manage AI crawler access and what these practices reveal about the challenges of protecting journalism in the AI era.
Untuk mendownload dokumen ini silahkan klik link berikut
