Preserving the history of the internet is getting harder.

From American Public Media, this is Marketplace Tech. I'm Stephanie Hughes.

The Way Back Machine is a project of the Internet Archive. It sends out web crawlers to take

snapshots to the Internet, basically creating a digital library of web pages. But now some

news publications are blocking its crawlers. They're worried that AI companies will access the

Wayback Machines publicly available archive and then train their AI models with the content.

I talked about this with Andrew Deck at Harvard's Neiman Lab.

We've seen time and time again, you know, how valuable news content is to generative AI companies.

Publisher archives house enormous amounts of informational, kind of edited text that has been used to train a lot of the large

language models on the market that most people use and also the commercial products that

are built on top of them. So there's potential revenue loss for news publishers if this were

to be happening. Is it actually happening? How real is this fear?

Yeah, I think it's important to say that in our conversations with news publishers,

a lot of them were taking this action preemptively out of a fear of proxy scraping

rather than direct evidence that it has happened to them already.

None of the publishers were able to point

to a particular AI company or other kinds of direct evidence that their content had already

been scraped by the way back machine. But I do think it's fair to say that news publishers feel

burned by the earlier development of large language models.

When we do know large swaths of their content were used by Silicon Valley companies to train models without upfront compensation and permission, you know, that tension over the value of journalism has played out over the last several years. It's led to

lawsuits. It's led to licensing deals. Yeah, I mean, with news sites, information is their currency.

And as you mentioned, some have struck deals to sell their content to AI companies.

...