The Internet Archive at 30: Can the web's memory bank withstand the AI era?

"If you were to travel back in time to 1996 with a 2TB thumb drive, you'd be able to fit the entire World Wide Web on it. Of course, that kind of storage didn't exist in the '90s, so it's never been that simple for the Internet Archive. The nonprofit site, which launched three decades ago this year, went from making copies of the web on tape drives to storing more than 1 trillion pages worth of Internet history at data centers around the world."

"Using its Wayback Machine, anyone can look back to what a web page used to look like, which means you can browse through old GeoCities websites, view Google's original Code of Conduct (back when it still said "Don't Be Evil"), or read the EPA's climate change indicators before the Trump administration scrubbed them. All that's on top of the Archive's vast collection of other digital resources, from live concert tapings and public domain e-books to troves of forgotten DOS games."

"Web publishers have started blocking the Wayback Machine out of fear that AI companies are scraping the material. A legal battle with book publishers ended with the Archive paying a settlement and removing more than 500,000 books from its collection. Meanwhile, the cost of storing humanity's digital footprint keeps going up, as demand from AI data centers drives up storage and memory prices."

""We want it all," says Brewster Kahle, the Internet Archive's founder and chairman. "We want all the public works of human beings. So if we don't have it, we want it." But while the Internet Archive hasn't fundamentally changed over the years, the Internet itself is transforming in ways that jeopardize the nonprofit's mission."

The Internet Archive preserves copies of the web and other digital materials for public access. It grew from storing web copies on tape drives to maintaining more than 1 trillion pages of internet history across data centers worldwide. The Wayback Machine lets people view how web pages looked in the past, including sites like GeoCities and older versions of major organizations’ pages. The archive also holds public domain ebooks, DOS games, and recordings of live concerts. About 2 million people access its resources daily. Web publishers increasingly block the Wayback Machine due to concerns about AI scraping. A legal dispute with book publishers led to a settlement and removal of over 500,000 books. Storage costs continue rising as AI data center demand increases memory and storage prices.

Read at Fast Company