The Truth Is Paywalled and the Lies Are Free: Brewster Kahle on the Internet of Forgetting

00:00:30 Andrew Keen: Hello, my name is Andrew Keen. Welcome to Keen on America, the daily interview show about the United States with world-leading commentators and thinkers. We are on the road today, down the road actually from my home in San Francisco, at the Internet Archive—the digital library for our online world—with its founder, Brewster Kahle. A legendary figure, part of the Internet Hall of Fame, and a man who's spent a lot of time thinking about how we can remember our digital content. A lot of talk these days in our age supposedly of surveillance capitalism about how the internet can't forget, how it remembers everything, but Brewster's concerned with its forgetting. Is that fair, Brewster?

00:01:55 Brewster Kahle: That's absolutely fair. Yes, there are some things that shouldn't live on that do, but there's an awful lot of people's writings and podcasts, memories and relationships, that just drop away because corporations change direction or want to do something else.

00:02:15 Andrew Keen: Are you concerned though with surveillance, or is that just not the kind of thing that librarians worry about?

00:02:22 Brewster Kahle: Surveillance is turning the internet into something people are very fearful of. They're feeling like the web has betrayed them. And that's terrible. If we lose trust in each other based on massive permitted bad behavior, it will make it so that we don't want to share. And we are, as a species, a sharing, happy, collaborative species—but you can put the kibosh on that. We've seen that happen in different parts of the world, and we're seeing it happen now to those in the online world.

00:02:52 Andrew Keen: The Internet Archive has a fascinating new pamphlet out; it's called Vanishing Culture. And at the end, in your summary piece, you note that—and I'm quoting you here—"our evolving digital age can either be our next Carnegie moment (referring to Andrew Carnegie and his libraries) or it can be a Library of Alexandria moment." In other words, the libraries burn down. Explain what the Carnegie moment is, and what the Alexandria moment is, and why we should fear the Alexandria moment.

00:03:20 Brewster Kahle: Andrew Carnegie had this spirit to invest in everyone to learn, grow, and be their best selves. Not by putting out textbooks that would tell people one truth, but by building libraries so that people can read, grow, and become as much as they can. During the early days of the 20th century, there was enormous investment in public libraries, public universities, public schools. And after World War II, when the United States was thrust into the big leagues of taking the world stage—because pretty much everywhere else was bombed flat—the United States was ready. Part of that was because of the public library system and the vision of Andrew Carnegie. Every town in America has a public library supported by the residents, and it's helped us grow into a great people in the late 20th century.

00:04:32 Andrew Keen: It's funny that you call it a Carnegie moment. Of course, Andrew Carnegie is a controversial figure; spent half his life, it seems, ripping people off or certainly making himself a very rich man, and then the other half giving the money back. Is that the model that the Bezoses, the Zuckerbergs, the Gates should be trying to emulate?

00:04:51 Brewster Kahle: They now have immense wealth that they could be giving away if they figured out how to give it away well. Whether they should have that amazing level of wealth in the first place, I actually doubt. I think wealth inequality tends to destroy societies. But that's a different subject. Andrew Carnegie, yes, is a controversial figure, but the Carnegie libraries were an investment in self-actualization, in public education, that has paid dividends we're still not able to live down. But right now we're in the process of smashing the public library system, the public education system, the public university system in the United States, in ways that are quite devastating.

00:05:37 Andrew Keen: You, in many ways, Brewster, have dedicated your life to librarianship, digital librarianship. As I said, many people call you the internet's librarian. Why?

00:05:48 Brewster Kahle: I'm a geek. I grew up just having a skill of being able to build computer systems. And the question is: what do you want to do with that skill? That was the puzzle set to me in college by a good friend, and I could only come up with a couple of answers. One was to try to protect people's privacy when they just, in general, throw it away. The other was to build the Library of Alexandria version 2: to make all the published works of humankind available to people. I tried to do the privacy one, but I couldn't make that cost little enough to help end users; it would mostly help large corporations, the mafia, and the intelligence groups, and they had their own people to help them. So I worked on this library thing for the last 45 years. We needed some things to work: we needed an open internet, so we had to transition the ARPANET to the internet—did that. We needed computers that could store a lot and make it searchable—I did help a lot in that, building supercomputers, Thinking Machines, back in the old days. We needed free and open-source software because the copyright laws had become so draconian that we needed a way of getting around what the publishers had done to the copyright system. So that was free and open-source software. Then we needed publishers on board making money to make this thing work. And I helped along those lines. And then by the early 1990s, we were kind of ready. We had people pouring their lives into this new internet and these new websites and sharing at this velocity. Then it was time to build the library. So in 1996 we started the Internet Archive and started by collecting the web, but now also television, radio, books, periodicals, and made them as available as possible. So it's that dream of the library of everything that I've dedicated my career to, but many, many others have as well.

00:07:52 Andrew Keen: Do you feel you've accomplished it?

00:07:53 Brewster Kahle: No.

00:07:54 Andrew Keen: Why?

00:07:55 Brewster Kahle: In many ways the library system we have now is worse than the library system I had growing up. It's faster to use the information available on Google—it's on your desktop, it's instantaneous, you have these phones that are connected. So you have access to something really, really fast. But if you know anything really well, you know it's not online. Or at least a lot of it's not online—there's maybe a Wikipedia page about it, but the depth is not there. So we've now brought up a generation when we could have brought up a generation with the library system at their fingertips, the Library of Congress at their fingertips. We don't have it. What we do have is people wanting to share and wanting to learn. We just hit one trillion web pages in the Internet Archive—maybe a billion people's voices are now in the library. People are awesome. People want to share. But whenever things touch the publishing system, it tends to come cratering down and controlled. So we have basically a trillion web pages—we have kind of everything except the Library of Congress, except the library system. It's just not there. And it's becoming more and more walled off. 60% of news organizations now put paywalls up. So the high-end newspapers are becoming newsletters for the rich. And that's not very good. We're starting to have a generation now that doesn't know what's true or not true about the 20th century because they haven't had access to the printed works, the published works, about the 20th century. People are really confused about whether the Holocaust happened. And that may sound sarcastic—how could you possibly not know, look online! But it's not like the library is there for people. There's all sorts of other things. I like the line from the Current Affairs editor: "the truth is paywalled, and the lies are free."

00:10:07 Andrew Keen: But Wikipedia is not paywalled, Brewster. If someone wants to check out whether or not the Holocaust happened, can't they just go to Wikipedia?

00:10:20 Brewster Kahle: Yes, Wikipedia is fantastic, but it's kind of shallow. But let me ask you a question: why did we have to do Wikipedia? It's not like we didn't have encyclopedias before. Why did we have to write it all again? The reason we had to was because the existing writings were under a copyright regime controlled by people that didn't want to share openly, even though they could have made real business money out of all of that—they didn't. So we had to write Wikipedia again with a new license to get around the copyright problems that were put in place in the United States in 1976. We've had 50 years of the copyright regime going from opt-in copyright—you used to have to put a little 'c' on things and send a copy into the Library of Congress—to opt-out: everything's under copyright and basically forever, and they keep lengthening how long it'll go for. So Wikipedia is tremendous. People are awesome. If you take other things like arXiv with an 'x'—"arkiv"—which is preprint servers, that was also a way around. Your academic paper went into the publishers such that people won't be able to see it anymore, so people would contribute it to this website, arXiv, and it's revolutionized how people do academic work. Because the knowledge is accessible. There's plenty of money to keep these publishers around, there's actually plenty of money to pay authors—but it's not going to authors, it tends to stop with a few very, very large media conglomerates. I was just at a symposium about the '76 Copyright Act at the Berkeley Law clinic and law school, and it had the Register of Copyrights, the appellate court, and they asked, "what happened out of this?" The thing they were surprised about was the conglomeration, the aggregation.

00:12:08 Andrew Keen: So you're talking about what—Penguin Random House, these types of—

00:12:12 Brewster Kahle: Yes, where there are on the order of four or five book publishers, not just in the United States but worldwide. We're not talking about just one state; this is everywhere, this is the West. In academic publishing, it's down to about three, and that really dominates. If you take the record industry, the music industry, it's down to a few, and it's worldwide. That was never allowed in the 20th-century America I grew up in. Antitrust would not have allowed that at all. And some of the repercussions of having that concentrated level of power of distribution of knowledge—but it's also the creation: what gets funded, and why people have basically moved to the coasts of the United States. Why did that happen? These large-scale conglomerates pushed the money into more and more concentrated places to find the greatest hits, the best-ofs, instead of a long tail of local publishing, local ideas.

00:13:11 Andrew Keen: Although Brewster, I have to say, I've got a friend—he will remain nameless—who runs one of the largest independent publishing companies. And he's not very keen on your Archive. I'm not sure how much he knows about it, but I told him I was coming in to talk to you, and he feels—and others in publishing feel—that you give away their stuff for free. How would you respond to that argument?

00:13:32 Brewster Kahle: Libraries have always bought materials from publishers, often local, such that it supports local creative environments. It's about 20% of the revenue of book publishing—about 20% of the trade publishers—that comes from library funding. That's a socialized injection of money into the creative industries, particularly in the United States, but worldwide too. So what libraries do is we buy things, we preserve them, we lend them in controlled ways, and we work with other libraries through inter-library loan. But in the digital era that has completely changed. In the digital era, libraries are not allowed to buy things—they're allowed to rent access in a surveillance environment of database products that are served by fewer and fewer corporations. They're not allowed to preserve the materials; they don't actually even get the bits in most cases—they're just customer service departments. Then there's lending—in the United States, lending of digitized books was ruled illegal. In Holland, and then in the rest of Europe, they ruled it legal. They actually want libraries to move forward buying and supporting and lending. And then inter-library loan—that's also highly, highly restricted in the United States. So we had a system that worked for centuries, and the movement against libraries funding and owning and supporting the populace and publishers and authors—that bargain is off.

00:15:02 Andrew Keen: So we know your view on the problems of Big Media. People don't vilify Big Media as much these days; the problem in many people's minds is Big Tech. How do these new tech conglomerates—from Google and Microsoft to Facebook and Amazon—how do they play into your narrative in this crisis of libraries?

00:15:25 Brewster Kahle: I think the model is this sort of large-scale conglomerate control of distribution, and I think they're kind of the same issue. The Murdoch structures around lots of different media types, the Sinclairs, the acquisition of more and more of the American media by politically oriented acquisitions—I think we're seeing it in both the publishing online, whether it's the Twitter/X kinds of moves, Facebook and the like, and also in other kinds of publishing all the way up and down. What we saw in the cable era was that those who controlled the distribution tended to buy up the upstream sources. That's why Comcast owns a bunch of movie studios. And now we have Amazon buying MGM. So that's kind of natural. I think they're not that distinct. If you read one publishing approach, they tend to point to another as a problem—but I think the consolidation of points of view into very few, controlled production and distribution of knowledge is not what you need for a prospering democracy, a capitalist structure with large numbers of small competitors, no barriers to entry, making better products. That's not what you see in a monopolistic, oligarchic environment.

00:16:39 Andrew Keen: Some people might say, "Well, I can go onto YouTube and if I'm willing to look at the ads I can see any video pretty much ever made, it seems, these days." Google offers their search engine for free; anyone can use it as long as you're willing to put up with adverts, which you don't always have to click on. What about that Web 2.0 advertising model, Brewster? Does that undermine content, libraries?

00:17:14 Brewster Kahle: I don't think you can see anything published on YouTube if you're willing to put up with ads. I mean, it's just not really there. There's a lot there—a tremendous amount of material that is available—but if you're looking for something in particular, or if you're looking to do deep research on something, it's just not there. YouTube is not a library. It's a large collection of materials that have been put up. But to answer your question on the ad model and what is creatively called surveillance capitalism—one of the problems with that ad-based model, and we knew this from early on trying to get the internet to go, is that you end up with very few winners. You end up with people who control the ad networks themselves, and they tend then to be able to buy up all the other magazines or all the other newspapers, as long as they're allowed by the governments. And more and more they're allowed to buy those things up because you're controlling ad distribution. So that tends to be a game with very few winners. Another approach is how we did publishing from, say, 1600 until the late 1900s—and that is a royalty model. You actually pay, and some of the money goes back upstream to things like authors. Often not as much money as you'd like, but there's a mechanism for taking money and making it go back upstream without any central points of control. As Lessig always points out: once you have a central point of control somewhere, they'll work backwards to make sure the whole chain becomes controlled. So the way to have democracy, a successful capitalist system, is to make it so that you can have lots of competition at every point in the system. That's the way to make an information ecosystem function.

00:19:01 Andrew Keen: Brewster, what's happened meanwhile, while we have this chaos of the online revolution—what's happened to the physical Carnegie library in America?

00:19:09 Brewster Kahle: They're still supported by their local taxpayers; they're still believed in, people love them. Out of civic institutions, I think that's one of the ones that have avoided the issues people have with the police or even the legislature. People love their libraries. But what we're seeing is book bannings, criminalization of librarianship, defundings—and the structural problem for libraries is this licensing model where copyright wasn't enough; now there are licenses such that libraries don't even own anything anymore. A book can be disappeared from all shelves all at once if it's in electronic form. And that happens, or books are changed. We've all read growing up what happens when you have that kind of level of control—and it's starting to happen as we've shifted from the physical to one where people want mostly digital. The physical world is still surviving—books are being bought, though from fewer and fewer publishers that don't necessarily represent the points of view of all the people in their environment—but at least the underlying laws around first sale and purchasing haven't been undermined in the physical world.

00:20:24 Andrew Keen: And meanwhile, we are putting ourselves online. Since you started the Internet Archive, more and more people define themselves according to social media, Facebook, X, all forms of social media. Is this vanishing too, Brewster? Is this part of the crisis—the way in which the owners of the social media platforms are somehow empowered to vanish culture, vanish our content, our photographs, our videos, our commentaries?

00:20:55 Brewster Kahle: Well, I got to give a fun talk at the Facebook Museum in the Netherlands, which was opening up a Facebook Museum, and it was sort of tongue-in-cheek—what happens if we look at Facebook in the rearview mirror? And so I gave a talk about what were the other forms of this during the digital era, going backwards to FidoNet and the bulletin board systems, GreenNet, PeaceNet, the WELL, through AOL and GeoCities, MySpace, LiveJournal—on and on and on. We have these systems where people have invested themselves, putting their ideas out there and putting them in the hands of companies that then get bought or change their position and it all goes away, over and over again. So we've invested ourselves, we've come to the party—but it feels like we're being betrayed. Our works are not being cared for in the ways they deserve. So I think we need a different approach. We need to make systems that work in a decentralized way, where there's more distributed control, but that's not clunky and janky. It's easier to make centralized systems than it is to make decentralized systems.

00:22:05 Andrew Keen: Are some of the centralized systems better than others? One that comes to mind is Apple. Many of us use an iPhone—I use one—Apple stores all our videos, our emails. I trust it; I certainly would trust it more than some of the other Big Tech companies. Are Apple doing something a bit better, or are they part of the problem too?

00:22:25 Brewster Kahle: I use Apple products; they work extremely well. What I really like about them—at least the Macintosh that I use—is I can write software or play other people's software without having it be gatekept. The iPhone, on the other hand, puts real restrictions on what apps you're allowed to run on the machine. That kind of level of control is unnerving. Even though it looks like an open ecosystem, there are a lot of things that aren't allowed onto the iPhone, though it hasn't gone through to its full abuse levels. But Apple comes out with hardware that's really tremendous; the level of engineering and effort that goes into it, the design of those machines, is quite good. So that's all good news. Now let's use it for different and interesting things. One thing is they've put a lot of GPUs out in the field; we could run local models, local AI, for doing most of what we want out of AI, and that wouldn't share your data back upstream at all. For a thousand dollars you can buy a computer with a terabyte of disk space on it. A terabyte is a lot. If you take all of the words in the Library of Congress, it's only 28 terabytes. So if you buy a laptop that has a terabyte, we're dealing with sizeable repositories, even if you're doing it with a lot of video. So I like the idea of having people empowered with devices they can control, that they can run the software they want on, that serves them rather than serves some other master.

00:24:00 Andrew Keen: Brewster, I want to come to AI in a second—it's always the final question in these kinds of interviews because it's on everyone's mind these days. But you're a figure who has been very optimistic about technology. You know Tim Berners-Lee and many of the other fathers of the digital revolution; you are indeed yourself a father. What do you make of this shift in the zeitgeist? Fewer and fewer people trust technology; there's more and more opposition to the so-called tech bros. You've never been a tech bro, but you were earlier a startup entrepreneur too. Are you concerned with that, or does it reflect the fact that the public is finally waking up to its digital fate?

00:24:44 Brewster Kahle: The AI tools are fantastic. Fantastically powerful, fantastically useful, but they're controlled by way too few people. One of the things we tried to do in the early internet was to make it so that lots of people could make services on this new internet, World Wide Web. You just had to play the protocols and you were in. That's not the way things are going now. In the United States, it looks like—because of just massive lawsuits—an appellate judge who spoke the other day said there are over 100 major lawsuits, most of them class actions, against these Big Tech companies. And you know how that ends up, right? This is where there are backroom deals that end up between publishing conglomerates and the tech companies, such that there aren't going to be new entrants—they're the only ones who can withstand 100-million-dollar attacks.

00:25:35 Andrew Keen: Sorry to jump in here, Brewster—but how do these backroom deals get done? I thought that—

00:25:40 Brewster Kahle: It's settlements. If you take the Google Books settlement, that was an attempt between Google and the Authors Guild in the United States. It was seen as an antitrust violation, a treaty violation, every which-way problem. So that settlement didn't pass muster as a class action rejiggering of the whole copyright scheme, and then it had to go back to court, and the judge determined what Google was doing was fair use. Ta-da! It was over. So they didn't end up with this bargain that was negotiated between those two. We're now seeing that kind of bargain trying to be struck again between the same basic players: Big Tech and the Big Publishing conglomerates and the collecting rights organizations—those folks trying to make a deal. And who's not at that table? The public, entrepreneurs, libraries. So you're going to end up with settlements negotiated between these players that are exclusionary of anybody not at that table.

00:26:35 Andrew Keen: I take your point, but I'm not sure you answered my question: are you disappointed with the way in which technology generally now is mistrusted by the public? Big Tech, small tech, doesn't matter what the tech is.

00:26:46 Brewster Kahle: Oh, I'm part of the group that distrusts what's going on! I think it's gone fantastically wrong. I think we've taken things that could be benefiting a very large number of people and made people feel like they're just cannon fodder, or they're the product, or they're being surveilled. It's a gateway of control mechanisms that make it possible for you to do authoritarian and totalitarian structures easily. Snowden called it "turnkey tyranny." All it needs is somebody motivated to think tyrannically, and all of the laws, the policies, the technologies are there. Is that a condemnation of the technology and the technologists? Yes, but I'd say it's really upon us because we allowed way too much centralization of control. So that theme, for me, is where the real promise is. The technology is moving along—it's how are we going to apply it? Out of AI, what I want is public AI. We want AI that serves the public good. Let's take this new tool that's fantastically powerful and have it read all of these things that we've been digitizing. We just got all of the agricultural records basically from the British Empire for the last century and digitized them. You're not going to read the Sri Lankan 1953 fish reports—but your computer can, and can see patterns in them, and answer the big climate issues that are going on. So ClimateGPT, which is an initiative in Europe supported by the Internet Archive Europe—because it's allowed in Europe to do public AI—is some way we can make real progress. What's going on in the United States is we may only see one, two, or three gigantic players.

00:28:41 Andrew Keen: Yeah, and speaking of gigantic players, we're in San Francisco—up the road are two of those: OpenAI and Anthropic. Have you reached out to a guy like, for example, Dario Amodei, who seems to want to do good? Have you talked to Sam Altman? He only lives up the road.

00:28:55 Brewster Kahle: I haven't met some of those players, but the Internet Archive has been working back and forth with those organizations to figure out how we can be helpful within the laws and structures and what is right within the library system. One thing the Internet Archive has done is to take open collections, like the old books that have fallen out of copyright, or all of the government information, and try to make that available not just to the big guys, but to lots and lots of small guys. So they can use their clusters to do things like ClimateGPT. Another thing that was demonstrated just last week at WikiCred—there was a conference held here at the Internet Archive to help build the credible references from Wikipedia by making a toolbox, AI-powered, to say, "Here are the assertions in a Wikipedia article—are they supported or not supported based on the literature?" And the AI, by reading all of this open journal publishing literature, can give fantastic support, or say it's not supported, and help build Wikipedia to be better. Those types of uses—ClimateGPT or a Wikipedia toolbox—are some of the great reasons why we need these AI tools in non-commercial hands, so that we're solving some of the big problems of our day: disinformation, climate, healthcare. That is for the public good.

00:30:20 Andrew Keen: Two very quick questions finally, Brewster. I know you've got to run—you've been very generous with your time, we appreciate it. There are people, of course, who argue that all this AI, whether it's Anthropic or Gemini or ChatGPT—all the wisdom, all the knowledge has been stolen. Where did they get the content to create their intelligence? Where do you stand on that?

00:30:41 Brewster Kahle: A lot of the AI that we're using, we're using to try to do optical character recognition—the conversation earlier today was trying to read microfiche and find the pages on it—so the AI tools that we're using are, I guess, much more mundane. In terms of the balance between large media conglomerates—call them AI companies—and large media conglomerates—call them publishers—how they should organize themselves, I guess we're going to see how that works out.

00:31:08 Andrew Keen: But in very simple—a yes or no answer—is the current AI landscape with these Big AI companies, now worth hundreds of billions of dollars—is it mass larceny or not? Yes or no?

00:31:22 Brewster Kahle: Mass larceny or not? I'd say the real loser out of a lot of this is going to be people.

00:31:28 Andrew Keen: So yes, it is mass larceny.

00:31:29 Brewster Kahle: But I think the usual answer there is whether that mass media conglomerate which is called an AI company is stealing from another mass media conglomerate, and I wouldn't say that's actually the dynamic I see playing out. I see it as these large corporations taking from people's goodwill, their creative thoughts, their output, their authorship, and it's landing in very few hands. So should it be the publishers or the AI companies—that's sort of like saying Coke versus Pepsi. Maybe the answer isn't either of those. I think we need a different alignment toward the public, and public good, and public AI, and public tech, to have a way through a dissolving of our institutions, our concept of what's true, who can we trust—that is being undermined at a phenomenal level.

00:32:19 Andrew Keen: Final question, Brewster. You have this wonderful new pamphlet, Vanishing Culture, many very interesting contributors, and you have the final note on it. To end our conversation, I think many people will be deeply concerned with your observations about vanishing culture, the amnesia online and in the digital world where everything gets forgotten. Could you end with a couple of very concrete ways that we can fight vanishing culture?

00:32:45 Brewster Kahle: So this book that the Internet Archive is putting out is a collection of essays from different users but also people interested in different media types over time that are getting washed away as we go through a digital transition and a transition of power.

00:33:02 Andrew Keen: No, concrete ways in which we can fight this vanishing culture, where the internet will remember but it won't be surveillance capitalism.

00:33:18 Brewster Kahle: Ah. I think the real source of where the information came from, and the passion for making sure it's passed down, is through individuals and small groups of people. It's the one-person museums, the small-scale libraries, the flourishing of creation and protection, but then making sure it moves down a generation. And places like Wikipedia, the Internet Archive, the Electronic Frontier Foundation, even the open-source software movement, are all mechanisms to try to take succession seriously, to make it so that we don't end up investing in a commercial property that then gets disbanded. How do we make things move on? So how to make this win? Take something seriously that you want to make sure makes it to the next generation and put some effort into it. People have expertise in railroad schedules, or the occult, or their family histories—make copies and hold on to them and move them forward from generation to generation. Make sure that people know and like and get exposed to the materials, because dark archives don't work very well—backups, if they're not used, don't work when you actually need them. So keep things alive and available and moving down—make copies of things you want to make sure your kids can learn from.

00:34:41 Andrew Keen: Can we make a copy of Brewster Kahle, Brewster?

00:34:44 Brewster Kahle: Ah, I'd rather not. One of the great things about humans is, at least at this point, we still die.

00:34:49 Andrew Keen: Well, we need you, Brewster Kahle. Wonderful conversation. A living legend, a member of the Internet Hall of Fame—real honor to talk. Thank you so much for your time.

00:35:01 Brewster Kahle: Thank you.

00:35:03 Andrew Keen: Hi, this is Andrew again. Thank you so much for listening or watching the show. If you enjoyed it, please subscribe on Substack, YouTube, Apple, Spotify—all the platforms.