Rich Skrenta and the problem of too much information

Above a trophy shop in Palo Alto, Rich Skrenta was building a business he hoped would let him stay put. It was 2004. Topix had six employees and a system for putting news into useful little compartments. A story acquired a subject, a location, a place to belong. The address offered an accidental running joke: a news startup upstairs, trophies downstairs. Upstairs, the ambition was to make a modest media company pay its own way.

The setting suits Skrenta better than a ceremonial garage. He has spent much of his working life sorting things: websites, news stories, search results, vast collections of pages. The companies have changed, and so have the customers. The recurring question is wonderfully stubborn. When information becomes abundant, how does anyone find the part worth reading?

Today he is executive director of Common Crawl, a nonprofit whose public web archive supplies researchers and developers with material they could scarcely collect on their own. Between the trophy shop and that archive lie volunteer editors, newspaper investors, a search engine called Blekko, and a considerable amount of crawling. There is also an Apple II prank that refuses to leave his biography. Software can be persistent. So can a good embarrassing story.

A joke that learned to travel

In 1982, while attending high school in the Pittsburgh area, Skrenta wrote Elk Cloner. It spread between Apple II computers on floppy disks, carrying a poem along with it. It became an early example of a personal computer virus circulating outside a laboratory. Decades later, he called it “some dumb little practical joke.” Few teenage jokes receive quite so many anniversary articles.

His later account has the mechanics of a school comedy. Friends had grown wary of the disks he handed them, so he worked out how code could copy itself without another personal delivery. Eventually a mathematics teacher confronted him about an affected computer. The prank had acquired a distribution department.

That episode is famous enough to obscure the rest of the programmer. Skrenta graduated from Northwestern University in 1989 with a computer science degree. His professional background includes Commodore’s Amiga Unix group, Unix Systems Laboratories and Sun Microsystems. These were years of operating systems and networks, the machinery beneath the friendly screen. He subsequently managed engineering work at Netscape and AOL, including search, music and shopping products.

He also made worlds people could inhabit together. His Olympia was a fantasy simulation played by email, with nobles, ships, castles, trade and magic. Its campaigns could continue for years. In 2009, he made the source code and background materials available after the game had been dormant. The old site still offers the peculiar pleasure of finding an elaborate imaginary civilization tucked inside an unassuming web page.

A directory with room for volunteers

In 1998, Skrenta and Bob Truel helped start the directory that became the Open Directory Project, also known as DMOZ. Truel was a connection from ninth grade. Across several later ventures, familiar collaborators would return. The organization charts changed more often than some of the names.

The directory began as GnuHoo, became NewHoo, and found a home at Netscape that same year. Its proposition was simple enough to explain over lunch: let people who care about a subject select and organize websites about it. A web directory could expand through the distributed attention of volunteers. Someone’s enthusiasm for a narrow topic became useful to strangers.

By the time Netscape announced the acquisition in November, NewHoo had more than 100,000 sites and over 4,700 volunteer editors. Those numbers give the idea its scale. The staff of a conventional directory had working hours. An international collection of enthusiasts had hobbies, opinions and an apparently durable appetite for classification.

This was an early statement of an idea Skrenta would revisit: computers can handle a great deal, while people remain useful judges of what belongs together. The directory made that judgment visible in its categories. Later projects would put it inside news pages and search filters. Each offered a different answer to the same practical irritation: a growing web can become harder to use even as it becomes more interesting.

The news had a ZIP code

Topix, founded in 2002 and publicly launched in 2004, brought that organizing instinct to news. Its system crawled articles and categorized them by subject and place. A reader could arrive at a page about a town or an industry rather than begin with the whole web and a hopeful search query.

In those early Palo Alto offices, Skrenta favored keeping the operation small and reaching self-sustaining revenue. He had spoken with venture capitalists and disliked how quickly their prescriptions arrived. “If you take their money, you have to buy that vision,” he said. A company’s funding, in his view, came with editorial notes on its future.

The relationship with the newspaper business became more substantial in March 2005. Gannett, Knight Ridder and Tribune jointly acquired 75 percent of Topix, with each taking a quarter and the founders retaining the remaining quarter. The deal’s financial terms were not disclosed. Newspapers were buying into a service that collected and sorted the kind of material they published.

Then the readers began contributing. Topix’s forums attracted eyewitness accounts and local conversation; a 2006 discussion about a tornado in Caruthersville, Missouri, drew more than 200 posts. Skrenta saw a place where people could converge when a location lacked an established online gathering point. His belief in readers’ ability to judge information was conspicuously strong, although he also acknowledged the work that required full-time reporters.

In June 2007, co-founder Chris Tolles succeeded him as CEO. Skrenta remained on the board and as an adviser. His later description of Topix included several changes of direction before finding a workable model. That admission matters. A founder’s finished biography tends to make every move look scheduled; running a business usually involves rather more pencil.

The slash was an editorial decision

Blekko, founded in 2007 and opened to the public in November 2010, took on search itself. Its distinctive device was the slashtag: a way to restrict a query to a selected collection of websites. Append a category such as /recipes, and the engine searched within that collection. People could build and share these filters.

The attraction was obvious to anyone who had watched a useful query disappear under pages assembled to catch it. Searching a smaller, chosen set could reduce the junk. The trade-off was equally real: selection can leave something worthwhile outside the fence. Blekko made the fence part of the product, something a user could understand and help construct.

“Search must serve an editorial function,” Skrenta said at launch. It is a compact explanation of his career. Ranking involves choices, even when those choices arrive dressed as mathematics. A directory editor, a news categorizer and a search engineer all influence what the reader encounters. Blekko let users participate in those choices through the search box.

Search must serve an editorial function.

Rich Skrenta, 2010

IBM acquired Blekko in 2015. Its public search service closed, and its technology and team joined IBM Watson. In IBM’s financial reporting, Blekko’s capabilities were described in terms of web crawling, categorization and intelligent filtering. The consumer destination disappeared; the work beneath it had another use. Skrenta’s subsequent experience included running crawling and machine learning teams at Meta.

The archive has acquired an audience

When Skrenta became Common Crawl’s executive director in 2023, he joined an organization founded by Gil Elbaz in 2007. It had already been collecting web data for years. Blekko had donated crawl metadata to it back in 2012, an earlier connection between the commercial search project and the public archive.

Common Crawl makes the collected material available for others to process. That distinction changes the relationship with the reader: researchers and developers can build their own tools, ask their own questions and apply their own filters. They receive the starting material for work that otherwise requires a substantial crawling operation. Its uses span language research, information retrieval and digital preservation.

Artificial intelligence brought the archive a much larger audience. In a September 2026 article he co-authored, Skrenta described Common Crawl’s long collaboration with Amazon Web Services, which has hosted the dataset at no cost since January 2012. The article put the archive at more than 300 billion pages. The scale makes the old office above a trophy shop feel pleasantly domestic.

Openness has also become contentious. In November 2025, a published investigation challenged Common Crawl’s handling of publishers’ removal requests and restricted articles. Skrenta issued a response disputing the allegations. He said the crawler respected access restrictions and explained that removing material from published archives involved technical limits. Preservation, permission and commercial reuse now sit uncomfortably close together.

In public podcast conversations, he has argued for open access to web data and discussed the difficulty of representing more of the world in these collections. His enthusiasm for AI is explicit. The aspiration comes with a familiar engineering question: who can afford the underlying work, and who gets to build on it?

There is a long distance between a volunteer adding a website to a category and a developer processing billions of archived pages. Skrenta has worked at both ends. He keeps returning to the arrangements that let information travel and become useful to someone else. The trophy shop offered one way to mark an achievement. His preferred medium has usually required a disk drive, a network connection, or somewhere to put the next page.