Internet Archive Amasses All TV News Since 2009

The following is an excerpt from an article in:


The New York Times
Tuesday, September 18, 2012

Internet Archive Amasses All TV News Since 2009

By BILL CARTER

Inspired by a pillar of antiquity, the Library of Alexandria, Brewster Kahle has a grand vision for the Internet Archive, the giant aggregator and digitizer of data, which he founded and leads.

“We want to collect all the books, music and video that has ever been produced by humans,” Mr. Kahle said.

As of Tuesday, the archive’s online collection will include every morsel of news produced in the last three years by 20 different channels, encompassing more than 1,000 news series that have generated more than 350,000 separate programs devoted to news.

The latest ambitious effort by the archive, which has already digitized millions of books and tried to collect everything published on every Web page for the last 15 years (that adds up to more than 150 billion Web pages), is intended not only for researchers, Mr. Kahle said, but also for average citizens who make up some of the site’s estimated two million visitors each day.  “The focus is to help the American voter to better be able to examine candidates and issues,” Mr. Kahle said. “If you want to know exactly what Mitt Romney said about health care in 2009, you’ll be able to find it.”

Of course, if you want to discredit or satirize a politician based on a clip showing some reversal of a position, that will be made easier as well. Or, as Mr. Kahle put it, “Let a thousand Jon Stewarts bloom.”

Many conventional news outlets will be available, including CNN, Fox News, NBC News, PBS, and every purveyor of eyewitness news on local television stations. And Mr. Stewart’s program, “The Daily Show” is one of those 1,000 series that is part of the new news archive.

For more, visit www.nytimes.com.

Words by the Millions, Sorted by Software

Excerpt from an article in

The New York Times (Business)

Sunday, March 25, 2012

Words by the Millions, Sorted by Software

By ANNE EISENBERG

IT just keeps growing — the vast electronic archive of books, journals and scholarly literature stored on the Web. But scientists are aiming to keep up with this trove of collective knowledge by devising computer-based tools to winnow and quantify it.

David M. Blei of Princeton University is among those who are teaching computers to sift through the digital pages of books and articles and categorize the contents by subject, even when that subject isn’t stated explicitly.

For decades, of course, librarians and many others have labeled books and documents with keywords. “But human categorization can only go so far,” said Dr. Blei, an associate professor in computer science. “We don’t have the human power to read and tag all this information.”

To cope with the information explosion, Dr. Blei and other researchers write algorithms so that computers can sift through millions of works and find their common themes by sorting related words into categories. It’s a field called probabilistic topic modeling.

Other research tools identify shifts in language over time that could signal important cultural, scientific or historical changes. At Harvard, Erez Lieberman Aiden and Jean-Baptiste Michel, who jointly lead a group there called the Cultural Observatory, will soon inaugurate a browser that searches for such language changes in a large online repository of scientific papers known as arXiv (pronounced like “archive”).

Users will be able to type in one or two words at the site, called Bookworm-arXiv, and immediately see a graph showing the ups and downs of the phrase’s use in the archive, Dr. Michel said. (A test version is at arxiv.culturomics.org.) Users can then click on the graph and drill down to read the original papers in which the terms appear, tracing ideas back toward their roots, or to spots where scientific ideas spread from one field to another.