The Googlization of Everything

http://www.theatlantic.com/technology/archive/2010/10/inside-the-google-books-algorithm/65422/
How the search algorithm works in Google Books

Rich Results is the latest in a series of smaller front-end tweaks that have been matched by backend improvements. Now, the book search algorithm takes into account more than 100 "signals," individual data categories that Google statistically integrates to rank your results. When you search for a book, Google Books doesn't just look at word frequency or how closely your query matches the title of a book. They now take into account web search frequency, recent book sales, the number of libraries that hold the title, and how often an older book has been reprinted.

So, if you search "Help" now, you get a big blow-up of Kathryn Stockett's 2009 book, not one of the dozens of other books with the same title. Or if you search "dragon tattoo," you get Stieg Larsson's blockbuster, not the 2008 children's book actually called Dragon Tattoo.

"One of the fundamental things we've learned is that the whole is greater than the sum of the parts," Gray said.

This is deeply Google thinking but without the dominant algorithm. It's a Google subspecies that evolved by feeding on a different corpus. There is less data about books than web pages, but there is more structure to it, and there's less spam to contend with. Yet the focus on optimizing an experience from vast amounts of data remains. "You want it to have the standard Google quality as much as possible," Gray said. "[You want it to be] a merger of relevance and utility based on all these things."

Via BoingBoing

You can read Chapter One here.

A Google employee was snooping around in the Gmail accounts of children. Google as fired him.

Of course, Google issued its standard mantra, "We take (insert problem here) very seriously."

See:

"We dismissed David Barksdale for breaking Google's strict internal privacy policies. We carefully control the number of employees who have access to our systems, and we regularly upgrade our security controls--for example, we are significantly increasing the amount of time we spend auditing our logs to ensure those controls are effective. That said, a limited number of people will always need to access these systems, if we are to operate them properly--which is why we take any breach so seriously," Google's Bill Coughran, senior vice president of engineering, said in a statement.

Here are some questions I would have for Google:

• How many Google employees would have access to the content of private accounts?

• What is their level of employment? What background checks does Google do on potential employees?

• Is there a "double key" system, by which one employee may not get access to a private account without another employee -- ideally a supervisor -- logging in as well and looking over his shoulder?

• Will Google call in federal investigators to see if Barksdale should face criminal prosecution? He likely violated federal computer hacking and theft laws.

• The amazing thing is that this sort of thing does not happen more often. Google has thousands of employees on six continents. Can we trust them all? Just as a chain is only as strong as its weakest link, the trustworthiness of Google is only as strong as its sleaziest employee.

• We saw how Google freaked out when people in China breeched security on GMail accounts in December. Why is it less of a crisis when Google employees do it?

From the University of California Press:

Siva Vaidhyanathan

(And Why We Should Worry)

In the beginning, the World Wide Web was exciting and open to the point of anarchy, a vast and intimidating repository of unindexed confusion. Into this creative chaos came Google with its dazzling mission--"To organize the world's information and make it universally accessible"--and its much-quoted motto, "Don't be Evil." In this provocative book, Siva Vaidhyanathan examines the ways we have used and embraced Google--and the growing resistance to its expansion across the globe. He exposes the dark side of our Google fantasies, raising red flags about issues of intellectual property and the much-touted Google Book Search. He assesses Google's global impact, particularly in China, and explains the insidious effect of Googlization on the way we think. Finally, Vaidhyanathan proposes the construction of an Internet ecosystem designed to benefit the whole world and keep one brilliant and powerful company from falling into the "evil" it pledged to avoid.

Siva Vaidhyanathan, Professor of Media Studies and Law at the University of Virginia, is the author of Copyrights and Copywrongs: The Rise of Intellectual Property and How It Threatens Creativity and The Anarchist in the Library: How the Clash between Freedom and Control is Hacking the Real World and Crashing the System.

Advance Praise:

"A powerful and gripping tour de force. Siva Vaidhyanathan uses Google
to examine our capacity for blind faith and to worship innovation as
an end in itself. You cannot read this book and remain unstirred."
-- Tim Wu, author of The Master Switch and Professor, Columbia Law School

"This is a critically important book because it's really about the
Googlization of All of Us. This is a brilliant meditation on
technology, information, and consumer inertia, as well as an ambitious
challenge to change how, where, why, and what we Google. Vaidhyanathan
forces us to think long and hard about taking responsibility for what
we all know and how we know it."
-- Dahlia Lithwick, Slate Magazine

"Vaidhyanathan is everything you could want in a cultural critic:
funny, fantastically readable, and insightful as hell. It's always a
treat when a new Vaidhyanathan comes out."
-- Cory Doctorow, co-editor of Boing Boing (boingboing.net)

"Siva Vaidhyanathan's lively, thoughtful, and wide-ranging book makes
clear, in detail, how Google is reshaping the way we live and work. He
finds much to admire, but also challenges us to not only use Google's
services, but to go beyond them to create a new and genuinely
democratic information order."
-- Anthony Grafton, author of Codex in Crisis

"This is an important and timely topic, and Vaidhyanathan's head and
heart are in the right place to guide the public through the thickets
of 'googlization.'"
-- Paul Duguid, co-author of The Social Life of Information

"Finely written and engaging, this is a book for anyone who has used Google."
-- Toby Miller, author of Makeover Nation: The United States of Reinvention

On the Media

Google is built to support a technocratic way of working. Its founders, Larry Page and Sergey Brin, and most of its early employees are computer scientists by training. It has always been the sort of place where those devoted to solving some of the biggest challenges in logic, mathematics, and linguistics can find a supportive yet challenging environment.41 It's the paradigm of the sort of practice that has emerged quickly over the past twenty years and that now dominates the scientific agenda in many fields: entrepreneurial science--the intersection of academic "pure" science and industrial technoscience.42

This technocratic mode of organization is anything but new. In The Engineers and the Price System, a book published in 1921 that fell into immediate obscurity, the iconoclastic economist Thorstein Veblen identified a new class of what we now call knowledge workers. In the late years of the American Industrial Revolution, Veblen saw that the increase in efficiency of the production and distribution of goods was creating tremendous wealth for the class that owned the means of production yet who were unable to do the mathematics necessary to understand the systems that enriched them. This situation would not stand for long, Veblen surmised. Unlike Karl Marx's unreliable proletariat, waiting to be sparked into revolutionary action by the sudden realization of historical exploitation, the engineering class might actually capture some of the wealth it created, Veblen argued. In fact, engineers could work together to disrupt American industry and bring it down within a matter of weeks. No one else could do that, especially not laborers, who could always be replaced. Because there would always be a shortage of engineers, they had real social and economic power if they chose to use it. If the engineering class succeeded well enough, it could reengineer society, politics, and government as well as the firms themselves. In that event, we might be ruled by a benevolent (or at least competent) "soviet of technicians."43

Google's position as both the dominant firm within its market and a model of how firms should behave in the world realizes Veblen's dream. And the ethos of the company meshes perfectly with one of the paradigmatic modern American values: merit conceived as technical competence. America, Walter Kirn writes, is run by "Aptocrats." These are people who excel at regimented procedures such as standardized tests and other forms of numerically quantifiable achievement. They conform to regimented expectations of excellence and clearly see every rung they must ascend on the ladder of success. "As defined by the institutions responsible for spotting and training America's brightest youth, this 'aptitude' is a curious quality," Kirn writes. "It doesn't reflect the knowledge in your head, let alone the wisdom in your soul, but some quotient of promise and raw mental agility thought to be crucial to academic success and, by extension, success in general. All of this makes for a self-fulfilling prophecy. The more aptitude that a young person displays, the more likely it is that she or he will have a chance to win the golden tickets--fine diplomas, elite appointments and so on--that permit you to lead the Aptocratic establishment and set the terms by which it operates."44 Aptocracy, on which Kirn elaborates in his funny memoir, Lost in the Meritocracy: The Undereducation of an Overachiever, rewards a large measure of gumption in addition to its strata of otherwise "fair" technologies of assessment (test scores, diplomas, and certifications)
Google may be the perfect realization of Aptocracy. Google hires the best of the best from America's top university technological programs. Even those who work in marketing and sales must demonstrate aptitude via tests and gamelike interview questions.45 This focus on standardized, predictable tasks as the measure of achievement is ostensibly fair. Success in America no longer depends so heavily on social status, ethnicity, or gender. Those things still matter, and once in a while a stunningly incompetent exception circumvents the Aptocracy and rises to power , as George W. Bush did. But the Aptocracy has transformed America largely for the better over the past forty years. It has also created the environment in which Google could gestate, grow, thrive, and dominate.46

Google shapes its products as well as its staff along Aptocratic ideological lines. In Web search, a link ends up high on the first page of search results if it has qualified in a mathematically demonstrable way. It must satisfy a number of tests of viability and quality. If it appears to have too many attributes that statistically correlate with untrustworthy pages--if, for example, it contains spam links or obvious attempts to game Google's ranking system--the algorithm will downgrade the page or omit it from the index. A page must have been reviewed and elected by other sites through the affirmative technology of the hyperlink to achieve a high ranking. As with the Aptocracy, members of the Internet elite have more power to determine the standards of excellence in the next iteration of Web search results. The system is always learning, just as the Aptocracy is always adjusting to new inputs and influences among high achievers.
This reliance on technologies to measure aptitude is part of what Neil Postman identified in 1992 as technopoly, or rule by and for technology. Postman was highly critical of what he saw as America's blind dependence on tools and its failure to apply critical thinking and deliberation. If it's new and shiny, Postman lamented, people will adopt it. Soon, the tools seem to set the priorities. They seem to demand more attention and further refinement. And thus real life, or what Postman called "culture," is evacuated of all meaning. It's all about the tools.47

Postman committed the fallacy of assuming that technologies are autonomous, that they have inordinate influence over our behaviors, values, and expectations. He did not appreciate the extent to which people influence and rework technologies.48 Google understands this better than Postman did. It's built to learn. It's designed to absorb influences, for better or worse. That's why the chief product the company delivers to users, the search-results page with links and advertisements, is contingent on the identity, history, and location of the user. The chief product Google sells, users' attention, is also contingent. It changes all the time because people's needs change and because people are fickle. Google is designed to absorb and respond to culture as much as it influences culture.

My latest article in MSNBC.COM about the Google-Verizon deal.

My talk at CUNY conference on "The Digital University":

I have been home since December. It's been great. I have been known to rack up 75k miles flying per year and give talks two or three times per month. But I decided to decline invitations for the first few months of the year.

But I will be back on the road next week. Here is a brief list of where I will be talking:

• March 23: McMaster University, Hamilton, Ontario.

• March 24: University at Buffalo School of Law.

• March 29: Boston University, College of Communications

• April 2: University of Virginia, A Colloquium on the Digital Encyclopédie.

• April 9: University of Virginia, Program on "The State of American Studies."

• April 15: Vanderbilt University, The Curb Program in Public Enterprise and Public Leadership.

• April 21: CUNY Graduate Center: Keynote speaker for "The Digital University"

• April 23: The New York Institute for the Humanities, New York University.

• April 26: The University of Pennsylvania, History and Sociology of Science

• April 30: University of Virginia, Local Support Partners Conference keynote speaker.

• May 12: University of Mary Washington, Faculty Academy keynote speaker.

• May 21: San Francisco Public Library, BayNet Annual Meeting, keynote speaker.

I have a lot more gigs planned in the fall. I am trying not to get overbooked. But if you have a good venue and are not too hard for me to reach from my tiny local airport, I will consider it.

I was on NPR show On the Media discussing the Italian conviction of three Google executives: