Brad on Watson

Brad's Ideas has a number of neat comments about the Watson Jeapordy match. One thing Brad mentions made me wonder about Jeopardy mechanics:

You can buzz in as soon as Trebek stops speaking. If you buzz early, you can’t buzz again for 0.2 seconds. Watson gets an electronic signal when it is time to buzz, and then physically presses the button. The humans get a light, but they don’t bother looking at it, they try timing when Trebek will finish. I think this is a serious advantage for Watson.

I agree that this is a big advantage to Watson. However, I don't understand why it works this way. The reason you can't buzz early is because it makes the game annoying to watch and unpleasant to participate in. Alex would constantly be interrupted as the contestants race to buzz in and cut him off.

However, making contestants hit the buzzer just when Alex finishes adds a substantial dexterity element to the game. Watson has an advantage in this dexterity game, but who cares? Everyone wants to know if Watson is smart, not if he can press a button with millisecond precision.

A better way to handle the buzzer, it seems to me, would be to let people buzz in early but not to count it until Alex finishes the question. At that point, if more than one person has buzzed in, the winner is selected randomly. Otherwise, it's a speed race, with no precision to the timing. This slight change in the algorithm should remove the precision timing from buzzing in and make it more of a race to figure out the answer the fastest.

Ken Jennings reflects on Watson

Ken Jennings, arguably the all-time champion at Jeopardy, has a great article on Slate. He reflects on his and Brad Rutter's match with Watson.

I expected Watson's bag of cognitive tricks to be fairly shallow, but I felt an uneasy sense of familiarity as its programmers briefed us before the big match: The computer's techniques for unraveling Jeopardy! clues sounded just like mine. That machine zeroes in on key words in a clue, then combs its memory (in Watson's case, a 15-terabyte data bank of human knowledge) for clusters of associations with those words. It rigorously checks the top hits against all the contextual information it can muster: the category name; the kind of answer being sought; the time, place, and gender hinted at in the clue; and so on. And when it feels "sure" enough, it decides to buzz. This is all an instant, intuitive process for a human Jeopardy! player, but I felt convinced that under the hood my brain was doing more or less the same thing.

Computers 1, Humans 0

One of the two Watson Jeopardy games has now been televised, and Watson won handily. Watson has $35,734, Rutter has $10,400, and Jennings has $4,800. Watson has more than twice the other two's scores combined. We'll find out tonight whether the computer can hold its lead.

It was an interesting match to watch. The audience was the most excited Jeopardy audience I've ever seen, and they were rooting for Watson. When he took a guess and got it right, there was a thunderclap of applause. When he had to make a wager, as with a daily double, they broke up laughing at Watson's odd-ball wagers such as $6,345.

The game was much closer than the score indicates. For many of the questions, all three contestants would know the answer, and it was a race to ring in the fastest. On many of them, if Watson had needed six seconds rather than five to figure out its answer, a human would have rung in first.

One thing I was surprised about was the Final Jeopardy question. The category was "U.S. cities", and I thought Watson would knock it out of the park. I thought it would bet high and answer it easily. The opposite was the case. The computer had no idea what the names of airports mean. Apparently, even with all the time contestants are given for Final Jeopardy, it couldn't connect the dots from "World War II" to "O'Hara" and "Midway". Yet, it still did okay in the end, because it only bet about $1000 on the question. Did it bet so low because of the category, or did the programmers have Watson be categorically cautious in Final Jeopardy?

I don't know, but one thing is clear. Humanity has met its trivia overlords, and they're made of silicon. This game show duel is just a spectacle, but take a moment to look what it means for the future. The way Watson is competing ultimately relies on a large, natural-language database. Unlike with Cyc or the Semantic Web, the computer doesn't need humans to explicitly re-encode information into a machine-friendly synthetic language. It is directly using the natural-language texts we wrote for communicating among ourselves.

The applications are far-reaching. At the simplest, Watson hints at greatly improved web searching, both when looking for pages and when looking for information. Other applications are in medicine. Imagine a Watson that had read a hospital's case files and a medical school's entire library, including the latest journal articles? No doctor can match that. For knowledge workers in any domain, imagine how much it would help to have a personal Watson that had read every email the person had ever sent or received? Natural-language processing has just passed a tipping point.

More about Watson is available on the Jeopardy web site.

Watson, the new Deep Blue

IBM worked long to build a chess computer that can rival the world's best chess players. Now they have built a computer that does something harder: answer pointless trivia questions. Harder for a computer, anyway. Airing starts tonight, and I can't wait.

Part of the fun of it is that this is an application that still requires one of the biggest computers that anyone on the planet can build:

Well the body of Watson, which is not on camera, is like 10 huge refrigerators, in an air-conditioned room.... Watson has the power of a couple thousand laptops working together.

On the down side, it's not a particularly elegant machine:

Instead of just trying one approach, saying 'We're going to do it statistically' or 'We're going to teach the computer a million different rules about the world,' IBM just tried a very pragmatic approach, which said that if there's anything that works, we're going to include it in this. So they've got hundreds of different algorithms all trying a different approach to understanding the clue and trying to figure out the answer. And then it's up to the computer to look at thousands of different candidate answers and pick out the one that's most likely to be right.