External validation

Robin Hanson points to an article by Vladimir M about when you can trust the results of an academic field.

When looking for information about some area outside of one’s expertise, it is usually a good idea to first ask what academic scholarship has to say on the subject. In many areas, there is no need to look elsewhere for answers: respectable academic authors are the richest and most reliable source of information, and people claiming things completely outside the academic mainstream are almost certain to be crackpots.

Robin's angle is Bayesian. He argues that we should trust academics by default, because they are experts, but that we should adjust our level of belief if the field is ideologically driven:

However, let us accept for the sake of argument that all else equal in ideological fields intellectual progress is slower, and claims tend to be make with more overconfidence. What exactly would this imply for your beliefs about this area?

I have a rather different perspective, and several of the commenters seem to agree. I think of academia as a club of intellectuals, but not one that is always motivated by the search for objective truth. Many academics don't seem to be bothered with objective truth at all, but more are interested in being at the forefront of some movement. As one commenter points out, such academics are more like lawyers advocating a cause than they are experts seeking the truth.

A better way to find experts in a field is to look for external validation. Look for people and ideas that have stood some test that you don't have to be an expert to verify. You don't have to know much about artificial intelligence to know that IBM is good at it., because they've proven it with their Watson Jeopardy player.

Brad on Watson

Brad's Ideas has a number of neat comments about the Watson Jeapordy match. One thing Brad mentions made me wonder about Jeopardy mechanics:

You can buzz in as soon as Trebek stops speaking. If you buzz early, you can’t buzz again for 0.2 seconds. Watson gets an electronic signal when it is time to buzz, and then physically presses the button. The humans get a light, but they don’t bother looking at it, they try timing when Trebek will finish. I think this is a serious advantage for Watson.

I agree that this is a big advantage to Watson. However, I don't understand why it works this way. The reason you can't buzz early is because it makes the game annoying to watch and unpleasant to participate in. Alex would constantly be interrupted as the contestants race to buzz in and cut him off.

However, making contestants hit the buzzer just when Alex finishes adds a substantial dexterity element to the game. Watson has an advantage in this dexterity game, but who cares? Everyone wants to know if Watson is smart, not if he can press a button with millisecond precision.

A better way to handle the buzzer, it seems to me, would be to let people buzz in early but not to count it until Alex finishes the question. At that point, if more than one person has buzzed in, the winner is selected randomly. Otherwise, it's a speed race, with no precision to the timing. This slight change in the algorithm should remove the precision timing from buzzing in and make it more of a race to figure out the answer the fastest.

Ken Jennings reflects on Watson

Ken Jennings, arguably the all-time champion at Jeopardy, has a great article on Slate. He reflects on his and Brad Rutter's match with Watson.

I expected Watson's bag of cognitive tricks to be fairly shallow, but I felt an uneasy sense of familiarity as its programmers briefed us before the big match: The computer's techniques for unraveling Jeopardy! clues sounded just like mine. That machine zeroes in on key words in a clue, then combs its memory (in Watson's case, a 15-terabyte data bank of human knowledge) for clusters of associations with those words. It rigorously checks the top hits against all the contextual information it can muster: the category name; the kind of answer being sought; the time, place, and gender hinted at in the clue; and so on. And when it feels "sure" enough, it decides to buzz. This is all an instant, intuitive process for a human Jeopardy! player, but I felt convinced that under the hood my brain was doing more or less the same thing.

Computers 1, Humans 0

One of the two Watson Jeopardy games has now been televised, and Watson won handily. Watson has $35,734, Rutter has $10,400, and Jennings has $4,800. Watson has more than twice the other two's scores combined. We'll find out tonight whether the computer can hold its lead.

It was an interesting match to watch. The audience was the most excited Jeopardy audience I've ever seen, and they were rooting for Watson. When he took a guess and got it right, there was a thunderclap of applause. When he had to make a wager, as with a daily double, they broke up laughing at Watson's odd-ball wagers such as $6,345.

The game was much closer than the score indicates. For many of the questions, all three contestants would know the answer, and it was a race to ring in the fastest. On many of them, if Watson had needed six seconds rather than five to figure out its answer, a human would have rung in first.

One thing I was surprised about was the Final Jeopardy question. The category was "U.S. cities", and I thought Watson would knock it out of the park. I thought it would bet high and answer it easily. The opposite was the case. The computer had no idea what the names of airports mean. Apparently, even with all the time contestants are given for Final Jeopardy, it couldn't connect the dots from "World War II" to "O'Hara" and "Midway". Yet, it still did okay in the end, because it only bet about $1000 on the question. Did it bet so low because of the category, or did the programmers have Watson be categorically cautious in Final Jeopardy?

I don't know, but one thing is clear. Humanity has met its trivia overlords, and they're made of silicon. This game show duel is just a spectacle, but take a moment to look what it means for the future. The way Watson is competing ultimately relies on a large, natural-language database. Unlike with Cyc or the Semantic Web, the computer doesn't need humans to explicitly re-encode information into a machine-friendly synthetic language. It is directly using the natural-language texts we wrote for communicating among ourselves.

The applications are far-reaching. At the simplest, Watson hints at greatly improved web searching, both when looking for pages and when looking for information. Other applications are in medicine. Imagine a Watson that had read a hospital's case files and a medical school's entire library, including the latest journal articles? No doctor can match that. For knowledge workers in any domain, imagine how much it would help to have a personal Watson that had read every email the person had ever sent or received? Natural-language processing has just passed a tipping point.

More about Watson is available on the Jeopardy web site.

Fight with fire and get burned

I almost wrote earlier about Google's promotion of other Google sites like Maps as part of its search result, but I didn't. It seemed too silly. Now I read that the issue might be heating up after all. In a recent interview, Google spokespersons claim that there is strong activity in D.C. to turn the issue into a serious anti-trust case.

One of Google's defenses is that there's no such thing as a neutral web search algorithm:

Google has recently brought in executives to discuss how neutrality can’t be achieved in search because the company’s algorithm is based on subjective factors to ensure users get accurate results.

I completely agree. A neutral web search is bound to be a bad web search. A good web search is heavily biased toward what the user wants, and any search provider is going to have to use their judgment on what the user will want.

However, I also feel the same about network neutrality. The Internet is not a star topology, where we route packets into a central router and then it spits them back out to other people connected to it. It's a complex network that no one in the world has a full map of. The routing decisions at each node can be quite complex, and good routers are almost never going to treat all packets the same. A neutral router is a bad router.

Additionally, I always felt the same way about browser neutrality. The best operating system shells have thorough integration with a web browser instead of just forking off a web browser application. It makes for a better user interface, and software engineers seem to agree if you look at what they build rather than their opinion on this case. There have been a number of good OSes with built-in browsers, including two by Google: Android and ChromeOS. A browser-neutral OS is a bad OS.

I hope Google does not get stuck with search neutrality provisions. I must say, though, that they invited the wolf into the hen house.

Watson, the new Deep Blue

IBM worked long to build a chess computer that can rival the world's best chess players. Now they have built a computer that does something harder: answer pointless trivia questions. Harder for a computer, anyway. Airing starts tonight, and I can't wait.

Part of the fun of it is that this is an application that still requires one of the biggest computers that anyone on the planet can build:

Well the body of Watson, which is not on camera, is like 10 huge refrigerators, in an air-conditioned room.... Watson has the power of a couple thousand laptops working together.

On the down side, it's not a particularly elegant machine:

Instead of just trying one approach, saying 'We're going to do it statistically' or 'We're going to teach the computer a million different rules about the world,' IBM just tried a very pragmatic approach, which said that if there's anything that works, we're going to include it in this. So they've got hundreds of different algorithms all trying a different approach to understanding the clue and trying to figure out the answer. And then it's up to the computer to look at thousands of different candidate answers and pick out the one that's most likely to be right.

POSIX for Java

I've long felt that Java sorely lacks for not having a standard POSIX interface. Every significant platform that the full version of Java runs on has POSIX, so it's no loss in portability to outright support it. Yet, the lack of POSIX support can really hurt applications that want to do something like scripting. Developers on the ground are left using crummy hacked-together scripting languages just because they provide POSIXY things, most especially process control. Process control is key because once you have good process control, you can shell out for the rest.

I'm over three years late to notice it, but JRuby has an excellent solution to this problem:

So the idea behind JNA (jna.dev.java.net) is to use a foreign function interface library (libffi in this case) to wire up C libraries programmatically. This allows loading a library, inspecting its contents, and providing interfaces for those functions at runtime. They get bound to the appropriate places in the Java interface implementation, and JNA does some magic under the covers to handle converting types around. And so far, it appears to do an outstanding job.

With this code in JRuby, we now have fully complete chmod and chown functionality. Before, we had to either shell out for chmod or use Java 6 APIs that wouldn't allow setting group bits, and we always had to shell out for chown because there's no Java API for it. Now, both work *flawlessly*.

I wonder how this is working out in practice? I wonder if anyone's tried it for Scala? I wonder if Oracle will adopt it into the Java mainline?

Interestingly, Squeak went through a similar history with its own native method interface. The initial version marked native methods with an integer index, and that index was looked up in a table of function pointers inside the VM. The next iteration was like JNI, and because it named native methods with strings, you could link in new ones with shared libraries. After that came an iteration much like JNA, called FFI. Like JNA, Squeak's FFI lets you access existing library directly without needing to write any glue code beyond a specification of the type signature of what you're calling.

The main differences for the Squeak version of this history are (1) they did it years earlier (2003 versus 2007), and (2) the platform maintainers immediately adopted it as a core feature. It would be great if Java could adopt the same thing as a core feature. Is there even a JSR for it, though? My initial Google searches aren't turning anything up.

Accreditation in software engineering

The Wall Street Journal has an article up on licensing. They make a claim that looks correct to me:

"Occupations prefer to be licensed because they can restrict competition and obtain higher wages," said Morris Kleiner, a labor professor at the University of Minnesota. "If you go to any statehouse, you'll see a line of occupations out the door wanting to be licensed."

The article covers a lot of the common issues, including mention of some of the more ridiculous license regimes such as for barbers and manicurists.

In software engineering, accreditation is much less prevalent than in other professions. The places I am aware of accreditation cropping up are for the more pigeon-hole parts of the craft, jobs where there's a well-defined set of tasks and a well-defined skill set for meeting those tasks. Examples are maintenance of Oracle databases or of Microsoft Exchange servers. Even in these cases, there is nothing illegal about performing the task without a license, and I'd hazard that more such systems are run without than with a formally certified technician.

The majority of software engineers I've encountered have no such license or certification at all. They are more jacks of all trades. They have all worked in some specialty or another at some point or another, but they've rarely completed an entire certification course. When hiring such an engineer, it's relatively complicated to evaluate them compared to licensed professions, but if you choose well, you typically get an enormously more productive worker than simply grabbing someone who has jumped through the right hoops.

All of this seems pretty good to me. Thus, unlike workers in most professions, I would argue against any formal licensing for software engineers. There are three big reasons:

• The track record of attempts is not good. Some large-scale efforts, such as those with UML-based death by paperwork, have been real productivity sinks for anyone who tried them. Other large-scale efforts, such as extreme programming, are so loose and hazy they are more rules of thumb than anything that could be taught and tested in a certification program. The best I've seen have been local customs adopted by individual development groups, but I've even seen a lot of these work out pretty miserably in practice and be retracted after a year or two.
  
  
• The profession is very young, and its practices are still being developed. For example, distributed version control has gone from theoretical to the norm in the last ten years. Garbage collection has gone from theoretical to the norm in the last thirty years. Even if we found some genius of software engineering who could describe today's best practices, it would quickly go out of date.
  
  
• The nature of the profession is to innovate. Unlike with plumbing or with shaving whiskers, a large proportion of new software written is something that's never been done before. It's wrong to think of most software as building a cog to fit into a well-specified gear in a well-specified machine. It's better to think of it like a business plan or a social arrangement or a new kind of tool. What kind of accreditation would make sense for the creators of Google, Facebook, or the Phillips screwdriver?
  

Will Exim support its users or its philosophy?

I wrote several years ago that I don't like when software lectures its users about how their suffering is going to help the greater good:

It is frustrating when programmers decide to lecture their users instead of supporting them. I just spent over an hour trying to get my mail to relay through EPFL's mail servers using the restrictive channels EPFL allows, only to find that Exim has its own restrictions. The two sets of restrictions are workable by themselves but do not have a solution together. The only way to make Exim and EPFL both happy is for them not to talk to each other. Bleh. I think I will stick with EPFL and replace Exim.

I still feel the same way. Both EPFL's admins and Exim's maintainers were unreasonably stubborn. Good software helps its users in their actual situations, not in some idealized alternate world. Good software works on crummy half-baked networks instead of needing everything around it to be in perfect condition.

Curiously, there's a chance now for Exim to change its mind and support SSMTP after all. Simon Arlott has written a drop-in patch that adds the support to Exim and linked it on my ancient feature request to Exim. Will Exim take the patch? The Exim of 2006 would have said no way. What will the Exim of 2010 say? Do they want users, or obscure philosophical purity?

DNS shutdowns are up and running

Via Freedom to Tinker, I read:

ICE executed seizure warrants against the 10, ATDHE.NET, CHANNELSURFING.NET, HQ-STREAMS.COM, HQSTREAMS.NET, FIRSTROW.NET, ILEMI.COM, IILEMI.COM, IILEMII.COM, ROJADIRECTA.ORG and ROJADIRECTA.COM, by demanding that registries redirect nameserver requests for the domains to 74.81.170.110, where a colorful "This domain name has been seized by ICE" graphic is displayed.

As I've written earlier, this is a bad way for people to civilly coexist on the Internet. Let me count a few of the ways:

• DNS is a crude weapon. Disabling a domain name is like disabling an entire postal zip code. The collateral damage can easily be larger than the intended damage.
  
  
• This option isn't needed if you convict the defendant, because then you have legal rights against the defendant's business assets, anyway. It's only useful if you are preemptively shutting down a business that you haven't had time to bother taking to court.
  
  
• There's no good legal theory where the U.S. government has authority over .com and .net addresses. If the U.S. has rights over .com and .net, why not Canada, or the state of Florida? Why not the Bahamas?
  
  
• One of the sites, Rojadirecta, has already successfully defended itself in court. In Spain. The U.S. has successfully turned off their DNS record anyway.
  

It all comes across as rather thuggish. To briefly try and outline how we might approach these issues in a more principled and civil way, it might go like this:

• Leave DNS and routing alone, much like we leave speech mostly free. This implies that law enforcement can't do much about people who are broadcasting copyrighted material from the North Pole, but realistically, they can't anyway. They can still convict the native citizens that download it.
  
  
• If a wrong has happened, then try the person or organization that did the deed. Don't go after a DNS provider, an ISP, a router manufacturer, a software author, or any other intermediary who merely provided a general-purpose tool.
  

Tech Talk on Scala+GWT

Grzegorz Kossakowski gave a tech talk on his Scala+GWT project that is now online.

Slides are available here.

The talk was around September, and there has been a lot of progress since then. For example, the system can now compile and run a much larger subset of Scala.

Does IPv6 have hope?

That probably sounds like a strange question. 6 > 4, and IPv6 is simply the newer version of IPv4. Everyone will eventually be upgraded, right?

That's not really accurate. IPv6 is not backwards compatible with IPv4. Despite the name, it's really a different protocol. From a technical perspective, asking if we will all upgrade to IPv6 is similar to asking if we will all sidegrade to IPX. The only difference is in the branding. IPv6 sounds like an upgrade.

Dan Bernstein has a great article up on the technical issues with migrating to IPv6. In general, the plans that IPv6 advocates are discussing involve every node on the Internet upgrading to support IPv6 and IPv4 simultaneously, and then we can make a big switchover. At first blush, this sounds like a good plan. It worked for HTTP, and as Bernstein points out, it worked for MX records for SMTP servers. However, so far at least, IPv6 isn't designed to work that way.

The problem is that, unlike with these other examples, it's not a simple software upgrade to simultaneously support IPv6 and IPv4. Most distressingly, every node on the Internet needs to additionally have an IPv6 address. That alone is a fatal flaw. It's simply not going to happen. With the MX record transition, nodes without an MX record simply fell back on their A record, which they already had. With the HTTP transition, HTTP/1.1 was and remains an optional extension. Every node can always fall back to HTTP/1.0 or even HTTP/0.9. With IPv6, however, any node that doesn't have an IPv6 address simply doesn't get to play.

The same thing is happening with DNS. In addition to allocating all those IPv6 addresses, you need to add them to DNS. Until every DNS record is updated to have both IPv6 and IPv4 addresses in it, it's not possible to flip the switch.

As a separate issue, does anyone really care about the other features of IPv6 other than the extended address space? IPv6 comes with a host of features, and some of the more complicated and computationally expensive ones like IPsec are mandatory. These strike me as things that would better as optional extensions, and indeed, most of these features except the larger address space are already being explored as such.

Overall, the main thing that IPv6 brings us over IPv4 is the larger address space. Why not make that an optional extension, too? The IP packet format already allows for extensions. Routers trying to forward an extended-address packet could simply ping each other before doing it, and if the next router in the chain doesn't support them, bounce back a "host not reachable" packet. Lots of software would need updating, e.g. the socket APIs used on end point software would need to support optional, longer addresses. However, the changes would be much smaller than are required to support IPv6.

So if the best transition to big address spaces is to extend IPv4 rather than stage a simultaneous leap to IPv6, what is going to happen? One possibility is that such an IPv4++ will be designed, it will be rebranded as IPv6, and everyone will simply ignore the failed experiments with a more radically different IPv6. Another possibility is that such an IPv4++ will be rebranded as IPv7. This is all branding and politics, though I must say that the most honest thing would be to simply call it IPv4.

Another possibility is that everyone interested in big addresses will get suckered into the IPv6 quagmire, and it just won't happen. This isn't clearly a bad thing. The "small" address space of IPv4 is plenty large if we continue to have an Internet that is a patchwork of interconnected networks rather than a true globally controlled network. With the smaller address space, my coffee maker can still send a packet to your coffee maker, but from each coffee maker's perspective it will be sending packets out into the cloud. That seems healthy, to me. Perhaps you want to support virtual coffee makers, or to have transparent coffee maker failover. It's not really my business exactly how you route my packet in your network. Why would you want to export a node address to me that tries to pinpoint a specific machine?

A more exotic possibility is that some other network gains market share. Some network that offers real, pressing advantages, unlike IPv6. A packet-switching network protocol is like a social network or an instant messaging system. New networks take over old ones by initially offering something attractive enough that people will operate on both networks simultaneously. Once enough people are on the new network, they can start taking it for granted, and the old network can deflate in usage very rapidly. Really, though, what possible improvement would the packet switching layer have that would encourage that initial batch of people to use the new one in parallel to the old one? Clearly it's nothing in the laundry list of features in IPv6, because adoption has been really tepid. Further, all the really good network improvements have been possible to retrograde onto IPv4.

My best guess is that we continue on with IPv4 plus extensions. More tentatively, I would guess that we never get around to extending it for larger address spaces. If larger address spaces do become a pressing concern, however, I'd expect IPv4 to be extended rather than for the whole world to waste time on switching to completely new protocol. It's just good engineering.