Friday, February 25, 2011

External validation

Robin Hanson points to an article by Vladimir M about when you can trust the results of an academic field.
When looking for information about some area outside of one’s expertise, it is usually a good idea to first ask what academic scholarship has to say on the subject. In many areas, there is no need to look elsewhere for answers: respectable academic authors are the richest and most reliable source of information, and people claiming things completely outside the academic mainstream are almost certain to be crackpots.

Robin's angle is Bayesian. He argues that we should trust academics by default, because they are experts, but that we should adjust our level of belief if the field is ideologically driven:
However, let us accept for the sake of argument that all else equal in ideological fields intellectual progress is slower, and claims tend to be make with more overconfidence. What exactly would this imply for your beliefs about this area?

I have a rather different perspective, and several of the commenters seem to agree. I think of academia as a club of intellectuals, but not one that is always motivated by the search for objective truth. Many academics don't seem to be bothered with objective truth at all, but more are interested in being at the forefront of some movement. As one commenter points out, such academics are more like lawyers advocating a cause than they are experts seeking the truth.

A better way to find experts in a field is to look for external validation. Look for people and ideas that have stood some test that you don't have to be an expert to verify. You don't have to know much about artificial intelligence to know that IBM is good at it., because they've proven it with their Watson Jeopardy player.

No comments: