Evaluating a scientific theory

I mentioned there is a second issue I'd like to raise about the leaked emails between Phil Jones and Michael Mann: how can scientific theories be evaluated by non-specialists?

It is often claimed that they can't, but I don't agree. One of the distinguishing characteristics of a scientific theory is that it makes objective claims that can be evaluated by anyone. You don't have to wear the right regalia or attune yourself with a spirit walk. The theory should make objective claims, and the experiments should be repeatable.

For that matter, full-time workers in an area have challenges compared to an outsider. One is that they tend to be highly specialized, and thus not familiar with the arguments supporting the foundational issues of a field. For example, practically no one working on programming language design spends a lot of time really studying why it is that computers have proven useful in the larger economy. It's just taken for granted. A second problem is that anyone making a living is highly biased in favor of any theory that will keep that living going. PL folks are eager to accept any reason that PL is important.

So outsiders are not only capable, but an important part of the process. What should they do, though? Let me describe two strategies.

First, anyone can evaluate the ultimate predictions of a theory. You don't have to be a Galileo to observe the results of his famous falling bodies experiment. More closely to my world, you don't have to be a football expert to know that the University of Georgia's football team is doing poorly this year. It takes a specialist to make a better theory of gravity, and it takes a specialist to make a good football team, but anybody can look at the results and see whether they are any good or not.

A second approach is to explore the logical implications and structure of a theory. As a simple example, a theory that contradicts itself is clearly problematic. Any theory that concludes both A and not A is a broken theory. More commonly, a theory makes an obscure claim A that implies a more accessible conclusion B. If someone tells you about some interesting property of phlogiston, try to find out what observable implications it has on real fires. It might be that the implication B is plainly false, in which case again the theory is broken. More frequently, B is contentious but not certainly false. Proponents need to either go further out on a limb and accept B, or they need to weaken A so that it no longer implies B. It takes a specialist to choose which way to go, but anybody can evaluate the whole package of logical implications.

That's two ways to evaluate a scientific theory from the outside. Overall, I believe such evaluation is not only possible, but that it's necessary for a scientific field to be healthy.

Players make poor referees

The recently leaked emails between Phil Jones and Michael Mann raise a number of issues about scientific progress. I'd like to address two of them.

As background, the emails are between major researchers and activists in the climate change debate. Here is a sample of what has observers excited:

In one e-mail, the center's director, Phil Jones, writes Pennsylvania State University's Michael E. Mann and questions whether the work of academics that question the link between human activities and global warming deserve to make it into the prestigious IPCC report, which represents the global consensus view on climate science.

"I can't see either of these papers being in the next IPCC report," Jones writes. "Kevin and I will keep them out somehow -- even if we have to redefine what the peer-review literature is!"

Here we see two people influential with the IPCC conspiring to eject papers that conflict with their preferred conclusions. As a result, we cannot believe that the IPCC is giving a balanced summary of the research that is outstanding, thus undermining what the IPCC claims to do.

What to make of it? What I'd like to emphasize in this post is that it's not bad, by itself, that Jones and Mann are taking sides. The problem is that they are trying to wear two hats, two hats that are particularly incompatible.

To make an analogy, think of scientific claims as sports teams. How do you find out whether a particular sports team is any good? Really, there's no other way than to field the team against other sports teams that are also believed to be good. No amount of bravado, no amount of popularity, is really going to convince an unbiased observer that the team is really good. Ultimately, it needs to play against good teams and win.

The tricky part is here: What counts as playing against a good team? To resolve this, sports have rules that are laid out to be as objective as possible, and they have referees adjudicate the games to make sure the rules are followed. Referees are monitored to make sure that they are applying the rules correctly and fairly, but since the rules are objective, this is a relatively straightforward task. The team players, meanwhile, can try a variety of strategies and techniques. It's hard to judge whether the strategies and techniques are good by themselves, but it's not hard at all to tell who won a fairly refereed sports game.

Bringing it back to science, if Jones and Mann are to be faulted, it's because they are claiming to act as referees even though they are actively taking sides. I don't know the particulars of how the IPCC is organized nor of what influence these two have in it, but it doesn't take a specialist to know that players make poor referees.

David Kravets on ACTA

David Kravets writes in Wired that he is not real happy with ACTA, especially with the way it's being developed:

Dan Glickman, the MPAA’s chairman, informs lawmakers that millions of film-related jobs are in peril because of internet piracy. Simply put, those who don’t back the proposed Anti-Counterfeiting and Trade Agreement don’t support intellectual property rights, he wrote.

“Opponents of ACTA are either indifferent to this situation, or actively hostile toward efforts to improve copyright enforcement worldwide,” Glickman wrote.

That’s an insultingly black-and-white viewpoint. It’s also not an accurate description of the treaty’s critics.

I am not hostile to intellectual property in general, but I oppose ACTA. I simply think we should take it as a constraint that we not give up basic liberties like being able to copy a CD for a friend. I don't think we necessarily have to protect incumbents as technology improves, but in this case copyrights can still be protected and made profitable. For example, we could loosen copyright but tighten rights on public performance.

More fundamentally, this issue should be discussed in public, and with representation by people trying to devise a better way. The ACTA negotiations appear to be between two groups of people: national officials and current copyright holders. I expect the first group is offering police work, the second group taxes, and they'll close the deal once they've agreed on how much of each.

David Pollak is happy about using Scala

David Pollak is happy about using Scala:

I've been a JVM guy since '96, so finding a language that was as on the JVM was a plus for me. I was looking for a statically typed language with high performance, but with the syntactic economy of Ruby. I bounced around a couple of language listing sites and found Scala. Three years ago, I fell in total love with Scala. That love continues today.

I can't say I blame him.

Google releases the Go language

The Go language's unique feature is very fast builds of large projects, due largely to careful support for intermediate libraries exporting a minimal API. Other aspects are: it is garbage collected, it has actors-like concurrency, it has objects, it uses structural typing for objects, it has updated syntax, and its syntax is downright quirky. It also leaves out a lot, including: inheritance, parametric types, and operator overloading.

Google releases Closure Compiler

From the announcement:

Closure Compiler is a JavaScript optimizer that compiles web apps down into compact, high-performance JavaScript code. The compiler removes dead code, then rewrites and minimizes what's left so that it will run fast on browsers' JavaScript engines. The compiler also checks syntax, variable references, and types, and warns about other common JavaScript pitfalls.

There's also a large library and a UI templating system.

Cory Doctorow on ACTA

Cory Doctorow is worried about what he finds. For example:

ISPs have to proactively police copyright on user-contributed material. This means that it will be impossible to run a service like Flickr or YouTube or Blogger, since hiring enough lawyers to ensure that the mountain of material uploaded every second isn't infringing will exceed any hope of profitability

.

Sites like YouTube have enough clout that I don't expect them to simply be shut down outright. On the contrary, their backing companies will be invited to the negotiating table, where they will help craft a treaty that has enough exceptions for them to operate.

That's great for YouTube, but there is a chilling problem for future innovation. It is unlikely that the next great network service will comply with whatever exceptions are carved out. Thus, if ACTA goes through, large swaths of potential society-changing services will end up requiring an act of congress to even get off the ground.

The Anti-Counterfeiting Trade Agreement (ACTA)

I believe copyright should be rethought in light of computers. We rethought it for the printing press, and computers are enough of a reason to rethink it again. Naturally, large incumbent companies, such as the ones represented by the RIAA, want to increase law enforcement enough that they can continue their business models even though the content is digitized and is distributed over the Internet.

That said, I even more strongly believe that the issue should be discussed in public, not worked out behind closed doors. The public should not be treated as masses of sheep on gaming board. The Anti-Counterfeiting Trade Agreement (ACTA) is being worked out, right now, in just that way. The only people invited are major government agents and their personal choices.

Naturally, the details of ACTA are hard to come by, but the emphasis seems to be on imposing liability on the third parties that distribute content. That's worrisome in general, because improvements in content distribution is one of the key ways that the Internet can continue to advance human well-being. For that matter, retiring dinosaur business models is also key to improving human well-being. I don't see anything good for the public coming from these negotiations. Perhaps that's why they are being held so quietly.

If the Obama Administration wants to prove its commitment to government transparency, there is an opportunity here.