Tikalon Blog by Dev Gualtieri

Tikalon Header

Peer Review

October 8, 2012

Peer review forces scientists to design their experiments well enough that there's no question of the result; or, craft their theories such that there are no divide by zero errors . Today, we only think of computers when we hear the phrase, "divide by zero," but before computers this was a subtle error that allows you to prove anything. For example,

a = b

a² = ab (multiplication by a)

a² - b² = ab - b² (subtracting b²)

(a - b)(a + b) = b(a - b) (factoring)

a + b = b (through division by (a - b))

b + b = b (since a = b)

2b = b

2 = 1

So, what happened? Our problem was at the division by (a - b), which is division by zero, since a = b. It's easy to see this in such simple equations , but the equations of physics are not that simple, so such problems are harder to find. It's reported that Einstein made such an error in one of his published papers, but I could find no definitive reference.

Stephen Wolfram thinks the universe is a computer; or, at least, it can be analyzed as one (perhaps I simplify). His ideas, presented in his book, A New Kind of Science ,[1-2] were a lightning rod for criticism by other scientists, so much so that I have a bound volume of critical articles about his book that's nearly as thick as the book, itself. This is quite impressive, since his book is nearly 1200 pages.[2]

The interesting thing is that all this criticism did not deter me from buying the book. After all, the topic was interesting to me since I've worked in science and with computers; and, something having that much buzz must be worthwhile reading, whether my opinion would eventually fall to one side, or the other. I recommend the book (freely available online, and in ebook format), but it's no Principia .

Title page from the 1687 first edition of Newton's Principia

Title page from the 1687 first edition of Isaac Newton's Philosophiae Naturalis Principia Mathematica , commonly called the "Principia."

In this book, Newton integrated (pun intended) physics and mathematics. Wolfram attempted to do the same with physics and computation.

(From the Université de Strasbourg, via Wikimedia Commons).

This idea, that criticism of scientific papers does not always detract from their popularity, is analyzed by Filippo Radicchi in an aptly named paper, "In science "there is no bad publicity": Papers criticized in technical comments have high scientific impact."[3] In this case, the comments analyzed are the formal, published comments on papers that are encouraged by some journals , sometimes as letters to the editor .[4]

Technical comments are worrisome to authors, since they are almost always negative. There's the idea that criticized papers will have low scientific impact , but Radicchi found otherwise. In analysis of commented papers in thirteen major publications, he found that commented papers have citation rates much higher than those of non-commented articles.

Not only that, but those papers are also the most cited in those journals, and Radicchi thinks that commentary on a paper is a predictor of its future impact. This is a validation of the popular proverb that "any publicity is good publicity ," in which even negative criticism is beneficial. Radicchi says this demonstrates "the fundamental importance of scientific disputes for the production and dissemination of knowledge."[3]

Citation rates for commented papers in Physical Review Letters and Science

Citation rates as a function of year for commented and uncommented papers in Physical Review Letters and Science . The average citation rates for commented papers are shown as bars, and the average of uncommented papers is indicated by the lines. A single commented paper in Physical Review Letters in 1996 had more than 20,000 citations.[5] (Fig. 1 (modified) of ref. 3, from the arXiv Preprint Server).

In any event, there's one comment that no scientist wants to hear. The comment, "...Not even wrong ," was first spoken by the theoretical physicist , Wolfgang Pauli . It means that what's proposed can't be falsified by an experiment , so it falls outside the realm of science.

References: