It Ain’t Necessarily So

“It ain’t necessarily so. It ain’t necessarily so….”

So begin the lyrics of a George Gershwin hit song from Porgy and Bess. The song goes on to dissect various tales told in the Bible: David slaying Goliath? Jonah staying alive in the belly of the whale? Methuselah living 900 years? Be skeptical, it warns.

Skepticism is healthy. If a sucker is born every minute, that leaves too much opportunity for those who would fool us. We don’t expect deception from our scientists, though, do we?

In his book, Wrong [Little Brown, New York, 2010], business and science writer David H. Freedman details the many ways scientific research can go astray, producing results and conclusions that we ought not believe. Prof. John Ioannidis, M.D., of Tufts University has presented a series of research publications demonstrating that a majority of research papers in top-tier scientific publications have subsequently been shown to be incorrect. Freedman gives numerous examples in the body of his book and an extensive appendix.

Why does science sometimes fail us?

People have goals that conflict with being scrupulously honest. Scientists want success: prestige or even fame, promotion, money. If their work produces an interesting new result, they are more likely to succeed than if the results are inconclusive or uninteresting. The best has to be new and good. A professor once devastated a doctoral candidate: “Some of your work is new and some of it good. Unfortunately, the new material is not good and the good material is not new.” The same can be said for many published studies.

Do scientists publish all their data? No. Often there are good reasons not to: the equipment broke mid-way; the conditions were not as specified for testing the hypothesis; some of the data were lost accidentally…. Then there are “obvious outliers,” results the researchers think clearly must be flawed, though they do not know why. Freedman cites a review of over one hundred drug studies published in eminent journals and found “in most of the studies some data were left out—and more often than not these were data that didn’t fit the conclusions.”

Freedman notes that the “ultimate form of data cleansing” is not submitting research results for publication because the results are somehow disappointing. He gives as an example a 2008 study showing 23 of 74 antidepressant drug trials were not published…all but one showing no benefit from the new drug. Somehow correcting for what never gets published is a major problem when “meta-analyses” are done, grand syntheses summarizing a multitude of published results to try to get at the over-all truth.

What do you do if your study has not turned out as hoped? If you have measured many things versus many other things, you may find a relationship or two or three merely by chance. Coincidence happens. You may “move the goalposts.” A commercial is currently running for a product that does not seem to have shown a medical improvement, but has “helped more people reach their goal” than did a competitor’s product. Freedman quotes British medical statistician Douglas Altman, “It’s rather like throwing darts on a wall and then drawing a dartboard around them.” Altman had found that in more than half the studies he reviewed, the stated focus of the work differed significantly from what the authors had originally proposed to do.

Correlation does not prove causation. Milk and egg production numbers are highly correlated from location to location, but cows are not producing eggs nor hens, milk. Where there’s smoke there’s fire, but the smoke does not produce the fire, so even in that connection, cause and effect can be confused. Measure enough aspects of the human condition and you will find interesting correlations that are deceptive. Does drinking diet soda make you fat or do fat people tend to drink diet soda in unsuccessful efforts to become thin? Healthy people exercise more, Freedman notes, but we do not know which is cause and which is effect.

Who or what is being tested? Psychology studies are often done with college students, but the results may be extended to the very different population at large. Medicine given to the ill may cause side effects not seen in the well. Behavior, physical characteristics and genetics, demographics (age, sex, ethnicity, income, wealth)…all potentially confound the data and possibly fool the analysts. Freedman writes “The Penn State biological anthropologist Kenneth Weiss and his colleagues have compile a long list of widely published epidemiological links that have failed to hold up….” Some of your favorites are probably there, including the effect of sunlight exposure on your risk of developing cancer. Eat butter or margarine? Prevent asthma with cleanliness or judicious dirt?

“There are three kinds of lies,” wrote American writer, novelist, and philosopher Mark Twain, “lies, damned lies, and statistics,” quoting a saying of his time. Statistics are useful in summarizing data, for example when computing the average or mean. Data variability can be summarized with the standard deviation, but most people will classify that as “too much information.” Used to draw conclusions, statistics are dangerous. They can help decide whether numbers sampled from Group A indicates the group likely is truly above or below some criterion or in comparison with another group. When used to determine causes or to make predictions, however, they are less reliable.

Does working for a government agency influence what questions you seek to answer and how you design a test and analyze its results? How about working at a university, relying on government or business money to support your research, your staff, your salary? Does working for a business? Will your boss support publishing results that contradict his own beliefs or those of your organization? Have these organizations employed saints or people? To ask these questions is to answer them. Are the significant associations of the researchers made clear in the descriptions of the authors that accompany their publications? Potential conflicts of interest abound, and Freedman cites data to that effect.

When the research is reviewed for potential publication, the reviewers are influenced by their own biases. A trendy paper will get less rigorous review than an unfashionable one. Does smoking cause cancer? Yes. Does second-hand smoke? Rarely, if ever, but try to get that published. [Calculate the amount of smoke inhaled second-hand and compare it with direct smoking dose/response relationships.]

What to do? Be skeptical. Freedman lists tip-offs to help you judge the more trustworthy advice.

Less trustworthy:

1. “It’s simplistic, universal, and definitive.”

2. “It’s supported by only a single study, or many small or less careful ones, or animal studies.”

3. “It’s groundbreaking.”

4. “it’s pushed by people or organizations that stand to benefit from its acceptance.”

5. “It’s geared toward preventing a future occurrence of a prominent recent failure or crisis.”

Advice we should ignore:

6. “It’s mildly resonant.” Sounds about right. Comfortable resolution, solution. Tempting.

7. “It’s provocative.” That gets your attention. Being fat is good for you! Maybe not.

8. “It gets a lot of positive attention.” This is the “bandwagon effect”…hop on to be with the crowd.

9. “Other experts embrace it.” How could that be a negative indicator? There are fads in science as elsewhere, though scientists try hard to prevent them. Who is an expert? Who has only a passing interest plus a Ph.D.?

10. “It appears in a prestigious journal.” This is a two-edged sword: the best stuff may be found there, and the most likely to mislead.

11. “It’s supported by a big, rigorous study.” How bad can that be? Wait until another, or even two more, studies back up the first.

12. “The experts backing it boast impressive credentials.” All that glitters is not gold.

More trustworthy advice:

13. “It doesn’t trip the other alarms.” In other words, items 1-12 above.

14. “It’s a negative finding.” This means the study concluded they did not find a relationship between x and y and possibly z, or they could not confirm their initial hypothesis.

15. “It’s heavy on qualifying statements.” The researchers avoid sweeping generalities and use, for example, “some” and sometimes” rather than “all” or “always.” Even just using “in our study” indicates the researchers have recognized some limits.

16. “It’s candid about refutation evidence.” Freedman quotes a droll paraphrase of Isaac Newton’s Third Law of Motion, “for every Ph.D., there is an equal and opposite Ph.D.” Studies that acknowledge conflicting data or opinions and deal with them are to be trusted over those that do not…generally.

17. “It provides some context for the research.” What are the major issues? What is known already? What supports this or does not and why?

18. “It provides perspective.” What does this mean to average readers and to those most deeply involved?

19. “It includes candid, blunt comments.” If it is true and important, then it should be willing to say so.

Why did I post this under “Culture” here at asiancemagazine.com? Because, as British novelist and chemist C. P. Snow influentially noted fifty-seven years ago, there are Two Cultures: The arts (literature, theater, music, painting, sculpture…) and the sciences (physics, chemistry, mathematics, engineering…), and the well-informed individual knows a substantial amount about both. In January 2013, the British newstatesman.com proudly announced “C P Snow’s epochal essay published online for the first time,” and epublish it they did. One erudite Brit said about culture “it is…ways of believing….” Yes, so much that we are asked to believe comes from scientific research, unfortunately often of doubtful value.

Keats wrote of the Grecian urn that, in its limited viewpoint, “beauty is truth and truth, beauty: that is all you know and all you need to know.” The arts aim to bring us beauty. The sciences aim to bring us truth. We want both, yet much art is ugly. As for much that science has proclaimed: well, it ain‘t necessarily so.

**********

Dr. Cooper is a retired scientist, now a writer, author and writing coach. His first book, Ting and I: A Memoir of Love, Courage and Devotion, was published by Outskirts Press in 2011 and is available from Outskirts Press, Amazon, and Barnes and Noble, in paperback and ebook formats, as are his subsequently co-authored memoirs The Shield of Gold and Ava Gardner‘s Daughter? and the memoir he recently edited, High Shoes and Bloomers. His writer-coaching web site is http://writeyourbookwithme.com.