Climate Audit

Louis Hissink

Folks,

invest inus, trust us, we will make you rich without ever having to do any work.

Or how some share brokers analyse data.

BradH

Doh! Should have read further down the front page before I pulled the posting trigger, shouldn’t I?

bender

From the link in #3:

Mr. Bradley notes that even though he has published with Mr. Jones and Mr. Mann, he disagrees with some of their recent work together in which they have pushed the temperature record back 2,000 years. “I don’t buy any of that, frankly,” he says. “I think there’s too much uncertainty.”

This shows why I have backed JMS’s view that Bradley is a highly competent dendroclimatologist. Now if only that BCP chronology were updated …

Steve Sadlov

RE:

“Question from Patrick Frank, Stanford University:

The original hockey stick has been shown not just flawed but wrong. Why was the NAS committee unable to clearly state that? A more basic question: There is no analytical theory that can extract a growth temperature from tree ring widths or tree ring densities. On what scientifically valid grounds, therefore, can anyone possibly “calibrate” a tree ring series using temperature vs. time data?

Gerald North:

I am not a tree ring expert, but we had an expert on our panel. We also listened to several experts and I read most of Fritts’s book on the subject. I concluded that there is something to the signals extracted from tree rings. I suppose we disagree on this matter. Tree rings have been studied for about a century now and I believe that they are probably the best indicator we have at this time. Other proxies will undoubtedly become better as they are developed. When this happens we will have a check on the tree rings. So far the tree ring estimates of surface temperature changes do agree with borehole temperatures in ground and ice and with glacial length data for the last 400 years.”

********

Notice how he completely dodged “The original hockey stick has been shown not just flawed but wrong. Why was the NAS committee unable to clearly state that?”

Steve McIntyre

Question from Stephen McIntyre:

The NRC Panel stated that strip-bark tree forms, such as found in bristlecones and foxtals, should be avoided in temprature reconstructions and that these proxies were used by Mann et al. Did the Panel carry out any due diligence to determine whether these proxies were used in any of the other studies illustrated in the NRC spaghetti graph?

Gerald North:

There was much discussion of this matter during our deliberations. We did not dissect each and every study in the report to see which trees were used. The tree ring people are well aware of the problem you bring up. I feel certain that the most recent studies by Cook, d’arrigo and others do take this into account. The strip-bark forms in the bristlecones do seem to be influenced by the recent rise in CO2 and are therefore not suitable for use in the reconstructions over the last 150 years. One reason we place much more reliance on our conclusions about the last 400 years is that we have several other proxies besides tree rings in this period.

Steve McIntyre

C’mon folks. Ask North a question.

Dave Dardinger

re: #11

Personally, I like the way Mann did it better than the others, because it is the long term stuff we want to check on.

What exactly does he mean here? I’m certainly not the expert here, but isn’t this precisely the sort of thing which gets messed up because of either overfitting or colored errors?

Mark T.

That is why the committee was reluctant to put error bars on the early part of the record (or even the late part).

This statement alone highlights the uncertainty. I.e., “we are so uncertain of the results we cannot make valid claims to the level of uncertainty!”

Mark

bender

North says:

(a) “The problem is that in these kinds of reconstructions, the errors are not always quantifiable as they are in purely statisical sampling errors where we can really quantify the error margins. Here we are really into the unknown and the biases are not well understood.”

(b) “When we put the forcing into our models (even with their uncertainties) we are able to link the cause and effect pretty certainly.”

How on earth did he put the uncertainties into the models in (b) if they are not in fact quantifiable in (a)? That was a clever dodge. Fact is, the uncertainties are probably so large that nothing conclusive can be said about the magnitude and precision of the estimate of the A in AGW.

Steve McIntyre

Question from Joel McDade, bystander:

Greetings Dr. North: I am curious what you thought of the primary part of the Wegman Report, that dealing with the statistical issues in Mann, et al. Specifically, the statement (or similar), “Incorrect mathematics + correct result = bad science.” I must say that the NAS Report appeared, to me, to find fault with the Mann methodology but then went on to seemingly endorse the result. The later was the media’s take, anyway. TIA

Gerald North:

There is a long history of making an inference from data using pretty crude methods and coming up with the right answer. Most of the great discoveries have been made this way. The Mann et al., results were not ‘wrong’ and the science was not ‘bad’. They simply made choices in their analysis which were not precisely the ones we (in hindsight) might have made. It turns out that their choices led them to essentially the right answer (at least as compared with later studies which used perhaps better choices).

Joel McDade

Question from Joel McDade, bystander:

Greetings Dr. North: I am curious what you thought of the primary part of the Wegman Report, that dealing with the statistical issues in Mann, et al. Specifically, the statement (or similar), “Incorrect mathematics + correct result = bad science.” I must say that the NAS Report appeared, to me, to find fault with the Mann methodology but then went on to seemingly endorse the result. The later was the media’s take, anyway. TIA

Gerald North:

There is a long history of making an inference from data using pretty crude methods and coming up with the right answer. Most of the great discoveries have been made this way. The Mann et al., results were not ‘wrong’ and the science was not ‘bad’. They simply made choices in their analysis which were not precisely the ones we (in hindsight) might have made. It turns out that their choices led them to essentially the right answer (at least as compared with later studies which used perhaps better choices).

Joel McDade

oops, already posted

Mark T.

If I were to ask a question, it would be:

Dr. North, you state “Despite finding some problems in the seven-year-old study that was the basis of the curve, the panel determined that it is basically correct in its conclusions, which have been corroborated by more-recent work,” however, more recent works such as Rutherford and Mann ’05 have again again corrupted the use of standard statistical analysis methods. In particular, RM05 redefines the expected value of a random vector to be the mean of the expected values of the vector, rather than a vector of individual expected values. I.e., they compute E{E{X}}, which results in a single value (say m_x), and then replicate that value for each random variable in X (which would be each individual proxy) and subtract to generate zero mean random variable. The proper method for “centering” a random vector is to take E{X}, which results in a vector M_x = [m_x1, m_x2, … m_xn], and subtract _that_ from the random vector. If each of the means is nearly the same, this corruption will have subtle effects, however, as in the case with the proxy data, not all means are created equally, or even created within the same ballpark, which can severely bias the results. How then, can “recent works” be used to corroborate previously corrupt methods if they employ equally corrupt methods themselves?

I’m not yet strong enough with the particulars of the impact to back this up. I am working on it, albeit slowly.

Mark

bender

The following questions did not make it to the discussion:

Q: It is often said that the scientific process, under peer-review, is “self-correcting”. As a scientist I recognize this. However it also seems to me that the rate of self-correction is painfully slow compared to the fast wheels of policy. If bad science is fast-tracked to the level of policy, and that science is ultimately reversed, should the victims of that policy be compensated?

Q: I sense an important contradiction on your statement about how uncertainties are handled. You say:

(a) “The problem is that in these kinds of reconstructions, the errors are not always quantifiable as they are in purely statisical sampling errors where we can really quantify the error margins. Here we are really into the unknown and the biases are not well understood.”

(b) “When we put the forcing into our models (even with their uncertainties) we are able to link the cause and effect pretty certainly.”

How do the uncertainties get put into the models in (b) if they are not in fact quantifiable in (a)? (And I realize that (a) was in reference to multiproxies, and (b) GCMs and the like. Still …) Don’t you agree that the uncertainties may be so large that nothing conclusive can be said about the magnitude and precision of the estimate of the A in AGW? For example a CO2 warming projection of 3±4°C dictates a very different policy than one of 3±1°C.

Jean S

This didn’t make it either:

You stated in an earlier answer:
“When we put the forcing into our models (even with their uncertainties) we are able to link the cause and effect pretty certainly.”
Since the climate models are extremely complex, how do you guard against overfitting with only 100 years of instrumental data? Wouldn’t it be exactly here where we need reliable temperature reconstructions?

Jean S

re #26: I think you are on a right track 😉

BKC

How ’bout inviting Dr. North to come and answer some of these questions that didn’t get answered (I had one myself)? After all, he’s not such a bad egg:)

John Hekman

I submitted a question, but the discussion was already closed.

I loved the question by Joel McDade:

Greetings Dr. North: I am curious what you thought of the primary part of the Wegman Report, that dealing with the statistical issues in Mann, et al. Specifically, the statement (or similar), “Incorrect mathematics + correct result = bad science.” I must say that the NAS Report appeared, to me, to find fault with the Mann methodology but then went on to seemingly endorse the result. The later was the media’s take, anyway. TIA

And North answered that, yes indeed, one actually can get the “right” answer from faulty methods. Unbelieveable!!

bender

Re #35
It’s easy to be an “OK guy” when you’re not the protaganist. The issue here is competence, not congeniality. And one should not be too hasty judging North’s competence. Everyone loves a “grandpa” figure. Everyone, that is, except the cold, hard scientist who loves only the facts.

bender

The nice thing is that no one need punch grandpa in this case. Because it’s his arguments that are flawed, not his being. Every rational scientist since Socrates understands that the reason arguments are made public is to maximize the opportunities for punching. So punch away. North has surely punched a few in his time. Maybe even some grandpas.

Michael Jankowski

Re#17:

Gerald North:

Sorry, I do not know these individuals more than acquaintances. Hence, I cannot answer any questions about motivation. I can say, however, that if they could prove the hockey stick or spaghetti graphics wrong, I am sure they would jump to the opportunity “¢’‚¬? and what scientist wouldn’t?

Could he possibly be serious? People are chomping at the bit to take-on the hockey stick and face the slander and attacks of the hockey team – not lining-up to publish the status-quo and be a part of the “consensous?”

Follow the money

“”At minute 55 or so, he describes panel operating procedure by saying that they “didn’t do any research”, that they just “took a look at papers”, that they got 12 “people around the table” and “just kind of winged it.”””

Probably they were led by the nose of permanent staffers juiced into the Kyoto lobbies. First question – who literally wrote the report? It’s narrative structure read like corporate public relations clean-up, not scientific study.

Mark T.

“”At minute 55 or so, he describes panel operating procedure by saying that they “didn’t do any research”, that they just “took a look at papers”, that they got 12 “people around the table” and “just kind of winged it.”””

And somehow this qualifies as a “peer reviewed” report yet Wegman’s does not. Oy vey.

Mark

Mark T.

Would they even be qualified to check the code that RM05 used anyway? Heck, it’s bad enough even for those of us that can read Matlab, and understand the methodology, let alone for those that have neither experience.

Mark

Barney Frank

“”At minute 55 or so, he describes panel operating procedure by saying that they “didn’t do any research”, that they just “took a look at papers”, that they got 12 “people around the table” and “just kind of winged it.”””

Wow! As a non-scientist I would like to say just how much confidence this inspires in me regarding the trillion dollar public policy decisions being based on this ‘research’.
I’m glad to know that they take the prospect of the world’s economy being stood on its ear serious enough to do no research and ‘wing it’. Why not just put on Karnak’s turban and hold the data up to their forehead? Amazing.

mark

Ergodicity w.r.t. which part of my statement? Do you mean the E{E{X}} portion? If so, then maybe they are, but it is not relevant to that implementation since each proxy a) measures something different (not all are tree-rings) and b) each has differing statistics (very easy to show).

As for the filtfilt, it works out to pseudo-sort of work. At least, what they refer to as the high and lowpass pairs really are uncorrelated after the filtfilt function. You can perform the function on white noise, for example, then subtract that result from the original data to get the highpass portion, and do a cross-correlation between them. The r is near 0 with a p near 1. Now, how that mucks with the data used later on, particularly for reconstruction, I have not investigated.

What filtfilt does is provide very steep filtering walls (response squared) which separates out the low frequency data better. However, typical dyadic decompositions use something akin to a quadrature-mirror filter, though their specific implementation obviously is not looking for an exact pi/2 cutoff (I don’t recall, offhand, their actual cutoff point). I.e. typically you would take the “mirror” highpass filter and apply it to the original data to get the proper output. Definitely a non-standard implementation and I think it needs proving to be of use.

Mark

mark

What line are you looking at in the script?

Look at their functions center and center2. They’ve taken Schneider’s original script, which seems to be correct, and modified it with an additional mean/nanmean on the output vector. I don’t recall the specifics offhand. I can probably comment further tomorrow evening when I’m home again (tonight is pool night).

Mark

TCO

Is that Beunadonna?

mark

Their initial “centering,” btw, is done on the years 1899-1959, not even the whole vector length.

That’s the jist of the “centering issue” that has been discussed here.

Mark

Jeff Norman

they got 1,000 “people around the table” and “just kind of winged it”

AR4 anyone?

John A

There is a long history of making an inference from data using pretty crude methods and coming up with the right answer. Most of the great discoveries have been made this way. The Mann et al., results were not “wrong’ and the science was not “bad’. They simply made choices in their analysis which were not precisely the ones we (in hindsight) might have made. It turns out that their choices led them to essentially the right answer (at least as compared with later studies which used perhaps better choices).

Dear Dr Gerald North,

That is a disgraceful answer and you of all people should be ashamed as to endorse faulty methodology which somehow gets the “right answer”. Would you allow any of your students to make such a statement? Would a PhD thesis ever get accepted if the candidate were to make such a statement?

The endorsement of bad scientific methodology that somehow gets the “right answer” is a statement of blinkered religious dogma and not of science. Such a response could equally endorse the cold fusion claims of Pons and Fleischmann or the stem cell results claimed by Hwang woo Suk.

Good science goes hand-in-hand with good scientific ethics, and it is simply unethical to endorse the demonstrably bad and wrong methodology of MBH98/99 because it got “the right answer”.

Steve McIntyre

Could somebody do me a favor – could someone make an audio-clip of NOrth’s comment about “just sort of winged it” which occurs at minute 55 or so, in a form that I can use in a Powerpoint presentation. If so email it to me. Thanks. (I’m editing my presentation and watching Federer v Roddick.)

welikerocks

I found this overview on a site unrelated to climate science; it sure fits:

FIRST, corrupt science is science that moves not from hypothesis and data to conclusion but from mandated or acceptable conclusion back to selected data in order to reach the mandated or acceptable conclusion. That is to say, it is science that uses selected data to reach the “right” conclusion, a conclusion that by the very nature of the data necessarily misrepresents reality.

SECOND, corrupt science is science that misrepresents not just reality, but its own process in arriving at its conclusions. Rather than acknowledging the selectivity of its process and the official necessity of demonstrating the right conclusion, and rather than admitting the complexity of the issue and the limits of its evidence, it invests both process and its conclusions with a mantle on indubitability.

THIRD, and perhaps most important, whereas normal science deals with dissent on the basis of the quality of its evidence and argument and considers ad hominem argument as inappropriate in science, corrupt science seeks to create formidable institutional barriers to dissent through excluding dissenters from the process of review and contriving to silence dissent not by challenging its quality but by questioning its character and motivation. In effect then, corrupt science is science that is flawed in both its substance and its process and that seeks to conceal these essential flaws. It is essentially science that wishes to claim the policy advantages of genuine science without doing the work of real science.

John Davis

#61
This keeps raising its head, and was the subject of a delightfully confused piece some while ago at RC. The argument is that if CO2 is the most significant factor, then large(~ish) changes in temperature over a period when we know CO2 to have been relatively stable imply absolutely horrendous changes in temperature in prospect from the current and future projected levels.
Looking at it another way, a significant MWP would cast huge doubt on the whole CO2-AGW hypothesis since, from the argument above, it would imply that temperatures should already be massively higher than they are in reality.
Looking at it yet another way, a significant MWP would imply that there is more than one thing that affects the global temperature. Maybe that big shiny thing in the sky that keeps us warm?