As you know, we are eagerly awaiting the publication of the following article by Wahl and Ammann reported here:
Wahl, E and C Ammann (In press). “Robustness of the Mann, Bradley, Hughes reconstruction of northern hemisphere surface temperatures: Examination of criticisms based on the nature and processing of proxy climate evidence.” Climatic Change (accepted).
May 10, 2005 — In review
September 27, 2005 — Revised
December 12, 2005 — Provisionally Accepted
February 28, 2006 — Accepted for Publication
I have made many posts on Wahl and Ammann – see here for Category and have been following the snail-like progress of this article for some time as it slouches towards Boulder. I objected to its inclusion in IPCC AR4 on the basis that it did not meet IPCC publication deadlines of acceptance by December 2005 and in print by February 2006. This objection was disregarded. Their revised version is the version that includes their confirmation of our result that MBH verification r2 was ~0 – although they and UCAR had issued a press release that our results were unfounded. As I reported previously, Wahl and Ammann withheld this information in their first draft and included the information in their revised version only after I filed an academic misconduct complaint against Ammann.
Their Climatic Change submission relies on a concurrent submission to GRL for their benchmarking of the RE significance. Their GRL submission, said to be “in review” in their version filed with IPCC, had in fact been rejected before they filed their revised version with IPCC.
I’m intrigued as to what the final Wahl and Ammann version will look like. They have an intriguing choice: the inclusion of a reference to this article in AR4 was premised on their article being “in press” which would prohibit them from re-working their article to deal with the GRL rejection. But the article needs to be re-worked since it will look pretty silly to describe their GRL article as “under review” over 18 months after it has been rejected.
I check Climatic Change from time to time to see how they are coming along. Today, to my great surprise, I noticed that Climatic Change had published an Ammann and Wahl article online on August 24, 2007 abstract . However, it’s not the Robustness article. It’s a different article: Caspar M. Ammann and Eugene R. Wahl, The importance of the geophysical context in statistical evaluations of climate reconstruction procedures. They continue to toil in MBH vineyards however.
Abstract A portion of the debate about climate reconstructions of the past millennium, and in particular about the well-known Mann-Bradley-Hughes (“MBH” 1998, 1999) reconstructions, has become disconnected from the goal of understanding natural climate variability. Here, we reflect on what can be learned from recent scientific exchanges and identify important challenges that remain to be addressed openly and productively by the community. One challenge arises from the real, underlying trend in temperatures during the instrumental period. This trend can affect regression-based reconstruction performance in cases where the calibration period does not appropriately cover the range of conditions encountered during the reconstruction. However, because it is tied to a unique spatial pattern driven by change in radiative balance, the trend cannot simply be removed in the method of climate field reconstruction used by MBH on the statistical argument of preserving degrees of freedom. More appropriately, the influence from the trend can be taken into account in some methods of significance testing. We illustrate these considerations as they apply to the MBH reconstruction and show that it remains robust back to AD 1450, and given other empirical information also back to AD 1000. However, there is now a need to move beyond hemispheric average temperatures and to focus instead on resolving climate variability at the socially more relevant regional scale
The submission-acceptance schedule says the following:
Received: 22 August 2000 Accepted: 13 June 2007 Published online: 24 August 2007
This seems like a long time for acceptance of this article, even for Ammann and Wahl; the topic also seems prescient for 2000; I presume that there is a misprint. It is also curious that this article, accepted only on June 13, 2007, has been published prior to their Robustness article “accepted” in February 2006.
Anyway I eagerly await the final version of this article to see exactly what decision Stephen Schneider, editor of Climatic Change, took, when faced with the Hobson’s Choice described above.
Climate Audit 
42 Comments
Caspar is in Antarctica with Waldo, who incidentally is wearing a white outfit instead of his usual stripes.
Caspar is eating a Wein-Berger.
I think this is a relevant statement:
Global warming as a global average, or even as a hemispheric average, is essentially meaningless since the various regions of the world show differing trends. The relevance of global warming is the impact on human societies. If Antarctica and Greenland are not significantly melting, then rising sea level will not be a major threat (even though Hansen now says “In stark contrast to estimates put forward by the IPCC, Prof. Hansen and his colleagues argue that rapidly melting ice caps in Antarctica and Greenland could cause oceans to swell several metres by 2100 “).
Regional trends are more important, however, as seen so often in the analysis of details, most regions of the world have a lack of data.
This loss of correlation should be essential part of the Hockey Stick course, http://www.7minds.org/climate/eas8001/syllabus_table.htm , in these two sessions
Multiproxy reconstructions : verification
Lab session 4 : Using pseudoproxies
My suggestion for the Lab session:
Use MBH98 algorithm (contact UC if you need a Matlab version)
1. Generate ideal pseudoproxies (no noise at all)
2. Step by step, add noise to pseudoproxies (say red noise with p=0.32). Add noise variance until verification r2s fall below 0.02
3. Repeat for each MBH98 period, report SNRs.
4. Discussion
The authors’ claimed chronology for the original paper does not add up. Looking at the journal, all of the recently published papers were accepted in 2007. As a journal editor myself, it is not plausible that one article would be kept aside for later publication, other than perhaps by an issue or two to get it into a particular special issue. I interpret that the claim by the authors “February 28, 2006 Accepted for Publication” is false in some way. But if it is false, where did all those dates (in review, revised, etc.) come from? Could they really make up a series of dates like that? If they are not made up, then what has happened?
Steve, if I were you, I would ask the editor what has happened to the paper.
It’s interesting that Caspar is no skeptic. The word (spelled “Kapar”) means “jester” in Czech. 😉
Mann’s 07 Robustness of proxy-based climate field reconstruction methods refers to the study in question several times. The following quotes are related to my comment in #3 :
#3 (UC): After seeing this presentation (see slide #12) by the instructor, I’m a bit skeptical if she has really understood how MBH9x was created or if she will implement any ideas for the course presented here.
Steve, the cited paper has this citation (my ephasis):
Wahl ER, Ammann CM (this volume) Robustness of the Mann, Bradley, Hughes reconstruction of Northern Hemisphere surface temperatures: Examination of criticisms based on the nature and processing of proxy climate evidence. Climatic Change, (in press)
Also the citation on page 4 (line 2) says “this issue”. So I think “Caspar” will appear any time soon.
Oh dear, here we go again:
WTF?!? Since other (part of) time series has a different mean, one should not use correlation?!?
Then the paper continues
Does not make any sense to me, can someone please translate it to plain English? Are they really saying that it does not matter if the verification r2 is zero (series are orthogonal) if the mean is correct?!!?!
Jean S, it’s worse than that: they say it’s expected and thus is not bad!
Re #9
What do you think would happen if they tried to publish stuff like that in a statistics based journal? Has anyone pointed out how easy it is to get an overfit on one degree of freedom?
Re #7
Wasn’t Kim Cobb one of those who SteveM thought could be an up and coming / clear thinking climate scientist? Very little in the way of clear thinking in that presentation, it is the usual list of unconvincing / weak claims. AlG would be proud of that presentation, I can’t say from a scientific viewpoint it is very impressive. I guess Kim has realised that clear, objective thinking is not a good way to get grant money in climate science. The 99.999% slide title is disgraceful and unscientific.
There is another related paper (cited by this “new WA”) out:
BO LI, DOUGLAS W. NYCHKA, CASPAR M. AMMANN: The hockey stick and the 1990s: a statistical perspective on reconstructing hemispheric temperatures, Tellus A (OnlineEarly Articles).
Unfortunately, I do not have rights to download it.
The first author (wow, a young statistician!) has some related material over here. I suppose they have a new reconstruction (shown on slides 31-32 here).
Re 12 – zoom in on the blue line where it merges with the instrumental. Guess what you find?
Jean S August 29th, 2007 at 2:29 am,
I’m as confused as you are. What the heck is the specific energy balance change they refer to?
Isn’t the energy balance always changing? Heat transfer time constants and all that. Varying input. Changing output.
Jean:
I think they are saying that if there is variability but no relationship the R2 is zero, which is true but bizarre and no basis for rejecting the use of R2.
I also have no idea why they are focused on the 1854 to 1901 period.
As for Bo Li’s presentation – it is way beyond me, but my instincts tell me to think carefully about slide #35. That seems like a caveat that includes the proverbial kitchen sink.
I am sure it would be chalked-up to an innocent oversight, but is that not as bad as, say, a little white lie on a resume – the kind of thing people are forced to resign for? Especially since they need to rely on that GRL submission as a supporting reference.
I find it hard to believe it wasn’t done unknowingly. Looks like a case of f-r-a-u-d to me. Or are we to believe Amman and Wahl are that stupid and irresponsible?
They are sunk without trace. If the temperature record is wrong (as it seems to be, after all the Hansen et al fiddles), then the calibration of the proxies to that record must also be wrong.
Also, why can’t they get a proper statistician to do the statistics? How many times do we have to read junk out of the Juckes/Mann school of voodoo statistics?
Also #12 Bo Li is a sometime student of statistics, see:
http://www.image.ucar.edu/~boli/
#16. There was an earlier precedent. For their first draft, I acted as an “anonymous reviewer” and in that capacity asked for information on the verification r2. They refused to provide this information to me as a reviewer and told Stephen Schneider that their GRL article had shown that we were “deeply” wrong or some such phrase. Unbeknowst to them, in my capacity as a GRL author, I had been informed by GRL that their first GRL submission had been rejected. So they were purporting to the editor that this article was still alive after it was rejected.
As a reviewer, I objected strongly to this deception in this review. In business, I would certainly have no truck with someone who pulled a stunt like that. How could you trust anything that they say? I also said that the article should not rely on results from unaccepted papers. I was fired as a reviewer.
Wahl and Ammann resubmitted their GRL article a 2nd time. I’ve posted our Replyhere. After making us do another Reply, it was rejected a 2nd time. Their Clim Chg re-submission once again relied on GRL re-submission.
Actually, now that I think about it, I think I can guess what’s going on here (and I haven’t seen the new article yet.) I’ll bet that this new Clim Chg article re-cycles their rejected GRL article and that they will change the citations to their “GRL, under review” to the new article. Pretty amazing for an article that was supposedly “in print” in February 2006.
Why is Climatic Change treated as a serious academic journal ?
I wonder how the hockeystick proxies (excluding the Bristlecone and other suspects) would look when calibrated against the raw, unadjusted data from the “good” rural temperature stations?
bernie (#15), I don’t there there are any hidden caveats in the methodology of this reconstruction. This (Bayesian ensamble approach) reconstruction might be even statistically defensible unlike the other reconstructions I’ve seen. I especially like the slides #5 (lays out the model assumptions) and #34 (explaines ways to improve reconstructions; some of these points have been discussed here also). However, they seem to be using (some version of) MBH99 proxies…
Anyhow, I won’t comment more before reading the actual paper.
#21. The proxy list are the 14 AD1000 MBH proxies using Mannian principal components and bristlecones. Nychka, one of the coauthors of this study, was on the NAS panel that said that Mannian principal components shouldn’t be used and bristlecones should be avoided.
So why is Nychka using them here? (Mannian PC calculatins have less impact on the AD1000 North American network because it is already dominated by bristlecones through longevity.)
At the NAS Panel, Christy asked Mann whether he calculated a verification r2 statistic for the AD1400 network and what was its value. Needless to say, Mann did not give a straightforward answer: he said that he had not calculated the verification r2 statistic as that would be a “silly and incorrect thing” to do. On the previous day, Ross and I had shown evidence that he had calculated a verificaiton r2 statistic – indeed the source code produced for the House Committee showed that it was calculated in the same step as the RE statistic.
As one of the statisticians on the Panel (And I had objected to Nychka being compromised), Nychka should have followed up on this, both on the falsehood that Mann had not calculated the statistic and the falsehood that it would be “foolish and incorrect”. Nychka sat silent like a bump on a log.
In the public session afterwards, I criticized the Panel for their passivity on this matter – which had been specifically raised in the original questions to Mann (with Mann storming out of the room as I got up to speak. Later it was said that he had to “catch a plane”, but he was observed chit-chatting with one of his handlers from NSF.)
Afterwards, Nychka came up to me and said that just because the Panel didn;t say anything didn’t mean that he hadn’t observed Mann’s failure to answer the question. Well, Nychka didn’t say that in the report and now he re-emerges as a coauthor with Caspar Ammann on this topic. (And BTW I had objected to Nuchka’s inclusion on the panel on precisely these grounds under NAS rules without ever getting an answer.)
Jean:
Slide #35 says:
My naive assumption is that the “e.g.” needs to be proved before using proxies and that proxies should have similar relationships to temperature – otherwise you get the bristlecone PC effect where the PC does not reflect some component of all proxies but a set or subset of proxies. Do I have this wrong?
California is a case in point vis a vis Pielke Sr’s thesis regarding the importance of land use changes on regional climate. Yes, there is a regional hockey stick here, having, possibily, nothing to do with the radiative balance of the atmosphere as a whole. Bristlecones may well reflect our regional hockey stick up through the 1980s, more recent divergence problems notwithstanding. I am not saying one way or the other, but I do find the clinging to Mannian 20th century proxies, which are overweighted by California, interesting.
Anyone interested in the Clim.Change paper who can’t download it can mail me at ewcz@seznam.cz.
And the Tellus paper also. Please omit the fullstop behind the email.
That means they are non-linear, and certainly not even close to stationary. Sheesh.
Mark
#27 Mark, your use of “non-linear” may be an incorrect use of the word for this situation. “If proxies change their realationship to temperature over time” doesn’t this mean that they can’t be used as proxies at all unless the change of relation with respect to time can be shown. However, the article is clear that they still had calibration problems.There are possiblilities that it is linear if they had a good measurement of “substance X”. I would assume upon reading the article, it is more about what is unknown; and a wake-up call to look for mystery substance X.
#21.
I downloaded the proxies and compared them to the MBH99 version archived at WDCP that I had downloaded previously. They are similar but none of them match. It’s not just a linear transformation as the correlations between versions are about 0.7 in nearly every case.
The authors say:
Jeez, why would anybody not check something that they got from Mann against other sources. Poor Bo Li may find that she’s been punked by Mann. Of course, the irony, as Jean S and UC realize, is that it doesn’t really matter what version you feed in to the Mannomatic. If you aren’t bothered by a verification r2 of ~0, just take one spurious regression, mix in noise, re-inflate the variance, and presto, you’ve got a hockey stick. They’ve got so focused on the PC trick that they aren’t watching the pea under the thimble in the regression phase.
John:#28
That was exactly my concern.
SteveMC.
is the temp series that Bo li has on her web page the one she used?
and that’s a NH series right?
What percentage of the Northern Hemisphere is covered by Bristlecone Pines?
Rather less than 2%, I guess. So according to Hansen they are insignificant in the great scheme of things.
Yes, I think that a time-varying relationship renders their use as proxies inappropriate since stationarity is generally assumed a requirement (at least, wide sense stationarity), though there are ways around this. However, linearity is still a problem. If a given set of inputs can create more than one output, then the system is necessarily non-linear, though it may be linear in the sense that there is still some one to one relationship at a given time, which is what I think you were referring to, right? Channel estimation schemes overcome this by shortening processing intervals to a period much shorter than that resulting from the channel variation. Either way, at least in the case of tree-rings, it is well known that the temperature to ring width relationship IS indeed non-linear.
Mark
It’s worse than ‘time varying’ though, isn’t it? I thought I saw a couple of references to Bristlecones being especially CO2 sensitive in the first place. Meaning they’re a truly daft choice for this particular problem.
Oh, I was only referring to that specific comment, i.e. “The Team” acknowledging the very thing that invalidates their method. By hypothesis, CO2 and temperature are correlated, but they are also inputs to tree growth, apparently moreso with BCPs. PCA is horribly unsuited to deal with correlated inputs. How on earth can you tell which of the two correlated inputs is generating the output? It’s absurd, absolutely absurd. Now we have Nychka actually publishing a paper in which he’s defending the very idea a panel he sat on refuted. What on earth was this guy thinking?
Mark
#12,
One symptom of overfitting, loss of r2 in verification is missing from the page 9 😉
#29
I just put that data through MBH machine, resulting 1000-1400 series has a correlation of 0.62 with the original (not exact correlation, but remarkable). Interestingly, in the calibration period, the correlation is 0.83.
That’s exactly what they mean by word robust.
#9 Jean
It is mostly panic-induced hand-waving, but let’s try to understand. First, if we assume that the reconstruction error is independent of the temperature, we’ll get a nice equation that relates (variance) Signal to Noise ratio and r2, r2=SNR/(SNR+1). Now, if signal variance goes down, but noise level doesn’t, r2 will be reduced. IOW:
In addition, verification reference temperature is presumably a little bit noisier than calibration reference temperature. The problem is that signal variance doesn’t really go down that much, and r2 drops significantly ( 1820-1980 step from 0.76 to 0.06 *)
Another explanation would be that proxies first low-pass filter global temperature, then add independent noise to the result. See Wahl-Ammann Figure S1, middle. The problem here is twofold; the assumption is silly and high r2 during calibration remains unexplained..
* My emulation result, Wahl and Ammann preprint claims 0.76 to 0.2
Having a theory concentrates the mind. The lower graph on slide 32 is in error. It smears the data so that the abrupt rise in temperature in 1939 is averaged out to about 1935.
Take the SST raw data. Remove the Folland and Parker correction. Try again. That’s my advice.
JF
RE3, it seems that Mann has almost done this, in
Reply to Comments on Testing the Fidelity of Methods Used in Proxy-based Reconstructions of Past Climate
Click to access MRWAZoritaReplyJClimate07.pdf
Here’s one example (Fig 1) with SNR 0.4,
( http://signals.auditblogs.com/files/2007/09/pseudopr.png )
I wonder what verification R2 is in this case. ( I don’t know the answer to #3, just interested)
The issue raised in this thread were contested by Julien E-G here.
Julien entirely misses the point of this thread.
Ammann shouldn’t have been “struggling” with a publication deadline if his article had actually been accepted in February 2006 as he claimed to IPCC in order to make it eligible for citation in AR4. The issue raised in the Where’s Caspar post was how the published Wahl and Ammann would deal with citations to their rejected article – an issue raised a long time ago. The answer came soon after when we saw that the “accepted” article had been changed. The citations to the rejected article were changed to citations to an article accepted only in summer 2007 (at the same journal LOL).
The reason for the delay became clear: even a journal as compliant as Climatic Change probably drew a line in the sand at citing a rejected article (especially after we’d made it an issue here) and made Wahl and Ammann get the rejected results published somewhere. As it turned out, they had to publish the rejected results in Climatic Change itself. After they did this, the article originally “accepted” in February 2006 was actually published – after a publication delay almost unprecedented (or at least unusually long) in the recent history of the publishing journal.
If one examined the correspondence on this file, there’s little doubt in my mind that, after the rejection of the GRL article, CC’s acceptance of Wahl and Ammann became conditional on them getting the cited results into some journal or other and in the end, Stephen Schneider wasn’t going to embarrass the process and eventually had to hold his nose and publish the companion paper containing the rejected results himself.
And the previously rejected results are …?
Read this one?
Ammann, C. M. and Wahl, E. R. 2007. Importance of the geophysical context for statistical evaluation of climate reconstruction procedures. Clim. Change, doi: 10.1007/S10584-007-9276-X
One Trackback
[…] general discussion Li et al 2007 never really got off the ground. (See also a few comments by Jean S and others in an unrelated CA […]