Climate Audit

Mann and realclimate have argued that MBH98 was replicable on the original record. Mann cites Wahl and Ammann as support for this, but Wahl and Ammann are close associates of Mann’s (Ammann is a realclimate contributor), working recently from a much different record, hardly “independent” and their code only addresses one area of MBH98 calculations. Cubasch is a better test; he was not able to replicate Mann – see here; also von Storch has reported Mann’s “shoddiness” here. As I pointed before, Wahl and Ammann’s replication does not do anything that we had not already done – their calculated RPCs under identical assumptions are identical to mine – see here, here and here. There are many issues which cannot be replicated even after one Corrigendum and an extensive Corrigendum SI, as well as issues that could not be replicated on the pre-Corrigendum record.

I’f you go to the Replication tab in the Categories frame, you will get some of my previous comments on this topic.

To clear up one misconception that many people have: there is a little code on Mann’s FTP site here – for the calculation of tree ring principal components. I know this; our GRL article discusses this code in detail. What’s missing is the rest of the code – for the calibration of proxies, the estimation of reconstructed PCs, the calculation of the NH temperature index, the application of Preisendorfer’s Rule N to tree ring networks, etc. etc. See the listing below. The existence of problems in the code already available makes it all the more mandatory to look at the rest of the code.

Here are matters which can now be mostly analyzed, but could not be replicated on the pre-Corrigendum record:

1. PC tree ring calculations. This is misrepresented in MBH98, but is not corrected in the Corrigendum itself. We’ve discussed this in our GRL and EE articles. This is the one bit of code that is at Mann’s FTP site, and, without inspecting code, this misrepresentation could never have been identified. The fact that a misrepresentation was identified in the only code available so far seems to me a good reason for inspecting other code.
2. The start date of the Gaspé series. MBH98 “edited” this series so that it would be included in early 15th century calculations, where it has a material impact. The start date was misrepresented in MBH98.
3. 35 series listed in the original SI as being used were not used. Mann et al say that the presence/absence of these 35 series “does not matter”. Why is it that the presence/absence of the bristlecones matters so much?
4. 2 series used in MBH98 were not listed in the original SI. The Corrigendum acknowledges this, but fails to give a location for the 2 series other than Stahle, pers. comm. Since the first 120-125 years of these 2 series are identical to 2 other series, it looks like near-duplicate versions of the same site (probably Spruce Canyon CO) have been used;
5. unreported truncation of the Central England, Central Europe and chin04 series. This truncation was carried forward from Bradley and Jones [1993] where it was also unreported.
6. use of “grey” versions of proxy data inconsistent with citations and versions archived by the authors;
7. erroneous geographical location of precipitation series. (These were not corrected in the Corrigendum. “The rain in Maine falls mainly in the Seine”) see MM03 and here
8. use of JJA data for Central England and Central Europe series instead of stated annual data;
9. calculation of temperature principal component series weighted correctly by the square root of the cosine of the latitude (instead of the cosine of the latitude) [von Storch et al 2004]. There has been much criticism of an error by McKitrick and Michaels pertaining to an incorrect calculation of cosine of the latitude, but silence by Mann et al. on this very similar error of their own. See here

Since we began examining MBH98, two substantial data archives have been made publicly available that were not publicly available in 2003. First, a directory at Mann’s FTP site at University of Virginia, not identified in prior correspondence and not previously publicly referenced, suddenly materialized in November 2003 after our first article. The FTP site itself had been established only in July 2002, long after publication after MBH98, and was not accessible to contemporary scientists or to IPCC TAR. Secondly, an extensive new archive was required by Nature in connection the July 2004 Corrigendum, which provided much new data and additional information on methods.

Even with all this new information, there are many items that cannot be replicated. Although Wahl and Ammann claim to have “reproduced” MBH98, there is a whole litany of things that their code shows no evidence of “reproducing”. Here is a quick list off the top of my head, which I’ll probably add to later:

1. the selection of proxies which was supposedly done according to “clear a priori” criteria. The “clear a priori” criteria were not reported and have not been disclosed in response to inquiries. Without a statement of these “clear a priori” criteria, it is obviously impossible to replicate the proxy selections of MBH98.
2. the selection of tree ring chronologies listed in the original SI according to the criteria listed in Mann et al [2000] (which expanded on MBH98 information); here
3. the explanation in the Corrigendum for the discrepancy between the tree ring sites listed as being used in the original SI and the tree ring sites actually used. The exclusion of the excluded sites (and the inclusion of included sites) cannot be replicated according to the stated criteria. here
4. the data set archived in July 2004 does not match the description provided in MBH98 or in the Corrigendum SI. here
5. the proxy rosters in each calculation step from pre-Corrigendum information, including a total of 159 series said to have been used in MBH98 [Mann et al , 2003] here
6. the use of 24 proxy series in the AD1450 step as reported in MBH98 here
7. failure to use 6 available proxy series in the AD1500 step (including 5 series used in the AD1450 step) here
8. the selection of the 1082 “dense” gridcells and 219 “sparse” gridcells according to the selection criteria stated (for the first time) in the Corrigendum SI. here
9. the archived “sparse” and “dense” instrumental series here
10. the number of retained temperature PC series in each calculation step. The number retained appears to depend on short-segment standardization, which we criticized in connection with tree ring series. The number retained cannot be replicated with non-erroneous PC methods.
11. the number of tree ring PC series retained in each network/calculation step according to the retention policy (Preisendorfer’s Rule N) reported at realclimate.org [link] in December 2004. No information was provided in MBH98, the Corrigendum talked about a scree test being used as well. I can replicate the illustration at realclimate for the AD1400 North American network, but as soon as you try other networks/periods, the criterion can’t be replicated. here
12. the Corrigendum states that PC series were re-calculated for each calculation step, but this is not correct. The actual selection of steps in which fresh calculations are made is impossible to replicate.
13. is there an unreported step commencing in 1650? If you plot the confidence intervals, there is a step here, but there is no mention anywhere of a step commencing in 1650 in MBH98 or the new SI. An archived reconstructed PC also begins in 1650: what’s going on here? here
14. the 5 archived RPCs cannot be replicated. Here I wish to emphasize that my emulations of the RPCs were identical to those of Wahl and Ammann. However, they are content if their emulation is roughly similar to MBH98; I am not. here
15. why does the RPC replication deteriorate in the early 15th century. The 15th century is obviously a problem area. Given other issues with the 15th century, I’m really interested to see what’s going on here. here
16. the reconstructed NH temperature series from the RPCs here
17. In MM03, we reported collation problems in the data set archived at Mann’s FTP site to which we were originally given access. After publication of MM03, Mann made the duplicate accounts available and a new FTP site appeared. Mann said that the collation errors in the previous accounts did not exist in the actual accounts. However, Rutherford referred to a file pcproxy (retrieved from the Wayback machine) long before our inquiry. I think that it’s quite possible that the collation errors in the first data set did not exist in the actual data set (and not much turns on this in terms of the final results), but, for good order’s sake, I’d like to see code demonstrating that collation errors were not made. I have a sneaking suspicion that they were made and that this is one of the reasons why Mann is so reluctant to show his code.
18. MBH98 and Mann et al [2000] both stated that MBH results were “robust” to presence/absence of all dendroclimatic indicators. A fortiori, this entails that MBH98 results are “robust” to the presence/absence of the bristlecones (and the PC4). Wahl and Ammann do not report that they have replicated this result — I wonder why not?
19. MBH98 stated that they had done R2 cross-validation tests and Mann told Natuurwetenschap that his reconstruction passed an R2 cross-validation test. Again we can surmise the answer — I presume that Wahl and Ammann have replicated the catastrophic failure of the R2 test and have similarly replicated MBH withholding of this information. This is not really the type of replication that one wants.
20. confidence interval calculations in MBH98 here and here
21. data citations for instrumental series. These are currently attributed only to NOAA, which is not an adequate citation.

There are two elements to replication: what could have been replicated on the record before our studies of MBH98 began and what can be replicated after the Corrigendum SI in July 2004. The two differ, but, as shown below, the Corrigendum SI is far from resolving many MBH98 issues.

Update (2023):  A small mystery has been solved: the identity of the two “unspecified” Stahle series used in MBH98, but still unidentified in the Corrigendum. The two series swmxdfew11 and swmxdfew11 are averages of three other Stahle SWM series: 1 (Spruce Canyon CO), 2 (Ditch Canyon B NM) and 3 (Pueblita Canyon NM) – respectively earlywood (e) and latewood (l).  The AD1400 SWM PC1 is calculated from earlywood and latewood versions of Spruce Canyon, Cerro Baraja (mexi027) and the above average.