Here are 5 graphics which show all the Mann 2008 proxies in a consistent format, highlighting two periods 950-1100 and 1850-1980. My software only permits 300 gif images per gif-movie, so 5 parts. The graphics show nicely how few series actually have low-freq variability and how few series are really contributing to any HS-ness. The series are shown in the alphabetical order of Mann’s data set. The Luterbacher series include instrumental information and their impact needs to be studied,
Part1 1-299
Part2 – 300-559
Part3- 560-779
Part4- 780-999
Part5 – 999-1209
Individual graphs are located as proxy001.gif-proxy099.gif and proxy0100.gif-proxy1209.gif in the directory http://data.climateaudit.org/data/images/mann.2008
Climate Audit 
123 Comments
Neat graphs! — Is there a way to pause these animations and then advance or back up?
Yes, this is striking, particularly if one comes at the matter from a business analysis background rather than from paleo or signal detection. The instinct, when confronted with claims about general trends which depend on a mass of data series some of which has the trend, but much of which does not, is not to resort to statistics.
Where the people you advise are going to be investing real money and their careers, you would normally want to throw out all the non trend confirming material – that is basically a counter-indicator, and focus the analysis on the confirming ones, with a view to understanding them better and either confirming or denying their significance. I think this is the point CA has made about the earlier HS work: it is not very interesting to know that when you have a few series with strong signals, and many with none, that in combination they produce a strong trend. What is interesting is the validity of the few.
When the CEO asks what you think in these situations, you don’t normally reply that there is a trend in the aggregated data. You reply that some series show this trend very strongly, and we do not know why these do, and the majority do not. Is it actually a real trend we can bet on? You would say, until I know the answer to the above question, I do not know. It might be. It might not be. Probably, you’d say, it does indicate an opportunity, but probably more of a niche than the proponents are claiming, and one they don’t understand in enough specificity to allow us to judge its merits.
There are cases like this in consumer behaviour. Usually they end up indicating market segmentation rather than a general trend. I guess the equivalent would be purely regional climate trends.
Perhaps this is a bubble-headed question from a non-scientist, but can any value be ascribed to the averaging of such diverse graphs? It kind of begs the question of how many other wildly diverse proxies are out there yet to be discovered.
Re: peter (#3),
That is a very good question.
It goes to the heart of the whole problem about the multiproxy strategy and beyond – for example, if a proxy shows a large excursion from the mean during the 20th Century, does it mean that the proxy is a good record of global temperature (whatever that means)?
There is more to Mann’s paper that disturbs me, which has less to do with statistics and more to do with the scientific method, and if Steve allows me I’ll try to put a post on it here sometime in the next few days.
Steve: John A, I’d rather not go into discussions of scientific method right now (if ever). I want to keep head posts technical and statistical and let the facts speak for themselves.
Re: John A (#6),
John,
If you post your essay elsewhere, could you let us know here?
Re: Neil McEvoy (#16),
Sure. I’d like Steve to see it first, because it’s really about Mann’s methodology and the justifications he makes for that methodology. It’s bizarre.
#1 There are quite a few slideshow plugins for word press that might be handy for pausing these types of graphs, rather than just using the animated GIF format and letting it roll. The site admin might want to have a look here at some of them:
http://wordpress.org/extend/plugins/tags/slideshow
– Ernie.
Almost none of the proxies seems to have a trend, especially not a hockey stick one, and those that have might very well have it by chance.
Even if there existed a correct method to extract some hockey-stick-like average, these proxies individually show that the global warming, even if it existed, is so small that it doesn’t actually influence anyone because normal things such as noise are much more significant.
In some sense, the individual proxies are more faithful descriptions of reality than their Mannomatic (or other) average because they more honestly describe the actual size of the natural variations which are more important than the trends, certainly at the regional level.
The noise! the noise! Turn down the noise!
There is a weak visual observation that the frequecy of cusps is roughly constant over a long series; and that a prevalent value is about a maximum every 20 years. Can one imagine that the frequency of cusps is related to the calibration periods being often of that magnitude? Might be interesting to do a first derivative of the graphs then count the number of zeros per unit of time. Then see if the “cyclicity” relates to the technique or is the same across techniques.
Thank you very much for showing the raw data sets to us. They are very impressive. It even seems to me that any temperature curve can be made from them if suitably selected.
Re: Kiminori Itoh (#8),
Appealing thought – like weighting and combining sinusoids in a Fourrier series. If can find the weights for combining these waverforms to get the desired HS, would it be fitting to call it a “Manic series”?
Re: Jordan (#22),
Perhaps we could refer to these proxies as the “Manic Wavelets.”
Re: Rusty Scott (#24),
As the HS shaft becomes more curved, perhaps the LIA will start to become recognised as “The Manic Depression”?
Interestingly varied, is it possible to add them all together and produce a simple average graph, taking account of course that there are different numbers of graphs covering each interval?
I’ve uploaded the 1209 individual graphs for people that want to inspect individual graphics. They are ordered in the same order as in Mann’s SI information SD1.xls.
#9. Right now, the “data” isn’t raw data, but contains values that Mann has created for the 20th century.
A few of the proxies that have over 1000 years of data seem to have an inverse temperature response. I didn’t get their names because they cycled so quickly. A comparison of the longer proxies could be interesting.
Here are some hopefully pertinent observations on Mann et al, especially the Supporting (sic) Information.
1. There are it seems (but see below) no regression analyses of the various proxies depicted in say SI Fig 4 (or Fig 2 of main paper) against the instrumental records.
2. Those Figs leaves out 1900-1950. Were there no proxies or instrumental temperatures then, or were the alignments even more obviously not close?
3. The so-called (by Mann et al in their SI)”data sets” are nothing of the kind, they do not show the raw data, only (in Sets #2-#4) the RE, CE and R2 of the absent data for e.g. the “Calib(1896-1995)/valid(1850-1895)–early-miss”. Those are not data, just calculations from data. Typical MannStats! The average R2s of all the individual series in each page in sets #2-#4 never reach 0.5; for the series cited, it is 0.3, for the “late miss” of that series it is even lower, 0.13. No wonder the main text never mentions R2. In short, Mann’s “data” show no “skill” against the instrumental record. Any measurement proxy that scores an R2 of less than 0.9 against a known overlapping instrumental record is an anti-proxy. Moreover, in the “no tree rings” set of proxies for NH in “data set” #2 against CRU temps, the best R2 is for the proxies in the century of Jesus Christ, at 0.49, while those proxies’ R2 falls to 0.05 for the 19th century. Seriously, it is clear the editors of the NAS do not know what is data and what not, certainly Mann et have provided no data despite claiming to have done so. It is true that SD#1 has some information about the proxies (lat., long., alt. etc) but again no data from them other than years covered. The ‘pearls’ in #SD1 include that of some 1200 proxies, only 17 begin before 1000 (one way of dealing with the MWP), and of these only 3 “pass screening over 1850 to 1995 (r)”. Yet those 3 are enough, we have to believe, to provide a continuous temperature series “over two millennia”, even though of the 3, one begins in 925, one in 1044, and one in 392. So we even have a new definition of “two millennia”, i.e. 1600 years. But then if one believes in fairies, like Australia’s Brook and Glickson, this is all kosher.
Re: Tim Curtin (#12), Form the PNAS website
http://www.pnas.org/site/misc/iforc.shtml#submission
(viii) Materials and Data Availability. To allow others to replicate and build on work published in PNAS, authors must make materials, data, and associated protocols available to readers. Authors must disclose upon submission of the manuscript any restrictions on the availability of materials or information.
Email PNAS and give them your concerns. I asked for reassurance that all data and code etc. would be available with this paper, and they replied thus :-
Related to:
I don’t understsand this comment in the SI and maybe I misunderstand the procedure.
Does this mean they are always running both screening steps or using the one relevant for the calibration step?
If they are always running both steps, doesn’t it mean the proxies effectively already include the instrumental data (the local grid data) for the validation period since they were selected based on a correlation with it?
Or put another way, isn’t the validation step testing the “skill” of the screening
(when it is used) and the selected proxy coverage rather than the reconstruction?
#14. Right now, it’s hard to know what they’ve actually done. HAve they used Luterbacher data to “infill” proxy values. HAving done that, do they then remove the Luterbacher data and say, Look, Ma, no hands? If so, then they’ve already embedded it. The archived proxy data is so massaged that it’s hard to find firm ground right now.
I think a quick look at the graph shows me 2 things. Temperature proxies think the temperature goes up and down. And the hockey stick is a figment of Mann’si magination.
Question:
Mann created his result using about 1/3 of these data sets.
What result would be achieved if the identical algorythm were applied to only the rejected 2/3?
If the selection process were designed to pick up-trend data, and the true result should have been no-trend, then analysis of only unselected data should result in an almost inverse down-trend.
Unless the algorythm and calibration processes adjust the trend artificially.
Re: Kevin (#18),
The more general question is (1) what sort of robustness test was done, and (2) is it appropriate. [Recall these are the MBH98 crew who did a robustness test, stored the result in a directory called “CENSORED”, chose not to report the result, chose not to relesase the result when probed, deleted that directory, and then lapsed into denial (“we’ve moved on”) ever since. It is fair to hold them now to a slightly higher level of accountability. Indeed, they should relish the opportunity to vindicate themselves.]
I just posted on real climate about the method they are using to build hockey sticks. My post was on topic, scientific and rejected into the electronic abyss.
So I made my own post here.
Re: Jeff Id (#19), Jeff Id, if you read the CA blog you will find out what is allowable at RC. The result you observed is what most here would have predicted. Be patient, it may surface. They screen some comments (those containing certain keywords) more carefully than others.
Re Jeff Id #19, the point you make in your comment to RC has much validity. However, as Mann & Co point out on p. 6 of the SI, only about 150 of the 1209 proxies would be expected to pass the screening if the test size is .128 (or 155, to be more precise). Since 484 did pass (over the full calibration interval), it would appear that many of the proxies, if not a majority, are valid temperature indictors, assuming that the critical value was correctly computed.
There is, however, still a selection bias problem, since many invalid proxies that only passed the screening by chance are probably included in the 484. Under the admittedly extreme assumption that the test has perfect power, so that it will never falsely reject a valid proxy, the 725 that were screened out are all invalid. However, there was a 1-.128 = .872 probability that an invalid proxy would be falsely accepted as valid (ie the null that it is invalid rejected), so that there must have been about 725/.872 = 106 invalid proxies that are included among the 484 that passed the screening.
Any reconstruction based only on the 484 accepted proxies should somehow have its confidence intervals inflated to take this selection bias into account. I don’t offhand see how to do this precisely, though something like multiplying the estimated variance by 484/(484-106) = 1.28 might be appropriate for the full sample. For the no-dendro and earlier reconstructions, where the sample is much smaller, the factor might be much different. I don’t see where the SI indicates how many of these smaller samples were included after screening.
Re: Hu McCulloch (#23),
725/1.00 = 725, so 725/.872 is greater than 725 which clearly can’t be right. So what is the proper math for what you’re saying?
Re: Hu McCulloch (#23),
Hu, if the rejected series were”invalid” was there any explanation of why they were “invalid”? Or was it just that they didn’t have a rising end in the 20th century/
In #23, I wrote, “…so that there must have been about 725/.872 = 106 invalid proxies that are included among the 484 that passed the screening. ”
Make that “about 725/.872 = 831 invalid proxies in all, of which about .128 X 831 = 106 are included among the 484 that passed the screening.”
Re: Hu McCulloch (#25),
Thanks for pre-answering me!
You can get away with this if you are a member of the social club as defined by Wegman. Any other individual publishing this would have calls from the Hockey Team for major penalties and a suspension.
snip – ha, ha, but it has to go.
Who here is reminded of Fred Hoyle’s ‘heroic’ attempts to rescue his ‘steady-state’ theory…?
I wonder what happened to the ‘include all models irrespective’ ensemble method? Isn’t that supposed to give a better picture?
Steve,
Just looking at the gif movies you made and I noticed that all of the proxies you’ve plotted end in 2000. Are all of the proxies so uniform? My understanding was that many of the proxies end at different periods. Some for instance back earlier in the 20th C. Are they the way you’ve shown them because Mann et al filled in the missing data? Or have you just assumed that they all finish in 2000?
Steve: I just plotted up Mann’s data. The provenance of the “data” after the series end is a mystery – we’ve been surprised, to say the least, that plugged information is appearing as “data”.
Hu McCulloch,
Thanks for the analysis, I think you are likely right but to me but the rejection % simply means there may be some temperature signal in the noise. I don’t see it as a valid reason for rejection of the other curves though because-
1 The noise level at the endpoints likely exists through the full history of all graphs.
2 By eliminating only graphs which have the most “noise” at the recent endpoint gives artificial clarity (and overall height) to the end peak while noise through the rest of the graph will filter the true temperature signal lower, guaranteeing a spike at the end.
I also am a bit suspect about the calibration, I am a newbie as they say but it happens I have been programming on the side for 26 years or so. I wonder if Steve would be kind enough to provide a link to his data, since he has obviously combined it well into a readable format, for me it would save weeks.
I am thinking of trying some algorithms myself, I’ll either convince myself that there is a real temp signal or at least be able to understand the trending of the data better after I run some simple understandable math.
Crazy idea:
What if we flip the data backwards and pick some arbitrary point like 1k years, recalibrate and fit today’s temperature curve, what % would actually be rejected. If we try a couple of different points and it’s the expected low % we can conclude there is some value to the data. I don’t know, my imagination has been running wild it might be a waste of time. Still I’m not sure that calibrated frequency limited data (low frequencies only) would fail the P test at the same %. I am going to investigate ‘calibration’ tonight.
jeff id, here is a simple R script to recreate the 1209 individual files. This downloads SteveM’s collation to d:\temp, then extracts each table and writes it under the original name.
#list of 1209 tables with columns headed year, proxy, count
download.file(“http://data.climateaudit.org/data/mann.2008/mann.tab”,”d:\\temp\\temp.dat”,mode=”wb”)
load(“d:\\temp\\temp.dat”)
length(mann) # returns “[1] 1209”
for (i in 1:1209)
{
fname = paste(“d:\\temp\\”, names(mann[i]), sep = “”) # concatenate to d:\temp\ak001 etc
sink(fname) # open the file for output
print(mann[i]) # write the table with headers
sink() # unsink, close file
}
Takes about 15 seconds to run on my machine.
I mentioned before that Gavin Schmidt would have foamed at the mouth if Loehle had invented data for proxies ending before the present. Look what Schmidt said at RC:
Uh, Gavin, the “data” at the SI is NOT raw data where it ends before 1995, as many key series do. Gavin’s statement that the “raw data (before any infilling) is available” is flat out wrong. Plus it’s been known to be wrong for a few days. The data at the SI site has been “infilled” and much of the modern data isn’t real. Is there any possibility that Gavin will admit this error. I’d put the over/under at 0%.
[Update – see http://www.climateaudit.org/?p=4064 for a discussion of this matter. It turns out that Mann changed his Supplementary Information a few minutes after I checked his website in the morning of Sep 4 and there is a time line by which Gavin Schmidt could have made his RC comment a few minutes after Mann changed his website. It would have required precise coordination between the two of them is not impossible.]
I noticed three folders at the site http://www.meteo.psu.edu/~mann/supplements/MultiproxyMeans07/data/proxy/
The in that directory Readme.txt says:
Does the allproxyoriginal folder also contain some infilled data? Or am I just looking in the wrong place?
– Ernie.
Re: Ernie (#40),
It appears those directoroes showed up less than 24 hours ago. The links Steve posted earlier for the proxy data no longer point to the proxy data (it is in the allproxy1209 subfolder now).
Re: Raven (#41)
Diligently attempting something approaching pseudo-diligence. Kudos. I guess.
Re: Ernie (#40),
Check the dates, Mann has added those now. Also the code for infilling seems to be added:
http://www.meteo.psu.edu/~mann/supplements/MultiproxyMeans07/code/codeprepdata/infillproxydata/
This is all a bit bewildering to me…
Mann puffs up his latest attempt at rescuing his version of the climate ups and downs during last 1300 years or so and uploads ‘all his data and code’ to a server located at the Penn State Department of Meteorology.
He is so confident of his findings that he feels it’s appropriate to announce his earth shattering conclusions via a global media release.
I read about it in the local paper published in my provincial Australian city.
I pop in here and find to my confusion that the posters here who can do the ‘sums’ have to fight hard to get access to his stuff and that the material provided by Mann changes by the hour as he appears to respond to criticisms from this blog.
It’s extraordinary – either he’s done the work and come to a supportable conclusion or it’s bulls.. and he’s busting his arse to try and rescue what’s left of his reputation in order to continue the sham he and Hansen promote.
Now this may seem to be a simple soldier’s complaint, but why is it so hard.
Either Mann can prove his version of climate history and can support his contentions by publishing every scrap of data and code organised in such a way as to facilitate others who might like to replicate his work or he is a liar.
Might sound harsh, but if he can’t organise a beer in his own brewery why does anyone take any notice of him – why is Penn State still paying his wages – are they complicit in his apparent deceit?
As Pachauri has mentioned, someone may have gotten their sums wrong. Pachauri is an engineer, and must be wondering. He is also Asian, and for both political and scientific reasons, Asia seems to have figured out that it is senseless to demonize CO2. I count Russia as Asian for purposes of this analysis.
====================================
I guess Gavin Schmidt won’t admit that he was full of crap when he said that the data at the SI did not contain infilling.
Lucky we copied this stuff. In 2003, in pre-Climate Audit days, we observed problems with the MBH98 data set at his website; we asked Mann to confirm that this was the data set actually used. He said that he was too busy and important to answer this question. After we criticized it, he said that we used the “wrong” data set; he deleted the “wrong” data set and substituted another one without any annotation or apology. The climate science community didn’t say boo; they thought that the old pro had taught the “amateurs” a lesson.
This time there are too many eyes on the data to let him do it again. That’s why they hate Climate Audit.
[SM Update Oct 6: I examined Mann’s website at 11.35 am EDT and non-infilled data wasn’t there. I incorrectly assumed that Gavin’s inline response was in error. However, it turns out that Mann amended his SI at 11.42 am EDT. There is a window of time between 11.42 am and 12.14 pm in which Gavin could have verified the existence of the website and then approved the 9.18 am comment adding an inline response.]
He WAS full of crap. But he’s since “moved” on.
🙂
Re: bender (#47),
Now he’s demanding an apology from Timo Hämeranta for saying:
Let’s copy the time stamps here, before they start playing with those. RC:
Mann’s archieve:
[SM Update Oct 6: I examined Mann’s website at 11.35 am EDT Sep 4 and the data wasn’t there. I incorrectly assumed that Gavin’s inline response was in error. However, it turns out that Mann amended his SI at 11.42 am EDT. There is a window of time between 11.42 am and 12.14 pm in which Gavin could have verified the existence of the website and then approved the 9.18 am comment adding an inline response.]
call it doodoo diligence
I eyeballed many of the individual graphs Steve provided picking about 100 at random twice and each time the number of sets in which the older temps generally spiked higher than the current period was more than half, another 15-20% had roughly equal highs leaving less than a third of the samples in agreement with the essential hockey stick ideal that today was significantly warmer than ever before.
Is my observation wrong? Did I miss something?
I agree with Luboš Motl’s comment that nothing in this data looks much like a pattern of any kind, much less a hockey stick. Of course, Motl is merely a physicist by profession so he probably doesn’t have Mann’s math skills. In contrast, my professional training is in law and I admit I stand in sheer awe of the ability of Team Hockey Stick to sling the brown stuff so well and so brazenly in the face of so much visually compelling contrary evidence. Mann is indeed the Johnny Cochrane of climate science.
“how few series are really contributing to any HS-ness”
Would any series be linear across all temperatures? You seem to be assuming they will be. I would assume any proxy will only give a response that is linear across a small range of temperatures. The broader the range of proxies, the better the ability to capture all ranges of temperatures.
Re: bugs (#51),
Bugs write:
One issue that has been repeatedly raised on this blog is eh supposed non-linearity of tree rings.
Is there a publication (paper or textbook) that details these non-linearities and the mans by which paleoclimate researchers and Mann in particular compensate for them?
Re: Stan Palmer (#58), Re: tree growth nonlinearities–see my paper next week (I hope) in Climatic Change.
Craig Loehle
“This time there are too many eyes on the data to let him do it again. That’s why they hate Climate Audit.”
My guess would be the continued, relentless and inherently personal attacks that originate here. The way science has traditionally worked, and worked well, was that if you had a better grasp of the science, if you had the breakthrough, you published the paper that demonstrated it. Scientists are inherently competitive, and always have been.
Steve: Instead of responding to our papers in peer-reviewed literature, Mann launched pre-emptive attacks at realclimate before our 2005 articles were even published, using material that he had obtained under confidentiality terms. The realclimate slagging was accepted by the climate science community. That’s why I started this blog. I try hard not to impute motives to anyone, ask readers not to impute motives to anyone and snip such comments when I notice them or they are brought to my attention. By doing so, this keeps a focus on what people are doing, rather than them as people. I don’t know whether there is a way to extract a temperature reconstruction from ragbags of “proxies” and have refrained from presenting an alternative until I’m satisfied that adequate data exists. However, it’s still worthwhile testing statistical claims on behalf of purported reconstructions. I mention the non-deletion of data this time, because I lived through a prior deletion of data with not a single objection by any scientists. I believe that something similar would have happened this time as well, without the audience.
Re: bugs (#52), Perhaps, Bugs, you would like to take some time here at CA to read back over the history of deceptions, dodges, refusals to release data and the insulting and contemptuous reponses resorted to by Mann and his cohorts in dealings with Steve; and then follow and try to comprehend the incredible statistical incompetence demonstrated by these people over the last several years. After that, take some time to write out and explain all the statistical and computational errors that you can show Steve has made during the same period in his dealings with the Stick and its adherents. That would be truly helpful. Thanks.
It is quite well known that the response is non-linear. Most reports I’ve seen show an upside-down bell-curve. It is complicated by other mitigating factors, however, particularly the availability of water. No matter how warm it is, trees don’t like to grow when they don’t get enough of the wet stuff.
Yeah, and he employs the Chewbacca defense regularly.
Mark
Nonsense. If anything Mann and his ilk are the source of continued, relentless and inherently personal attacks.
Mark
George Tobin: “Of course, Motl is merely a physicist by profession so he probably doesn’t have Mann’s math skills.”
Ah, but as Rasmus said at RC about Steve: “The person (Steve McIntyre) behind Climate Audit is – to the best of my knowledge – not a physicist, but an economist. I personally doubt he understands the physics of the atmosphere/oceans.”
So obviously, economists (congratulations on your new career, Steve) can’t know anything about atmospheric/oceanic physics. But Lubos must; he’s a physicist.
Glad to clear it all up. 🙂
Now there is an objective measurement of “skill?” 30 degrees of skill is better than 5 degrees, perhaps?
Re: jae (#57),
[quote]
Mannomatic
[/quote]?
It seems that every topic to do with Mann involves some form of personal abuse. The rational response of any person in that situation is to avoid such treatment.
Re: bugs (#60), “Mannomatic” is a term of endearment for a highly idiosyncratic algorithm that was shown by Dr. Arthur Wegman to produce bunkem. What else do you call a method that is so “innovative” that it has no name and is in fact considered “intellectual property”? I used to think it perjorative. It isn’t. It’s descriptive.
Re: bender (#61),
I don’t think so.
Re: jae (#57),
There indeed is a measure of skill that seems to have escaped the notice of paleoclimatologists. It is the cross-spectral coherence squared, which shows the fraction of variance in each frequency band that is inter-related between two time series. I’ve already offered Steve Mc to cross-spectrum analyze any pair of proxy and calibration data series that he sends me. A by-product of such analysis is the cross-covariance matrix for optimal reconstruction filters (Kalman-Bucy) along with an explicit measure of their potential power. I’m expanding my offer to analyze any two proxy series (the longer the better) to reveal their respective spectral structures and phase inter-relationships. That should answer the questions about fairly regular oscillations evident in many proxies raised by other commenters.
Anyone who thinks that algorithms that are used to drive global public policy are protected under private IP laws is so delusional that they deserve a great deal of sympathy. Dr Mann has my sympathy. To me “Mannomatic” is a very sympathetic term. In fact, using that algorithm to buy and sell stocks, I truly hope he gets what he’s fully owed.
For the record, I never mentioned the mannomatic. However, I like it, since it is a very concise way of describing a very complex buncha questionable maneuvers.
bugs’s opinion is irrelevant. It’s the originator and frequent users of the term that can best explain its intended meaning. Fact: the dear professor’s algorithm is proven junk.
bugs seems to view this as an ad hominem attack on a person. It isn’t. It is no one’s fault but the innovator’s that the algorithm is junk. That doesn’t mean the inventer himself is junk. Most people have no difficulty distinguishing between the inventor and the invention. For bugs, this seems to be a challenge.
Here’s a thought experiment. If I were, say, Ronco, what would I call such a morsel of patentable junk? The bugs-o-matic? Of course not. The bender-o-matic? Maybe. The Mann-o-matic? Of course! Turn the crank and get the results you want and expect and deserve.
Dale C #37
I just realize I never thanked you for the script. It worked great. There goes my weekend.
jeff id #66
The script is now redundant (but a nice example of quick and dirty ascii-fication to a file) because Mann yesterday posted single zips for both the infilled and the original proxy sets. However, the zips have differing numbers of files (1355 for infilled, 1357 for original) which needs to be reconciled to the claimed 1209.
See posts 40, 41 and 42 above for URL.
#41 Seems like the date stamps on Mann’s data changed again on the weekend. 05-Sep-2008 15:29 is now says on the allproxyoriginal/ folder, it was 04-Sep-2008 15:42 when I looked on Friday.
– Ernie.
Steve,
It seems that you jumped the gun when you accused gavin of not being truthful about the posting of raw proxy data. Did you first check the directory before your comments on Sept 4th and 5th? If you didn’t perhaps an apology on RC would be in order. I’d like both you and the Mann team to focus on the important stuff instead of pointing fingers at each other.
Steve: Yes, I checked; that’s the sort of thing that I do. If you look at the earlier comments, you will see references to Mann changing the contents of the directory to add in un-“filled in” versions. Please examine the record and I’m sure that you’ll withdraw the small insuation against me. I submitted a contemporaneous comment on the matter at RC which they did not publish. The only person to make an untrue statement on this matter with Gavin; and instead of making a simple correction, he chose to expurgate matters. I have nothing to apologize for in this matter; the only person who has been less than straightforward is NASA employee Gavin Schmidt. If you examine the posts in the past week, you may notice a number of substantive posts on the topic as I try to decode what Mann et al have done; I’ve spent a negligible amount of time on Gavin Schmidt’s lack of straightforwardness.
[SM Update Oct 6: I examined Mann’s website at 11.35 am EDT Sep 4 and the data wasn’t there. I incorrectly assumed that Gavin’s inline response was in error. However, it turns out that Mann amended his SI at 11.42 am EDT. There is a window of time between 11.42 am and 12.14 pm in which Gavin could have verified the existence of the website and then approved the 9.18 am comment adding an inline response]
Re #69 – Did you verify it, Jonathan?
Regarding the “original proxy” brouhaha:
Steve McIntyre: September 4th, 2008 at 11:33 pm
Ernie: September 5th, 2008 at 2:06 am (2 1/2 hours later)
Raven: September 5th, 2008 at 2:22 am
Gavin’s statement was also less than twenty-four hours previously. So it may well have been correct at the time it was made (presumably some time after the original post it was replying to), and was almost certainly true at the time Steve posted. Or are you claiming that the original proxies showed up some time between 11:30 pm and 2 am?
[SM Update Oct 6: I examined Mann’s website at 11.35 am EDT and the data wasn’t there. I incorrectly assumed that Gavin’s inline response was in error. However, it turns out that Mann amended his SI at 11.42 am EDT. There is a window of time between 11.42 am and 12.14 pm in which Gavin could have verified the existence of the website and then approved the 9.18 am comment adding an inline response.]
bugs:
Steve:
Well, the bar is set pretty low for “personal attacks” that don’t “impute motives”, if Steve’s recent comments are anything to go by. Gavin Schmidt “was full of crap” and Mann et al are “ass…es” who “blocked” access to directories. Not to mention that the accusations appear to be unfounded.
Re: Dave Clarke (#72),
Don’t feed the troll
He’s not feeding the troll, Stan, he’s playing one on TV. See Steve’s follow-up comments in jonathan (#69).
Mark
Jean S #50 at 7:58 am Sept. 5
Actually, PSU appears to be running some flavour of UNIX. My hunch is that the time stamps are using UTC. And guess what?
9:14 EDT = 15:14 UTC
So it looks like the raw proxies were in place well before Steve posted his first attack on Gavin at 11:38 pm later that day. It also looks like they were being copied in at the same time as Gavin posted his response, even if the RC post and response appeared at the same time.
Now it well may be that the raw proxies only appeared after objections from Steve and others. But, of course, that still does not support the specific allegation that was made against Gavin Schmidt in this thread, and nor does it excuse the vituperative attacks that are all too common here.
[SM Update Oct 6: I examined Mann’s website at 11.35 am EDT Sep 4 and the data wasn’t there. I incorrectly assumed that Gavin’s inline response was in error. However, it turns out that Mann amended his SI at 11.42 am EDT. There is a window of time between 11.42 am and 12.14 pm in which Gavin could have verified the existence of the website and then approved the 9.18 am comment adding an inline response]
Re: Dave Clarke (#75),
Um, 9:14 EDT = 13:14 UTC (EDT = UTC-4, EST = UTC-5). Also, even if the PSU servers were using Unix (more than likely Linux, but immaterial to my point), there’s no reason to suspect they are using UTC rather than local time. It is not an inherent feature of Unix/Linux to use UTC.
Mark
Re: Dave Clarke (#75),
This is not true. During US Daylight Savings Time, EDT is -4 hours different from UTC. So 9:14 AM EDT would be 13:14 UTC. So you are off by two hours.
Re: Richard Patton (#92),
This is correct. If you select and drag or Ctrl-C/Ctrl-V, the actual hyperlink won’t carry as a link but the text will most certainly be there. So, if Steve copied the entire section where the link exists and the link text isn’t included, then it was added after the fact.
Steve: I experimented and, using my usual copy methods, I would have had to manually add the link. It’s something that I often do. However, I can’t guarantee that the matter either way. It would definitely surprise me if I missed a link that was there in my initial reading, but it’s possible. If Gavin makes a categorical statement that the present hyperlink was inserted at the time of his original inline comment, I’ll take him at his word.
Re: Jonathan Schafer (#104), In fairness, Dave caught his error in (#76), which he submitted while I was typing (#77).
Mark
Re: Mark T. (#106),
Ok, guess I missed that. Hard to keep up with all the threads and comments.
Correction (yes I do admit my mistakes):
9:14 EDT = 13:14 UTC, which is admittedly earlier than the proxy time stamp.
But, of course, as I pointed out the 9:14 timestamp was the time of the original RC post and not necessarily Gavin’s response (in fact that was probably much later). And there is no question that the proxies were there when Steve posted against Gavin.
So, yes, Steve, you do owe Gavin Schmidt an apology.
Re: Dave Clarke (#76),
You might have a point if instead of saying, “The raw data (before any infilling) is also available on the SI site, and so you can look for yourself.” Gavin had said, “now” between also and available. As it is it appears he’s trying to pull a fast one. And I don’t think you can convince me that an ardent member of the team like Gavin didn’t know know what he was saying or wasn’t apprised of the change.
Mark T et al:
I may well have been wrong about UTC, but there is no question that the raw proxies were there when Steve posted his comment, and Gavin has confirmed my supposition about the lag between the posting time and his response/approval.
See Steve’s comment at RC and Gavin’s response.
That says it all.
The good news for CA readers is that I don’t have time to check out every wild accusation made here, so I’ll be laying off for a while.
[SM Update Oct 6: I examined Mann’s website at 11.35 am EDT Sep 4 and the data wasn’t there. I incorrectly assumed that Gavin’s inline response was in error. However, it turns out that Mann amended his SI at 11.42 am EDT. There is a window of time between 11.42 am and 12.14 pm in which Gavin could have verified the existence of the website and then approved the 9.18 am comment adding an inline response]
Not to belabor this but Gavin’s claim was that the raw proxies were available at the time Steve posted both here and at RC. He also claimed that when he responded to the first inquiry at RC about it he verified that it was available and that posting’s time actually was much later than the recorded time because of the delay in moderating and response.
Steve’s response to my question was that indeed he did check the availability of the raw proxies at the time of his postings.
So we have a he said-she said.
We all can pick who we believe. My observation is that the time between the posting of the first inquiry at RC and the purported actual response from Gavin seems excessively long and doesn’t jive with the sequence of subsequent unrelated postings. So advantage: Steve
[SM Update Oct 6: I examined Mann’s website at 11.35 am EDT Sep 4 and the data wasn’t there. I incorrectly assumed that Gavin’s inline response was in error. However, it turns out that Mann amended his SI at 11.42 am EDT. There is a window of time between 11.42 am and 12.14 pm in which Gavin could have verified the existence of the website and then approved the 9.18 am comment adding an inline response]
#80
Time stamp of the raw proxies from # 50:
04-Sep-2008 15:15 (3:15 pm)
Time stamp of Steve’s first post on this here:
04-Sep-2008 11:38 pm
So either:
a) Steve did not check properly (by following Gavin’s link)
or
b) The proxies arrived some time between 11:38 pm and 2:06 am (confirmed sighting by Ernie #40) and then the time stamp was changed to the earlier time before Jean S’s post at 7:58 am (time stamp in #50).
So it’s more of a “he said, it [the time stamp] said”. I’m betting on the time stamp.
[SM Update Oct 6: I examined Mann’s website at 11.35 am EDT and the data wasn’t there. I incorrectly assumed that Gavin’s inline response was in error. However, it turns out that Mann amended his SI at 11.42 am EDT. An individual file timestamped 15:16 (11.16 EDT) was not publicly available on the server until then. There is a window of time between 11.42 am and 12.14 pm in which Gavin could have verified the existence of the website and then approved the 9.18 am comment adding an inline response.]
Re: Dave Clarke (#81),
There is a lot of troll feeding going on.
Re: Stan Palmer (#82),
Yeah, I misread your post. I now see the light.
Yes, btw, time stamps are usually when the file was created, not when it was moved into a specific directory, or when the directory containing the files was made publicly available.
Mark
I’m going to look at the timing of the various comments on another occasion.
But first, let’s keep in mind the obvious: Mann did not archive “original” data in the first instance, but massaged data. This was reported at CA leading Mann to alter his archive without providing any notice and without thanking CA for pointing out the problem.
Second, Gavin’s inline response to comment #29 (comment at Sep 4 9:18 am EDT) currently links to a data set that supposedly contains the original proxies.
Please note that Gavin’s inline response to RC comment #29 was copied onto CA in the morning of Sep 5 by both myself and Jean S and neither of our copies contains the hyperlnk now present in the inline comment. Now it’s possible that such a hyperlink might not have carried forward through the copy onto CA, but it’s the sort of thing that I pay attention to. Note that my RC comment, also in the morning of Sep 5, refers to a hyperlink in comment #45, noting a difference between #45 and #29. So there’s circumstantial evidence that the hyperlink currently in the inline response to #29 did not exist on the morning of Sep 5, but was merely a reference to the website. I’m not 100% sure of this, but I raise it as a possibility and will defer to evidence from any other contemporary copies of the page in question.
Be that as it may, there’s a further problem. The data set, what are now said to be the original unmassaged proxies, presently linked in Gavin’s inline response to #29, did not exist online on the evening of Sep 4 or the morning of Sep 5 when this discussion was taking place. It was not placed online until Sep 5 15:18, after the various exchanges.
There is, of course, a third version, a version posted up by Mann on Sep 4 without notice and promptly deleted, also without notice. Perhaps Gavin is relying on this deleted data set as support for his self-righteous demands, but presumably the deleted data set was not the original data set. Plus the cavalier insertion and deletion of data sets without notice raises other questions.
[SM Update Oct 6: I examined Mann’s website at 11.35 am EDT Sep 4 and the data wasn’t there. I incorrectly assumed that Gavin’s inline response was in error. However, it turns out that Mann amended his SI at 11.42 am EDT. There is a window of time between 11.42 am and 12.14 pm in which Gavin could have verified the existence of the website and then approved the 9.18 am comment adding an inline response]
Dave Clarke (#81),
It is a big assumption to accept the online file timestamp as the upload time. My FTP/SCP software as a default sets the online date to be the same as the original file/folder date. That’s a common feature, common practice.
All we can say from the proxy file times is that they were at some point dated 3:15pm.
What we know from Steve’s postings: the original information placed online was different. When Steve downloaded it, his script pulled in the available data based on a “pointer” file that is no longer available. Yes, it has since been changed. So I do not see how Steve’s calling for the “real” data was in any way improper.
Yes, we would hope they intended to upload all of it all along.
Yes, we can celebrate that there is a lot more data available now.
Yes, in the future we can change the assumptions, no longer assuming that inability to access files is due to blocking…
However, THIS time, that was a pretty good assumption, based on much experience.
[SM Update Oct 6: I examined Mann’s website at 11.35 am EDT and the data wasn’t there. I incorrectly assumed that Gavin’s inline response was in error. However, it turns out that Mann amended his SI at 11.42 am EDT. There is a window of time between 11.42 am and 12.14 pm in which Gavin could have verified the existence of the website and then approved the 9.18 am comment adding an inline response.
The files turn out to have probably not been created at 3:15 pm EDT, but at 15:15 UTC (11.15 EDT), though they were not accessible until 11.42 EDT and were not accessible at 11.35 am.]
#84. I can confirm that this is the case with Mann’s FTP site management from personal experience. In first half 2004, I was monitoring his website closely as we were trying to get information on the what exactly was used in MBH98 (leading up to the July 2004 Corrigendum.) In mid-2004, Mann placed data on his public server that was timestamped December 2003, but which had definitely not been in any public area of his website.
Indeed, this reminds me of the controversy over the “wrong” data set for MBH98, when, after the publication of MM2003, a data set at Mann’s website suddenly materialized – never having been linked to previously – and bore timestamps prior to Mann even being at the University of Virginia.
The first explicit notice of Mann’s Sep 4 data version (which as noted above was promptly deleted) was by a CA reader in #40 above at server time Sep 5 2:06 am (4:06 am EDT). So it was online by then.
As noted above, neither my copy nor Jean S’s copy of Gavin’s inline comment to #29 contained the hyperlink that presently appears in this comment – see above post for considerations on this. One reason why I think that the hyperlink has been added is that I check these sorts of things out. Whatever my many faults, it’s pretty hard for even my severest critic to try to sustain a claim that I’m not pretty diligent about chasing down loose ends. I looked at the hyperlink in #43 when the inline comment appeared – why wouldn’t I have also chased a hyperlink in #29?
[SM Update Oct 6: I examined Mann’s website at 11.35 am EDT Sep 4 and the data wasn’t there. I incorrectly assumed that Gavin’s inline response was in error. However, it turns out that Mann amended his SI at 11.42 am EDT. There is a window of time between 11.42 am and 12.14 pm in which Gavin could have verified the existence of the website and then approved the 9.18 am comment adding an inline response]
Re: Steve McIntyre (#85),
Steve, for what it is worth, the Google cache of that page from Sep 8, 2008 13:29:51 GMT (Sept 8 9:30 am EDT, if I converted right) has the links as they appear now, with both pointing to http://www.meteo.psu.edu/~mann/supplements/MultiproxyMeans07/data/proxy/allproxyoriginal/. I agree about your diligence, but perhaps you missed the link the first time through? It does seem normal for links not to be copied when using drag and drop.
I realize you don’t need my advice, but I’d advise you to write a narrow apology to Gavin for the name-calling, acknowledging that the raw data was available at the time he posted the links. Perhaps he knew nothing about the FTP update, and perhaps the raw data was there the first time he looked at the link. Calling him names is not going to help to get to the bottom of the science. General questions about what the notification procedure should be for data updates are certainly fair game, though.
Steve: I don’t know when Gavin posted the links. I agree that drag and drop won’t carry a link forward, but there’s more to it than that. I don’t recall a link being in the inline comment when I first saw it – I would have followed it, it’s what I do. A Sep 8 cache is immaterial to the situation as at Sep 5. But regardless the present data set was not there on the morning of Sep 5.
Gavin is being very cute in all of this – in #45, he asked Timo Hameranta to apologize (he seems to be demanding a lot of apologies.) Timo could hardly be expected to check Mann’s website every few hours to see if it changed. Gavin could easily have observed that Mann had responded to criticism and coopered up his website. Instead of doing that, Gavin acted as though the data had always been there.
[SM Update Oct 6: I examined Mann’s website at 11.35 am EDT Sep 4 and the data wasn’t there. I incorrectly assumed that Gavin’s inline response was in error. However, it turns out that Mann amended his SI at 11.42 am EDT. There is a window of time between 11.42 am and 12.14 pm in which Gavin could have verified the existence of the website and then approved the 9.18 am comment adding an inline response]
#86. On this basis, consider the fact that all of the massaged data versions bear timestamps of December 2007, while the “original” data versions all have timestamps of September 5, 2008. If #86 is right, then the “original” data version was made after the massaged data. Odd.
Expanding on Mark’s point above, the ‘file creation’ on Unix is usually the time that the file was first created on that particular machine. It’s possible to keep the timestamps across machines, but uploading by FTP usually assigns a new timestamp since the file is new to that machine. Once created, the timestamp usually remains the same even if moved from directory to directory.
The most likely explanation here is that the infilled data was uploaded to the SI site in December, and the raw version was uploaded very recently. It is not possible to make any conclusions about the relative creation times of the original files from this information, only of when they were transferred to the server.
It’s not just “possible” that the hyperlink didn’t carry, it’s pretty much certain. At least, hyperlinks are not copied here in my version of FireFox, even the ones in other CA posts.
So again, we have two hypotheses:
a) Steve missed the hyperlink and didn’t notice that the proxies were present
b) The link was copied in later (for which there is absolutely no evidence)
Again (for at least the third time), the original proxy files *were* certainly there at 2:06 am, a mere 2 1/2 hours after Steve posted (even Steve admits this). And they bore a time stamp of 15:15 Sep. 4, or at least they did the next morning.
Regarding timestamps, some file systems have two (the “created” date and the “last updated” time). On my Windows system, though, I only see the time of the last change (“last updated”). Also note that when a file is copied it often will bear the time of copying, not the creation time of the original file (again depending on the system). That’s the default in my FTP software, by the way.
Now it does appear that the original proxy files were recopied on Sept. 5. But I assume someone from CA downloaded all the Sept. 4 timestamp files. So you could compare them to the Sept. 5 file set and see if there are any differences, or you could ask nicely as to why they were recopied. But what you should not do is make unwarranted assumptions about what these changes mean.
Steve, it’s time to acknowledge the obvious: you made a mistake and the original proxies were indeed present at a time you claimed they were not.
Steve: Dave, you say that “the original proxy files *were* certainly there at 2:06 am, a mere 2 1/2 hours after Steve posted (even Steve admits this).” No, I don’t. As I observed above, the “original” proxy files were not archived until Sep 5. What was archived on Sep 4 was something else, which has been deleted. Now I acknowledge that something which was not the “original” proxy files was archived a few hours after my post and, in all likelihood, it was there at the time of my post. This gives a little ammunition to both sides – Mann had done something as at the time of the post, but he subsequently undid it and what’s there now wasnt there then. In addition, before you and Gavin get too self-righteous, please acknowledge for your part that Mann originally archived massaged data and only archived “original” data after we made an issue of it. When Gavin learned that Mann had revised his website, Gavin should have reported the amendment (as Mann should have.) It is unreasonable to expect people to check Mann’s website every few hours to see if he’s changed it.
As to when the hyperlink was placed in comment #29, contrary to your assertion, there is evidence that it was not there in the morning of Sep 5, though the evidence is not proof. As noted above, the hyperlink is not embedded in copies at CA posted both by myself and Jean S. As I had noted (and you also observe), it is possible that both Jean S and I failed to carry the hyperlink forward in drop and drag copying and this is possible. But it is also possible that we accurately copied the inline comment as it then appeared and that one or both of us would have attended to any embedded hyperlink in our copying, especially as we had each noted the need for care in such copying. My recollection – that there was no hyperlink when I initially read the inline comment – is also evidence, though it is not proof. So don’t say that there’s no evidence that the hyperlink was not there on the morning of Sep 5; there is evidence, but it doesn’t rise to proof. IT’s too bad that I shut my computer down in between or else I might have a version in between the RC copy and the CA copy; but maybe Jean S has such a version, in which case I’ll defer to what he has.
[SM Update Oct 6: I examined Mann’s website at 11.35 am EDT and the data wasn’t there. I incorrectly assumed that Gavin’s inline response was in error. However, it turns out that Mann amended his SI at 11.42 am EDT. There is a window of time between 11.42 am and 12.14 pm in which Gavin could have verified the existence of the website and then approved the 9.18 am comment adding an inline response.
The files turn out to have probably not been created at 3:15 pm EDT, but at 15:15 UTC (11.15 EDT), though they were not accessible until 11.42 EDT and were not accessible at 11.35 am.]
Re: Dave Clarke (#90),
I suppose that the phrase “tempest in a teapot” is appropriate to this discussion
A link won’t copy “as a link” but the text itself copies just fine.
Impressively, Google has another cached version of that page still up from the morning of September 5th, dated “Sep 5, 2008 10:40:22 GMT”, with a last post of 5:42 am Sept 5th. It also has the SI Link in both #27 and #45 as they currently stand. This cached page will probably disappear shortly, but I think this makes it very likely that the links have not been changed since the time Gavin made them.
I also realize that this says nothing about the contents of those links, and I agree with you completely that it is awkward at best to expect readers to check the contents of each link to see if those contents have silently changed. But it at least seems possible that Gavin was not aware of these recent changes, and I therefore suggest that you give him the benefit of the doubt. The ‘gotcha’ approach benefits no one.
Dave, I asked you a question about this at RealClimate, but am still waiting for moderator approval before the comment appears. I agree that Steve should apologize for the allegations that the raw proxies were not available. But do you, Dave, agree that there should be better notification when the data changes, and that if Gavin did know about these changes, this could have been acknowledged in a clearer manner?
ps. In an odd aside, the link I made to the word ‘cache’ in my parent post does not appear with standard link decoration. On my browser, it looks exactly like plain text until I hover over it. I presume this is just some temporary glitch with the WordPress updates, but it seemed remarkably apropos to a discussion of whether a link might be missed on a first reading.
Mann has been making more changes to his Supplementary Information.
On Sep 6 or so, I read the following file:
This was not the list of 1209 proxies. There was another candidate file with a different name which I read as follows:
When I attempted to execute the second read just now, it failed. But when I tried the first read, it yielded 1209 proxies.
In addition, this file now has a date of Dec 2007 even though it is in a directory that was only created on Sep 4 or Sep 5 2008 – and it is different from the file of the same name that was there on Sep 6 2008.
The Mannomatic seems to be much like a Swiss Army knife in its versatility. Is this the first use of the time machine function?
http://www.ncdc.noaa.gov/paleo/pubs/mann2008/mann2008.html dated 9/9/2008 has a version of the original proxies which is different (the zip files have different sizes) from the “original proxies” presently at Mann’s website dated 9/5/2008 which was different than the version dated 9/4/2008.
It’s a full time job keeping up with the different versions. Perhaps the version at WDCP is the same as the version deleted from Mann’s website. IF so, will there be another version still to come.
We have some posters more interested in the time differences of a few minutes than data swaps. Maybe they could question at RC??
I do not need to follow the intricate details of this episode of changing SIs to conclude that the Team has once again tended toward handling important data in a sloppy and vague manner. It almost makes one think that the authors conclusions (as vague as they might be) are what the authors judge to be most important not the data and methodologies that got them there.
I can see Steve M wanting to reply to the detail of those that want to argue on the margins of what this all means, but I have seen enough.
For those of you unaware, this might be of use.
In Windows in a DOS prompt in XP, you can sort the files by time; dir /tc /ta or /tw for created, accessed, written. If you want to see it in the GUI, right click the file and go to properties, it lists created, modified and accessed on the General tab.
In Linux, ls -al –time= will let you list by atime, ctime, access, use, or status.
#80
Steve replied:
I can’t speak for Gavin and I have even less connection to RC than I have here. But I’ve already commented on this in #75:
Steve also said:
Nathan said (#93):
It’s clear that there was some sort of problem with the first Sept. 4 “originalallproxy” archive, and so it was updated on Sept. 5. My personal opinion is that these kinds of issues should be documented in a directory “readme” and resolved if necessary by respectful, professional dialogue between investigators.
The continual unwarranted presumption at CA of bad faith, expressed publicly in unprofessional and abusive language, is a severe impediment to meaningful dialogue about these kinds of issues, or indeed any scientific issues.
I’ll be crossposting a reduced, edited version of this comment at RC. I’ve got a very busy couple of weeks coming up, so future comments will be sporadic at best, much to the relief of most CA readers, I’m sure.
Re: Dave Clarke (#100),
To the contrary, Dave. I appreciate your responses, and I think your presence is generally appreciated here. Despite disagreeing, you’ve been respectful and coherent. Regardless of which sites they post to, we need more people who can bridge the gap between the ‘warring’ factions.
I’d like to get one more response from you, though, before you head off. I’ve been posting under the assumption, I think justified, that Gavin was posting links to the data in good faith. I’m presuming that if he knew that the data had been changed in the hours just before his replies, or if he had been involved in making the changes, he would have indicated this directly.
Do you agree that if Gavin had known that the data had been recently changed, there would have been a responsibility to mention this change in his replies? I think this is all that Steve is asking for: either an acknowledgment that the initial data was flawed, or a clear statement from Gavin that he didn’t know of any chnges to the data at the time of his replies. If you could help to produce either, or any clear explication of which data went where, it would be a fine service toward future communication. Thanks!
Re: Dave Clarke (#100),
As Steve would say, puhleeze. No-one wants to get into a childish slanging match, but have you read Gavin’s comments on RC? He is shrill, openly abusive of commentators, has directly questioned the integrity of respected scientist (Douglass et. al. among others), and uses language closer to playground snark than professional discourse. I have not heard you claim that this is an impediment to meaningful dialogue.
I look forward to you applying the same standard to RC as you do to CA. Until then, many will choose to discount your comments.
Everybody loves a good argument, but isn’t this completely irrelevant to the validity of Mann et. al 2008?
Dave Clarke:
Steve M:
These two statements are incompatible and unless it can be resolved which one is correct it will be impossible to resolve the issue (as minor as it might be).
Since Steve seems preternaturally methodical and Mann has a proven history of being an accomplished pea and thimble man I would say Dave Clarke is plowing some pretty hard ground.
Unfortuntely it is not unwarranted, but in the case of the persons in question is well earned.
Indeed, that’s why (I think) this is one of the areas that Steve seems to give speculation a little more room than in others. Perhaps the hope is that somehow the data defenders will be embarrassed into actually behaving scientifically w.r.t. their data handling practices.
Mark
Re #79 – Dave Clarke **The good news for CA readers is that I don’t have time to check out every wild accusation made here, so I’ll be laying off for a while.**
If all you are looking for is errors and accusations, you could have a gold mine at RC.
Re #100 – Dave Clarke **I’ll be cross posting a reduced, edited version of this comment at RC. I’ve got a very busy couple of weeks coming up, so future comments will be sporadic at best, much to the relief of most CA readers, I’m sure.**
What is your point? You said you do not spend much time at RC. Now, if you got a reply from Lonnie Thompson as to when we can expect him to log his data (original would be nice) we could get a bit excited here.
We could use more scientific evaluations. Or are the statistics too much?
I think all the nitpicking on the availability of the data misses the main point. There were many sets of data over the week on Manns site, I witnessed two version of them myself, and at no point was there a changelog as either a seperate document or amendment to the Readme.txt to explain why the data files had been replaced. This of course creates more work for everyone that is trying to use this public data. It’s very sloppy, and wastes everyone’s time.
– Ernie.
Of course, it misses the point. But that’s also the point. Their intent is to divert attention from the undocumented substitutions to sneering at any users who got wrongfooted by the substitutions. I’ve seen this movie.
You’ve starred in it, Steve. I think you still have the lead role, too.
Mark
I asked on the parallel Real Climate thread if Gavin could offer any more insight into the update of the data. I was treated kindly, but unfortunately don’t think I managed to do much to open communication channels. My final post there seems to be caught up in moderation, so I’ll repost it here in case anyone is interested. I’ve expanded the quote a bit to make the context clearer, and removed a response to John Mashey that seemed less relevant here.
It’s obvious that Gavin is as frustrated with this as you are, Steve. But he’s also intelligent and influential. I encourage you to figure out a way to move beyond the presumptions of bad faith, regardless of how strong the evidence may feel. It’s also apparent that my usefulness as a mediator is minimal, so I’ll happily bow out now.
Re #113 Nathan: You are up against a brick wall. Forget trying. For a site that still goes along with the Hockey stick and a lot of other unfounded proxies and assumptions, do not expect Gavin or Mann to change anything even though there is a chance he was not far off on timing this time. I gave up posting there long ago.
Nathan, as a matter of interest, can you provide me with examples “going a long way back” where my “first and only instinct is to impugn [Schmidt’s] motives and honesty the minute they can’t work something out (and this goes back a long way).” I can’t think of a single such example. In this particular case, I asserted that Schmidt’s inline comment was erroneous at the time that he made it. Regardless of whether that comment is right or wrong, it is neither a case where I “impugned” either Schmidt’s motives or honesty or, for that matter, was a comment arising from an occasion where I “couldn’t work something out”. On the contrary, I think that I’m singularly patient about working things out.
Mann’s errors and shiftiness are probably embarrassing to have to continually defend. I feel sorry for Schmidt for having chosen such an undependable character as a trusted colleague. RC should throw Mann overboard.
Re: bender (#116),
Agree. The game gets tough for Gavin. Quite a work to go through different blogs such as
http://wmbriggs.com/blog/2008/09/06/do-not-smooth-times-series-you-hockey-puck/
and try to defend Mann’s methods. And this is just the beginning, Mann08 requires some time to comprehend. Code with lines
pp=load(‘c:\scozztemann3\newtemp\CRU_NH_reform’);
and
nnn=load(‘/holocene/s1/zuz10/work1/temann/newtemp/HAD_NH_reform’)
makes the task difficult. Specially, I’d like to find
c:\scozztemann3\newtemp\nhhinfxxxhad
( Funny, variance matching is well alive, SI:
)
I don’t feel sorry for him at all. He is complicit in this. He made the bed and continually feels it necessary to blame everyone else for his problems.
Mark
More problems with the code, gridboxcps.m,
reg_ts seems to be too big for my computer
and seems to be almost full of NaNs. Any volunteers to optimize the code by applying sparse matrices ?
First match, quick way to reduce memory consumption is to run fewer proxy sets at a time,
replace
by
etc, for example. Now NH_whole_newgrid_HAD_gbeachyear.mat gbcpsfull(:,4) agrees with archived NH_had.mat gbcps for 1500-1600, so I guess I’m on the right track.
What does gridboxcps.m do ?
* Each gridbox instrumental is filtered with Mann’s smoothing algorithm
* Proxies are standardized (they seem to be Mann-smoothed already in griproxy.m; cutfreq 0.1 for ‘annual’ and 0.05 for ‘decadal’ proxies)
* All proxies in 5×5 grid are averaged
* Within the grid, proxy set is ‘calibrated’ with variance matching to temperature. Smooth first, calibrate then. What would happen vice versa ? I have to read ‘The Foundations of Variance Matching’ again..
(* Data is regridded to 15X15)
* Grid-reconstructions are area-weighted to get the hemispheric means, and variance matched again (!!) (with CRU or HAD as target).
This is probably inappropriate thread, #119-#121 might need a transfer. Anyway, gridboxcps.m is very interesting program to play with. For example, you can replace proxy data with instrumental data to see the effect of sampling errors in the NH mean. Leaves not much room for proxy errors.. Fig 3. CPS uncertainties seem to be about +- 0.15 C ( and not shown after 1850. Probably because there would be a conflict with CRU 😉 )
UC. I’ll set up a replication thread. I’m having trouble moving posts.
There is no doubt this data shows us moving into a cooling trend,you had better add some data to turn that around or you will quickly lose all you funding!
One Trackback
[…] it spawned. Why not? Compare to the BBC’s lead with the history. Climate Audit is already critiqueing the proxies used by Mann et al (proxies are the ’stand-ins’ for actually having records of […]