A little progress on this front since my last post on this issue.
I received a pleasant note from Glenn McGregor acknowledging the changes to my note, in which he said that he would ask the publisher of IJC, the Royal Meteorological Society, to provide a statement of their data archiving policy. Given that the policy is set by the Royal Meteorological Society, rather than the editor, our issue is really with the Royal Meteorological Society.
Bishop Hill (and a couple of others) wrote to the Royal Meteorological Society about their lack of policy and it is now on the agenda of their next meeting. He reports the following answer:
Thanks for your note. I’ve had a couple of emails relating to this discussion and the position currently is as Prof McGregor mentioned. As I have mentioned to others who have emailed in, I’m very happy to consider the requirement for a clear policy statement and as such I have put this on the agenda for the next meeting of the Society’s Scientific Publishing Committee, at which all the Editors of the Society’s journals are members.
It seems odd that they have no policy, but they don’t. So let’s hope that they develop one. For what it’s worth, Climatic Change instituted a data policy as a result of my acting as a reviewer.
In early 2004, in my capacity as a reviewer, I asked for supporting data and code. Schneider said that, in 28 years of the journal, no reviewer had ever made such a request. Needless to say, that didn’t impress me as a reason not to make the request. He said that it would require a decision of their editorial board, so I asked him to obtain such a decision. They then agreed on data but not code. I then asked for supporting data under the new policy, which the authors (MBH) refused to provide and the manuscript disappeared from sight (in the mean-time, they made a check-kiting citation to it in Jones and Mann 2004, so that accomplished what they wanted.) It’s such a stupid game.
However, the exercise was not entirely pointless. I asked for data from Thompson and this resulted in the scraps of information on Dunde, Dasuopu and Guliya now available digitally.
A few more letters to the Royal Meteorological Society wouldn’t hurt. And give a little support to Bishop Hill as well.
Climate Audit 
34 Comments
So, the reviewers are reviewing the conclusions but not the data and code on which the conclusions are presumably based? snip
Re: Tom Vaughn (#1), This is the situation everywhere. It comes as a surprise to people outside academia that reviewers don’t check the data and calculations. It comes as a surprise to people inside academia that anyone thought they did. Replication work does not begin until after a study has been published, and even then it is rare. When authors refuse to release their data and code it makes replication work difficult or impossible, meaning nobody checks the results ever.
Re: Ross McKitrick (#7),
Amen Ross.
Ross explains why no governmental body should ever rely on science as currently practiced. This is really all just ground hog days for what started Steve on this path. Mann’s response to McIntyre’s inquiry for data by saying he didn’t know where it was wraps it all up in a single anecdote that should set off screeching alarms for every intelligent person who hears the story. Under the circumstances, it is that answer, by itself, which tells us all we need to know about the state of climate science.
The scientific community’s understanding of everything we thought we knew about the earth’s temperature history for the last thousand years was completely overturned by a study which no ever bothered to check.
Re: stan (#8),
Well actually thankfully someone did bother to check otherwise this web site wouldn’t exists and we would probably all still be none the wiser as to the shenanighans that goes on and passes as peer reviewed science amongst the (so-called) climate science community. Sadly despite all Steve and Ross’s hardwork the sad saga of using ‘novel’ statistical techniques in combination with ‘cherry picked’ data continues but hopefully not for much longer.
KevinUK
Re: Ross McKitrick (#7), “It comes as a surprise to people outside academia that reviewers don’t check the data and calculations.”
I’ve taken data from figures in manuscripts that I’ve reviewed, and checked the results. Admittedly, I only do that when something looks suspect, but also admittedly am merely scientific staff and not of professorial elevation. I’ve known others who have taken similar or greater care, but also know of some amazing passes. So, reviewing is indeed spotty.
Thanks Steve. I’ve had some further information from Prof Hardaker which I’ve posted as an update to the original article. The meeting to consider a new policy will not be until May, but Prof H will be putting comments received from interested outsiders to the committee.
Steve:
Do you think it would be helpful for you to prepare a short written presentation to go (via the Secretary or Prof McGregor) to all members on the Society’s Scientific Publishing Committee prior to the Meeting so that they understand clearly what you would wish to see and the benefits to them? One of them would be to raise the scientific status of the publications.
When I have been on technical committees it has always been helpful to have a case summarised rather than starting cold on a subject. That would often be a recipe for kicking a subject into touch!
#3. it’s not on my to-do list. I already have too many things to do.
Re: Steve McIntyre (#4),
This is an opportunity for regular commenters to compile a list of SI items and reasons. Here’s a start:…
Items:
1. Data (source and processed or intermediate datasets)
2. Code (with metadata related to programming language)
3. Methods (statistical and other procedures not elaborated in paper)
4.
Reasons:
1. Permits easier replication and detection of errors.
2. Reduces the likelihood that unsupported conclusions will be published.
3. Promotes transparency of motives and agendas.
4. Provides a resource for new analysis in the future.
5.
Re: Gary (#12),
I like your idea. Based on this article, I would like to see:
Items:
4. Code on all verification tests run.
5. Code showing how uncertainty was calculated.
Reasons:
5. It would make it very simple for reviewers to confirm all necessary verification tests were actually done and nothing was hidden.
6. It would make plain whether all points where uncertainty enters was accounted for.
Wow, this issue was brought up in early 2004, and it is noooow 2009!
I’m only claiming that the sky is falling and you want to see my data? How rude!
Open science is needed now more than ever. Open it all up, what’s the Internet for.
snip (original comment in #1 snipped)
During my career in the nuclear industry it was normal practice for all analysis to be independently verified and all data and code to be archived before any ensuing report could be issued. Similarly the report would also be independently verified and approved. Not only would this ensure that all work could be repeated at any time, but it also ensured that the work would be carried out to a very high standard, since nobody likes having mistakes uncovered. It was so easy to archive data and code and added little in time and cost.
I shall be writing to the Royal Meteorological Society.
Re #12:
Another reason: Enhances credibility of climate scientists and the statements that they make.
And another: Ends the bitter polarisation between “alarmists” and “denialists” since issues can be discussed on the basis of facts.
And another: Gives governments a sound basis on which to decide policy.
I’m genuinely astonished by the number of institutions that don’t archive data as a matter of course.
ever feel like you’re fighting the war to gain just an inch?
According to D.A.
“The IPCC is transparent. It gives press conferences when each of their ARs are released. Every one of their reports lists all of the authors of every chapter. All of them have Web sites where they publish their phone numbers and email addresses and publications, and anyone may contact them at any time to ask questions. He has never once had an IPCC scientist decline an interview when asked.”
There are IPCC scientists?
It’s nice to know that your efforts are paying off. Transparency in regards to the data and code can only be a good thing. I hope that at some point all of the journals institute data archiving requirements.
As an industrial scientist working in regulatory toxicology, I follow GxP requirements relating to Good Laboratory, Clinical and Manufacturing Practices. This is not rocket science but provides a codification of Good Scientific Practice relating to record keeping. We preserve in an inspectable archive everything from the original raw data to the final report, as a consequence we can return to any study since 1980 and reproduce exactly what was done, how it was done and who did it.
My facility was audited, at short notice every few years when the audit team would selct studies from the archive (at random) and proceed to do a detailed check on all the data, including interviews with the relevant staff to ensure that they understood the procedures that they had signed off.
All this simply to confirm data with relatively insignificant consequences. I continue to be apalled at the way in which the academic climate science community appears top operate.
Re: Arthur Dent (#22),
Ok, we have Ford Prefect and Authur Dent on the site. Considering we’re looking at Finnish chronologies, where’s Slartibartfast?
Re: Dave Dardinger (#24),
Until your post I did not realize Ford Prefect was a Hitchhiker character. I had to look it up on Wikipedia. Kind of funny now that I know the joke!
Calvin, et al. about the consensus process in climate science.
http://www.gocomics.com/calvinandhobbes/2009/01/05/
http://www.gocomics.com/calvinandhobbes/2009/01/06/
Be warned, the publisher GC has a special data archiving policy and will remove all data older than two weeks from public access.
I’d rather have Marvin’s take on the whole thing…
It wouldn’t matter. You’d just ignore it anyway. Sigh.
Marvin
Sigh?
Are you Gavin in disguise?
I think you missed the joke, Bishop. Marvin is the depressed robot in The Hitchhiker’s Guide to the Galaxy.
Mark
Have data archive, credibility will travel. Don’t leave home without it or your towel.
===================================================
Mark
No, I got it, I was just moving it on a bit.
Ah, then apparently I missed the follow-up joke! 🙂
Mark
Re: Mark T. (#32),
BTW, I’m thinking that our genial host and his pub partner (i.e. M&M) are clearly Zaphod Beeblebrox. Of course it does have the disadvantage of making Phil here some sort of sock-puppet.
And I note there is another potential character here from HHG, namely Deep Climate. Though this nom de plume could have been from other sources.
Nah, Zaphod had a huuuuuge ego… hardly Ross or Steve.
Mark
2 Trackbacks
[…] Another fellow blogger added an interesting post today on Royal Meteorological Society Considers Data Archiving « Climate AuditHere’s a small readingEvery one of their reports lists all of the authors of every chapter. All of them have Web sites where they publish their phone numbers and email addresses and publications, and anyone may contact them at any time to ask questions. … […]
[…] McIntyre notes (bold emphasis added): It seems odd that they {the Royal Met Society} have no policy, but they […]