I’ve spent some time (an inordinate amount of time) trying to figure out why GISS uses some GHCN stations and not others. Doing so has required a lot of work on GISS data sets which are nastily organized and with many seemingly ad hoc inclusions, exclusions and sloppinesses. Does any of this matter to world peace? Probably not. But it matters to anyone who’s trying to see what they did and I’ve documented some of the relevant information here, mostly to ensure that I don’t forget it.
Hansen et al 1999 describes the provenance of their station data as follows:
The source of monthly mean station temperatures for our present analysis is the Global Historical Climatology Network (GHCN) version 2 of Peterson and Vose [1997]. This is a compilation of 31 data sets, which include data from more than 7200 independent stations... We use the version of the GHCN without homogeneity adjustment, as we carry out our own adjustment described below.
Hansen et al 2001 uses very similar language to describe station provenance, noting that, in this case, they did not use the GHCN unadjusted version of USHCN data, but used a version that had already been adjusted and then overlaid NASA adjustments. (I think that the two adjustments interact to screw up Hansen’s “unlit” criterion but that’s another story.)
The source of the monthly mean station temperatures for the GISS analysis is the Global Historical Climatology Network (GHCN) of Peterson and Vose [1997] and updates, available electronically, from the National Climatic Data Center (NCDC). This is a compilation of 31 data sets, which include data from more than 7200 independent stations. One of the 31 data sets is the
U.S. Historical Climatology Network (USHCN), which includes about 1200 stations in the United States. The USHCN [Karl et al.,1990; Easterling et al., 1996a] is composed of stations with nearly complete records in the 20th century and with metadata that aid homogeneity adjustments. The GISS analysis uses the version of the GHCN without homogeneity adjustments, as adjustments are carried out independently in the GISS analysis… For USHCN stations the time-of-observation and station history adjustments of Karl et al. [1990] are applied before the urban adjustment is made…
The current NASA webpage (front page) contains substantially similar language, but refers to the inclusion of Antarctic data. I wonder if this was included in Hansen et al 2001? Or whether it was added in subsequent online versions? They don’t say, although maybe the maps will show something.
The NASA GISS Surface Temperature Analysis (GISTEMP) provides a measure of the changing global surface temperature with monthly resolution for the period since 1880, when a reasonably global distribution of meteorological stations was established. Input data for the analysis, collected by many national meteorological services around the world, is the unadjusted data of the Global Historical Climatology Network (Peterson and Vose, 1997 and 1998) except that the USHCN station records up to 1999 were replaced by a version of USHCN data with further corrections [after an adjustment computed by comparing the common 1990-1999 period of the two data sets. (We wish to thank Stephen McIntyre for bringing to our attention that such an adjustment is necessary to prevent creating an artificial jump in year 2000.)] These data were augmented by SCAR data from Antarctic stations not present in GHCN. Documentation of our analysis is provided by Hansen et al. (1999), with several modifications described by Hansen et al. (2001). The GISS analysis is updated monthly.
In the next GISS webpage we get a somewhat different story:
In our analysis, we can only use stations with reasonably long, consistently measured time records. For a list of the stations actually used click here, for the full list (copied from GHCN’s website and augmented from SCAR) click here.
Here’s an example of typical mealy-mouthed climate science methodological description: “stations with reasonably long, consistently measured time records.” What is the definition of “reasonably long“? Or “consistently measured”? This type of loose language is used over and over again in their “descriptions” of their algorithms and, typically, it is impossible to discern any operational definitions that implement the descriptions. It’s very Mannian.
The “Full List”
The “Full List” is located at http://data.giss.nasa.gov/gistemp/station_data/v2.temperature.inv.txt . This contains 7364 unique records. Later we see that, in the list of stations “actually used”, there were 6257 records, of which there were 21 inexplicable duplicates, reducing the number of unique stations to 6236. I’ll discuss the differences. Records in the Full List look like the ones shown below. The first 3 digits of the ID are a country code, the next 5 digits are the WMO code, and the next 3 digits are a station code. These appear to yield unique identification.
42574500003 MARYSVILLE 39.15 -121.60 17 19S 12FLxxno-9x-9WARM IRRIGATED C
22220744000 MALYE KARMAKU 72.37 52.70 15 8R -9HIxxCO 1x-9TUNDRA A
Crosschecking to Original GHCN
Comparing the GHCN data list to original listings at GHCN ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.temperature.inv : there were 7280 unique stations at the original GHCN listing. All 7280 stations were in the GISS “full” list. All but two of the 84 GISS “full list” stations not at GHCN were from Antarctica (as noted in one of the webpages, but not in the original article) plus two Southern Ocean islands. The format of the GHCN information matched the GISS version.
42574500003 MARYSVILLE 39.15 -121.60 17 19S 12FLxxno-9x-9WARM IRRIGATED C
22220744000 MALYE KARMAKU 72.37 52.70 15 8R -9HIxxCO 1x-9TUNDRA A
Scraping “Combined” Sources
There are 3 different data sets at GISS: dset =0 gives different versions for stations; dset=1 is said to be “after combining sources at the same location” and dset=2 is their adjusted version. Using the 7364 unique station ID numbers, I scraped the dset=1 (“after combining sources at the same location”) data versions which took about a day on a high-speed network. The resulting file size is about 8.5 MB, giving some idea of how crappy Hansen’s system is for providing data in an organized form for data analysis.
I was only able to download values for 7249 out of 7364 stations in a first pass. I cross-checked one of the 115 unavailable stations (MOSTAGANEM) on the manual NASA data retrieval system http://data.giss.nasa.gov/gistemp/station_data/ and did not get a value here either. I identified one read failure as due to the fact that one of two Irgiz series (211355420003) has no values in it – why, I don’t know. I patched this by reading only 211355420000 – every step is a struggle with NASA, leaving 114 unavailable stations. Maybe a few of them are Irgiz types rather than Mostaganem types, but even I got weary with this crap.
Now another problem arose. For 335 of the 7249 stations with values, there was more than one version in the dset=1 file even “”after combining sources at the same location”. I manually checked one of these stations “Ponferrada” and observed that this data file had three different versions which, in this case, had not been combined. Why: who knows? There is no clue in the “methodology” that Gavin tells us to read. Maybe it’s because the record is so crappy. It had substantial discontinuities in the station record and I have no idea how one could plausibly adjust for homogeneities in this station. Nevertheless, there was an adjusted version of Ponferrada which picked out one of the three versions plus one value from a later year seemingly plucked out of the air. This left 6915 stations with only one version in their file.
Stations “Actually Used”
The list of stations “actually used” is at http://data.giss.nasa.gov/gistemp/station_data/station_list.txt . This file has a format as shown below. The identification number on the left is tricky: it has one extra digit on the right as compared to the GHCN list – the extra digit appearing to represent the station version (I don’t know whether this number is assigned by GISS or GHCN). Also identification numbers can overlap between countries so you have to include the country code (cc=) as well to be careful. This file also has a brightness index that doesn’t appear in the other data set. On another occasion, I collated this information into the USHCN data set.
—ID—- Legend: R/S/U=rural/sm.town/urban A/B/C=dark/dim/bright cc=country-code brightness-index
745000030 MARYSVILLE lat,lon (.1deg) 392 -1216 S3C cc=425 39
This file only lists 6257 stations. When I tried to do some calculations using this list, I encountered a problem with duplicate IDs. While the file has 6257 records, it only has 6236 unique identification codes. In 21 cases, different versions of stations have been entered twice – although in nearly all cases, different versions do not give rise to different station identifications. Here’s an example:
#[1] “207440000 MALYE KARMAKU lat,lon (.1deg) 724 527 R A cc=222 0”
#[2] “207440001 MALYE KARMAKU lat,lon (.1deg) 724 527 R A cc=222 0”
Why are there duplicate IDs for these 21 stations but not for others? It appears to be simple carelessness. Does it “matter” for world peace? No, but it matters when you’re trying to program as some kind of patch needs to be done. Here’s a plot of excluded and included series. The minimum number of measurements necessary for inclusion seems to be 240 measurements. But there are a number of excluded sites with more values than this including a few with many more. Why did NASA exclude them? Who knows.
Figure 1. Distribution of length of records. left – not used; right – used.
GISS Adjusted Version
I separately scraped the GISS adjusted version, again taking about a day operating non-stop on high-speed internet to download net 8 MB of data. In this case, I was able to download 5990 stations which contained adjusted data. I spot checked a few stations identified as being absent in the scraped version (e.g. Cap Carbon) against the manual GISS system and did not obtain any values from the GISS adjusted webpage. So there are another 246 stations that got lost between the “As Used” and the adjusted version – why? Who knows. Hey, it’s climate science. Hey, it’s only NASA.
If there’s a demand, I’ll upload collated versions of the scraped data.
In the cases where I’ve identified duplicates and omissions, I can assure you that I didn’t initially think to check for stupid duplications. I checked only after crashes of one type or another and then cross-checked back – each NASA sloppiness takes a lot of time to diagnose. It’s trivial afterwards, but why does NASA organize its data so stupidly?
Climate Audit 
79 Comments
There is an interesting notion in the Malye Karmaku and Marysville examples – “Tundra” and “Warm Irrigated.” I presume this refers to the innate general characteristics of the bulk regional environment. Of course, in the case of Marysville, we now know that that particular station is not highly representative of the bulk regional environment – it’s certainly nothing like Orland.
I am totally in awe of your persistence. It can be tough being an auditor. Do you intend to try some different statistics, like maybe, principle components, or just try to reproduce what NASA has published?
Steve, I’d be very interested in a collated version of the data. I want to do a rural vs urban comparison.
w.
I submitted this 8/18. I don’t know if it will help or if I will even get the right data. However, if you have a copy, perhaps the two can be compared, if they even respond.
Willis:
You should go ahead and look at what this super set of data suggests, but from my look at the Brazilian data, their designation of Rural and non-rural is very suspect. I used official Brazilian population census estimates and found significant discrepancies (factors of 3 on more than 50% of the 47 stations.) It looks like the population data that is included in the data set is circa 1990. The designation of rural, suburban and urban is largely based on their population estimate – though they do not report the actual population numbers for designated rural stations – it is coded -9.
My earlier comment from the Brazil thread:
I shared the spreadsheets with SteveM so he may be able to say whether he agrees with what I think I found.
Steve: Having fooled around trying to match up 47 stations, your achievement is pretty incredible.
#5. Matching the USHCN station numbers to GHCN station numbers was tricky. There’s no published concordance that I’m aware of (other than the one that I did.)
TCO, RE #5,
I think Steve is being very open in his analysis of information that he has downloaded from NASA. If he makes a mistake, then others will be able to point it out to him…potentially saving him a bunch of time. In addition, he noted things in the data files that he doesn’t understand. The open analysis might find people who can explain what he doesn’t understand.
If there is any outrage, it is this: My tax dollars is funding NASA’s effort to determine temperature trends. NASA’s past mistakes in determining temperature trends and their unwillingness to make their analysis and corrections “unvague” is outrageous. My representatives are writing legislation that will have a negative economic impact on the United States and this is based on untrustworthy analysis from NASA. If there is any “skewed views” in this debate, just look at those who lead’s NASA’s effort (James Hansen, devout Democrat).
Keep up the good work Steve.
Re:TCO
I for one am very interested to see what Steve is doing as he is doing it. I also believe that it provides a good opportunity for in processs commentary that may catch faults in his work or suggest alternative avenues of examination. Your constant niggling about his publishing habits is, frankly, tiresome and your references to the hoi polloi even more so. If you’re so interested in pushing Steve in a certain direction, offer to help with your time!
This blog has become what it is due largely to Steve and the way he does things – he is inclusive. OTOH, NASA and the AGW proponents….
SteveMC.
Glad to see I wasnt the only one who was perplexed by the station list stuff.
FWIW. ushcn output is SAS generated. ( when it craps out I get a SAS server error)
GISS output, I imagine, is Fortran.
Everything that hansen 1999 or 2001 describes is doable in SAS. But if you
are a scientist trained in fortran, SAS will make you CRAZY. ( learning SAS
definately took 6.2 points off my IQ)
This implies that they read in the SAS files and then apply homegrown stats
coded in fortran ( matlab prolly as gavin has said) and output accordingly.
A programming disaster waiting to happen.
Let’s see. Hockey stick was at one time fully finished, undocumented, unaddressed by core scientists. Then with some forensic work it was finished off, documented to be a statistical embarrassment, and returned to sender, address unknown, undeliverable even by IPCC core scientists. Given that Hansen’s work is evidently not yet finished (pesky jesters having recently caused continental cooling by spying an oddity), it’s clearly undocumented, and just waiting for a forensic inspection. I’d go with keeping an open record of the evidence uncovered, speculations, findings and so on, finished or not. It’s up to the reader to decide at what point the weight of the evidence starts to fall on one side or the other sufficiently so that some personal investment might be made. Those irritated with this site, with emotional investments in being part of saving climatic paradise, will just have to deal with their own anxiety about the forensic process as it unfolds before their eyes.
RE 1.
Ya, the description is not particularized. once I posted
a post on RC that compared the Metadata description to the site photo.
( just to put their nipple in the wringer)
STILL there is the issue of nightlights accuracy. Some GIS wennie here
should be all over that.
Audit the damn pixels
Steve Mcintyre,
It may be possible to make them show their hand by designing your own correction algorithm and letting the snipers have at it.
That may force them to point out your “errors”.
Yes. It will be a long hard slog.
Steve:
You are right. I had to use edit functions in Excel to split at least one set of ID numbers apart so that I could effectively sort them. How you managed to work with 1000s, as I said is pretty amazing.
A cleaned up data set would be very useful, particularly to decode some of the “brightness” designations. I was thinking yesterday about that particular index: If you have been to Brazil you know that street and external lighting is not a high priority and, therefore, many urban areas will show up much “less bright” than in the US. I am assuming the brightness is actual optically measured rather than some kind of heat signature.
Did TCO go off on a tangent again?
Steve, I second Willis’ request. I’d like to get the dataset for some analysis, particularly in the tropics. I’d like to know if Christy’s LT tropics trend of 0.05 C/decade is consistent with the surface trend of 0.02 – 0.04 C/decade predicted by CO2 forcing.
You know I was just looking at the station data for Canada last night in the GHCN stations. I found these amusing the country code for Canada is 403:
40309021000 49.0200 6.7000 252.0 CA GROSTENQUIN A FRANCE
40309022000 49.4500 5.4200 265.0 CA MARVILLE A FRANCE
40309030500 48.3700 7.8300 157.0 CA LAHR A GERMANY
40309031000 48.7700 8.1000 123.0 CA SOELLINGEN A GERMANY
40309032000 49.2200 7.4200 337.0 CA ZWEIBRUCKEN A GERMANY
From
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/ghcnd-stations.txt
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/ghcnd-countries.txt
I’m glad to see we’ve started conquering new lands, HAHA! And what are you going to do tell the UN? Canada just attacked and has taken over Lahr, Germany! Not Canada they’d say, Canada is too nice. That’s what makes the plan so insidious!
Jim B: all these stations refer to former Canadian Air Force bases in France and Germany.
Re: 15
Grostenquin,Soelligen and Zweibrucken were Canadian Nato air bases way back in ’54 when I served in Europe. Murray
#6 I am astounded that a concordance is not published. It must exist, otherwise one cannot readily organise things.
On several occassins I have asked Gavin for simple data.
example: Hansen 2001 CULLS the entire crater lake NPS HQ record because it
is inconsistent with stations “in the region” I asked for a list of stations.
Nothing.
I asked for a Radius around crater lake so I can figure ut what “region” means.
no answer.
I think its not documented.
Re: Jim B:
The monthly data for Canada does not seem to include does bases.
Sinan
Re: 9
Off-topic: SAS was designed with the purpose of manipulating large data sets with limited memory (when data were entered using punchcards). It has changed a lot in outward appearance since early versions I used to use, but that philosophy is there.
Well-written SAS scripts can serve as documentation. Unfortunately, it is hard to find well written SAS scripts.
Sinan
RE 21
Yes. I am speculating based on my own experience. USHCN holds the stuff
in SAS ( fine with me, I did my time with that),but I can be fairly confident
that GISS take this SAS file and muck about with Fortran.
1. The SAS learning curve is huge.
2. The programming model in SAS is something programmers who use proceedural langauges
dont get right off the bat. its more like stream processing
3. Gavin said GISSTEMP was two pages of matlab.
4. The output of GISSTEMP versus USHCN is REORDERED. USHCN is a SAS like
date feild feild feild feild feild
date feild feild feild field feild structure.. GISS restructures this.
SO, I surmise they read in SAS files from USHCN and then processed in home grown code.
Frankly the the first time I worked with SAS I wanted to do the same. Then I became
a SAS god.. ( just kidding)
Anyway, Gavin’s notion that we can just take a flow diagram or verbal description
and “duplicate” is just silly.
Ive been examining the station record for Erbogacen in Russian Siberia (GISS ID 222248170006). I originally picked this site randomly by clicking on what looked like the middle of Siberia on the map at the GISS website. GISS indicates the record is the combination of seven sets of records over the period of 1913 through 2007, although only station 0 had records back to 1913, and included a gap from 1914 to 1936. Therefore, for all practical purposes, the record begins in 1936.
When I saw that seven separate records were combined into a single record, I was curious as to how that was done. I noted that many of the overlap periods had small differences in temperature, so some homogeneity algorithm had to be applied. I would have simplistically averaged the overlap periods across all stations, but Hansen et al. 1999 says they use the reference station, or bias method:
I could understand the rationale, but after scribbling a few sample calculations I realized that the selection of the reference station was important, because the other stations would be adjusted upward or downward so that their mean during the period of overlap would be the same as that of the reference station. So I became curious as to how the reference is selected. HL87 acknowledges that the result of combining the station records in this way depends on the order stations are combined, but says that they begin with the station containing the longest record to minimize the effect.
So is this what was done with Erbogacen? Well, I am not sure yet because I have not completed my program to do this analysis, but a look at the averages across the seven stations makes me suspicious this was not done.
What I did was take each stations original data and subtract from it the combined data to get a sense of the sign and magnitude of the adjustment for each station. I noticed that the early years seemed to have a significant number of months where the combined record was colder than the individual station record for the same period. So I went a step further, and averaged the difference across all stations. Here I noticed that the average across all stations was always higher than the combined record up until 1991, when the difference became and stayed 0. This indicated to me that the reference station is the station that covers the period 1991 to the present.
Looking back at the individual stations, the only one that covers that period is station 6. However, five of the other stations have significantly longer records than station 6. As I said, I need to finish my program to apply the bias method to this data and see where things really fall out, but indications are that an inconsistency exists.
Thanks for the input on the Germany, France stations. I didn’t think it had been used, but I just found it very funny.
I’m I was playing with Google API and was wondering if some one could use a program that takes GISS station data and does a quit Google Maps Mashup?
http://www.surfacestations.ca/GISSCANADA.php – 800 stations
#24: I’ve imported GISTEMP into Google Earth. The data set is a few weeks old right now so it does not have the recent adjustment. I’m fixing that as I write this…
http://dev.edgcm.columbia.edu/wiki/GISTEMP/
#25. Ken and others, the giss.dset1 file was re-scraped, but I’m re-scraping the giss.adj file as we speak. Wait for the re-issue.
Here’s a simple script for patching the new USHCN versions, which I’m running right now. It depends on a prior concordance of USHCN and GHCN id numbers which took time, but which I have in hand.
re:23
John,
Great job! It looks like you may have found something publishable.
#25: Thanks, Ken! Would it be possible to set up a web page where people could post errors in GISS metadata? Here’s one that caught my eye: Helsinki/Seutula (614029740000) AKA Helsinki-Vantaa Airport is located at an airport serving 12 million passangers (2006).
RE 23.
This sftware package ( freely available) might help you.
he has done the work for creating a reference station
just drp the guy an email
Click to access Prezentation_SW.pdf
http://www.climahom.eu/
Nice powerpint on czech data and issues with homogenizatin
Click to access Prezentation_T_homogenization.pdf
Slide 30 shows 3 approaches for reference series creation.
then a nice presentatin on three methods of homogenization
including Vincent method ( easterling peterson) and a comparison of their perfrmance
RE 23.
A few more things.
Click to access Prezentation_T_homogenization.pdf
A large number of czech stations ( might be fun to compare with GISS) going back
over 100 years.
Shows difficulting integrating instrument changes that happened in 1997
( slide 54 )
Have a look
Steve M – thanks for the links. The presentations were quite informative. When I find a chunk of time I may try to download and test drive their software for kicks.
The quality control discussions reminded me of something I ran into when downloading the Erbogacen data. GHCN keeps a file of data that fails their automatic quality-control tests, so I looked to see what was in there from Erbogacen:
Station ID Year Mo Temp
222248170000 1965 10 34
222248170004 1970 6 189
222248170004 1970 7 116
222248170004 1970 8 54
222248170004 1970 10 76
222248170004 1970 11 194
When I compared that data with the valid station data, I realized that the data was mis-entered by a keypunch operator and actually was valid. This is a common mistake and is acknowledged by Peterson in his paper on GHCN quality control. The mistake I found was that three months were missing the minus sign, and part of the 1970 sequence above was shifted early one month. This was easily confirmed by looking at the collocated stations. The data should read:
222248170000 1965 10 -34
222248170004 1970 7 189
222248170004 1970 8 116
222248170004 1970 9 54
222248170004 1970 10 -76
222248170004 1970 11 -194
I will be the first to say that this omission is not significant. However, it does tell me that no one appears to be doing a visual inspection of the results to check the quality of the automated processes.
While we are at it, I will throw a question out there that maybe someone can point me to an answer:
How does GISS calculate an annual average when some of the monthly data is missing? For example:
1997 -29.4 -23.5 -17 2.9 999.9 999.9 17.7 14.1 6.8 -3 999.9 999.9 -4.93
Four of the twelve months are missing data, but an average annual value of -4.93 C is somehow calculated. I’d like to understand the algorithm for doing that. I cannot find a reference in the GISS material to the procedure used. Note that the average of the valid stations is -3.9C, so something far more complex is going on here.
Isn’t that 13 data items ?
RE 32.
I sent the guy an email and had the package in a couple minutes.
Re #33
John Goetz,
I’m wondering if the reference station values are used to fill in the gaps. You might try that and see if you get something close to the calculated average.
RE 32.
I swear I have seen that one month shift problem in some series that have passed
QC.
I load up a sequence. x1.x2.x3.xn
Then a nearby site y1.y2.y3.yn
Then I just look. And more often then not you’ll see series that track pretty well
ESPECIALLY at the extrema. But on rare occasions I’m pretty sure I saw what looked
to be one off errors. should be easy to detect.
John Goetz,
The GISS rule for the getting a yearly mean appears to require no three month grouping with more than one missing month. Since they calculate the value by starting from the previous December and dropping the most recent one, the example you show is missing one month from each of three groups. MAM, JJA, SON, with the missing December value being carried over to the next year. At least that’s the way I see them doing it.
RE 33.
There are several appraches to filling in missing data.
I’ll see if I can reverse engineer it.
I suspect the reason for using the previous December value is simply to give them more talking points to report unusual temperatures.
Now if the month of January make for a warm DJF they can say winter was warmer than usual. On the other hand if a warm January make for a warm JFM they can say the first quarter of the year was warmer than usual.
Athough winter starts in December, it starts closer to January. So I think January-March is more reflective of seasonal temperatures.
Same applies for the groupings.
Ooop. Should say Same applies for the other groupings
RE 40.
Hansen discusses seasnal analysis in his 1999 paper…
Havent parsed it yet. be my guest.
RE #42
No thanks. Just saying what I observed. Didn’t investigate in depth and only looked at raw not combined data.
Mosher #39
I tried but the closest I got was doing valley to peak, peak to valley linearization. But to do that for the year given, you need the next year, since December is one of the points missing. Also, this assumes it is in a year and not winter cold to summer high to winter cold using data in up to 3 years.
RE 43.
I am beginning to doubt my Tom Sawyer abilities.
Steve, relative to your comment in the initial discussion-
“Hansen et al 2001 uses very similar language to describe station provenance, noting that, in this case, they did not use the GHCN unadjusted version of USHCN data, but used a version that had already been adjusted and then overlaid NASA adjustments. (I think that the two adjustments interact to screw up Hansens unlit criterion but thats another story.)” – you might want to read the two Hansen remarks you show again.
Hansen, et al 1999- We use the version of the GHCN without homogeneity adjustment, as we carry out our own adjustment described below.
Hansen, et al 2001- The GISS analysis uses the version of the GHCN without homogeneity adjustments, as adjustments are carried out independently in the GISS analysis.
I am an amateur at this, but to me the description of the data used is the same for both. Perhaps your parenthetical expression needs removed or revised?
#46. You missed the phrase: “For USHCN stations the time-of-observation and station history adjustments of Karl et al. [1990] are applied before the urban adjustment is made.” Karl’s adjustment appears to me to blend urban trends into the “rural” sites in the name of fixing station discontinuities. It will take some time and some statistical analysis to figure out how it does this, but from the Grand Canyon and Tucson examples, I’m convinced that it does. This is just in the US.
In the ROW, it’s hard to say. I suspect that some services have pretty good sites and other services don’t.
I see some mention of using SAS to analyze & visualize
climate data, and I wanted to share some sas code with
you guys, in case it’s useful:
http://robslink.com/SAS/nasa/aaaindex.htm
In particular, if you’re trying to use SAS/Graph to
plot some data onto a map, these samples should provide
a good starting point.
I’m open to questions & suggestions about the code 🙂
(there’s an email link at the bottom of the page)
Some timeline information for those who may want to know:
When, in the first month or two of a new year, GISS publishes a USA
(contiguous 48) temperature analysis of the previous year, they are not
using USHCN data for that analysis, because they do not yet have the
USHCN data for the previous year, because GHCN does not yet have the
USHCN data for the previous year.
USHCN station data for the year 2002 were added to GHCN between November
8, and December 10, 2003.
USHCN station data for the year 2003 were added to GHCN between April 10,
and May 6, 2004.
USHCN station data for the years 2004, 2005, and the first three months
of 2006, were added to GHCN between August 13, and September 11, 2006.
By the end of February of each year, GHCN will usually have data for the
full previous year from only 120 (non USHCN) stations in the 48
contiguous USA states. In some years GHCN will also have partial
previous year data from perhaps up to another 150 such stations by March.
I’m not sure which thread is best for the above info, so I hope this one will do.
As of this morning, the only US (contiguous 48) stations for which more than
three months of 2006 data are in the GHCN mean temperature file, are the
120 non-USHCN stations which might be called the rapid reporters.
Thus, the GISS temperature analysis for the US 48 for 2006 would seem still to
be based on those non-USHCN stations.
I seem to recall some comment to the effect that the recent GISS patch to USHCN
station data seemed not to affect their 2006 numbers for the US.
If that recollection is correct, and if that comment was correct, that lack of
affect might be due to the their 2006 numbers being based entirely on non-USHCN
station data.
Here is something interesting, which someone may want to scrape, before it gets updated into oblivion. Very current data, with not only a few months missing, but also, during the days prior to the missing data, a stuck temperature sensor input:
http://www.wrh.noaa.gov/rev/remotedata/sqwhic.php
Hello. I was looking some blogs mentioning that a large number of weather monitoring stations from the former soviet union have been lost in recent decades, and implying that the recent “global warming” data might be a result of this.
I was wondering, has anyone published, or looked at data from the past 130 years which uses only weather stations that have existed for the entire 130 years. In other words, if a particular weather station was lost in 1985, remove all of that weather stations data completely from the graph, so that the error of lost weather stations is removed. You would think this would show a more accurate trend in “global” temperatures. (although the total average of the tempurature might be less accurate).
I’m neither a “skeptic” nor a “believer”. I’d just like to get to the bottom if it.
Thanks
RE 52 Hi KEN
“Hello. I was looking some blogs mentioning that a large number of weather monitoring stations
from the former soviet union have been lost in recent decades, and implying that the recent
global warming data might be a result of this.”
Well, there is a coincidental relationship between a decrease in the number of stations and the
increase in temperature. Some have suggested this is more than a coincidence, but it has not been
established in any study I have seen. Still, when I first looked at the “number of stations” chart
and the “temperature” chart, the coincidence was striking. So, there is a potential clue there,
but no proof.
“I was wondering, has anyone published, or looked at data from the past 130 years which uses only
weather stations that have existed for the entire 130 years.
In other words, if a particular weather station was lost in 1985,
remove all of that weather stations data completely from the graph,
so that the error of lost weather stations is removed.”
would that it Were that simple. Stations change over time, they move. they start. they stop. they change instruments.
Speaking for GISS ( nasa) they select stations with the LONGEST RECORDS and then they use these as an
anchor of sorts to patch together an entire record using nearby stations.
” You would think this would show a more accurate trend in global temperatures.
(although the total average of the tempurature might be less accurate).”
It’s not entirely clear what a smaller sample of longer measurements would show.
The problem is complex because you have time series over an area. and the time series
are not spatially uniform and not temporally complete.
Where do you take the temperature in the pool? and for how long? And what about
the kid with a smile on his face? Is he smiling because the water is warm? or
is the water warm because he is smiling.
Im neither a skeptic nor a believer. Id just like to get to the bottom if it.
Thanks
Steven
Thanks for the reply and insight. I think I’ve seen that same chart. It’s kind of mind-boggling to me that they would let so much partial data be included on what is supposed to be a complete record which is being used as evidence for average temperature changes.
You would think if they started with say, 5000 weather stations collecting data in 1900, but had only say, 1000 of the original 5000 remaining by 2007, which had not moved or been altered. etc, it would be a relatively simple matter to produce a graph of just data from the 1000 “consistent” weather stations over time, without trying to outsmart the raw data by using “anchors” or “fixing” data points. All that does for me is make things more confusing and complicated. Why does quantity have to be more important than quality. Even if there were only 12 “perfect” weather stations on the whole planet for the whole durations, I would like to know what a graph of those 12 looks like.
Especially when they are talking about numbers in the magnitude of .1 degrees Centigrade temperature change over time, one would think that eliminating all the data from Russia would make a huge difference in the apparent amount of warming taking place.
Anyways, thanks again for the response.
RE 54. Well Ken for GISS you can go get the data for yourself and look at it.
And there is a really long record from central london that goes bak a couple hundred
years, but that’s one spot.
I just checked it out. Wow they do have a lot of data. I can see your point. Too bad theres no way to grab all the data in just one big excel file. I think a spreadsheet set up correctly would make it easy, but trying to hunt and pick one by one off thier site would take weeks.
I did notice one of the graphs shows there are about 1000 stations that go back 100 years, but there doesnt appear to be any way of getting a filtered list of just the long ones.
Ken
what is important in this context is the fact, that GISS seems to have thrown out many rural stations. I just can speak for Switzerland. Currently they use just four stations of which only one can be seen als rural. Many other rural stations have not been used since about 1980, when the number of stations decreased. It would be very interesting to check this for other countries and see what those omitted stations show as trend (up or douwn or flat).
This is a lot of work, and I don’t think we will ever see a paper from the mainstream research investigating on this.
re 56. Ken you can grab all the files either from USHCN or from GISS. actually steveMC has links to
both. Or ask JohnV he can help you. You will need to program, however.
Gaundez
Are you trying to account for the “urban heat sources” I’ve been reading about?
Steven
It looks To me like theres a seperate file for every weather station. (yuck!) I was hoping to find one giant spreadsheet with all the data. Actually that would be too gigantic of a file, lets make that one giant spreadsheet with all the data summarized (yearly average temps only), and a column for the date ranges. That would only have about 140 columns and 7000 rows. (todays PC’s can deal with those sizes.) Then one could just filter out all except the roughly 1000 stations which have consistent data and plot a graph. Unfortunately, I suspect such a file does not exist, and it would take a full-time data cruncher months to try creating such a beast.
I’ve forgotten 99% of what I ever learned about programming (which wasn’t much), but here’s my newbee-ish theory. If all you wanted to get was temp trends, you don’t need some elaborate computer code to help “guess” the temps to fill in around the raw data. Just plot the raw data. It won’t give a true average (is there really such a thing as true average, but it should be far more accurate for showing temperature trends because its based on 100% real world information and zero guesswork. Is this reasoning flawed?
I have grabbed nearly all the USCHN files for historical temperatures (Min/Max/Mean)in the several adjustment data sets (from Raw to Urban). If you know R use it. If you use and are limited to Excel like I am, the downloads and manipulations can be rather easily done by downloading into Notepad then to Excel (into several separate worksheets as they do not fit in raw form into a single worksheet) and using Excel functions such as Pivot Tables and Lookup Tables to manipulate the data and fit it into a single worksheet and more useable form.
Ken, I think you will learn more by digging through all these files and the Readme files on your own. I found it a major revelation and was most astounded by the amounts of missing data from the early periods. Some of that data was filled in using neraby stations with good correlations but even then a goodly percentaged of very early data has missing data points. I have major reservations going back past 1920 with these data, but you will find others here using the data as far back as it goes and without caveats.
Variability of temperature trends in nearby stations is another feature of the data that can surprise one who has viewed global warming rather exclusively in terms of global and national averages.
RE 59 Hi ken. The other Kenneth here is also a great resource.
Let’s look at your question:
“but heres my newbee-ish theory. If all you wanted to get was temp trends,
you dont need some elaborate computer code to help guess the temps to fill in around the raw data. ”
Different groups and methods handle missing data differently.
“Just plot the raw data. It wont give a true average (is there really such a thing as true average,
but it should be far more accurate for showing temperature trends because its based on 100% real world information and zero guessw
Is this reasoning flawed?”
Flawed: Won’t give a true average, BUT will be more accurate. Presupposes knowledge of the “true” average.
Lets see if I can give you a simple example. There are two thermometers in the world, A and B. Both recording
since 1807. Thermometer A has 200 years of data. Each year is 10C. for 200 straight years.
Thermometer B is halfway around the globe. It also started in 1807. It has 100 years showing a temp of 0C.
Then 10 years of missing data. Then 90 years showing 0C.
Compute the average temperature? Do you ignore the 0C because of the missing 10 years? Do you fill those years
in with 0C? what’s the justification? what kind of error do you have? Do you say the average is 5C for the
first 100 years, and then the average is 10C for 10 years, and then it decreases back down to 5?
Simple cartoon examples can illustrate some of the issues
But what if all I wanted to know were the most reliable temperature change and not the actual temp. For example, I’m not interesting in figuring out whether the ave temp at weather station B in your example was 0C in 1908 or 2C. I don’t even care whether the average of both station in 1908 was 5 or 6C I just want the best estimate of how much the average for the globe has changed. So I could take the data from station A, (which would show the temperature change is zero) and run with it, and in your example, I would still be accurate but with better certainty than if I had to use guesses.
Your example is simplified, so it’s easy for one to say with reasonble confidence that the missing data might be filled in with 0. In reality from what I’ve observed, there are thousands of stations with very short real data. How does a researcher go about filling in the gaps in 1980 for a station that went down in 1905 and stayed down? The bigger the gaps, the more we must rely on the researcher’s skill. But if we just remove the entire series for all the stations with piecemeal data, we are removing the guesswork so that we are left with only real data. Then we don’t even have to argure about whether a gap should be filled in with 0C’s or 1C’s or 5C’s or whatever.
I’m suggesting “yes” we ignore it. Because I think we still have at least 1000 weather stations which span the entire 100 or so years. And I would rather see a graph showing temp changes based on 1000 weather stations that we know are using real measurements, than having 7500 stations, but with about 6500 of them being supplemented with a lot of guesswork which may or may not be accurate depending on the skill of the researcher or on the a computer model.
Kenneth
I’ll look into that. I was under the impression USCHN were regional to the US and I was more concerned with that scary looking graph of global temps being used by NASA, you know the one with temps shooting up like a rocket in the 1990s.
RE 63… Stick with the case I presented. 2 thermometers.
Or if you insist on the 1000 thermometer case, what if all those thermometers are in one country?
one state? one county? The issue is not simple. you have a spatial sample and a temporal sample.
I’m not suggesting your approach is wrong ( I rather like it) I am suggesting you think through
the implications, flaws, shortcomings, etc
Re #64
USCHN covers the lower 48 states. While I have not downloaded the global data sets I am assuming that it would be no different for GHCN.
One has to be careful using Raw USCHN and I assume Raw GHCN (if that data set carries through to global) in that in the early history it contains data points that are not only missing but obviously incorrect. The progressively more corrected versions fill in a good portion of missing data points, but certainly not all, and in some cases attempt to correct the obvious errors. A number of these error points evidently cannot be corrected and are therefore left blank even in the more corrected versions.
Even a low percentage of missing data for a station can throw off a calculation of temperature anomalies so one has to either ignore the entire station in the calculation or attempt to apply a reasonable algorithm to fill in the missing points. And remember that even in the USCHN there are missing data points in even the most corrected version and these corrections have been treated with an algorithm to fill in missing data. If these people responsible for the data sets choose not to use their algorithm to fill in missing points I would be cautious about using a homemade algorithm. One can minimize the effects of missing station data on calculating trends by normalizing the temperatures for all stations involved in the calculation. This is what John V did for his calculations here.
If you are simply attempting to confirm a global average trend with your own calculations, I would not bother since it has already been calculated for you. If you want to do it to learn something than you need to be aware of the limitations of the data. I personally think that global and national average temperatures are not most exciting temperature measurements/calculations to be looking at.
The US temperature measuring stations have become a much larger percentage of the global total over recent decades since other countries have cut back significanly on the number of official stations.
Probably a more exciting analysis would be looking at the differences between Satellite(MSU)/Balloon(Radio Sondes) temperature measurements and surface recorded temperatures measurenments over the past 30 to 40 years.
Another place to look would be the station to station variations in temperatures and trends and attempting to determine whether and how much of the variations are true manifestations of the climate and how much is measurement error. Many climate scientists seem more interested in measuring a global average than looking at real local differences. Accurate measurements of local variations I would assume should be important when calibrating models for temperature proxy reconstructions even though these calibrations sometimes get teleconnected to more regional temperatures.
Guys here the facts I am having a lot of trouble with.
1. NASA has a graph for US temps and a graph for global temps. The one which seems to be used repeatedly as evidence for AGW is the global one, which seems to be where they get get the verticle part of the “hockey stick”. The US graph, on the other other hand, does not look like any reason to be alarmed at all. The two graphs I am talking about are here:
http://data.giss.nasa.gov/gistemp/graphs/
2. By the year 2000, the global temps graph shows an “anomoly” of roughly 600% more than the one from US data.
3. We know that in the 90’s, a very significant portion of the weather stations from Europe were lost, leaving, as Kenneth states above, a much larger portion of the weather stations in the US. I found this graphical animation showing locations of weather stations over time
http://climate.geog.udel.edu/~climate/html_pages/ghcn_T_stn.html
Good grief! Look at 1992, there’s still 1000’s of stations in the US and, I’ll take a guess here, maybe 200 total stations everywhere else? So how did they fill in all that missing data? Guys, that is a LOT of data to be coming from some computer model or researchers best guesstimate. I work in the construction business. A lot of times I have to compile data to come up with the cost of a change order for the owner. We do not ever use data as justification for a change order that is not real. snip
Steven, to offer my opinion on “what if all the thermometers are in one country”, looks to me like they are! I think the shape of the graph of global temps should more resemble the one for US temps, since a very large portion of the data is coming from the US. I think that ultimately, if they are really missing that much data from the rest of the world, the only conclusion is that it’s impossible to arrive at any meaningful graph of global temps, and therefore we must accept the US graph as the next best thing for an accuracte look at the shape, that is, the rate of temperature change. (but not for an accurate look at the actual temperature, since the US is a small percentage of the total globe.)
I’m standing behind my original theory. I wish I could see a nice, clean and simple graph only the raw data.
Ken (#67) writes,
I’m not sure why that map runs out in 1992, but the following map from the same site:
http://climate.geog.udel.edu/~climate/html_pages/Global_ts_2007/T2000_2006_loc.html
shows a lot more stations in ROW. If you back up a directory or two there is a documentation file that explains what’s included, plus maps for other decades.
Steve: I’m persuaded that the dropoff in stations in the early 1990s (China, Russia, Australia as well) is due to GHCN failing to collect data other than from urban airports, not because of weather stations closing as people often think.
RE 67. Ken you should read the “where’s Waldo” threads. The problem you note
has been discussed here, “where ” is all the global warming? geographically.
The US is over sampled, but the measurements are supposed to be area weighted.
As you note the warming takes place were the measurments are most sparse.
It’s an ongoing concern here. Keep asking questions.. folks will help
Re: #67
Unfortunately the graphs you are comparing have different y axis scales and it compares global land and sea with US land. You need to eyeball the trends by accounting for the scales. The global surface temperatures trends reported for land are consistently higher than for the oceans. The NH also has a higher trend than the SH part of which is due to the land/sea ratio differences in the hemispheres.
It is however true that the US surface temperature trends have been different than the global average surface temperature trend– less and having a higher peak in the 1930s. Those in love with using global averages will remind you that the US surface makes up approximately 6% of the global surface and less than 3% of the total global area.
The temperature trends by themselves are only an indication of GW and not AGW.
As I recall the USCHN has a total of 1221 stations in the US while the total stations world-wide for GHCN are around 3000 (including USCHN).
RE #68, the animation for the 90’s is at
http://climate.geog.udel.edu/~climate/html_pages/Global_ts_2007/T1990_1999_loc.html.
There is some falloff in the early 90’s, but not a complete evaporation as in the
animation from the same U. Del. site that Ken linked in #67. I suspect that the
animation Ken linked was intended to be for the 80’s, but was inadvertently continued into the
90’s with most of the data missing? Or were ROW sites reclassified in some way that
excluded them from Ken’s animation but not this one?
To my mind there would be a lot of advantage in selecting a small number of stations that appear to have “good” data and making a trend comparison of their simple arithmetic mean with the overall published trends.
Why?
Well it would be a quick and dirty way of getting a handle on whether there is a substantial possibility that all the adjustments and interpolations in the published trends affected the result.
“Good” data stations would be ones that are least likely to have a UHI effect, and have complete or nearly complete records over a long span. These can be very sparse, say 20 stations worldwide. Where data is missing, just remove it from the calculation of the average.
What will this comparison of trends tell you?
Well, if the trends between this “good” data and the published trends are similar, it strongly suggests that all the adjustments and interpolations of data in the published trends don’t make a lot of difference. It doesn’t prove it . Your “good” stations would have all kinds of biases and these might just happen to yield a result similar to the published trends, but the chances of that seem remote.
On the other hand, if the trends are different, a more rigorous investigation might be warranted. In that case some thinking about the differences might give you an intuition about where to look.
Just a suggestion from someone not doing the work, but appreciative of what you all are doing.
agree
NIce article Hans. Did Luterbacher reply anymore? His handwaving (I assume)
My question is what did his paper say about loss of variance, was it stated correctly? Did the explanation make sense and can be supported?
http://www.giub.unibe.ch/~dmarta/publications.dir/Luterbacher2007.pdf His new one has R2 of .17 and .14 for hazels and snowdrop flowering. If this is the Pearson product, isn’t this pretty low for making conclusions?
This paper seems to continue the 2004 paper. The 1540 reconstruction does not (visually) approach the Pfister and van Engelen reconstructions.
I was wondering if you were going to comment on this paper.
#73
To the simple eye and brain that graph of central European temperatures from 1780 has a strong 10-13 year signal. As the central tenet of AGW is that historic temperatures can not be understood without CO2 forcing, how does AGW theory explain that signal?
Thanks for pointing that out. I actually did look at the Y-axis scale, but had not noticed the decimal point in front of the global data. I was reading 6 and not .6. So my 600% mentioned earlier was wrong.
That is a lot different from my graph. I wonder which one most resembles what GISS used.
In looking at GISS’s info on stations,
http://data.giss.nasa.gov/gistemp/station_data/
You can see clearly from the graph in the middle, that the number of stations used in thier published global temperature graph peaks from about 6000 in the 1970’s, all the down to about 1000 by the year 2000. You can also see from the graph on the left, that there are about 1000 stations which have records spanning about 100 years. Here is what I’m proposing, if they made a graph of only those 1000 stations, with no tinkering of data or filling in the blanks, and that graph showed a rate of temperature change (the slope of the line, not it’s actual Y-axis value) that was any different from thier final published product of global temps, than someone over there needs to go back to the drawing board.
I had another question to put out there. What is “forcing”? I found this at GISS site.
http://data.giss.nasa.gov/modelforce/
But I cannot find a good explaination of what it is. What is the Y-axis unit of F(W/m^2) representing? For CO2, they show a forcing of “1.50 W/m^2)”. I guess the big question is: Is this something they fed back into to the computer program to help them reconstruct temperatures? Or is it just the results of some analysis they did which they are presenting to us for our information?
Steve: “Forcing” is a term used in climate literature. It is not used in GISS temperature calculations.
I recently scraped page two of the GISS station data for each station they use.(the page with the chart link) I was able to get 6249 of the 6257 pages. There they provide the start and final year of each station record. There was an understandably large downturn in stations circa 1991 with the collapse of the Soviet Union. But what’s the story with 2007 which lost nearly as many stations as 1991? Was there a massive collapse in the US that no one told me about?
The stations used in their analysis of 2007 were reduced from 2113 in 2006 to 1051 in 2007. Mostly (98%) by removal of US stations. The stations classified as rural have dropped from 48% to 28% world-wide. US stations dropped from 57% to 13% of the world.
World figures include USA.
Steve: I’ve talked about the stations before – it is presently my view, although I’m not 100% certain, that there was no collapse of station population in 1991-1992 as many people think. All that happened is that GHCN did a big historical collection in the early 1990s and hasn’t kept them up to date. As to the 2007 population, it’s probably a lag in including the USHCN data. The first tranche is only the MCDW primarily-airport network.
We’ll know for sure in the next few weeks. I’ll do it again next month just for current back to 2006. Can’t say I’ve paid close attention to the dates on that page, but if monthly data were available during 2007 I think they would have adjusted that page since the 2007 seasonal year concluded with Novemeber.
Your initial query for this post was “why GISS uses some GHCN stations and not others”?
I have been interested to see how the GHCN data sets became GISS data sets. I’ve had a brief look at Western Australia (the largest Australian state). I used the GHCN V1 station list from ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v1/ press.sta.stat.inv (a BitZipped file).
The list only seems to contain 12 Western Australian stations and only data for the period 1951 to 1988:
Western Australia has a land area of approximately 2.6 million square kilometres. The V1 GHCN data was released in 1992 (from approximately 6000 stations). The Western Australian data therefore represented 1/1000 of the total stations.
Was this GHCN data commencing in 1951 used at the time of Hansen’s 1981 paper?
The V1 GHCN data was the subject of some cleaning up and review by the NCDC in 1998 which led to GHCN v2 data (see Peterson et al (1998) International Journal of Climatology, Volume 18, pages 1169 to 1179).
I looked at the GHCN station inventory file ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.temperature.inv file on 9 October 2008. There were 65 Western Australian stations in the V2 temperature station inventory file. The representation has increased ten fold from v1 GHCN data.
As you have noted above, the type of information contained in the v2 files has also changed slightly from the v1 station file.
Sample Data from v2.inventory.inv:
In reply to the original note and comment 1, above, re TUNDRA I confirm that some other vegetation codes seem inappropriate for their location. For example, for the Western Australia data, ROTTNEST ISLAND and MANDURAH stations contain vegetation codes for “WARM CROPS”. No cropping goes on there and those stations are both practically on the Indian Ocean. Similarly, I do not understand why CAPE LEEUWIN and CAPE NATURALISTE would be materially different (both south western coastal).
I am curious as to why this data is relevant and why it was included in v2? It is not in v1 Station Inventory file, so it represents additional GISS data and not modified GHCN data?
To investigate this, I have looked at ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.temperature.readme and it states:
Whilst looking at this, I noticed that many of the stations in the vicinity of Perth, Western Australia have a string “FLxxCO”. The information in the little Fortran program (read.inv.f) explains that “FL” as part of the string “FLxxCO” for PERTH AIRPORT would be referrring to flat topography and “CO” means the station location is coastal (i.e. within 30 kilometres of coast). It then goes on to offer some information on the the last vegetation description:
It looks like there is another vegetation data base out there somewhere to explain the codes in the GISS data.