In his comment to How much Estimation is too much Estimation?, Anthony Watts suggested I create a scatter plot showing station distribution with latitude/longitude. It turned out not to be the ordeal I thought it might be, so I have posted some of the results in this thread. I started with 1885 and created a plot every 20 years, ending in 2005. I deliberately ended with 2005 because this is the final year in the GHCN record prior to the US station die-off of 2006.
Every dot on a plot represents a station, not a scribal record. Stations may be comprised of multiple records. A blue dot represents a station with an annual average that was fully calculated from existing monthly averages. A red dot represents a station that had missing monthly averages for that year, so the annual average had to be estimated. Stations that had insufficient data to estimate an annual average are not shown.
In the case where multiple scribal records exist for a station in the given year, I assigned a blue dot if all records were fully calculated from existing averages, a red dot if at least one record was estimated, and no dot if none of the records could produce an estimate. I believe this errs in the direction of assigning more blue dots than is deserved. Hansen’s bias method mathematically forces estimation to occur during the period of scribal record overlap.
The first plot shows coverage in 1885, five years into the GHCN record.
1905 shows improved coverage across the continental US, Japan and parts of Australia. A few stations have appeared in Africa.
1925 shows increased density in the western US, southern Canada, and the coast of Australia.
At the end of WWII, not a lot of change is noticeable other than improved coverage in Africa and South America as well as central China and Siberia.
In 1965 we see considerable increases inChina, parts of Europe, Turkey, Africa and South America.
A decline in quality seems to be apparent in 1985, as many more stations show as red, indicating their averages are estimated due to missing monthly data.
A huge drop in stations is visible in the 2005 plot, notably Australia, China, and Canada. 2005 was the warmest year in over a century. Not surprising, as the Earth hadn’t seen station coverage like that in over a century.
The final plot illustrates the world-wide station coverage used to tell us “2006 Was Earth’s Fifth Warmest Year“.
Update (Steve Mc): USHCN station information gets added into GHCN with a lag of almost a year (noted in comments below). Jerry Brennan, who’s followed this for some time, reports the following update schedule in the past:
USHCN station data for the year 2002 were published in the USHCN website by May 2003, and added to GHCN between November 8, and December 10, 2003.
USHCN station data for the year 2003 were added to GHCN between April 10, and May 6, 2004, and published in the USHCN website by January 2005.
USHCN station data for the years 2004, 2005, and the first three months of 2006, were added to GHCN between August 13, and September 11, 2006, and published in the USHCN website, with data through October 2006 by March 2007. The additional months of data were not added to GHCN.
USHCN station data through May 2007 were published in the USHCN website in October 2007, but the “new” data have not been added to GHCN as of this date.
By the end of February of each year, GHCN will usually have data for the full previous year from only 120 (non USHCN) stations in the 48 contiguous USA states.
I might add that there are two locations for USHCN data, one at NOAA and one at CDIAC. The NOAA version is more updated than the CDIAC version – perhaps there are other differences as well. I personally confirmed that the NOAA version (Oct 2007 edition) is updated to May 2007 for most USHCN stations. There was a USHCN update in May or June 2007 which updated to late 2006 for most USHCN stations.
I personally confirmed that the most recent GHCN version (ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2 Feb 2008 edition), as noted by Jerry, contains USHCN updates only to March 2006 or so. Thus, GHCN is two USHCN updates behind at present (May 2007, Oct 2007) and its USHCN version is at least 14 months stale relative to what it could be using.
Climate Audit 
114 Comments
Nice job, John. What are the details on how you made the maps? Not that I’m asking you to do it, but a time lapse movie showing the stations winking on and off would add the time dimension in a revealing way. Somebody who visits here could do it, I’m sure. The rapid decline in the 90s would really be apparent.
Wow! That is shocking! Worse than I had thought, even though there has been a lot to cause doubts to exist. How can the “experts” really expect anyone to trust the “global average temperature”?
What happened between 2005 and 2006 to account for the
large drop in stations in the USA?
losing spatial coverage and temporal coverage..
losing spatial coverage and temporal coverage..
So by Rabbit’s logic, 20-century global warming is a local phenomenon limited to the continental US.
The drop from 2005 to 2006 has to be a lack data reporting. There is no other explanation that makes sense
The video idea would be very helpful. I remembered this one from Statistics Canada. The 50 year census data provides one example.
Technical question: how do they take into account the non-uniform distribution of the stations when calculating the “global” mean temperature? Regions like the Arctic, Siberia, large parts of South America and Africa have little or no coverage. Surely this introduces an error in the global mean. What is the estimate for this error?
Follow up to my post #9
If most of the warming has occured in the Arctic (including Greenland) and Siberia, as the data seem to show, surely the lack of stations there introduces a large uncertainty into the warming trend. In other words, if the regions where the trend is the strongest have the poorest coverage, then the error on the trend is larger than if the trend was uniformly distributed across all stations.
Assuming that the satellite data now provides more complete coverage, is there a way of determining how well the satellite data may be calibrated against the ground station data? Is there enough overlap and stability in the satellite methods with sufficient numbers of ground stations?
John, Do you have the List of stations for the US in 2005…
I’m wondering how many of those stations anthony has surveyed .
Francois–
The averages are supposed to be surface area weighted. That must mean that in areas with low thermometer density, each thermometer measurement contributes a lot to the average. To the extent that that thermometer is inaccurate or temperature varies spacially, this can add quite a bit of uncertainty to the GMST.
In places like the US which, until recently was entirely covered with dots, each individual thermometer would contribute less to the average. In these areas, uncertainties in individual thermometer measurements are likely to cancel, and uncertainties due to real spacial variations will be smaller.
Ideally, to minimize error, you have thermometers distributed more or less evenly by area. ( Though, if there are large regions that are climatically similar, you could deviate from this a bit.)
It’s interesting to see Canada never quite jumped into engaged in the temperature measurement frenzy that appeared well under way in the US by 1885. I can’t help wonder what caused us to sprinkle thermometers everywhere?
I’m squinting– it looks like Guatamala city had a station in 1905; that general area had many in 1965 (around when my family lived there.)
Squinting again– are there no stations in Greenland?
#12 Steve
Yes, I have lists now of the entire world, by year, and whether or not the station reported an average and if it was estimated. I also have summary lists by continent / by month, and can create them easily enough by country if I need to.
#1 Gary
I have considered creating an AVI or some other movie file. If I have time I will do it. The movie is easy enough. It is generating each graph that can be tedious.
John:
Can you change the color of the dots to make them stand out from the coastline. I would hate for Lucia to hurt her eyes! 🙂
Lucia, there are stations in Greenland but they are on the coast and are masked in the coastline. You can see them if you look at 1965.
My recollection that much of the network of weather stations was associated with agriculture. Hence the more temperate regions of Canada have coverage comparable to the US lower 48, but once you get outside of the areas where there is substantial farming, there isn’t much need for the coverage.
Although John’s map doesn’t seem to have any Greenland Stations, GISS has several. Here is one .
It’s unfortunate that between 1985 and 2005 so many thermometers in Canada disappeared.
#13 Thanks Lucia,
I understand the idea of weigthing by surface area. The point I was trying to make is that if you’re looking for a trend (up or down), and that the trend is not the same everywhere, the extent of the coverage will affect the error on the trend differently for different locations.
Say there’s no warming in the tropics, but you’ve got one station every kilometer, but there is a presumably strong trend in the Arctic, where you’ve only got a handful of stations. All those stations you have in the tropic are basically useless to determine what the trend is. You might as well not use them. So if you only use the Arctic stations, you will find that your uncertainty on the trend is mighty large. So having a large number of stations globally can give you a false sense of confidence in the actual uncertainty of the trend itself.
For example, some data seem to show that Greenland was as warm in the 1930’s as it is today. But how are we to know: there were precious few stations, if any at all, during that period.
I’m not too concerned about the data after, say, 1960. But I’d say the error on the first half of the 20th century may be larger than a simple estimate based on the sheer number of stations would give. I’m not quite sure how you would take that into account in making your error estimate. That’s basically the question I am asking to this board!
This is important, not so much because the 1930’s, or the MWP or whatever other period may have been as warm as today. But if you’re going to calibrate your GCM based on the past, and our knowledge of the forcings (solar, volcanic, GHG), you need to take into account that uncertainty. Larger “natural” variability may mean that solar forcing, for example, has a larger effect than currently estimated.
lucia says:
lucia says:
The gridded data is fixed 5×5 lat/long blocks. This implies that a much smaller surface area near the poles would have a more significant impact on the global average. Do you know if they really use surface area or it the GMT a simple average of the gridded dataset?
Steve: They say that they area-weight and I have no reason to think otherwise.
Raven–
Well,I’m not certain. But it isn’t necessarily either/or. They could easily attribute an average temperature to each block, and then, later when finding the average for the planet, weight by the area for the block.
I’d be a bit surprised if computations of GMST used temperatures weighting them like a distorted flat map. That makes no sense physically, and doesn’t strike me as the sort of thing either GISS or Hadley would be likely to do. (I guess we could break down and read the papers? There are links at each entities sites.)
13, until fairly recently, someone had to live there to have a weather station. That’s why most of Canada and all of Greenland are empty. Even now, if it’s unmanned, a source of power and a communication link of some sort is necessary. That can be done way out in the boonies with solar panels and a satellite uplink, but that’s a bunch of money.
The GHCN station collation (used in turn by CRU and NASA-GISS) has a delay in incorporating USHCN network. It looks like the 2006 list lacks the USHCN stations, but they will get into the 2006 list at some point fairly soon.
Also and I’ve mentioned this many times, GHCN collated a lot of historical data in the early 1990s that it promised to update “irregularly”. For the most part, the “irregular update” hasn’t occurred since then and subsequent data is mostly airport data. It’s my surmise that data is actually available in local services for many stations that “disappear” in the early 1990s and that the reduction shown here mainly reflects the fact that GHCN collation has been very lackadaisical. There may also be station closures, but I know for a fact that some Russian and Chinese stations unavailable after the GHCN last date in the early 1990s continued to functioun.
This animation may be of interest (a bit dramatic):
http://climate.geog.udel.edu/~climate/html_pages/Ghcn2_images/air_loc.mpg
Francois– Yes. It’s the large areas with no stations that are a problem with respect to uncertainty. (That is, unless we can assume temperature vary much less spacially in those regions rather than other regions for some reason. Can we? I think there are spacial temperature variations in northern Canada. No reason to think otherwise. )
Lucia says:
From http://data.giss.nasa.gov/gistemp/sources/gistemp.html
“A grid of 8000 grid boxes of equal area is used. Time series are changed to series of anomalies. For each grid box, the stations within that grid box and also any station within 1200km of the center of that box are combined using the reference station method.
A similar method is also used to find a series of anomalies for 80 regions consisting of 100 boxes from the series for those boxes, and again to find the series for 6 latitudinal zones from those regional series, and finally to find the hemispheric and global series from the zonal series.”
It appears they measure area with lat/long and attempt to compensate for the different physical areas by computing the zonal averages. However, they effectively negate this step by using the zonal means to calculate the global means. This implies the polar data makes up 1/3 of the GMST if they calculate a simple average of the zones.
Re #26
Here are the GISS anomaly values for 2005, as a test case:
Equ to 24N: +0.53C
24N to 44N: +0.50C
44N to 64N: +1.19C
64N to 90N: +2.17C
This data, adjusted for the area of the NH which each band represents, should give the GISS value for the Northern Hemisphere (+0.82C).
I have been trying to determine that the GISS definition of grid boxes with “equal area” mean and grid boxes with equal number of degrees on each side rather than an equal number of meters.
I found this that seems to say that they do mean equal, as in distance:
http://badc.nerc.ac.uk/data/gedex/detail/giss_tmp.det
That link has break down of station coverage. I did some calculations to estimate the effect that a station has on the global average using this data:
64.2 – 90.0 N – 1.12
44.4 – 64.2 N – 0.43
23.6 – 44.4 N – 0.60
0.0 – 23.6 N – 0.90
0.0 – 23.6 S – 1.51
23.6 – 44.4 S – 1.65
44.4 – 64.2 S – 6.78
64.2 – 90.0 S – 5.46
I calculated this number by diving the % grid boxes per band by the % stations per band
Initially, war needs; the ‘wx services’ of the time were originally in the ‘Signal Service’ (think ‘telegraph services’ back then) branch of the US Army with subsequent reporting changes since then:
David Laskin’s book, “Braving the elements: the stormy history of American weather” pg 143
They don’t compute a simple average, and I haven’t been able to come up with a weighting scheme that gets me from the zonal averages to the global averages.
Here’s the last seven years:
24N 24S 90S 64N 44N 24N EQU 24S 44S 64S 90S
Year Glob NHem SHem -90N -24N -24S -90N -64N -44N -24N -EQU -24S -44S -64S Year
2001 48 59 36 75 37 36 101 79 64 36 38 43 21 43 2001
2002 56 66 46 79 51 40 123 95 53 48 55 37 32 74 2002
2003 55 69 41 81 54 31 144 101 46 51 56 31 24 48 2003
2004 49 63 34 72 45 30 75 89 60 49 41 35 23 26 2004
2005 62 82 43 100 55 33 217 119 50 53 57 37 19 53 2005
2006 54 74 34 92 46 27 172 103 58 47 44 39 14 19 2006
2007 57 80 34 109 35 34 221 129 60 36 34 36 6 96 2007
Maybe somebody else will have better luck.
Hansen says the adjustments he quietly made because of Steve’s discovery of an error in the post-2000 data was restricted to the US. These maps indicate the extent to which US stations dominate the global total. It is hard to imagine the correction didn’t have some effect on the global number. We also need to know how many stations Hansen uses for his global annual average are from outside the US.
The other issue to consider is that in 1885 most stations were close to big cities. It is likely they would have experienced the most amount of Urban Heat Island Effect, which raises questions about which stations P.D Jones used to create his 130+ year claim of an unnatural rise in the annual average temperature.
RE 31. Dr. Ball, Hansens method wont Overweight the US. He divides the world into equal area chucks
and then estimates the chuck based on the stations within the chuck. So, densely sampled areas get
1 number with a small varience and less dense areas get a number with a bigger varience.
Still, the lack of any semblance of uniformity either in the spatial dimension or the temporal dimension
Should raise some legitamate doubt.
Grant me some literary licence in the following: With ‘cold air’ being produced in the polar regions and warm, humid air in the tropics (we will ignore oceans for the moemnt) under the influence of the global ‘three-cell GCM’ (GCM as in General Ciculation Model as the term is used introduced to those studying meteorology) with a sufficiently long average (multi-year averaging) I as an engineer would be somewhat content that we would be measuring climate scale temperatures (and any changes) of that region and not simply weather variability.
re 28. Raven go read Hansen 87 ( i think) where he describes the equal area tiling of the sphere.
Within each equal area tile he makes further tilings. I think its hansen 87. if you cant find it
or I’m wrong, throw a rock at me, and I’ll search my stack of junk
RE #30 For 2005 Northern Hemisphere I get –
Eq to 24N: +0.53C for 40.7% of NH area
24N to 44N; +0.50C for 28.8%
44N to 64N: +1.19C for 20.4%
64N to 90N: +2.17C for 10.1%
which comes out to the +0.82C reported by GISS for the Northern Hemisphere
#33 Not sure if it is on topic but from what I gather you are suggesting the use of time averages to make for the lack of ensemble run. I think this is not a hopeless suggestion but it clearly has its problems.
RE 25. Lucia. There is something VERY FISHY about the 1200KM figure that hansen uses
to “estimate” or adjust stations.
In Hansen 87 ( I’ll check if I’m right) he looked at Correlations between stations.
At 1200KM he found a correlation of .6 in the NH .5 in the SH.
He tested a relatively small number of stations.
I find this utterly insane. Especially given the teleconnection argument where A grid
square on one side of the planet is correlated with a grid square on the other side.
more later
steven– he found a correlation of 0.6 between what and what?
Not all the methods in Hansen are the same as those in CRU. Be careful to specify.
What is CRU?
#24
Of course in the arctic and antarctic much of the winter months are without sunlight to power a solar powered device.
From HL87:
Thanks John for doing this, this is splendid!
OT I’m a bit late in joining the conversation again, as I just spent most of the weekend developing confirmation that a volcano near the Ross ice shelf once thought to be extinct, is now becoming active again. I welcome any thoughts anyone might have.
http://wattsupwiththat.wordpress.com/2008/02/10/photo-evidence-for-eruption-of-an-extinct-volcano-near-the-ross-ice-shelf/
#13 Lucia
The Greenland stations are there. Any station on a coast will be hard to pick out, because the color will be hidden by the grayscale of the coastal outline. I toyed with larger station marks, but they gave the false impression of really great global coverage. I tried with different color schemes, but I am sensitive to those that are color-blind. Rest assured, though, the stations are there.
#45 How about some kind of contour plot to indicate station density. Given stations occupy a discrete point some kind of low pas filter would need to be used. Perhaps a Gaussian shaped low pas filter. I’m not sure how you would decide on what the best spatial width of the filter should be.
Anthony: If there was an eruption there would likely be earthquake activity. Have you checked this yet?
RE47, yes lots of it, but please carry on the conversation over there rather than OT here. Just wanted to advise people here who are good at pointing out flaws so I could strengthen the presentation or go down in flames gracefully.
I’ve combined John’s graphics into an animated GIF here:
http://www.flickr.com/photos/toptechwriter/2257020940/
Re #41;
CRU is the Hadley Cenre Climate Resarch Unit in East Anglia, UK.
Between 1890 and 1940 the Weather Bureau was a part of the Dept. of Agriculture. Temperatures were important because the daily max and min temps affected crops, especially around 32°F and in the 100+°F ranges. Because of the need for detailed and frequent observations for aircraft operations, the Weather Bureau moved into the Dept of Commerce (FAA) and the interest in parameters such as temperature and quantitative precipitatiion were of low priority as compared to wind direction and speed, ceiling heights and horizontal visibilities. None of the parameters most important to agriculture and today’s climate studies was of much concern to the FAA observers. The switch from manned observations by specially trained NWS employees to automated observations, notably ASOS and the replacement of CRS platforms with ASOS and MMTS equipment occurred in the late 70s and early 80s. The reliability and precision of thermometry in the automated systems was sharply lower than the certified error range of mercury-in-class thermometers used in the CRS screens. Certified mercury-in-glass thermometers were calibated to about 0.1°C. The acceptable range of calibration for ASOS instrumentation was plus or minus 1.2°C.
The development of the North American Radar Defense network, usually referred to as the DEW (Distant Early Warning) Line, brought manned weather observations along the Arctic Circle in Canada. Some 58 Dew Line stations were constructed during the 1954-1957 period. These were usually the only regular observations available from these remote locations. The end of the cold war saw a reduction and essentially an elimation of most of these observations. I suspect that similar intstallations in the FSU fell by the wayside as well. A great history and a map of the Dew Line stations with other informative goodies can be found here.
With today’s observations mostly taken from airports in or close to urban areas, there are very few continuous observations taken from rural locations left to provide reliable climate data. Urban areas amount to well under 2% (some estimations put them, at 0.2%) of the Earth’s surface. It would be surprising if, as these urbam areas continue to expand, there were not upward trends in the average of currently observed land temperatures. To refer to these averages as representing global average temperatures is patently absurd.
Re#43 & 47:
Tony, you might want to check out this NSF article released in May 2004 re Ross Shelf volcanic acivity.
Just a quick note: Stations seem to proliferate where humans are. More humans means more heat from metabolism, (irregardless of urbanisation).
Hence increase in global surface temperature simply means that life proliferated.
Over to you
Some timeline information regarding USHCN data and GHCN.
About 1200 US stations in GHCN are USHCN stations, and their data
are handled differently than those of other stations.
USHCN station data for the year 2002 were published in the USHCN website
by May 2003, and added to GHCN between November 8, and December 10, 2003.
USHCN station data for the year 2003 were added to GHCN between April 10,
and May 6, 2004, and published in the USHCN website by January 2005.
USHCN station data for the years 2004, 2005, and the first three months
of 2006, were added to GHCN between August 13, and September 11, 2006,
and published in the USHCN website, with data through October 2006 by
March 2007. The additional months of data were not added to GHCN.
USHCN station data through May 2007 were published in the USHCN website
in October 2007, but the “new” data have not been added to GHCN as of
this date.
By the end of February of each year, GHCN will usually have data for the
full previous year from only 120 (non USHCN) stations in the 48
contiguous USA states.
#52 JerryB
Thanks. This confirms what Steve has been saying about the missing US data (what I like to refer to as “die off”).
When the data for 2007 and 2006 appears, it will be fun to see if / how it changes the results for those two years here. Right now those two year’s anomalies are calculated without the USHCN stations.
Re: 53,
John,
NCDC has a USHCN version 2 in the works, and the next addtion
of USHCN data to GHCN may not occur until USHCN v 2 gets
published. The to be published date has already passed (July 2007)
so it’s anyone’s guess as to when it will be.
C’mon Steve, tell us about your trip. Are you or are you not now a Ramblin’ Wreck fromm Georgia Tech?
Hello all:
Steve, congratulations on the invitation. I am looking forward to hearing how it went.
As for the coverage issue, it is something I brought up some time ago on CA before. For example, see http://www.climateaudit.org/?p=1985#comment-131636 (I remember making other posts on the topic as well but I can’t find them).
There is also some numbers of contributing stations and countries at http://www.unur.com/sinan/outbox/070816-us-is-only-two-percent.html
As someone else suggested, it might be worthwhile to put together an animation of station locations from scratch. I’ll take a stab at it but I probably won’t have any time before March.
— Sinan
Thanks David. I was missing the sine function factor there and was too distracted to track it down.
#532. Jerry B, thanks for this. I’ve added this info to the post.
I might add that there are two locations for USHCN data, one at NOAA and one at CDIAC. The NOAA version is more updated than the CDIAC version – perhaps there are other differences as well. I personally confirmed Jerry’s observation that the current NOAA version of USHCN (Oct 2007 edition) is updated to May 2007 for most USHCN stations. In addition, there was a USHCN update in May or June 2007 which updated to late 2006 for most USHCN stations.
I also confirmed Jerry’s observation that the most recent GHCN version (ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2 Feb 2008 edition) contains USHCN updates only to March 2006 or so.
Thus, GHCN is two USHCN updates behind at present (May 2007, Oct 2007) and its USHCN version is at least 14 months stale relative to what it could be using.
I wonder what accounts for the delay which seems completely unnecessary. It takes only a few minutes to do the programming to perform the update.
RE 54. Jerry. V2 has employed some change point analysis, but the cited papers are behind the green
wall. Do you have the relevant papers?
Re: 58,
Steve,
I believe the delay is due to the expectation that v 2 will
be published real soon. In the late summer of 2002 I spoke
with someone working on USHCN v 2 at NCDC, and he then
expressed the opinion that it would be published by early 2003.
BTW, the two developers of GHCN v 2, Peterson and Vose, do
not participate in the month to month updating process.
Re: 59,
steven,
I do not have any of those papers.
Indeed. When I checked Czech stations, some of these also disappeared, although they are run by Hydrometeorological Institute, properly stationed and well and alive.
Great graphs, John!
If the updating lag is part of the appearance of recent station loss, it might be useful to go back a couple of years before 2005, say 2000, when reporting is presumably as complete as it is ever going to be.
John’s graphs (and many other such climate graphs) are equirectangular, meaning that the vertical axis is proportional to latitude. This eliminates the distance distortion of the Mercator projection in the vertical direction, but leaves a horizontal distortion proportional to secant(longitude). As a consequence, polar areas like Greenland and Antarctica still have too much area, though not as badly as Mercator.
This area distortion can be eliminated by an appropriate compression of the vertical axes as the poles are approached. I believe making the vertical axis proportional to sin(latudide) would have this effect, since the derivative of this function is cos(latitude), the reciprocal of the secant(latitude) horizontal distortion. This projection would doubly mash shapes near the poles, but would show relative areas at a glance, without the N-S directional distortion of ovoid projections.
Has anyone in Climatology ever used such an equiareal rectangular projection? Is it available on canned mapping routines? The standard equirectangular projection maps make a little bit of local Arctic warming look like the whole earth is on fire!
The real question of interest in USHCNv2 is in how they’re going to handle UHI effects. Their UHI corrections as applied in part-4 of version one are totally ridiculous. See the NYC Central Park version of this which is completely untenable. Despite this, except for a brief presentation of the NYC data on this site and a few other blog sites, the silence from the AGW community has not only been been deafening, but also revealing. The numbers are so outrageous that any attempt to support these numbers would justifiably provoke ridicule. On the other hand, the raw data has been generally faithful to the actual observations. Only in the world of climate science could this be considered a novel approach.
I understand that they have made some sort of “adjustment” to the raw data in v2 to provide a uniform UHI adjustment, but they may be spending some time on figuring out how to justify these changes when the data is released to the masses. Whatever; when the data finally is released, look for the adjusted version to become the focus of reinvigorated controversey.
#62 The Lambert Equal Area Projection avoids some spacial distortion, although it’s not rectangular.
http://en.wikipedia.org/wiki/Lambert_azimuthal_equal-area_projection
Warm 1930s may well be global. And any appearance that is is not, may be a result of poor non US coverage.
#62. Equi-area projections are used in many contexts. In terms of graphic presentations, you’re 100% right and anything other than equi-areal should be banned from scientific journals.
In practical terms at my level, the only issue is the availability of routines in R (and for someone else Matlab). Doug Nychka of NCAR maintains the R-package fields. He’s very responsive to inquiries. I’ll check the manual and otherwise check with him.
Steve – A very important add-on to this analysis would be to quantify the actual overlap in the GISS, NCDC and CRU data sets in terms of the raw station data that they draw from. Phil Jones told us several years ago that about 90-95% of the sites are from the same raw data, but this has not been confirmed by quantitative analysis. If the data sets are so interdependent, this means that it is misleading to present the trends from the separate analyses as independent assessments. Roger
Steve Mc,
Long time reader, first time commenter. Kudos for all your, et al, work.
I’ve not seen any discussion in the great debate that deals with the accuracy of the sensor that
records the temperature. I’m not talking about an external interference, such as rooftop or parking lot locations,
but the inherent accuracy of the gauge. Suffice it to say that thermometers have gotten better since biblical times.
Is there any way to account for this in the modeling?
re 63… Ushcnv2 will do NO UHI adjustments, If I read the posting correctly.
1. peterson ( 2003) beleives them unecessary ( cool parks thesis)
2. Change point analysis solves everything..
Just when you guys thought you had it all figured out, the method changes.
re 63… Ushcnv2 will do NO UHI adjustments, If I read the posting correctly.
1. peterson ( 2003) beleives them unecessary ( cool parks thesis)
2. Change point analysis solves everything..
Just when you guys thought you had it all figured out, the method changes.
http://cran.r-project.org/doc/packages/mapproj.pdf has a package with equi-area maps.
Somebody should be able to locate parameters that yield a “good” map for representing areas in climate contexts.
#67. I agree entirely that they are not “independent”. My guess right now is that GISS has more smoothing at a gridcell level than CRU3. For example, the CRU3 gridcell containing Barabinsk can be tracked to Barabinsk, but the GISS gridcell will be a complicated blend.
The allocation of critical enterprise between land SST is totally skewed. Hardly any critical effort has been expended on SST and IMO it’s a greater priority.
Re # 62, 64, 66, after a little Wikisurfing, it appears that what I am after is known as The Lambert Cylindrical Equal Area projection. The following graph is lifted from http://www.progonos.com/furuti/MapProj/Normal/ProjCyl/ProjCEA/projCEA.html:
The Lambert Equal Area that Gary refers to in #64 is his Azimuthal, rather than Cylindrical projection.
Another good option (but one that distorts N-S directions) is the Mollweide projection:
A good example of the distortion of impression that is caused by the equirectangular projection are Hansen’s graphs in the 2/3 thread “Hansen in Antarctica” (#2658). His “Fig. 3.19” copied there is somewhat like Mollweide, though I don’t think it’s quite equal-area.
#32
I appreciate the averaging approach, but the problem is what do you do when you have no station in an area? In fact, there are many areas where you have many rectangles over a large area with no stations. As I recall they used to take averages for the four corners and then produce an average for the entire area. Again as I recall there were several problems with this because many rectangles had no stations but were various proportions of land and ocean or freshwater. I also recall questions about differences in elevation. One debate I had with a modeler involved a rectangle that was half flat prairie and half high mountains and there were no stations in the mountainous half. As I have noted many times the surface data is inadequate as the basis for determining global annual average temperature let alone as the basis for models, it is made much less than adequate with the lack of above surface data.
Discussion about the arctic, lack of data, and the equi-rectangular problem is exacerbated by the lack of data. Take a look at the map in the Arctic Climate Impact Assessment Report (ACIA)(the main source of information for the IPCC) here:
Click to access ACIA_Ch01_Final.pdf
and on page 3 of Chapter one you will see a map depicting the lack of data. Elsewhere in the report you will see maps showing the paucity of data throughout the arctic region, however you define it.
RE 74. Hi Dr. Ball. I’m not sure about hadcru, but Hansens method is to average over a 1200km radius.
( after carving the globe into equal area tiles)
I think Hansen 87 has the relevant graphs and analysis. In essence Hansen studied the correlation
in temp between some 50 or so sites and found that at 1200KM the correlation was .6
.6 being a nice number and 1200km being a nice round number ( in metric, of course in the
english system it’s downright ugly) the science was settled. So, thou shalt average stations within
1200km of each other, even when said 1200km crosses oceans, mountains,deserts, and polar ice caps.
sarcasm off.
If you modify latitude by sin(lat in radians) you get the simplest equal-area projection. No need to use fancier things for basic display.
One possible reason for some of the reduction in stations is that the FAA had an automation program in the 1980’s that involved closing Flight Service Stations. They were mostly consolidated into 64 AFSS facilities. One of the FSS Specialist jobs was taking local weather observations, so that may have caused some of the US reduction.
Something intuitively bothers me about lumping such diverse climates into these tiles. I can’t quite put it into words what seems so wrong about, for example, lumping all of California and it’s surrounding land and ocean into one tile, but it seems awfully coarse. Can anyone either show why I’m wrong, or why I’m right?
Next time you need to leave your sarcasm on long enough to note that a correlation coefficient of 0.6 is equivalent to an R-squared of 0.36. IOW, the 1200km radus explains 36% of the temperature variation. Science Settled. Moving on.
#74 Tim Ball, They should really use satellite data to help to interpolate the missing pieces.
Hi,
John #80,
I think they don’t use satellite data is that the satellite and GISS have diverged lately. Current January 2008 RSS -.629, UAH -.588, GISS +.12. That is the problem. As for coverage I see the tropic and desert seem to have few stations. That is where the AGW group says we should have changes along with the well known Arctic and Antarctic coverage problems. GISS has yet to explain the divergence from the satellite data. Some say its the polar regions but if you do or do not include them it still doesn’t seem to match anything.
re 79. I think we’re in violent agreement, unless you want to fight about it
Here is a graph with much higher station-to-station correlation in high lattitudes, the correlation is worst near the equator, which is logical as he value of annual average temperature is dominated by winter temperature.
which is actually not bad for interpolation to 1200 km at high lattitudes.
A similar interpolation is done with global heat flow measurements.
http://en.wikipedia.org/wiki/Spherical_harmonics
RE: Greenland data – From my review, there are two stations with long-term data (at least 1930 – present), Gadthab Nuuk and Angmagssalik, in the GISS station data. For both of these sites, the temp anomaly in the 1930s/40s was about the same as the past ten years or so. The two long-term sites in Iceland (Reykjevik and Akureyri) as well as one (Jan Mayen) on the big island north of Norway and Sweden show the same pattern. So for this part of the Artic, the current temperatures are not much different than historic temps.
Re: Central Park, NYC – If you look at the pre-homgeneization data from this site (i.e., not corrected for UHI), there is very little, if any increase in temperatures since the 1940s. For this specific site, UHI may not be that big a deal in the latter half of the 20th century since the entire area around Central Park was already built up by that time. Since we are looking at anomalies instead of actual temperature, I don’t know that it would play that big of a deal in locations that have been heavily urbanized throughout the temperature record.
Re: lumping different climatic zones into one grid – Again – since what they are looking at are the temperature anomalies for each station, I don’t think that lumping the sites together makes that big a difference.
The easiest way to evaluate the question of whether there is long-term global warming would actually to calculate the long-term temp anomaly trend for an spatially representative set of stations (probably 1000 would be enough) with the most reliable data set and determine if there is a statistically significant increasing trend at each location. If 90% of these stations showed a statistically significant increase, some showed no change, and less than 5% showed cooling, we could be pretty damn certain that there has been world-wide warming. I am sure it would be possible to develop specific cutoff percentages (e.g., 70% warming, 20% no trend, 10% cooling) to determine the answer. Of course, you don’t get a global average temp. anomaly to worry about with this method.
Bob North
Then why grid the earth at all? Why not one big lump? I can’t see any more rationale for lumping the Mojave desert in with the Pacific ocean than lumping the Andes in with the Indian ocean.
RE Hans Erren #83,
Very interesting! However, these correlations must be sensitive to time interval. At a daily frequency, 1200 KM is much too large, though it evidently works well at annual or even monthly frequencies.
Incidentally, Hans’s maps appear to be based on the 1805 Mollweide projection, which I’m coming to appreciate more and more as I think about this. The 1772 Lambert projection is a big improvement over the 200x Hansen projection, but it makes NAm and Eurasia look like they have been worked over with a meat tenderizer, while SAm is horribly crippled. Mollweide is just as equi-areal, yet much more elegant.
RE 83.. thanks Hans, I wonder why hansen did not adjust the range ( 1200km) by latitude.
Its just code.
re 85. If all you care about is global “trend”, use one thermometer.
central england has a friggin long record. the biases will all work out in the long term.
I agree. Put up your dukes. 😉
The grids that are a mix of land and water are factored to remove one or the other, supposedly. My beef is that if I’m tracking a grid and the variation between coldest and warmest in the year is 100F, why do I care if the global anomaly is off by 1.2 or so, when you can probably pick any day in a year and have much much more of a spread between the high and low of the day? What’s 1.2 if the temperature changes 40 in a day? Plus, do they track and adjust for heat flow from one 2×2 or 5×5 grid and another? I just don’t see it.
Anyway, if you want the data, it’s on the GISS site. GLB.Ts+dSST.txt
Global-mean monthly, annual and seasonal land-ocean temperature index, 1880-present, updated through most recent month on http://data.giss.nasa.gov/gistemp/
Different than what’s here (although it seems it’s not supposed to be) but it’s the NOAA site NESDIS:
http://www.ncdc.noaa.gov/gcag/gcagmerged.html
My question is why isn’t the GISS, and other like temperature data compared to the RSS and UAH data over the time period Jan 1979 to Jan 2008. RSS and UAH are probably the best and most consistant (within themselves and reasonably between them) temperature data that we have. The problems with these satellite data sets are probably well known.
I realize that there are different baselines etc, etc, but surely the statistical geniuses here can compensate for that (no slight intended). It seems to me that too much time is spent trying to fix the GISS type of data.Lets just compare the two and see what comes out of it I might be interesting.
For example, I came across this site and what struck me about the bar graphs was what a different impression they gave about the period 1979 to 1997 and how dissimiliar the different regions were. I seemed to me, that to a visual approximation, the years between 1979 and 1998 showed no temperature increase. So I asked William Briggs what he thought. His rather completereply was very interesting:
Most graphs of temperature show an simple regression like this:
Dr. Briggs look into the trends more deeply and one of the results was this
Note: No warming from 1979 to 1996 (or 98 depending on the transects). Warming to 2004 and then cooling again.
He also, indicates some cycles, and other patterns of unknown origonand in need of more study to see if they mean anything. Dr. Briggs wisely makes no conclusions (except one) for this analysis, but a layperson like myself sure finds them interesting. I would like to see more comment on them and the same type of analysis done on GISS type data to see if it is at all the same, somewhat the same or completely different.
My question is why isn’t the GISS, and other similar temperature data compared to the RSS and UAH data over the time period Jan 1979 to Jan 2008. RSS and UAH are probably the best and most consistant (within themselves and reasonably between them) temperature data that we have. The problems with these satellite data sets are probably well known.
I realize that there are different baselines etc, etc, but surely the statistical geniuses here can compensate for that (no slight intended). It seems to me that too much time is spent trying to fix the GISS type of data. Lets just compare the two and see what comes out of it. It might be interesting.
For example, I came across this site http://mclean.ch/climate/Tropos_temps.htm and what struck me about the bar graphs was what a different impression they gave about the period 1979 to 1997 and how dissimilar the different transects were. I seemed to me, that to a visual approximation, the years between 1979 and 1998 showed very little no temperature increase. So I asked William Briggs http://wmbriggs.com/blog/ what he thought. His rather complete reply was very interesting:
Most graphs of temperature show an simple regression like this:
Dr. Briggs look into the trends more deeply and one of the results was this:
Note: No warming from 1979 to 1996 (or 98 depending on the transects). Warming to 2004 and then cooling again.
He also, indicates some cycles, and other patterns of unknown origon in need of more study to see if they mean anything. Dr. Briggs wisely makes no conclusions (except one) for this analysis, but a layperson like myself sure finds them interesting. I would like to see more comment on them and the same type of analysis done on GISS type data to see if it is at all the same, somewhat the same or completely different. After all, I would think that at least the trens between the two types of data should be at least similiar.
Re #73:
Thanks Steve for the clarification. The “cool parks” theory doesn’t seem relevant unless you only analyze stations where data is collected in cool parks.
The change-point analysis is designed to find inflection points in a temporal data stream rather than the trend itself. It can only be analyzed well after the fact and perhaps that’s its attraction; it can’t be used for real-time tracking. It’s also a little cumbersome to use, even with large data streams, unless one presumes that the large data stream is in some way systematically correlated.
Re #84 and NYC data:
That’s my point. The raw data, or even the TOB corrected or Filnet data shows little warming over the past several decades. However, the UHI corrected data in that same data file differs from the raw data by as much as 7°F (cooler) for some of the months between 1960-1990, but this “correction” is gradually reduced to less than 2°F by the early 2000s. The effect of this “correction” is to create a hockey-stick trace in the past 40+ years where no such curve is detectable in the raw data. In other words, it’s an absurd manipulation of the data base under the guise of being an urban heat island correction in a city that has been urbanized for over 100 years. Still, I hear nothing but the sound of crickets from the AGW crowd on this particular USHCNv1 data series.
Aghh
Sorry everyone but kept on getting errors on the web server so two separate tries at sending got combined into one.
88, you just made the ergodic assumption. In the corner. Wih the dunce hat.
For a quick station coverage animation you can use Microsoft PowerPoint and
paste each one of John’s Station coverage GIF images onto a seperate slide.
You can then step forward or back through the years using the up and down
arrow keys. Down and dirty but works well.
Still, my UHI corrected average annual temperature in Central Europe does show significant warming over the 20th century.
http://home.casema.nl/errenwijlens/co2/europe.htm
Waldo is here.
Hans
but when you compare it to temperatures in the late 18th century there’s only minimal warming. When you start in 1880 as Giss and HadCruT do, there is significant warming
Re:grids, equirectangular projections, etc. Some years ago two solar physicists had an angry debate (thru regular mail and scientific magazines, no internet at that time). One of them had published a paper showing a 14 day period solar oscillation. The other one complained the oscillation detected was half the Sun rotation period so what the first one was actually measuring was an harmonic not filtered by the instruments or the data analysis. They finally agreed on building a computer model (CM) simulating the rotation of the sun to avoid instrumental bias. They built inside the CM a grid for the star feeding each cell of the grid with the actual luminosity (or whatever they were measuring) and then made the Sun rotate using the CM to see the average result for the whole Sun. They discovered everything was useless. The method used to map the grid within the CM had too much influence on the final results and afaik they finally switched – side by side-to a different strategy. Sorry if too much off topic. Best
re 98:
Indeed, The station data for the graph comes from http://data.giss.nasa.gov/gistemp/station_data/ before the truncation to post-1880 values in january 2005.
In january 2005 GISS decided to truncate temperature records at 1880…
This has been a very excellent thread. Despite the widespread issues with the land-based data, I do think we can say that there has been a warming trend over many parts of the world in the latter half of the 20th century. However, as shown in #96 above, the current temperatures do not appear to be wildly different than in many periods in the past, at least based on available instrument records (which probably have issues of their own). What gets me is why don’t the GISS folks just come clean with the many issues surrounding the temperature record and put some much bigger error bars on the trend estimates? Also, why do so many insist on keeping hundreds of records of qeustionable quality, yet when it comes to the pre-instrumental records, they are willing to rely on very sparsely distributed proxies?
Bob North
If you look at the CRU Central England Temperatures for the month, do you see anything worth spending 20 trillion dollars to “fix”?
http://www.cru.uea.ac.uk/cru/climon/data/cet/
The trend is telling us anything other than some averaged average of averages of sampled tMeans at random locations. Really, there’s is no evidence it has any meaning at all one way or the other.
Well, I asked the question (in the Estimation thread) about whether we were to believe there has been 80+% coverage of the Southern Hemisphere, but I already knew the answer.
I cannot post it, because it’s on paper, but somewhere I have a page from Science around 1999 of a study claiming to tease out a human component of GW. It is accompanied by a map, which shows white for areas with not enough coverage to assess.
As I recall, in addition to the Roaring ’40s, areas not assessed including all of Africa except the coasts, all of souhwest Asia, almost all of the Amazon basin, all of the Tibetan plateau and Sinkiang, and all of Antarctica.
In other words, this ‘global’ average disregarded the three hottest and the three coldest parts of the globe. I call shenanigans.
The graphs are cute but don’t tell me much I didn’t already know.
Sam post #102, that was exactly the gist of my post that was removed. When I posted “corect me if I am wrong” I was expecting to be corrected and learn but not shut.
In the end, this thread and the previous one is displaying the complexity of the disparate temperature data and of the statistical treatment the data is submitted to. Simply put, my question is: is the signal really better than the noise? Is it why this blog is so critical to make scientists who are not statisticians understand the temperature data quality problem in relation to the climate change issue? Isn’t that the reason why as suggested by Leroux, adding or substracting temperatures is meaningless at explaining climate change?Thank you for an answer.
Bob North, (84) I am not sure you are aware of all the issues for the Central Park, NYC site. Yes, very little, if any increase in unadjusted temperatures since the 1940s. However, in the seventies and eighties, the HCN adjusted temperature was about 7 degrees less than actual reading. Now, the HCN adjusted data is 2 degrees less. The decreased differential adds about 5 degrees (F) of reported warming. Meanwhile, the GHCN adjusted temperature(apparently used for global temperature) is almost seven degrees warmer than the HCN adjusted data — and both come from the same thermometer. Moreover, it is not clear that the UHI effect is no stronger today than it was 30 years ago. While inhabitants may have not increased in numbers, it seems to me in my trips to the area that number of workers in the area has increased, visitors to the park have increased, there are more offices in the area, and and traffic has increased. (Again, personal observation — no empirical study.) It would not surprise me if there had been some increased UHI effect even if the population has not increased.
Neverheless, even if NYC has not had a warming effect, I am comfortable with the conclusion that on average temperature are higher than they were two or three hundred years ago. Higher temperatures would explain glacial melt and extended growing seasons. Yet, because of proposed regulations, it is important to get an handle on whether current trends match projections of Global Climate Models if these Models are the basis of adopting regulations.
Re #105;
I agree. See my point in the last paragraph of #92.
The USHCN v2 announcement at
The USHCN v2 announcement
appears to have been updated, deleting mention of when
the data are to be made available.
Steve: Interesting. It used to say that the data would be available in July 2007. It’s hard not to think that the delays are related to their awareness that scrutiny of this release is likely to be more substantial than the prior release. Maybe they’ll even have proper documentation and technical reports accompanying the release.
global warming skips Vermont
I did a quick comparison of the sites with historic data prior to 2006 at co2 science http://www.co2science.org/scripts/CO2ScienceB2C/data/ushcn/ushcn.jsp
and compared them to the current Giss set.
at c02 science you can look by state and record the latitudes. If I remember correctly all the sites in vermont save one or two which show some kind of warming actually show cooling . AT co2 science, the state of vermont has escaped all the controversy!
Then I looked over at the Giss data. It is impossible to find most of these sites: they have been eliminated. But the one or two which remain are the warming ones. One station, enosberg looks very different at Giss than at co2 science. most troubling.
perhaps stations which were not behaving have gotten the boot
Belated, but perhaps useful:
Various organizations have collected temperature data from meteorological
organizations around the world. Some continue to do so. Some pass on
much of that data in more or less convenient manners. One such
organization is the US NCDC, a part of NOAA.
Among their collections of data is GHCN V2, a combination of previously
collected (i.e. mostly old) data from numerous sources, and relatively
recent data from some US locations, and some non-US locations that are
MCDW locations, i.e. many fewer locations than were among the collections
of old data. The apparent “drop” in the numbers of GHCN stations was
largely due to the large number of non-MCDW stations that were included
among the collections of old data.
See the paper:
Peterson, T.C., and R.S. Vose, 1997: An overview of the Global
Historical Climatology Network temperature database. Bulletin of the
American Meteorological Society, 78 (12), 2837-2849. (PDF Version)
linked at:
http://www.ncdc.noaa.gov/oa/climate/ghcn-monthly/index.php?name=temperature
for more information on GHCN V2 sources.
Non-US locations which are not MCDW locations are not included in the
GHCN process for gathering relatively recent data.
RE 108. YUP! and the link to the “visual” ftp site, the one with the folders has vanished.
Hmmm. Changes are a foot laddies. Also, GISS changed their monthly .txt output creating some minor annoyances for excell users of data. Adding text headers, adding additional headers
every 20 years… Silly boys. what a piece of code can add another peice of code can subtract
Off topic but of interest to the excell Wennies: Switch to Openoffice. you can extend it
with R!
re 108. Well, Menne’s paper is available but it was kinda milk toast
Sooooo, where is the science behind “man-made” global warming?
I am going to use this sequence of graphs by John Goetz for a presentation in a couple of days but don’t really understand the following ”
“In the case where multiple … records exist for a station in the given year, I assigned a blue dot if all records were fully calculated from existing averages, a red dot if at least one record was estimated, and no dot if none of the records could produce an estimate.“
Even if one of the multiple records at a location exists continuously without breaks, surely the station deserves a blue dot?
Tom Harris
tom.harris@sympatico.ca
3 Trackbacks
[…] questo link su ClimateAudit, il blog di Steve McIntyre, ci sono delle mappe molto interessanti sulla distribuzione delle stazioni di osservazione sul […]
[…] are both from comments in Climate Audit, where there is this telling map about the distribution of actual measurement stations across the […]
[…] are both from comments in Climate Audit, where there is this telling map about the distribution of actual measurement stations across the […]