OK, so I got interested in decoding the binary data sets at ftp://data.giss.nasa.gov/pub/gistemp/download/ as well. Wrote some Perl to slice and dice the data set into various series. I now have fully 1.6Gb less free hard drive space and I cannot figure out where my Sunday went 🙂
I’ll tidy up the various scripts and post on my web site when I get a chance. The result of my attempt at visualizing TSurf1200 and SSTHadR2 combined is available on Google Video.
Enjoy.
Sinan
Climate Audit 
45 Comments
Amazing. As you watch what jumps out at you?
It seemed to me that an incredible amount of the variation occurs in the very high latitudes especially in Siberia. Unravelling those data series seems to be key. I also looked to see if there was any obvious impact from the Great Depression since that should clearly have reduced all ongoing CO2 generation with the massive reduction in industrial output in the Western World. I did not see anything.
Is there anything else we should look for?
That is fun. I, too, did not see much in the 40s. I saw warming post 80s, but no real hockey stick.
I was interested if the video indicated a discernable temperature variance during the 1998 El Nià ±o. It went by very quickly. Visually it didn’t look like an anomalous year compared to all the others. The huge variability in the Polar Regions and Asia was clearly noticeable. Nice job Sinan.
Thank you for the kind words and thanks to Steve for posting this. I am really keeping my fingers crossed hoping that I have not made a gross error in putting together the pictures.
One thing that really bugs me is the fact that data contains “temperature anomalies” rather than actual levels. I like to start with raw data and apply whatever transformations I think are appropriate to the data myself.
If anyone knows of an equi-rectangular projection outline map of the world in the public domain, I would really appreciate it. I derived the map background used here from http://commons.wikimedia.org/wiki/Image:Equirectangular-projection.jpg and I think the solid areas of the map alter the colors used a little bit and obscure the locations where data are missing (large parts of South America, Africa and the poles) for extended periods of time.
Anyway, I am in the process of putting together a page explaining the process of decoding the data and generating the frames.
Sinan, the R-package “fields” has lots of interesting things in it relating to maps.
Re #4 & #5:
Check the R-packages maps, mapdata, and mapproj. There IS a world map in there but I’m not sure if it’s an equi-rectangular projection.
alternative names for equirectangular are “plate carrĂ© projection” and “geographic projection”
http://projections.mgis.psu.edu/
Re: #4
Sinan, I was happy to see it displayed as an anomaly. If the anomalies were based on an average temperature for the period and for a particular grid cell, what I visuallized was much variation (in anomaly) from month to month and over the entire period of years that was large compared to some global average that was slowly drifting up and down.
In other words, these old eyes could not pick out the “reddening” of the overall display, but could see the rapid “bluing” to “reddening” changes over short time periods.
Re: 5
I am downloaded the maps and mapproj packages and will look into that. Thanks for the tip.
As promised, here is the code and any other information I was able to put together: http://www.unur.com/climate/
Let me know if anyone has any comments either on this thread or using my contact form. I promise, I am not going to spam anyone 🙂
Sinan
Re: #6
Thank you John. I did download maps and mapproj and there is a rectangular projection. My first attempt, however, at using the generated map as a background image resulted in something that is offset quite a bit, so, I’ll take a look again.
Re: #7
Ken, I agree with you on the point you make about variability. But, it would have been nice to have the levels and then compute deviations from whatever level we wished, including but not limited to the hemisphere averages in a given month, deviations from a linear grid trend, deviations from a linear world trend etc etc.
Sinan, if you have some time, this would be fun:
You can download Jones’ gridded temperatures, version 1.0 (1991) from here:
http://cdiac.ornl.gov/ndps/ndp020.html
Take the difference between that and the appropriate (variance unadjusted, I think)
version of CRUTEM3
http://www.cru.uea.ac.uk/cru/data/temperature/
and produce the same video from that!
BTW, if someone is interested, Jones’ 1991 station data is also available from the above link!
Re: #11
Jean S., I am downloading those as I type. I don’t think I’ll get around to doing anything until next weekend but thanks for the pointer.
Sinan
I posted a new version of the video using a plainer map image for the background. I’ll stop tinkering now.
GISS Global Temperature Anomalies 1880 – 2006
Sinan,
Good job.
Here are a couple equi-rectangular cylindrical projections of the world. They are land-ocean masks made from one of the maps from NASA’s Blue Marble Project. For non-private use their images are free of license fees. They don’t actually have the images in this form. Their’s are color. Nice pictures.
I use the large one for plotting. At 3600×1800 it allows 1 pixel per tenth of a degree. Then I reduce the size.
You may prefer the smaller one 800×400. They are 100% black and 100% white, so if you wish to change the colors most graphics programs should be able to do a ‘change all matching color’. The grid is a light-gray.
Large map
Small map
Thanks for all this Sinan,
In 2000 or 2001 I found a 6MB gif animation on a NOAA/NCDC ftp site, showing ANNUAL mean T changes for GHCN global grid points, there is a pane for every year 1880 to 1998 or 1999.
Quite a few anomalously wrong looking grid points flick past during the runs.
If you go to;
http://www.warwickhughes.com/climate/movie.htm
in Internet Explorer (or Netscape), right click on the map and you can download the file. (movie_mntmp_annavg_web75.gif)
It will animate all day for me offline if opened in Netscape 7.1
If you left click on the map it links to the file but will not animate for me on the net.
Maybe some other readers can devise other ways to make the animation work.
Hello again:
Google Video was failing to process the new version of the GISS (with lighter, less cluttered background), so I switched to YouTube. The link to the GISS animation is at http://www.youtube.com/watch?v=eC8JC2b3fCw (with explanation at http://www.unur.com/climate/giss-1880-2006.html).
In addition, I had promised myself I would stay away from this for a while, but could not resist. So, here is the animation of global land temperature anomalies using CRUTEM3. The video should soon be available at http://www.youtube.com/watch?v=E6sMA-cBIhY. Code and explanation at http://www.unur.com/climate/crutem3-1850-2007.html.
While looking at the generated frames, I noticed a few odd discrepancies between the GISS and CRUT3EM versions. I am not sure what the explanation for this is and the fact that most frames look fairly similar in the areas which CRUTEM3 covers leads me to hope that the discrepancies are not due to errors on my part.
Re: #15
Warwick Hughes says:
First off, thanks for the kind words. The animation plays fine on Firefox 2.0.2 for me on Windows. Since animated GIFs are usually used for ads, there might be some ad-blocker or accessibility setting on your browser preventing non-local animations from looping. Dunno.
Re: #15
Warwick, I converted the video to AVI format. You can download it from http://www.unur.com/climate/mntmp_annavg.avi.
I will remove it in a couple of days.
Sinan
Jean #11
Just so I don’t make a total ass of myself can you explain the difference between the Jones data you referenced and the Jones data that Steve is after? The data set seems to be documented though I have not tried to unravel it all since there are no field headers that I can immediately see.
Sorry for imposing
I have uploaded the HADCRUT3 animation to YouTube as well. You can find it at: http://www.youtube.com/watch?v=mJgh8fIY-_Q. I used the same scripts for this, so there is not much interesting on my site.
I also set up a playlist for these and any future animations I might put together. It does not seem to be active yet, but the URL is http://www.youtube.com/view_play_list?p=A9288DB433D94F9B. This way, I can stop from cluttering the sidebar every time I play with some data.
I also noticed the following animation on YouTube: http://www.youtube.com/watch?v=jIIAyLmY5Yk which is purportedly based on some animation released by GISS. I also could not find anything resembling that on the GISTEMP web site. What I find odd is the fact that there are no observations from Antarctica in the GISS until the fifties.
Where is all the blue over the south pole region coming from?
#19
Looks like http://www.youtube.com/watch?v=jIIAyLmY5Yk is playing with a completely different data set. Who?, what? and why? are the questions that spring to mind.
Re: #19, 20
I found the source:
http://svs.gsfc.nasa.gov/vis/a000000/a003300/a003375/
They don’t specify where the data come from but Hansen is the lead scientist.
I am not claiming my color choices are best but I don’t … I really don’t know what to say. This animation makes it look like there is no missing temperature data for any part of the planet.
Sinan
Bernie (#1 above) notes the high variability in the high latitudes in this animation.
Now, if I recall correctly, the CGMs are wrong on polar regions – mayby not surprising, since data is sparse; wrong on lower troposphere temp rises; the only correct predictions are for higher latitudes.
(Help me out here: what else am I missing in CGM success/failure?)
Could high variability and CGM “success” on higher latitudes be more a matter of chance than parsing out underlying causation?Anyone with other alternate ideas?
Re: #18
Bernie, IIUC, Steve is trying to find out the stations used in Jones’ UHI study using Chinese data.
Hope this helps.
Sinan
Sinan:
Got it – you mean the rural stations with populations about the size of Cambridge, Massachusetts!!!
Jones (1990): I have uploaded an animation of the data from this paper of global temperature anomalies from Jan 1851 – Dec 1990 to YouTube. Here is the link: http://www.youtube.com/watch?v=MKep0C7Aybk.
Since grid coordinates and sizes differ between Jones (1990) and CRUTEM3, I am having a bit of difficulty coming up with a good way of presenting the differences between the two data sets. I subtracted some random frames from Jones from the corresponding frames from CRUTEM3. If there is no substantial difference, the resulting frame should be mostly black. Yet, the subtracted frames show a lot of colors. However, I would like to come up with a more meaningful depiction of differences than that.
So, Jean S., and any others, who have suggestions, please let me know. FYI, I am working with transformed data sets which have one line per gridbox per year per month. I.e., the files look like this:
and
As a randomly selected example, here is what happens when I subtract the Jan 1970 CRUTEM3 frame from the same frame using Jones’ data:
1970-01-jones-minus-crutem3.png
I don’t know if image embedding works or how to do it so here is a link just in case:
Jones – CRUTEM3 (January 1970)
Visually, the two frames side-by-side look very similar, so I would like to first get proper numerical differences and colorize them based on which data set is warmer at that point (and by how much).
Thanks Sinan! This is exactly what I’d like to see the visualization for… how much “adjustments” to the old data has been done since 1991.
Since CRUTEM3 is supposed to be just an average of the selected station anomalities within each grid cell, can’t you just take an average of two adjacent 5×5 cells in order to get the 5×10 gridding as in 1991 version? Also, I would use some other color, maybe white, for representing “no change”, and maybe black for “no value”.
Re: 26
Jean S.: I’ll do something along the lines you suggest. However, note that Jones’ data are centered on coordinates divisible by 5 whereas the CRUTEM3 grids have their corners at such points. I need to sit and draw some boxes on paper.
Sinan
Re: 26
Jean S., here is what the grids look like superimposed (green=Jones, red=CRUTEM3):
Given that stations are not uniformly distributed, my timid mind does not see a straightforward way to compare anomalies in Jones’ grid versus the CRUTEM3 grid.
Right now, I am be more inclined to re-grid Jones’ data using jonesnh.dat and jonessh.dat from the web site. I’ll see if I can do that.
Sinan
Re 17 Sinan, I wondered how large that avi file is.
Could you try emailing it to inward@warwickhughes.com if you have time, anytime.
When I go to your link with IE6, it opens Windows Media Player and I can not download. I have tried unblocking all popup blockers.
Re your 28, I love your digression into these historic files of Jones.
I seem to recall that his Southern Hemisphere station files were not adjusted in that version given to CDIAC.
re #28: That’s nasty, I think there is no “nice” solution for this. Does anyone know why they changed the grid locations?
re #29: Warwick, are you saying that the SH data file (jonessh.dat) contains “raw” (=no adjustments done by Jones) data?
Re: #31
Here is what the documentation says:
All stations anomalies from Jones (1990)
The animation should be available on YouTube at http://www.youtube.com/watch?v=DmYAVfFYlgM soon.
For this animation, I manually calculated the station averages for all stations (except for a few which did not have series past the fifties) for each month over the period 1951-1970 and subtracted that from the monthly average for that station. It turns out that Jones (1990) has a few stations with series going back to the 18th century, so included those years in this animation as well.
The blocks in the animation corresponding to 1-degree square areas around the station coordinates (this is done for visibility only). I put up a temporary page comparing Jan 1933 in CRUTEM3 (gridded), Jones, 1990 (gridded) and Jones, 1990 (all station anomalies).
There is nothing shocking in the animation one way or another but this was the first step in re-gridding Jones, 1990 data so as to be able to compare it with CRUTEM3.
Sinan
Re 31, I note that 32 indicates the Jones SH stations were uncorrected in that file, as I said.
I assume they were corrected before gridding.
Does it show a surprising casual attitude to record keeping ?
Why would CDIAC have not reminded PDJ to update the SH station file with corrected stations, bearing in mind the DoE has paid millions for Jones/CRU services over the years ?
The corrections are all listed in the DoE documentation book, TR027 but what a job to pull them out.
All station anomalies (base=1950-1980) from Jones (1990)
Everything else is pretty much the same except that the animation starts in Jan 1850 (not much happening before then). The main difference, of course, is using the same base period 1950-1980 for calculating ‘normals’ as CRUTEM3.
I am going to do the re-gridding to 5×5 grid boxes later.
Sinan
Ahem, forgot the YouTube link for that last one:
http://www.youtube.com/watch?v=O7cqGHKhozw
I am getting miffed. Here are some odd things I discovered while looking at jonesnh.dat and jonessh.dat:
724070 -999 -1999 -999 ATLANTIC CITY USA 1 1874 1991 20 1884
The first series of digits is station id, the second is latitude and the third is the longitude. Why are the coordinates missing?
718680 -999 -1999 -999 HUDSON BAY SASK CANADA 1 1943 1991 10 1944
Ditto.
719660 -999 -1999 -999 DAWSON YUK CANADA 1 1900 1991 10 1901
Ditto.
724390 -999 -1999 -999 SPRINGFIELD ILL USA 1 1879 1991 20 1879
Well, we know why Springfield is missing.
727931 -999 -1999 -999 SEATTLE USA 1 1890 1991 10 1941
These are the ones I have been able to detect so far. There might be others.
Now, some pictures for these stations (no comment on the meaning of this, I just found out about this coordinate issue as I was trying to re-grid Jones, 1990, and I was curious how the data looked for stations whose coordinates were missing).
The last graph above should have been:
Sorry for the mess.
Sinan
Atlantic City and Seattle are two interesting cities. Both have had recent building booms, with Seattle becoming more of megalopolis – extending out to Bellevue, Redmond, Kirkland and down through Tacoma (the traffic is awful). Wasn’t Atlantic City redeveloped in the 80s. Obviously I’m thinking of changes in the UHI profiles for these two locations. Now I think about it, has anyone looked at the profiles of known high growth cities to see how these changes are reflected in the anomalies?
Sinan:
Your animations are really intriguing. I had one thought. I was looking at Statistics Canada site and I came across a somewhat similar chart of changes in population. If you look, perhaps some ideas of how to marry the two might pop into your head. I was looking at the growth in the Artic settlement areas since this could have an impact on the readings – especilaly if we are talking anomalies. These Arctic settlements in Nanvut seem to be growing fairly rapidly in the last 15 to 20 years.
Map
It is a pretty big file though.
Re: 40
Bernie, that is a link to an executable file. My guess is you went through some form submissions to get to what you wanted me to see but I cannot see that because I don’t know what steps are needed. Just trying to open the URL results in the web browser wanting to download the executable.
Sinan
THE USUAL BULLSH*T
Well, NOAA is at it again, claiming that this most recent winter was “the warmest since 1880“.
This highlights the difference between the HadCRUT3 and the GISS dataset. Here’s the HadCRUT3 data, including errors:
A couple of things of interest about this graph:
1) Error estimates are crucial in any discussion. The constant neglecting of error estimates by the AGW folks is starting to get irritating. Including the error estimates shows that there’s not much we can say about last winter at all.
2) According to HadCRUT3, the winter of 1997-1998 was warmer than the most recent winter.
Here’s the difference in the HadCRUT and GISS winter temps:
Note that the difference is as high as 0.4°C at times, and that it increases almost linearly since 1965.
It is problems like this that makes climate “science” no more than a guessing game.
w.
Sinan: Sorry about that. Try this.
Map
I assume that the US might have a similar population display.
Bernie
MORE OF THE USUAL BOVINE EXCREMENT …
… from the usual suspects. NOAA says this is the warmest winter on record. But that’s the GISS dataset … what does HadCRUT3 say?
Well, it says the following:
A couple of notes:
1) As usual, they’re different, with GISS being the more photogenic for the media. HadCRUT3 says that the winter of 97/98 was warmer than 06/07.
2) As usual, NOAA has managed to forget the error bars … probably just got overlooked in the shuffle. Including errors, we can’t even say if last winter was warmer than any other winter in the last decade, much less in the record … this is not science, this is unadulterated hype.
How different are GISS and HadCRUT3 in this regard? Here’s the year by year comparison:
Note that the difference is as much as four tenths of a degree. Also note how the good GISS folks have tweaked the data so that it rises vis-a-vis HadCRUT3 from about 1965 onwards.
Ahhh, the sad state of climate “science” … my wife just asked me why I was sighing … because it beats weeping for the death of science.
w.
RE #43 It’s remarkable that the RSS satellite-derived temperatures do not support the claim that 2006-2007 winter was a record. In fact, the winter of 2006-2007 looks rather vanilla.