Repository navigation
cluster
Three approaches are present -- kmeans clustering, gaussian mixtures, and decision trees. Programs are kmeans.py, gauss.py, and decision.py, respectively. Also present are rekmeans.py, regauss.py, and redecision.py. The former set run from raw inputs. The latter set run on the output of the first, enabling re-testing or testing with different parameters to the clustering algorithms without the slow process of matching up to external data sets.
does the bulk of the io work. 'read' (the primary function) expects inputs:
- an ascii file of brightness temperatures with their latitude-longitude location
- a netcdf file 'posteriori.nc' which has flags per Grumbine, 2009
- a grib file of sst for the day
- a grib file of ice concentrations for the day
as an sst filter is applied to the operational analysis, points which are warmer than 275.3 (the cutoff in Grumbine, 1997 et seq. are pre-emptively removed.
You may remove the sst filter in 'read' and set sst and flag to 0. They must be retained for indexing in ary, more below. But the values aren't used elsewhere in current versions of the programs.
read then augments the brightness temperature vector with 'delta ratios' -- difference in temperature between two channels divided by their sum. This is applied to all pairs of channels i,j where i != j.
At the end, we have **ary **-- the 12 brightness temperatures along with flag, sst, ice concentration, latitude, longitude and **dr **-- the 66 delta ratios augmented by the 12 brightness temperatures themselves.
'read2' is the same except that the delta ratio is (Ti^2 - Tj^2)/(Ti^2 + Tj^2)
It turned out to not lead to very different, much less to better, answers.
'reread' reads in **ary **from the output of decision, gauss, or kmeans and fills out the dr array
Each program reads in the data, executes its particular type of ML clustering, and writes out ary plus the cluster or leaf it is identified with. Each also evaluates the bayesian conditional P(ice | id) where id is the cluster identifier which shows how good the cluster is at being sea ice (or not-ice). And the P(id | ice), which identifies what fraction of ice winds up in this class. Ideally, these would both be 1 -- everything in the class is ice, and all ice is in the class. This does not happen in practice. But the P(ice | id) could be used in establishing quality flags. If it's very high or very low, there's high confidence in the classification (ice or ocean, respectively). If it's moderately high or moderately low, only moderate confidence. But for some users this may be enough.
decision.py
Runs sklearn's decision tree for requested tree depths. Maximum depth currently is 4, but that may be changed. The greater the depth, the more time it takes. But this is a modest term.
The maximum count is fairly arbitrary, limited mostly by machine memory. With a limit of 30 million, the memory requirement approaches 30 Gb. Runtime is approximately linear in number of data points
'thin' is the thinning applied. With thin = 16 (about the practical limit on my desk), every 16th line of the input is used. Run in the Antarctic for a year, it takes a little over an hour to find decision trees for depths 1,2,3, and 4.
kmeans.py
Run sklearn's kmeans clustering for requested (nlim) numbers of clusters. this program is structured to take all the data for each day on the nth of the month. No thinning, so it isn't possible to run all days of the year in one go.
gaussian.py
Run sklearn's gaussian mixture models for requested (nlim) numbers of clusters. Like kmeans, it is structured to take all the data for each day on the nth of the month. No thinning, so it isn't possible to run all days of the year in one go. This is markedly slower than kmeans and decision. For given numbers of clusters, it is also typically inferior to kmeans cluster identification.
rekmeans, regauss, redecision
Rerun the given algorithm on output from an initial run of it. Output files can be composited. And depth or number of cluster limits need not be the same as in the original runs.