Download and Organize Example Classifications¶
These are preliminary classifications that demonstrate how to organize your own classifications or use the ones posted on svrimg.org¶
By default, the function "get_pred_tables" will download the example classifications. You just have to provide a directory in which to save it.¶
In [1]:
from svrimg.utils.get_tables import get_pred_tables
egs = get_pred_tables(data_dir="../data/csvs/", remove_first_row=True)
egs.head()
Out[1]:
You can also organize your local predictions in yearly csv files. Look at the documentation to see how it works.¶
In [2]:
help(get_pred_tables)
The following example saves a combined csv in "../data/csvs/" named "eg_classes_96-17.csv" and will search that directory for your csvs with the filename format "*table*", which basically says it will accept any value before or after "table", or, in other words, any csv file with that format. This will work for all files that were downloaded and unmodified from svrimg.org. Additionally, the first row is removed if remove_first_row is set to True. This should always be the case for svrimg.org tables, because the first row is "example data".¶
E.g., 1997_table_1598590650584.csv
In [3]:
egs = get_pred_tables(data_dir="../data/csvs/", example=False, remove_first_row=True)
egs.head()
Out[3]:
You can then use other helper functions to plot representative means for each classification subset. The image shown is the probability matched mean for each subset.¶
In [4]:
from svrimg.utils.get_images import get_img_list
from svrimg.utils.map_helper import draw_box_plot
from svrimg.utils.pmmean import _run_pmm_one_variable
import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = 25, 25
plt.rcParams['xtick.labelsize'] = 20
plt.rcParams['ytick.labelsize'] = 20
plt.rcParams['axes.labelsize'] = 20
i = [8, 7, 5, 4, 2, 1]
for cls in ['Cellular', 'QLCS', 'Tropical', 'Other', 'Noise', 'Missing']:
sub = egs[egs["Class Name"]==cls].copy()
imgs = get_img_list(sub.index.values, "../data/tor/")
img = _run_pmm_one_variable(imgs)
ax = plt.subplot(4, 3, i.pop())
ax = draw_box_plot(ax, img, cbar_shrink=0.8)
ax.set_title(cls, fontsize=20)
ax.text(0, 130, "n={}".format(len(imgs)), fontsize=20)
plt.tight_layout()
In [5]:
from svrimg.utils.get_tables import get_svrgis_table
svrgis = get_svrgis_table(data_dir="../data/csvs/")
egs_join = egs.join(svrgis)
egs_join.head()
Out[5]:
Now you can query the classifications by year.¶
In [6]:
plt.rcParams['figure.figsize'] = 40, 40
plt.rcParams['xtick.labelsize'] = 20
plt.rcParams['ytick.labelsize'] = 20
plt.rcParams['axes.labelsize'] = 20
i = 1
for subset in [(1996, 2011), (2012, 2013), (2014, 2017)]:
sub_ = egs_join[(egs_join.yr >= subset[0]) & (egs_join.yr <= subset[1])].copy()
for cls in ['Cellular', 'QLCS', 'Tropical', 'Other', 'Noise', 'Missing']:
class_ = sub_[sub_["Class Name"]==cls].copy()
imgs = get_img_list(class_.index.values, "../data/tor/")
img = _run_pmm_one_variable(imgs)
ax = plt.subplot(6, 6, i)
ax = draw_box_plot(ax, img, cbar_shrink=0.8)
ax.set_title(cls + "\n{}-{}".format(subset[0], subset[1]), fontsize=30)
ax.text(0, 130, "n={}".format(len(imgs)), fontsize=35)
i += 1
plt.tight_layout()
Save the image data and classification data in pickles¶
In [7]:
import numpy as np
import pickle
for subset, name in zip([(1996, 2011), (2012, 2013), (2014, 2017)], ("train", "validation", "test")):
sub_ = egs_join[(egs_join.yr >= subset[0]) & (egs_join.yr <= subset[1])].copy()
imgs = get_img_list(sub_.index.values, "../data/tor/")
imgs = np.expand_dims(imgs, axis=3)
classes = sub_['Class Code'].values
## Assumes this folder exists
pickle.dump([imgs, classes], open("../data/pkls/{}_{}_{}.pkl".format(subset[0], subset[1], name), "wb"))
Read in the image data to be ready for Keras¶
In [8]:
x_train, y_train = pickle.load(open("../data/pkls/1996_2011_train.pkl", "rb"))
x_val, y_val = pickle.load(open("../data/pkls/2012_2013_validation.pkl", "rb"))
x_test, y_test = pickle.load(open("../data/pkls/2014_2017_test.pkl", "rb"))