Python tools used at the SMC for data processing

Roger Veciana i Rovira

@rveciana

GeoExamples

Python: What is it

An interpreted language
Dynamic types / Strong types
Object oriented
Multi-purpose
Thought to be fast (to learn, to use, to understand)
Thought to create a clean an uniform code

Python: Some advantages

Allows to address simple problems in a simple way, but also complex problems
- Script like coding
- Object oriented programming with exceptions, etc
Big standard library: "batteries included"
Lots of libraries and frameworks that extend it
- Web creation
- System administration
- Scientific and geographic libraries
- Parallel programming
Python glues libraries coded in different languages.
Software that allows using it interactively: ipython, SAGE

Python basic instructions

a = 3
b = 4
a + b

Conditions

if b > a:
    print "b is bigger than a"

Iterations

for i in range(10):
    print "Iteration number " + str(i)

NumPy for matrix calculations

Open the ipython console

import numpy
a = numpy.ones((20,10)) -> 20x10 array filled with 1
a * 23 -> 20x10 array filled with 23
a + 1 -> 20x10 array filled with 2
a.sum() -> 200
a[3:6,4:8] = 2 -> Fills the selected cells with 2
a[[2,3],[4,4]] -> Returns the selected cells

Try

Other functions: max(), mean() (try to pass axis=1 as an argument), reshape(), numpy.gradient(a), numpy.arange()
Other array indexation: a[b], a[b,c]
Logical operations: a>2

GDAL/OGR

Geospatial Data Abstraction Library

Abstract data model for reading raster and vectorial GIS formats
More than 150 supported formats:
- GeoTIFF
- ENVI
- Shapefile
- PostGIS
- GRIB
- NetCDF
Used by many applications and written in C, has Python bindings
Reading/Writting/Coordinate changes/GIS Operations

VAISALA IRIS format

Proprietary format made by SIGMET/VAISALA for weather radars
Difficult to get the data
GDAL driver made by the SMC

Basemap

Based on Matplotlib
Ready to use
Many projections
Adapted to meteorology
Integrated with numpy (and GDAL)
You can check the tutorial http://basemaptutorial.readthedocs.org/en/latest/

Basemap code example

from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt

map = Basemap()
 
map.drawcoastlines()
 
plt.show()

Basemap examples

Scipy

Extends NumPy
Functions as in MATLAB, IDL, Octave, R-Lab and SciLab
Algebraic functions
Statistical functions
Interpolation functions
Clustering functions

Scipy example

from scipy.interpolate import interp1d
import matplotlib.pyplot as plt

x = np.linspace(0, 10, 10)
y = np.cos(-x**2/8.0)
f = interp1d(x, y)
f2 = interp1d(x, y, kind='cubic')

xnew = np.linspace(0, 10, 40)

plt.plot(x,y,'o',xnew,f(xnew),'-', xnew, f2(xnew),'--')
plt.legend(['data', 'linear', 'cubic'], loc='best')
plt.show()

Scikit-learn

Machine learning algorithms
Integrated with NumPy
Classification
Regression
Clustering
Normalization

Interpolating station data

SMC has about 160 automatic weather stations
Temperature depends on the altitude and the distance from the sea
Scikit-learn multilinear regression (or scipy)
Residues calculation
Final result with NumPy

Interpolating station data: results

MOS

Model Output Statistics

Input: WRF model output + station data (years)

Scikit-learn multilinear regression / Scikit-learn classification

Forecast at station locations (interpolable)

Viamet

Roadcasting
METRo model
Virtual points
Thermal mapping

More information