Plotting Geo Data using Basemap

Matplotlib has an interesting toolkit called Basemap for plotting geo data and maps. Basemap handles all the possible map projections and uses Matplotlib to do the plotting. Basemap also includes data like countries, coastlines and background satellite images from NASA Blue Marble.

Plotting geo data using Basemap is actually very simple. Here is an example with geo data from Twitter using the script in my previous post.

from mpl_toolkits.basemap import Basemap
from pylab import *
import twitter_geo

# setup Lambert Conformal basemap.
m = Basemap(width=400000,height=400000,projection='lcc',resolution='f',lat_0=55.5,lon_0=13)
m.bluemarble() #background
m.drawcoastlines() #coastlines

# get some geo data from Twitter
geo = '55.583333,13.033333,100km' #Malmo, Sweden
tweets = twitter_geo.parse(twitter_geo.search(geo=geo))

for t in tweets:
    coords = t[2]['coordinates']
    x,y = m(coords[1],coords[0]) #project to map coords
    plot(x,y,'ro')
    
show() #you might not need this

This example takes tweets in a 100km radius around Malmö, Sweden and plots the location with red dots on top of a NASA Blue Marble image with coastlines drawn. The result looks like this:

Pretty simple. If you had some dense data like weather or topographic data, that could be placed on the map (e.g. using Matplotlib's contour()) and opens up interesting possibilities for data visualization.

Geo Data from Twitter using Python

I wanted to write an example of plotting geo data with maps but I didn't have any good example data. Having played around with Twitter's API before, I decided to use that. There are several Python libraries for interfacing with Twitter but to use the Twitter Search API you actually don't need any of them. Since the result is returned in JSON you just need something like simplejson (or the built in JSON modules in later versions of Python).

Here is a simple example (twitter_geo.py) that searches for tweets with a query string and/or a geographic location. Pretty simple.

import urllib2
import simplejson as json

def search(q='',geo=''):
    """ use search api to find tweets matching a query string and/or 
        location as string of "lat,lon,radius", see api documentation"""
    query = 'http://search.twitter.com/search?q=%s&format=json&rpp=100&result_type=recent&geocode=%s' % (q,geo)
    f = urllib2.urlopen(query)
    r = json.loads(f.read())
    return r["results"]

def parse(res):
    """ take the relevant parts of the result"""
    #list of tuples (user,time,geo)
    return [(r['from_user'],r['created_at'],r['geo']) 
            for r in res if r['geo'] != None]


if __name__ == "__main__":
    tag = '%23fail' 
    geo = '55.583333,13.033333,100km' #Malmo, Sweden
    
    #query Search API
    print parse(search(q=tag,geo=geo))

Installing Numpy and Matplotlib on Mac OS X 10.6 Snow Leopard

This is partly an update to my previous post on installing for Leopard, partly a note to myself. After upgrading to Snow Leopard (and 64bit) I could still use my MacPort installs but not upgrade or install new ones (since all previous builds were for 32bit). So, time for another fresh install.

With MacPorts:

sudo port install python26
sudo port install py26-numpy
sudo port install py26-matplotlib

After this I can start python and load pylab but I get problems with the plotting window totally missing. My colleague Jerome Piovano found a fix over at stackoverflow. Replace the last line with

sudo port install py26-matplotlib +cairo+gtk2

Then edit ~/.matplotlib/matplotlibrc (it should be in your home directory, if no such file exists just make a new file) by adding:

backend: MacOSX

Now everything works as it should.

Computational Mathematics with Python

Posting has been a little slow the last months. I have been busy at Polar Rose and in parallel I've been teaching a course in computational mathematics with python. I hope to increase posting soon as I have a couple of unfinished posts and examples I'd like to write about.

If you are interested in numerical analysis or how to program mathematics using python, check out the course page. Lecture slides and course material is available here.

Computer Vision and the New iPhone SDK

A look at the newly released iPhone SDK for iPhone OS 4 shows some very interesting new features for people building computer vision and augmented reality applications.

Video playback & Capture
"You now have full programmatic control over video playback and capture, using new APIs in the AV Foundation framework." This is the one AR people have been waiting for. No more hacks to get the video feed. Should give more video based apps in the open and better video performance.

Photo Library Access
"Applications now have direct access to user photos and videos with the Media Library APIs." Also very useful. Build auto-tagging tools or photo browsers for your on-phone photos! (Cooliris, are you watching this? You finally have access.)

And finally...

Accelerate
"Accelerate provides hundreds of mathematical functions optimized for iPhone and iPod touch, including signal-processing routines, fast Fourier transforms, basic vector and matrix operations, and industry-standard functions for factoring matrices and solving systems of linear equations." Will need to try it to find out how well it works but this is potentially extremely useful. Some basic optimized linear algebra will take computer vision or AR far on this device! Is it using OpenCL? I don't know. Do you?

sparray - sparse n-dimensional arrays in Python

Yesterday I wrote about sparse arrays for Python using dictionaries. I couldn't get it out of my head so tonight I actually made a complete module containing a class for sparse n-dimensional arrays. The module supports any number of dimensions and any size. Sparray has all basic operations like adding, subtracting, multiplication, division, mod, pow, printing, sum() and can easily be converted to full NumPy arrays.

The sparray module can be downloaded from here. Running sparray.py will show a series of examples and print the results in the console.

Enjoy. Bug reports and feedback welcome!

Below is some output from the example:

adding...
[[  3.   3.   3.]
 [  3.  13.   3.]
 [  3.   3.   3.]]
subtracting...
[[-3. -3. -3.]
 [-3.  7. -3.]
 [-3. -3. -3.]]
multiplication...
[[  0.   0.   0.]
 [  0.  30.   0.]
 [  0.   0.   0.]]
[[ 9.  9.  9.]
 [ 9.  9.  9.]
 [ 9.  9.  9.]]
division...
[[ 0.  0.  0.]
 [ 0.  3.  0.]
 [ 0.  0.  0.]]
mod...
[[ 0.  0.  0.]
 [ 0.  1.  0.]
 [ 0.  0.  0.]]
power...
[[    0.     0.     0.]
 [    0.  1000.     0.]
 [    0.     0.     0.]]
iadd...
[[  3.   3.   3.]
 [  3.  13.   3.]
 [  3.   3.   3.]]
sum of elements...
74
mix with NumPy arrays...
[[  6.   6.   6.]
 [  6.  26.   6.]
 [  6.   6.   6.]]
Frobenius norm...
601.0

Sparse arrays in Python using dictionaries

UPDATE: I made a complete module for sparse n-dimensional array. See the next post.

A great application of Python's dictionary type is to use it for representing sparse arrays and matrices (in arbitrary dimensions). Consider the following code example:

a = {}
a[0,3] = 12
a[2,1] = 5
a[3,3] = 7

Now, this dictionary looks like this:

{(0, 3): 12, (3, 3): 7, (2, 1): 5}

It is easy to define operations on these dictionaries. For example, the Euclidean (L2) distance between two arrays (Frobenius norm in the case of matrices) is simply:

def L2(a1,a2):
    ndx = set(a1.keys()+a2.keys())
    return sum([ (a1.get(i,0)-a2.get(i,0))**2 for i in ndx])

Here we used set() to remove duplicates after concatenating the keys and the dictionary get() method that returns a default value (in this case zero) if a key is missing.

With another example array like this:

b = {}
b[0,3] = 10
b[1,1] = 15
b[1,3] = 4

the value of L2(a,b) is 319.

There are some interesting efforts for sparse multidimensional arrays in Python out there. Ndsparse is a new one, which uses lists instead of dictionaries and looks very nice and clean. Unfortunately I found lots of basic things broken when testing today. Hope to see a new release with bugs fixed soon! There is also the SciPy sparse module with data structures for 2D sparse matrices but that looks a little messy (and is only 2D).