Using Ipython Notebooks on external servers when behind restrictive firewalls

I recently held an introduction to Python for a group of colleagues (LINK), which was quite well received.
However, in testing out the setup (IPython notebooks running on a server in the cloud), I had trouble connecting to the server. This was because IPython notebooks use the nonstandard port 8080 for serving notebook traffic, and this port, along with all other ports outside the standard range, is blocked by the university's network.

Since the external server also runs an Apache web server, starting the notebook service with -port 80 gave an error. Rather than uninstall the web server with all the headaches that follow, I found a simple solution on Stackoverflow. (I've unfortunately lost the original source)

As root, run the command:
$ iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to 8080

This redirects all incoming traffic on port 80 (the standard for http traffic) to port 8080 (on which the ipython server resides), and allows users to connect via the IP directly.

When the session was over, I simply restarted the web server to remove the redirection.

On choosing a license for open scientific data

I (and my collaborators) am currently in the process of releasing an open-access and open-source data set. In doing so, I've had my first brush with the semi-logical process of selecting a license to stamp on that data. To make the process of selecting a license easier for others who may be in the same situation, I'll walk through my thought process when deciding which license to pick, and which elements to include in the license.
This is only a single example; any generalizations are drawn at the reader's own discretion. (Also, be wary of taking legal advice from a physicist!)

Using LFTP to bulk download from dongle-secured servers

For some recent data work, I had to download about 500 gigs worth of weather data from a server secured by a dongle authentication system. Due to some logistics of the server setup, I was unable to attain SSH access, and had to connect via FTP.

Using browser or FileZilla yielded download speeds of 10-20 kB/s - obviously too slow for that amount of data! Connecting via LFTP gave me good speeds, but the mirror command didn't work due to the dongle authentication system; 'mirror' tries to open a new connection for each file which doesn't work when the passwords are one-time use only. I had to come up with a solution which would allow me to download the data without having to type in 60,000 individual GET commands by hand, and achieved this using a quirk of the Linux terminal.

It's a bodge, but it works.

Authenticating Gurobi Academic Licenses on an external server

I've been using Gurobi to program optimization problems directly in Python, as they provide an excellent pythonic API, as well as free academic licenses with very little fuss.

In running a tutorial to teach my colleagues how to use Python and Gurobi, I wanted to write my tutorials in an iPython notebook and serve this up on a web server. This would skip all the mess that can occur with installation of Python + packages on Windows/Mac, waiting for the entire room to install everything, and getting Gurobi installed and running for everyone.

However, the server I'm running the iPython notebook server on is in the cloud, and thus not located at an academic IP. This causes Gurobi's validation program to fail.

The solution below solves this issue, and the equivalent problem of authenticating on a server without direct internet access.

Inverting a sort the easy way

Suppose you are working with an array:

> X = [10, 40, 20]

And you'd like to do some operations on a sorted version of X, then translate those operations back into X's original order.
The simple way to do this is using Numpy's argsort:

> s = np.argsort(X)
> invs = np.argsort(s)

For instance, to get the cumulative sum of X's elements, when summing up from lowest to highest, do:

> scX = X[s].cumsum()
> cX = scX[invs]
> print cX

[10, 70, 30]

Inverting dictionaries

A quick one today; I had a dictionary for a graph which keyed nodes to the community they belong to, and wanted to invert this dictionary to yield a list of the nodes for each community. The code below is the small helper function set up for this task.

Bivariate polynomial fitting

I recently had to convert some geographical data given as points on a map projection (x,y) to geographical coordinates (lon, lat). As the map projection was unknown, I resorted to fitting a polynomial to some selected points, and transforming the remainder of the data set using the fit. This can be achieved easily using linear least squares, but writing out terms in the bivariate expansion can quickly get tedious.

The code below implements the LLS algorithm for bivariate polynomials, along with a helper function to calculate the results of the fit.