Using LFTP to bulk download from dongle-secured servers

For some recent data work, I had to download about 500 gigs worth of weather data from a server secured by a dongle authentication system. Due to some logistics of the server setup, I was unable to attain SSH access, and had to connect via FTP.

Using browser or FileZilla yielded download speeds of 10-20 kB/s - obviously too slow for that amount of data! Connecting via LFTP gave me good speeds, but the mirror command didn't work due to the dongle authentication system; 'mirror' tries to open a new connection for each file which doesn't work when the passwords are one-time use only. I had to come up with a solution which would allow me to download the data without having to type in 60,000 individual GET commands by hand, and achieved this using a quirk of the Linux terminal.

It's a bodge, but it works.

Authenticating Gurobi Academic Licenses on an external server

I've been using Gurobi to program optimization problems directly in Python, as they provide an excellent pythonic API, as well as free academic licenses with very little fuss.

In running a tutorial to teach my colleagues how to use Python and Gurobi, I wanted to write my tutorials in an iPython notebook and serve this up on a web server. This would skip all the mess that can occur with installation of Python + packages on Windows/Mac, waiting for the entire room to install everything, and getting Gurobi installed and running for everyone.

However, the server I'm running the iPython notebook server on is in the cloud, and thus not located at an academic IP. This causes Gurobi's validation program to fail.

The solution below solves this issue, and the equivalent problem of authenticating on a server without direct internet access.