Using LFTP to bulk download from dongle-secured servers

For some recent data work, I had to download about 500 gigs worth of weather data from a server secured by a dongle authentication system. Due to some logistics of the server setup, I was unable to attain SSH access, and had to connect via FTP.

Using browser or FileZilla yielded download speeds of 10-20 kB/s - obviously too slow for that amount of data! Connecting via LFTP gave me good speeds, but the mirror command didn't work due to the dongle authentication system; 'mirror' tries to open a new connection for each file which doesn't work when the passwords are one-time use only. I had to come up with a solution which would allow me to download the data without having to type in 60,000 individual GET commands by hand, and achieved this using a quirk of the Linux terminal.

It's a bodge, but it works.

Open a terminal and navigate to the directory below the one you want to download your data to (for this example, I want to put my data in ~/temp/data):

$ cd ~/temp

$ mkdir data

Use lftp to login to the FTP server, and navigate to the folder containing the data (For this example, I'll be using a publically available FTP, if the server is dongle-secured, use a password from the dongle):

$ lftp ftp.somewhere.com -u username,password

Let's assume the data is in /pub/CFCA/, use the find command and output to a local file:

$ cd pub

$ find CFCA >> dir.txt

Open up ~/temp/dir.txt in your favorite text editor, it will have lines like:

CFCA/
CFCA/123456720151013124049.txt
CFCA/123456720151013124209.txt

 Now, remove all lines which end in a "/", and add "get " to the front of the remaining lines:

get CFCA/123456720151013124049.txt
get CFCA/123456720151013124209.txt

In the terminal window, make sure you are in the folder you want the files to go to:
$ lcd data

Select all lines in the file and middle-click on the terminal window - your downloads should be going. Remember to press enter after the last download in order to get the last file.

Preserving directory structure

The above commands will download all files directly to the data directory. If you want to preserve the directory structure, a bit more work is needed.

Firstly, after find'ing to dir.txt, copy all lines ending in a "/" to a separate file, and add "mkdir " to the front of each:

mkdir CFCA/

 Use another terminal, navigate to the data directory ($ cd ~/temp/data/), select all these commands and middle-click them in there. This will build the directory structure on your local machine.

In ~/temp/dir.txt again delete all lines ending with a "/" and format the lines as follows:

get CFCA/123456720151013124049.txt -o CFCA/123456720151013124049.txt
get CFCA/123456720151013124209.txt -o  CFCA/123456720151013124209.txt

I.e. add "get " to the front, duplicate each file name, and add " -o " between them. Middle-clicking this list of files into the lftp window (after using $ lcd data) will copy all the files over to the correct directory.

And to think all this could be avoided if lftp's mirror had a --serial switch...