Data Download Script (Perl)
Script
Purpose
A generic data download script that can be used to download data files from Earthdata Login enabled servers. Data files are identified using URLs, and may be provided on the command line or in a file. User credentials are required for authentication, and the applications from which the files are being downloaded must have been pre-authorized.
If a file is returned from the server with a 'Content-Disposition' header, the filename given in that header will be used as the name of the downloaded file, otherwise the filename in the URL will be used.
Options
| Option | Description |
|---|---|
-h / --help |
Print out the command syntax |
-u <uid> |
The Earthdata Login username to use for downloading the URLs. |
-n / --netrc |
Pull Earthdata Login user credentials from user's .netrc file. |
-i <file> |
The name of a file that contains URLs to download |
-d <directory> |
Save downloaded files into directory. If the directory does not exist, the script will attempt to create it. |
-q / --quiet |
Quiet mode - will suppress error messages about invalid URLs. |
-v / --verbose |
Verbose mode - will output additional debugging information about the progress during operation. |
Examples
Download the files whos URLs are given in the file 'urllist' to the current directory using the user profile 'peter.l.smith'. Prompt for the user password. Verbose (debugging) output will be generated.
get_data.pl -u peter.l.smith -i urllist -v
Download the file given by the URL on the command line using the user profile 'peter.l.smith' and save it into the directory 'data'. Prompt for the user password. Note that you need send your request via HTTPS or your passwords will be sent in the clear. Make sure your links are modified from the metadata. All EOSDIS DAACs should support HTTPS even if the metadata hasn't been updated.
get_data.pl -u peter.l.smith -d data https://e4ftl01.cr.usgs.gov/ASTT/AST_L1T.003/2016.09.11/AST_L1T_00309112016020239_20160912100405_5264_QA.txt
Download the files whos URLs are given in the file 'urllist' and save into the directory 'data'. Earthdata Login credentials will be retrieved from the .netrc file.
get_data.pl -n -d data -i urllist