Hoffman2:Software Tools: NDATools: Difference between revisions
Jump to navigation
Jump to search
(Created page with "[https://github.com/NDAR/nda-tools | For more information] NDATools is for downloadig or uploading data to NDA. Here is how to use it in Hoffman ==Load modules== module lo...") |
(No difference)
|
Revision as of 23:51, 8 March 2021
NDATools is for downloadig or uploading data to NDA. Here is how to use it in Hoffman
Load modules
module load python/3.6.1_shared module load ndatools
Generate Temporary Tokens
To access NDA data in AWS S3, you need temporary token generated by your NDA user credential
generate_token.sh ‘USERID’ ‘PASSWORD’ 'https://nda.nih.gov/DataManager/dataManager'
Then you'll get something like the following
Beginning token request... Access Key: AAAAAAAAAAAAAAA Secret Key: SSSSSSSSSSSSSSSSSSSSSSSS Session Token: ABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCD Expiration: 2021-03-04T12:55:00Z
Download Data
Use command downloadcmd to download data
To check the usage of command downloadcmd
downloadcmd -h
To start downloading the package, you'll need the packageID. The packageID can be found in NDA's website after you login and submit your request for data access.
downloadcmd <packageID> -dp -d /u/project/MYGROUP/MYNDADATA_FOLDER
If the download got interrupted and you want to resume the download
downloadcmd <packageID> -dp -d /u/project/MYGROUP/MYNDADATA_FOLDER -r /u/project/MYGROUP/MYNDADATA_FOLDER