Hoffman2:Software Tools: NDATools - Revision history

Hwang: /* Start an interactive mode */

2024-03-28T22:03:39Z

Start an interactive mode

← Older revision		Revision as of 22:03, 28 March 2024
Line 9:		Line 9:
	qrsh -l h_rt=20:00:00,h_data=4G -pe shared 2		qrsh -l h_rt=20:00:00,h_data=4G -pe shared 2

	==Use conda env==		==Use conda env==

	module load anaconda3/2023.03		module load anaconda3/2023.03

Hwang: /* Use conda env */

2024-03-28T22:03:19Z

Use conda env

← Older revision		Revision as of 22:03, 28 March 2024
Line 9:		Line 9:
	qrsh -l h_rt=20:00:00,h_data=4G -pe shared 2		qrsh -l h_rt=20:00:00,h_data=4G -pe shared 2

	==Use conda env==		==Use conda env==

	module load anaconda3/2023.03		module load anaconda3/2023.03
	conda activate /u/project/CCN/apps/nda-tools/0.2.21		conda activate /u/project/CCN/apps/nda-tools/0.2.21

	~~'''~~==Generate Temporary Tokens==		<!--
			==Generate Temporary Tokens==

	To access NDA data in AWS S3, you need temporary token generated by your NDA user credential.		To access NDA data in AWS S3, you need temporary token generated by your NDA user credential.
Line 29:		Line 30:
	ABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRST		ABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRST
	Expiration: 2021-03-04T12:55:00Z		Expiration: 2021-03-04T12:55:00Z
	~~'''~~		-->

	==Download Data==		==Download Data==

Hwang: /* Generate Temporary Tokens */

2024-03-28T21:59:45Z

Generate Temporary Tokens

← Older revision		Revision as of 21:59, 28 March 2024
Line 14:		Line 14:
	conda activate /u/project/CCN/apps/nda-tools/0.2.21		conda activate /u/project/CCN/apps/nda-tools/0.2.21

	==Generate Temporary Tokens==		'''==Generate Temporary Tokens==

	To access NDA data in AWS S3, you need temporary token generated by your NDA user credential.		To access NDA data in AWS S3, you need temporary token generated by your NDA user credential.
Line 29:		Line 29:
	ABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRST		ABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRST
	Expiration: 2021-03-04T12:55:00Z		Expiration: 2021-03-04T12:55:00Z
			'''

	==Download Data==		==Download Data==

Hwang: /* Load modules */

2024-03-28T21:59:03Z

Load modules

← Older revision		Revision as of 21:59, 28 March 2024
Line 9:		Line 9:
	qrsh -l h_rt=20:00:00,h_data=4G -pe shared 2		qrsh -l h_rt=20:00:00,h_data=4G -pe shared 2

	==~~Load modules~~==		==Use conda env==

	module load ~~python~~/3.~~9.6~~		module load anaconda3/2023.03
	~~module load ndatools~~		conda activate /u/project/CCN/apps/nda-tools/0.2.21

	==Generate Temporary Tokens==		==Generate Temporary Tokens==

Hwang: /* Download Data */

2022-12-07T20:42:08Z

Download Data

Hwang: /* Download Data */

2022-12-07T20:40:31Z

Download Data

← Older revision		Revision as of 20:40, 7 December 2022
Line 59:		Line 59:
	Number of worker threads		Number of worker threads
	--file-regex <regular expression>		--file-regex <regular expression>
	Option can be used to download only a subset of the files in a package. This command line arg can be used with		Option can be used to download only a subset of the files in a package.
	the -ds, -dp or -t flags.		This command line arg can be used with the -ds, -dp or -t flags.
	Examples -		Examples -
	1) To download all files with a ".txt" extension, you can use the regular expression .*.txt		1) To download all files with a ".txt" extension, you can use the regular expression .*.txt
Line 70:		Line 70:
	--verify When this option is provided a download is not initiated. Instead, a csv file is produced that contains a record of		--verify When this option is provided a download is not initiated. Instead, a csv file is produced that contains a record of
	the files in the download, along with information about the file-size if the file could be found on the computer.		the files in the download, along with information about the file-size if the file could be found on the computer.

	-v, --verbose Option to print out more detailed messages as the		-v, --verbose Option to print out more detailed messages as the
	program runs.		program runs.

Hwang: /* Download Data */

2022-12-07T20:38:30Z

Download Data

← Older revision		Revision as of 20:38, 7 December 2022
Line 68:		Line 68:
	3) Finally to download all files underneath a folder called "T1w" you can use the regular expression ./T1w/.		3) Finally to download all files underneath a folder called "T1w" you can use the regular expression ./T1w/.
	downloadcmd -dp 12345 -t s3-links.txt --file-regex ./T1w/.		downloadcmd -dp 12345 -t s3-links.txt --file-regex ./T1w/.
	--verify When this option is provided a download is not initiated. Instead, a csv file is produced that contains a record of the files in the download, along with information about the file-size if the file could be found on the computer.		--verify When this option is provided a download is not initiated. Instead, a csv file is produced that contains a record of
			the files in the download, along with information about the file-size if the file could be found on the computer.

	-v, --verbose Option to print out more detailed messages as the		-v, --verbose Option to print out more detailed messages as the

Hwang: /* Download Data */

2022-12-07T20:37:50Z

Download Data

← Older revision		Revision as of 20:37, 7 December 2022
Line 68:		Line 68:
	3) Finally to download all files underneath a folder called "T1w" you can use the regular expression ./T1w/.		3) Finally to download all files underneath a folder called "T1w" you can use the regular expression ./T1w/.
	downloadcmd -dp 12345 -t s3-links.txt --file-regex ./T1w/.		downloadcmd -dp 12345 -t s3-links.txt --file-regex ./T1w/.
	--verify When this option is provided a download is not initiated. Instead, a csv file is produced that contains a record of		--verify When this option is provided a download is not initiated. Instead, a csv file is produced that contains a record of the files in the download, along with information about the file-size if the file could be found on the computer.
	the files in the download, along with information about the file-size if the file could be found on the computer.

	-v, --verbose Option to print out more detailed messages as the		-v, --verbose Option to print out more detailed messages as the

Hwang: /* Download Data */

2022-12-07T20:35:03Z

Download Data

← Older revision		Revision as of 20:35, 7 December 2022
Line 64:		Line 64:
	1) To download all files with a ".txt" extension, you can use the regular expression .*.txt		1) To download all files with a ".txt" extension, you can use the regular expression .*.txt
	downloadcmd -dp 12345 --file-regex .*.txt		downloadcmd -dp 12345 --file-regex .*.txt

	2) To download all files that contain "NDARINVZLHFUAF0" in the name, you can use the regular expression NDARINVZLHFUAF0		2) To download all files that contain "NDARINVZLHFUAF0" in the name, you can use the regular expression NDARINVZLHFUAF0
	downloadcmd -dp 12345 -ds image03 --file-regex NDARINVZLHFUAF0		downloadcmd -dp 12345 -ds image03 --file-regex NDARINVZLHFUAF0

	3) Finally to download all files underneath a folder called "T1w" you can use the regular expression ./T1w/.		3) Finally to download all files underneath a folder called "T1w" you can use the regular expression ./T1w/.
	downloadcmd -dp 12345 -t s3-links.txt --file-regex ./T1w/.		downloadcmd -dp 12345 -t s3-links.txt --file-regex ./T1w/.
			--verify When this option is provided a download is not initiated. Instead, a csv file is produced that contains a record of
			the files in the download, along with information about the file-size if the file could be found on the computer.

	-v, --verbose Option to print out more detailed messages as the		-v, --verbose Option to print out more detailed messages as the

Hwang: /* Download Data */

2022-12-07T20:33:26Z

Download Data

← Older revision		Revision as of 20:33, 7 December 2022
Line 61:		Line 61:
	Option can be used to download only a subset of the files in a package. This command line arg can be used with		Option can be used to download only a subset of the files in a package. This command line arg can be used with
	the -ds, -dp or -t flags.		the -ds, -dp or -t flags.

	Examples -		Examples -
	1) To download all files with a ".txt" extension, you can use the regular expression .*.txt		1) To download all files with a ".txt" extension, you can use the regular expression .*.txt
	downloadcmd -dp 12345 --file-regex .*.txt		downloadcmd -dp 12345 --file-regex .*.txt

	2) To download all files that contain "NDARINVZLHFUAF0" in the name, you can use the regular expression NDARINVZLHFUAF0		2) To download all files that contain "NDARINVZLHFUAF0" in the name, you can use the regular expression NDARINVZLHFUAF0
	downloadcmd -dp 12345 -ds image03 --file-regex NDARINVZLHFUAF0		downloadcmd -dp 12345 -ds image03 --file-regex NDARINVZLHFUAF0

	3) Finally to download all files underneath a folder called "T1w" you can use the regular expression ./T1w/.		3) Finally to download all files underneath a folder called "T1w" you can use the regular expression ./T1w/.
	downloadcmd -dp 12345 -t s3-links.txt --file-regex ./T1w/.		downloadcmd -dp 12345 -t s3-links.txt --file-regex ./T1w/.

← Older revision		Revision as of 20:42, 7 December 2022
Line 62:		Line 62:
	This command line arg can be used with the -ds, -dp or -t flags.		This command line arg can be used with the -ds, -dp or -t flags.
	Examples -		Examples -
	1) To download all files with a ".txt" extension, ~~you can use the regular expression .*.txt~~		1) To download all files with a ".txt" extension,
	downloadcmd -dp 12345 --file-regex .*.txt		downloadcmd -dp 12345 --file-regex .*.txt
	2) To download all files that contain "NDARINVZLHFUAF0" in the name, ~~you can use the regular expression NDARINVZLHFUAF0~~		2) To download all files that contain "NDARINVZLHFUAF0" in the name,
	downloadcmd -dp 12345 -ds image03 --file-regex NDARINVZLHFUAF0		downloadcmd -dp 12345 -ds image03 --file-regex NDARINVZLHFUAF0
	3) Finally to download all files underneath a folder called "T1w" you can use the regular expression ./T1w/.		3) Finally to download all files underneath a folder called "T1w"
	downloadcmd -dp 12345 -t s3-links.txt --file-regex ./T1w/.		downloadcmd -dp 12345 -t s3-links.txt --file-regex ./T1w/.
	--verify When this option is provided a download is not initiated. Instead, a csv file is produced that contains a record of		--verify When this option is provided a download is not initiated. Instead,
	the files in the download, along with information about the file-size if the file could be found on the computer.		a csv file is produced that contains a record of the files in the download,
			along with information about the file-size if the file could be found on the computer.
	-v, --verbose Option to print out more detailed messages as the		-v, --verbose Option to print out more detailed messages as the
	program runs.		program runs.