DownloadClient

__init__

Initialises the basic settings for an DownloadClient object

Arguments:

client: Optional: rucio.client.client.Client object. If None, a new object will be created. external_traces: Optional: reference to a list where traces can be added logger: Optional: logging.Logger object. If None, default logger will be used.

download_pfns

Download items with a given PFN. This function can only download files, no datasets.

Arguments:	items: List of dictionaries. Each dictionary describing a file to download. Keys: pfn - PFN string of this file did - DID string of this file (e.g. 'scope:file.name'). Wildcards are not allowed rse - rse name (e.g. 'CERN-PROD_DATADISK'). RSE Expressions are not allowed base_dir - Optional: Base directory where the downloaded files will be stored. (Default: '.') no_subdir - Optional: If true, files are written directly into base_dir. (Default: False) adler32 - Optional: The adler32 checmsum to compare the downloaded files adler32 checksum with md5 - Optional: The md5 checksum to compare the downloaded files md5 checksum with transfer_timeout - Optional: Timeout time for the download protocols. (Default: None) check_local_with_filesize_only - Optional: If true, already downloaded files will not be validated by checksum. num_threads: Suggestion of number of threads to use for the download. It will be lowered if it's too high. trace_custom_fields: Custom key value pairs to send with the traces traces_copy_out: reference to an external list, where the traces should be uploaded deactivate_file_download_exceptions: Boolean, if file download exceptions shouldn't be raised
Raises:	InputValidationError: if one of the input items is in the wrong format NoFilesDownloaded: if no files could be downloaded NotAllFilesDownloaded: if not all files could be downloaded RucioException: if something unexpected went wrong during the download
Returns:	a list of dictionaries with an entry for each file, containing the input options, the did, and the clientState clientState can be one of the following: ALREADY_DONE, DONE, FILE_NOT_FOUND, FAIL_VALIDATE, FAILED

download_dids

Download items with given DIDs. This function can also download datasets and wildcarded DIDs.

Arguments:	items: List of dictionaries. Each dictionary describing an item to download. Keys: did - DID string of this file (e.g. 'scope:file.name') filters - Filter to select DIDs for download. Optional if DID is given rse - Optional: rse name (e.g. 'CERN-PROD_DATADISK') or rse expression from where to download impl - Optional: name of the protocol implementation to be used to download this item. no_resolve_archives - Optional: bool indicating whether archives should not be considered for download (Default: False) resolve_archives - Deprecated: Use no_resolve_archives instead force_scheme - Optional: force a specific scheme to download this item. (Default: None) base_dir - Optional: base directory where the downloaded files will be stored. (Default: '.') no_subdir - Optional: If true, files are written directly into base_dir. (Default: False) nrandom - Optional: if the DID addresses a dataset, nrandom files will be randomly chosen for download from the dataset ignore_checksum - Optional: If true, skips the checksum validation between the downloaded file and the rucio catalouge. (Default: False) transfer_timeout - Optional: Timeout time for the download protocols. (Default: None) transfer_speed_timeout - Optional: Minimum allowed transfer speed (in KBps). Ignored if transfer_timeout set. Otherwise, used to compute default timeout (Default: 500) check_local_with_filesize_only - Optional: If true, already downloaded files will not be validated by checksum. num_threads: Suggestion of number of threads to use for the download. It will be lowered if it's too high. trace_custom_fields: Custom key value pairs to send with the traces. traces_copy_out: reference to an external list, where the traces should be uploaded deactivate_file_download_exceptions: Boolean, if file download exceptions shouldn't be raised sort: Select best replica by replica sorting algorithm. Available algorithms: geoip - based on src/dst IP topographical distance closeness - based on src/dst closeness dynamic - Rucio Dynamic Smart Sort (tm)
Raises:	InputValidationError: if one of the input items is in the wrong format NoFilesDownloaded: if no files could be downloaded NotAllFilesDownloaded: if not all files could be downloaded RucioException: if something unexpected went wrong during the download
Returns:	a list of dictionaries with an entry for each file, containing the input options, the did, and the clientState

download_from_metalink_file

Download items using a given metalink file.

Arguments:	item: dictionary describing an item to download. Keys: base_dir - Optional: base directory where the downloaded files will be stored. (Default: '.') no_subdir - Optional: If true, files are written directly into base_dir. (Default: False) ignore_checksum - Optional: If true, skips the checksum validation between the downloaded file and the rucio catalouge. (Default: False) transfer_timeout - Optional: Timeout time for the download protocols. (Default: None) check_local_with_filesize_only - Optional: If true, already downloaded files will not be validated by checksum. num_threads: Suggestion of number of threads to use for the download. It will be lowered if it's too high. trace_custom_fields: Custom key value pairs to send with the traces. traces_copy_out: reference to an external list, where the traces should be uploaded deactivate_file_download_exceptions: Boolean, if file download exceptions shouldn't be raised
Raises:	InputValidationError: if one of the input items is in the wrong format NoFilesDownloaded: if no files could be downloaded NotAllFilesDownloaded: if not all files could be downloaded RucioException: if something unexpected went wrong during the download
Returns:	a list of dictionaries with an entry for each file, containing the input options, the did, and the clientState

download_aria2c

Uses aria2c to download the items with given DIDs. This function can also download datasets and wildcarded DIDs.

It only can download files that are available via https/davs. Aria2c needs to be installed and X509_USER_PROXY needs to be set!

Arguments:	items: List of dictionaries. Each dictionary describing an item to download. Keys: did - DID string of this file (e.g. 'scope:file.name'). Wildcards are not allowed rse - Optional: rse name (e.g. 'CERN-PROD_DATADISK') or rse expression from where to download base_dir - Optional: base directory where the downloaded files will be stored. (Default: '.') no_subdir - Optional: If true, files are written directly into base_dir. (Default: False) nrandom - Optional: if the DID addresses a dataset, nrandom files will be randomly chosen for download from the dataset ignore_checksum - Optional: If true, skips the checksum validation between the downloaded file and the rucio catalouge. (Default: False) check_local_with_filesize_only - Optional: If true, already downloaded files will not be validated by checksum. trace_custom_fields: Custom key value pairs to send with the traces filters: dictionary containing filter options deactivate_file_download_exceptions: Boolean, if file download exceptions shouldn't be raised sort: Select best replica by replica sorting algorithm. Available algorithms: geoip - based on src/dst IP topographical distance closeness - based on src/dst closeness dynamic - Rucio Dynamic Smart Sort (tm)
Raises:	InputValidationError: if one of the input items is in the wrong format NoFilesDownloaded: if no files could be downloaded NotAllFilesDownloaded: if not all files could be downloaded RucioException: if something went wrong during the download (e.g. aria2c could not be started)
Returns:	a list of dictionaries with an entry for each file, containing the input options, the did, and the clientState

preferred_impl

Finds the optimum protocol impl preferred by the client and

supported by the remote RSE.

Arguments:	sources: List of sources for a given DID
Raises:	RucioException(msg): general exception with msg for more details.