ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-01-31 14:37:18 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Dataset: Include another use case for add_external_files (#356)

Browse Source
This commit is contained in:
Fred DeSerres-Blanchet 2022-11-02 10:10:56 -04:00 committed by GitHub
parent 1fa97a6b55
commit 03d3b014a8
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 10 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
docs/clearml_data/clearml_data_sdk.md
View File

@ -169,15 +169,22 @@ dataset.add_files(
### add_external_files()
To add files or folders to the current dataset, leaving them in their original location, use the [`Dataset.add_external_files`](../references/sdk/dataset.md#add_external_files)
method. Input the `source_url` argument, which can be a link from cloud storage (`s3://`, `gs://`, `azure://`)
or local / network storage (`file://`).
method. Input the `source_url` argument, which can be a link or a list of links from cloud storage (`s3://`, `gs://`, `azure://`)
or local / network storage (`file://`).
```python
dataset = Dataset.create(dataset_name="my dataset", dataset_project="example project")
dataset.add_external_files(
source_url="s3://my/bucket/path_to_folder_or_file",
dataset_path="/my_dataset/new_folder/"
)
)
dataset.add_external_files(
source_url=[
"s3://my/bucket/path_to_folder_or_file",
"s3://my/bucket/path_to_another_folder_or_file",
],
dataset_path="/my_dataset/new_folder/"
)
```
There is an option to add a set of files based on wildcard matching of a single string or a list of wildcards, using the