ClearML/clearml
1
0
Fork 0
mirror of https://github.com/clearml/clearml synced 2025-06-26 18:16:07 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
2,526 Commits 4 Branches 181 Tags 56 MiB
2507fdc1f0
Commit Graph

106 Commits

Author SHA1 Message Date
clearml
2507fdc1f0 Version bump to v2.0.0
Some checks failed
CodeQL / Analyze (python) (push) Has been cancelled
Details
2025-05-22 13:45:04 +03:00
clearml
66a352b587 Fix the verify configuration option for S3 storage (boto3) is not used when testing buckets 2025-05-21 10:04:26 +03:00
clearml
1a2d4749ad Fix TQDM upload/download reporting, remove warning 2025-04-25 16:13:58 +03:00
vignesh
7f822e7626
Fix local file uploads without scheme (#1326)
Some checks failed
CodeQL / Analyze (python) (push) Has been cancelled
Details 
* handle scenarios of local file upload without file:// scheme. Fetch the absolute path for the file in such cases and use it as source url

* Update README gifs (#1327)

* skip adding scheme to the path if the scheme is already provided

* handle scenarios of local file upload without file:// scheme. Fetch the absolute path for the file in such cases and use it as source url

* skip adding scheme to the path if the scheme is already provided

---------

Co-authored-by: Vignesh D <vignesh_d@epam.com>
Co-authored-by: pollfly <75068813+pollfly@users.noreply.github.com>
 2025-04-18 16:11:47 +03:00
clearml
90d3578aeb Add function annotations 
Remove support for PY2.x and PY3.5
 2025-04-17 22:26:06 +03:00
clearml
1d012de0b5 Add silent_on_errors argument to Task.delete_artifacts (default False) 2024-09-12 09:38:34 +03:00
allegroai
2811931d69 Support path to certificate files to be specified as a downloadable URL 2024-08-27 10:33:05 +03:00
allegroai
7dc601598b Add support for HTTP file upload progress reporting 2024-07-04 15:26:45 +03:00
nfzd
7fcd315c4e
Fix path substitution for file:// URIs (#1251) 
Co-authored-by: Michael Mueller <michael.mueller@wsa.com>
 2024-05-31 09:07:45 +03:00
allegroai
09ed70a313 Fix merge conflict 2024-02-15 19:30:05 +02:00
Olivier Dulcy
2e084688be
Fix forward secure in StorageHelper for S3 (#1205) 2024-02-15 19:26:22 +02:00
allegroai
87f0e63b28 Fix setting output_uri when a config section with secure=false exists will raise an error 2024-02-15 19:02:01 +02:00
allegroai
233f94f741 Support AWS profile when specifying bucket storage credentials 2024-02-04 19:31:30 +02:00
allegroai
0a928c24ca Fix unsafe usage of tempfile.mktemp 2024-01-30 19:20:22 +02:00
allegroai
0e0763d566 Report upload/download with tqdm if installed 2024-01-18 16:06:46 +02:00
allegroai
e746b3e535 Support GS credentials as JSON string if not a JSON file reference 2023-11-05 21:07:32 +02:00
Matteo Destro
a8ec1076da
Remove subdir path from bucket url (#1117) 2023-10-17 20:13:20 +03:00
allegroai
e467c05fb4 Fix S3 uploads fail with LazyEvalWrapper type error (#1081) 2023-07-30 13:04:42 +03:00
Alex Burlacu
cc875f0fbe Fix linter issues 2023-07-19 17:28:58 +03:00
Alex Burlacu
75b4015fdb Fix misconfigured boto3 bucket credentials verify 2023-07-19 02:09:49 +03:00
alex-burlacu-clear-ml
bae4886412
Merge pull request #1059 from cgaudreau-ubisoft/s3-configure-multipart 
Added boto3 parameters for multipart configurations
 2023-07-14 14:13:21 +03:00
allegroai
5aa80267d6 Version bump to v1.11.2rc0 
Small refactoring
 2023-07-10 11:20:27 +03:00
Alex Burlacu
6dc2c8ca8c Fix GCS quoting issue 2023-07-06 12:23:06 +03:00
AbdulHamid Merii
9e8ed16b3e Added boto3 parameters for multipart configurations 
- multipart threshold and chunksize
 2023-06-26 12:07:40 -05:00
Alex Burlacu
60c3a5ef98 Fix downloading artifacts from GCP when the download URI includes unsafe characters 2023-05-25 18:14:17 +03:00
allegroai
7ab1168532 Fix multiple local parallel experiments conflict on storage access validation (#897) 2023-05-11 16:05:19 +03:00
allegroai
4cd8857c0d Refactor code 2023-03-27 13:38:11 +03:00
Alex Burlacu
067c817f30 Allow deleting files when deleting datasets stored with clearml-data 2023-03-23 18:03:40 +02:00
john-zielke-snkeos
df5d317c06 Fix typo 2023-02-24 19:13:13 +01:00
john-zielke-snkeos
8f57c5132c Added fix to store unsubstituted url for model 
Added canonize_url for .delete()
 2023-02-24 18:59:15 +01:00
allegroai
311e05d938 Fix StorageManager.get_file_size_bytes() and StorageManager.get_metadata() might raise an error even with silence_errors=True 2023-02-14 12:25:59 +02:00
allegroai
a48f1c4b58 Keep original upload URI when reporting events when URI is changed due to path substitution 2023-02-05 10:19:44 +02:00
john-zielke-snkeos
7d80406290
Bugfix for storage with path substitutions (#877) 
* Bugfix: canonize url in StorageHelper.exists_file()

* Added fixes for path substitutions:
StorageManager to use orginal path in metadata
Helper to use path rewrites in list operations

* Fixed a copy-paste error

Co-authored-by: eajechiloae <97950284+eugen-ajechiloae-clearml@users.noreply.github.com>
 2023-01-23 15:51:21 +02:00
allegroai
b4cb44bf1b Fix auth headers are not set on substituted fileserver urls 2023-01-23 14:09:46 +02:00
allegroai
1a62d4b6e1 Fix StorageManager.list() does not return size metadata (#865) 2023-01-23 14:07:02 +02:00
allegroai
812d22a3c7 Fix clearml-data previews are saved on file server even when output_uri is specified 2023-01-19 11:16:21 +02:00
Colin Gaudreau
d0fa72ee1a
Add signature version to boto3 config (#884) 2023-01-18 19:13:03 +02:00
pollfly
b39829a011
Edit docstrings (#875) 2023-01-10 11:12:47 +02:00
allegroai
c75c83c21d Fix casting None to int fails uploads and permission checks 2022-12-29 13:18:39 +02:00
allegroai
ab73447cbd Add StorageManager.set_report_upload_chunk_size() and StorageManager.set_report_download_chunk_size() to set chunk size for upload and download 2022-12-22 21:57:15 +02:00
allegroai
3dee5854bd Fix StorageManager.get_file_size_bytes() returns ClientError instead of None for invalid S3 links 2022-12-22 21:52:24 +02:00
allegroai
1dd98ecf8d Fix lazyloaderproxy casting 2022-12-18 22:57:31 +02:00
allegroai
dd6e4f1598 Fix StorageManager doesn't support uploading files to minio (#763) 2022-12-13 16:07:44 +02:00
allegroai
16c8a037cc Fix can't add S3 external links (#845) 2022-12-13 15:48:18 +02:00
allegroai
62a5ef102a Fix uploading files to S3 when _at_exit() is called fails (work around https://github.com/boto/boto3/issues/3113) 2022-12-13 15:43:18 +02:00
john-zielke-snkeos
e709e46d91
Fix StorageHelper.delete() does not respect path substitutions (#838) 
Fixes allegroai/clearml#825
Make StorageHelper.get_object() call StorageHelper._canonized_url() to make path substitutions work.
 2022-12-06 14:26:05 +02:00
Fred DeSerres-Blanchet
911a72f561
Set GCS credentials to None if invalid service account creds are provided (#841) 
Related to https://github.com/allegroai/clearml/issues/840
 2022-12-03 12:08:36 +02:00
allegroai
ed2b8ed850 Support num_workers in dataset operations 
Support max connections setting for Azure storage using the `sdk.azure.storage.max_connection` setting
 2022-11-21 17:12:02 +02:00
allegroai
0b0fd55329 Fix StorageManager.list(“s3://..”, with_metadata=True) doesn't work 2022-11-19 00:58:05 +02:00
allegroai
a4d83c30ba Fixed Python 2.7 support, refactor, PEP8 2022-11-09 11:42:06 +02:00