Commit Graph

226 Commits

Author SHA1 Message Date
allegroai
0c9e2f92ee Add server-side support for deleting files from fileserver on task delete 2022-09-29 19:34:24 +03:00
allegroai
e6dc4b7557 Set cloned task parent to original task if original task has no parent 2022-09-29 19:30:13 +03:00
allegroai
238a47a197 Add created field to backend.user 2022-09-29 19:29:36 +03:00
allegroai
04e7076628 Add support for specifying a specific task ID in queues.get_next_task 2022-09-29 19:27:42 +03:00
allegroai
0531612bf4 Fix deleting a queue should dequeue all enqueued tasks 2022-09-29 19:27:09 +03:00
allegroai
3ae410a1e9 Remove the ThreadsManager.terminating flag 2022-09-29 19:23:26 +03:00
allegroai
8d4c02fc3c Add support for hidden internal queues 2022-09-29 19:20:24 +03:00
allegroai
a4fa567be2 Fix task stats update 2022-09-29 19:18:22 +03:00
allegroai
ddb91f226a Add Task Unique Metrics to task object 2022-09-29 19:16:56 +03:00
allegroai
efd56e085e Fix threaded jobs management (invoke only from AppSequence) 2022-09-29 19:13:22 +03:00
allegroai
ab495ae586 Fix archived projects handling 2022-07-08 17:55:02 +03:00
allegroai
b058770af1 Fix handling of empty hyperparam/configuration keys 2022-07-08 17:54:19 +03:00
allegroai
f7e833bf6f Fix loading expired worker entries from Redis 2022-07-08 17:53:12 +03:00
allegroai
ec0436d0da Fix task cleanup 2022-07-08 17:50:49 +03:00
allegroai
0f6c4e75b7 Fix debug images URL handling and task routing 2022-07-08 17:50:26 +03:00
allegroai
c28f478ea8 Fix worker Id is used instead of worker key when processing report 2022-07-08 17:48:17 +03:00
allegroai
3a60f00d93 Add support for Dataset projects 2022-07-08 17:45:03 +03:00
allegroai
ee87778548 Better support for PyJWT 2.4 2022-07-08 17:44:17 +03:00
allegroai
52c0c4d438 Add model task cleanup on tasks.reset 2022-07-08 17:43:54 +03:00
allegroai
d117a4f022 Support max_task_entries option in queues.get_by_id/get_all
Add queues.peek_task and queues.get_num_entries
2022-07-08 17:42:20 +03:00
allegroai
6683d2d7a9 Fix task cleanup 2022-07-08 17:40:55 +03:00
allegroai
adc1825843 Add support for model statistics 2022-07-08 17:39:41 +03:00
allegroai
b2feafac09 Support workers filtering with tags 2022-07-08 17:37:33 +03:00
allegroai
b41ab8c550 Better support for queue metrics and queue metrics refresh on sample 2022-07-08 17:36:46 +03:00
allegroai
62d5779bd5 Count own tasks/models for projects 2022-07-08 17:35:01 +03:00
allegroai
dd8a1503b0 Add support for navigate_current_metric in events.get_debug_image_sample 2022-07-08 17:31:44 +03:00
allegroai
cff98ae900 Add support for events.get_task_single_value_metrics, events.plots, events.get_plot_sample and events.next_plot_sample 2022-07-08 17:29:39 +03:00
allegroai
9b108740da Bump PyJWT version due to "Key confusion through non-blocklisted public key formats" vulnerability 2022-05-25 16:50:19 +03:00
allegroai
08a7bc7c9f Fix not all the event logs are returned from sharded ES 2022-05-20 15:11:05 +03:00
allegroai
710443b078 Fix move task to trash is not thread-safe 2022-05-18 10:31:20 +03:00
allegroai
e0cde2f7c9 Add support for deleting pipeline projects 2022-05-18 10:30:21 +03:00
allegroai
60b9c8de14 Allow arbitrary task fields in project statistics filter 2022-05-18 10:29:36 +03:00
allegroai
2570bd9e26 Fix ES issue with capital letters in index name 2022-05-18 10:18:23 +03:00
allegroai
174f84514a Fix no destination when merging projects 2022-05-18 10:17:34 +03:00
allegroai
65cb8d7b43 Refactor method name 2022-05-18 10:16:41 +03:00
allegroai
5f8ef808a3 Fix ES issue with capital letters in index name 2022-05-18 10:16:19 +03:00
allegroai
4941ac70e0 Add events.clear_task_log 2022-05-17 16:09:23 +03:00
allegroai
92b5fc6f9a Fix handling hidden sub-projects 2022-05-17 16:06:34 +03:00
allegroai
b90165b4e4 Support queue_name in tasks enqueue 2022-05-17 16:04:34 +03:00
allegroai
5131b17901 Support not returning hidden sub-projects when include_stats is specified without search_hidden 2022-04-18 16:36:14 +03:00
allegroai
2350ac64ed Fix internal error on count task events if there is no events index 2022-04-18 16:31:02 +03:00
allegroai
d146127c18 Add events.clear_scroll endpoint to clear event search scrolls 2022-04-18 16:29:57 +03:00
allegroai
c04e2e498b Support credentials label and last_used_from fields 2022-03-15 16:29:37 +02:00
allegroai
da8a45072f Add pipelines support 2022-03-15 16:28:59 +02:00
allegroai
e1992e2054 Fix queue metrics calculation 2022-03-15 16:28:49 +02:00
allegroai
b6ad8f8790 Add support for worker auto-unregister (instead of raising an error) 2022-03-15 16:25:14 +02:00
allegroai
af09fba755 Add metadata dict support for models, queues
Add more info for projects
2022-03-15 16:18:57 +02:00
allegroai
b9996e2c1a Protect against multiple connects to the update server from different processes
Code cleanup
2022-02-13 20:12:12 +02:00
allegroai
afdc56f37c Use task active duration for worker task running time 2022-02-13 20:01:47 +02:00
allegroai
a25cd5dae8 Fix version conflicts when deleting task events cause an error 2022-02-13 20:01:25 +02:00
allegroai
447adb9090 Add support for credentials label
Support no_scroll in events.get_task_plots
Support better project stats
Fix Redis required on mongodb initialization
Update tests
2022-02-13 19:59:58 +02:00
allegroai
ea3b6e955f Optimize nested_get() 2022-02-13 19:32:22 +02:00
allegroai
e334246b46 Add support for project stats with children flag 2022-02-13 19:26:47 +02:00
allegroai
36e013b40c Add support for events.scalar_metrics_iter_raw 2022-02-13 19:26:03 +02:00
allegroai
f20cd6536e Add scroll support to *.get_* 2022-02-13 19:23:29 +02:00
allegroai
a377a7e315 Support status_message and status_reason in tasks.delete 2022-02-13 19:20:31 +02:00
allegroai
3d046ac282 Fix project should not be merged into itself 2022-02-13 19:18:08 +02:00
allegroai
694dbc31c4 Fix incorrect ES query (merge issue) 2021-07-25 14:40:49 +03:00
allegroai
6488dc54e6 Better handling of stack trace report on 500 error 2021-07-25 14:39:59 +03:00
allegroai
465e270342 Fix queued task is not dequeued on tasks.stop 2021-07-25 14:32:09 +03:00
allegroai
6705aff56f Allow requesting plots and iter_histograms for all variants 2021-07-25 14:30:38 +03:00
allegroai
9069cfe1da Support querying task events per specific metrics and variants 2021-07-25 14:29:41 +03:00
allegroai
677bb3ba6d Add force parameter to tasks.enqueue 2021-07-25 14:27:46 +03:00
allegroai
d4edeaaf1b Add projects.validate_delete 2021-07-25 14:17:29 +03:00
allegroai
56aea1ffb8 Fix filtering on hyperparams (https://github.com/allegroai/clearml/issues/385, https://clearml.slack.com/archives/CTK20V944/p1626600582284700) 2021-07-25 13:55:09 +03:00
allegroai
ad72a435f1 Clean Task runtime on reset 2021-05-27 16:56:03 +03:00
allegroai
1cef03b8c2 Add check_contents flag for projects.get_all_ex 2021-05-03 18:12:44 +03:00
allegroai
ee90b0b024 Remove "Auto-generated while cloning" project description 2021-05-03 18:10:32 +03:00
allegroai
b55fad1b59 Remove "Auto-generated during move" project description 2021-05-03 18:09:31 +03:00
allegroai
b67aa05d6f Return results per task iterations in debug images request 2021-05-03 18:08:14 +03:00
allegroai
6b0c45a861 Fix batch operations results 2021-05-03 18:07:37 +03:00
allegroai
dc9623e964 Fix docker_cmd projection in backwards compatibility
Fix support to clear input/output models and docker_cmd in backwards compatibility mode
Fix schema
2021-05-03 18:06:39 +03:00
allegroai
3d73d60826 Better handling of invalid iterations on add_batch 2021-05-03 18:05:24 +03:00
allegroai
b99f620073 Added unarchive APIs 2021-05-03 18:04:17 +03:00
allegroai
e2f265b4bc Unify batch operations 2021-05-03 18:03:54 +03:00
allegroai
251ee57ffd Fix rapidjson dumps does not support ensure_ascii, only Encoder initialization does
Add task enqueue status
2021-05-03 18:03:17 +03:00
allegroai
7e03104f1c Add Model last_update field 2021-05-03 18:02:25 +03:00
allegroai
179661a0d4 Rename default input and output models
Better handling of backwards compatibility in task models
Code cleanup
2021-05-03 17:56:50 +03:00
allegroai
3d22ca1888 Escape task.container and task.execution.model_labels fields in DB 2021-05-03 17:56:17 +03:00
allegroai
fdf6798d0c Don't unset Task's execution.queue on dequeue 2021-05-03 17:54:16 +03:00
allegroai
9d9a44b927 Add skip_empty parameter in get_configuration_names 2021-05-03 17:53:56 +03:00
allegroai
a75534ec34 Add batch operations support 2021-05-03 17:52:54 +03:00
allegroai
29de110abb Add support for queue and model metadata 2021-05-03 17:50:25 +03:00
allegroai
2e7f418ee2 Fix Task.container backwards-compatibility
Fix sub-projects
2021-05-03 17:49:48 +03:00
allegroai
5a438e8435 Fix projects.move 2021-05-03 17:47:11 +03:00
allegroai
ef42d0265d Add multi-models support 2021-05-03 17:46:00 +03:00
allegroai
3c5195028e More sub-projects support and fixes 2021-05-03 17:44:54 +03:00
allegroai
0d5174c453 Support iterating over all task metrics in task debug images 2021-05-03 17:43:02 +03:00
allegroai
c034c1a986 Add sub-projects support 2021-05-03 17:42:10 +03:00
allegroai
f5008d80ad Optimize and improve tasks/models/projects.delete 2021-05-03 17:39:13 +03:00
allegroai
8b464e7ae6 Return file urls for tasks.delete/reset and models.delete 2021-05-03 17:38:09 +03:00
allegroai
d029d56508 Support active users in projects 2021-05-03 17:36:04 +03:00
allegroai
7f4ad0d1ca Support projects.get_hyperparam_values 2021-05-03 17:34:40 +03:00
allegroai
8f646043bb Allow enqueueing stopped tasks
More clearml stuff
2021-05-03 17:31:02 +03:00
allegroai
4b11a6efcd Move apiserver to clearml 2021-05-03 17:26:44 +03:00
allegroai
7d6918ecb0 Fix large plots comparison 2021-05-03 16:55:59 +03:00
allegroai
a8d90887e2 Fix task execution queue is not cleared on clone 2021-01-05 20:07:04 +02:00
allegroai
16430a6636 Support query by task state in projects.get_tasks_parents, return task project names 2021-01-05 19:02:48 +02:00
allegroai
ba03924cb4 Fix archived and development system tags should not be cleaned up 2021-01-05 18:52:12 +02:00
allegroai
64c63d2560 Add projects.get_task_parents 2021-01-05 18:49:25 +02:00
allegroai
03094076c8 Fix tags handling fail reloading datetime-like strings 2021-01-05 18:32:18 +02:00
allegroai
4707647c92 Use EVENT_TYPE enum instead of string 2021-01-05 18:21:11 +02:00
allegroai
6974aa3a99 Improve internal events implementation 2021-01-05 18:20:38 +02:00
allegroai
e2deff4eef Fix update project time on task changes
Fix project time in non responsive tasks watchdog
2021-01-05 18:19:45 +02:00
allegroai
59994ccf9c Fix task and model last_change handling
Improve db model index
Improve db model infrastructure
2021-01-05 18:17:29 +02:00
allegroai
29c792d459 Fix tasks.clone 2021-01-05 18:15:01 +02:00
allegroai
3272d0f31f Rename migration script
Support refresh flag in debug image samples
Remove silent_dequeue_fail param to prevent status change in case task wasn't queued
Add organizations.get_user_companies
Fix reset should also reset active_duration
Add api_version to server.info
2021-01-05 18:09:34 +02:00
allegroai
618a0b9473 Do not set Task.last_update when moving or updating (i.e. changing name, comment, tags etc.) 2021-01-05 18:06:04 +02:00
allegroai
bca3a6e556 Set default task active duration to None
Move endpoints to new API version
Support tasks.move and models.move for moving tasks and models into projects
Support new project name in tasks.clone
Improve task active duration migration
2021-01-05 18:05:44 +02:00
allegroai
8b0afd47a6 Set configurable and consistent limits on variants and metrics across different iterators 2021-01-05 18:02:01 +02:00
allegroai
0303c3525f API version bump
Update internal tests
Allow edit/delete task artifacts/hyperparams/configs using force flag
Improve lists query support for get_all calls
2021-01-05 17:57:58 +02:00
allegroai
0ad0495733 Add tasks.archive support 2021-01-05 17:49:08 +02:00
allegroai
be788965e0 Fix using reserved keywords as atrifact/hyperparams/configuration names
Replace events.get_debug_image_event and event.get_debug_image_iterations with events.get_debug_image_sample and events.next_debug_image_sample
2021-01-05 17:47:27 +02:00
allegroai
b89de43373 Support sorting by task active duration 2021-01-05 17:44:17 +02:00
allegroai
323b5db07c Add support for debug images history using events.get_debug_image_event and events.get_debug_image_iterations
Remove untracked files
2021-01-05 17:42:05 +02:00
allegroai
f084f6b9e7 Fix get_hyperparams and unique_metric_params handling of public tasks 2021-01-05 17:38:50 +02:00
allegroai
eb4c9f0b13 Fix batch events counting 2021-01-05 17:37:40 +02:00
allegroai
37e5d8a7e0 Fix ParseError import with new luqum version
Fix incorrect strip to task diff and requirements
Add missing property to server.report_stats_option response
Add active_duration parameter for tasks
Move artifacts info dictionary structure
2021-01-05 17:07:14 +02:00
allegroai
1c7de3a86e Add worker runtime properties support
Refactor login and add guest mode
Support artifacts in prepopulate
2021-01-05 16:56:08 +02:00
allegroai
e24389fda9 Add configuration loader 2021-01-05 16:44:31 +02:00
allegroai
f4ead86449 Add support for returning only valid plot events 2021-01-05 16:41:55 +02:00
allegroai
171969c5ea Optimize task artifacts 2021-01-05 16:40:35 +02:00
allegroai
c7bbac73d0 Refactor es_factory into a separate class 2021-01-05 16:29:25 +02:00
allegroai
f832ea565a Use apiserver namespace 2021-01-05 16:28:49 +02:00
allegroai
22e9c2b7eb Fix type annotations
Fix obtaining events for tasks moved from private to public
Fix assert_exists() to return company_origin if requested
2021-01-05 16:27:38 +02:00
allegroai
df65e1c7ad Rename server to apiserver 2021-01-05 16:22:34 +02:00