mirror of
https://github.com/clearml/clearml-docs
synced 2025-03-13 07:28:42 +00:00
Small edits
This commit is contained in:
revital
parent
001ae8e5ed
commit
9e275de492
@ -17,7 +17,7 @@ title: ClearML Agent
|
||||
|
||||
**ClearML Agent** is a virtual environment and execution manager for DL / ML solutions on GPU machines. It integrates with the **ClearML Python Package** and ClearML Server to provide a full AI cluster solution. <br/>
|
||||
Its main focus is around:
|
||||
- Reproducing tasks, including their complete environments.
|
||||
- Reproducing task runs, including their complete environments.
|
||||
- Scaling workflows on multiple target machines.
|
||||
|
||||
ClearML Agent executes a task or other workflow by reproducing the state of the code from the original machine
|
||||
@ -46,7 +46,7 @@ install Python, so make sure to use a container or environment with the version
|
||||
While the agent is running, it continuously reports system metrics to the ClearML Server (these can be monitored in the
|
||||
[**Orchestration**](webapp/webapp_workers_queues.md) page).
|
||||
|
||||
Continue using ClearML Agent once it is running on a target machine. Reproduce tasks and execute
|
||||
Continue using ClearML Agent once it is running on a target machine. Reproducing task runs and execute
|
||||
automated workflows in one (or both) of the following ways:
|
||||
* Programmatically (using [`Task.enqueue()`](references/sdk/task.md#taskenqueue) or [`Task.execute_remotely()`](references/sdk/task.md#execute_remotely))
|
||||
* Through the ClearML Web UI (without working directly with code), by cloning tasks and enqueuing them to the
|
||||
|
||||
@ -14,7 +14,7 @@ powerful remote machine. This is useful for:
|
||||
* Managing execution through ClearML's queue system.
|
||||
|
||||
This guide focuses on transitioning a locally executed process to a remote machine for scalable execution. To learn how
|
||||
to reproduce a previously executed process on a remote machine, see [Reproducing Tasks](reproduce_tasks.md).
|
||||
to reproduce a previously executed process on a remote machine, see [Reproducing Task Runs](reproduce_tasks.md).
|
||||
|
||||
## Running a Task Remotely
|
||||
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
---
|
||||
title: Reproducing Tasks
|
||||
title: Reproducing Task Runs
|
||||
---
|
||||
|
||||
:::note
|
||||
|
||||
@ -77,7 +77,7 @@ See more information about explicitly logging information to a ClearML Task:
|
||||
See [Explicit Reporting Tutorial](../guides/reporting/explicit_reporting.md).
|
||||
|
||||
## Remote Execution
|
||||
ClearML logs all the information required to reproduce a task on a different machine (installed packages,
|
||||
ClearML logs all the information required to reproduce a task run on a different machine (installed packages,
|
||||
uncommitted changes etc.). The [ClearML Agent](../clearml_agent.md) listens to designated queues and when a task is enqueued,
|
||||
the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the
|
||||
task manager.
|
||||
@ -93,7 +93,7 @@ Use the ClearML [Autoscalers](../cloud_autoscaling/autoscaling_overview.md) to h
|
||||
cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents: the autoscaler automatically spins up
|
||||
and shuts down instances as needed, according to a resource budget that you set.
|
||||
|
||||
### Reproducing Tasks
|
||||
### Reproducing Task Runs
|
||||
|
||||
|
||||
|
||||
|
||||
@ -75,7 +75,7 @@ See more information about explicitly logging information to a ClearML Task:
|
||||
See [Explicit Reporting Tutorial](../guides/reporting/explicit_reporting.md).
|
||||
|
||||
## Remote Execution
|
||||
ClearML logs all the information required to reproduce a task on a different machine (installed packages,
|
||||
ClearML logs all the information required to reproduce a task run on a different machine (installed packages,
|
||||
uncommitted changes etc.). The [ClearML Agent](../clearml_agent.md) listens to designated queues and when a task is enqueued,
|
||||
the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the
|
||||
task manager.
|
||||
@ -91,7 +91,7 @@ Use the ClearML [Autoscalers](../cloud_autoscaling/autoscaling_overview.md) to h
|
||||
cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents: the autoscaler automatically spins up
|
||||
and shuts down instances as needed, according to a resource budget that you set.
|
||||
|
||||
### Reproducing Tasks
|
||||
### Reproducing Task Runs
|
||||
|
||||
|
||||
|
||||
|
||||
@ -74,7 +74,7 @@ See more information about explicitly logging information to a ClearML Task:
|
||||
See [Explicit Reporting Tutorial](../guides/reporting/explicit_reporting.md).
|
||||
|
||||
## Remote Execution
|
||||
ClearML logs all the information required to reproduce a task on a different machine (installed packages,
|
||||
ClearML logs all the information required to reproduce a task run on a different machine (installed packages,
|
||||
uncommitted changes etc.). The [ClearML Agent](../clearml_agent.md) listens to designated queues and when a task is enqueued,
|
||||
the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the
|
||||
task manager.
|
||||
@ -90,7 +90,7 @@ Use the ClearML [Autoscalers](../cloud_autoscaling/autoscaling_overview.md) to h
|
||||
cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents: the autoscaler automatically spins up
|
||||
and shuts down instances as needed, according to a resource budget that you set.
|
||||
|
||||
### Reproducing Tasks
|
||||
### Reproducing Task Runs
|
||||
|
||||
|
||||
|
||||
|
||||
@ -87,7 +87,7 @@ and debug samples, plots, and scalars logged to TensorBoard
|
||||
|
||||
|
||||
## Remote Execution
|
||||
ClearML logs all the information required to reproduce a task on a different machine (installed packages,
|
||||
ClearML logs all the information required to reproduce a task run on a different machine (installed packages,
|
||||
uncommitted changes etc.). The [ClearML Agent](../clearml_agent.md) listens to designated queues and when a task is enqueued,
|
||||
the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the
|
||||
task manager.
|
||||
@ -103,7 +103,7 @@ Use the ClearML [Autoscalers](../cloud_autoscaling/autoscaling_overview.md) to h
|
||||
cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents: the autoscaler automatically spins up
|
||||
and shuts down instances as needed, according to a resource budget that you set.
|
||||
|
||||
### Reproducing Tasks
|
||||
### Reproducing Task Runs
|
||||
|
||||
|
||||
|
||||
|
||||
@ -76,7 +76,7 @@ See more information about explicitly logging information to a ClearML Task:
|
||||
See [Explicit Reporting Tutorial](../guides/reporting/explicit_reporting.md).
|
||||
|
||||
## Remote Execution
|
||||
ClearML logs all the information required to reproduce a task on a different machine (installed packages,
|
||||
ClearML logs all the information required to reproduce a task run on a different machine (installed packages,
|
||||
uncommitted changes etc.). The [ClearML Agent](../clearml_agent.md) listens to designated queues and when a task is enqueued,
|
||||
the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the
|
||||
task manager.
|
||||
@ -92,7 +92,7 @@ Use the ClearML [Autoscalers](../cloud_autoscaling/autoscaling_overview.md) to h
|
||||
cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents: the autoscaler automatically spins up
|
||||
and shuts down instances as needed, according to a resource budget that you set.
|
||||
|
||||
### Reproducing Tasks
|
||||
### Reproducing Task Runs
|
||||
|
||||
|
||||
|
||||
|
||||
@ -72,7 +72,7 @@ See more information about explicitly logging information to a ClearML Task:
|
||||
See [Explicit Reporting Tutorial](../guides/reporting/explicit_reporting.md).
|
||||
|
||||
## Remote Execution
|
||||
ClearML logs all the information required to reproduce a task on a different machine (installed packages,
|
||||
ClearML logs all the information required to reproduce a task run on a different machine (installed packages,
|
||||
uncommitted changes etc.). The [ClearML Agent](../clearml_agent.md) listens to designated queues and when a task is enqueued,
|
||||
the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the
|
||||
task manager.
|
||||
@ -88,7 +88,7 @@ Use the ClearML [Autoscalers](../cloud_autoscaling/autoscaling_overview.md) to h
|
||||
cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents: the autoscaler automatically spins up
|
||||
and shuts down instances as needed, according to a resource budget that you set.
|
||||
|
||||
### Reproducing Tasks
|
||||
### Reproducing Task Runs
|
||||
|
||||
|
||||
|
||||
|
||||
@ -96,7 +96,7 @@ additional tools, like argparse, TensorBoard, and matplotlib:
|
||||
* [PyTorch Distributed](../guides/frameworks/pytorch/pytorch_distributed_example.md) - Demonstrates using ClearML with the [PyTorch Distributed Communications Package (`torch.distributed`)](https://pytorch.org/tutorials/beginner/dist_overview.html)
|
||||
|
||||
## Remote Execution
|
||||
ClearML logs all the information required to reproduce a task on a different machine (installed packages,
|
||||
ClearML logs all the information required to reproduce a task run on a different machine (installed packages,
|
||||
uncommitted changes etc.). The [ClearML Agent](../clearml_agent.md) listens to designated queues and when a task is enqueued,
|
||||
the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the
|
||||
task manager.
|
||||
@ -112,7 +112,7 @@ Use the ClearML [Autoscalers](../cloud_autoscaling/autoscaling_overview.md) to h
|
||||
cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents: the autoscaler automatically spins up
|
||||
and shuts down instances as needed, according to a resource budget that you set.
|
||||
|
||||
### Reproducing Tasks
|
||||
### Reproducing Task Runs
|
||||
|
||||
|
||||
|
||||
|
||||
@ -102,7 +102,7 @@ See more information about explicitly logging information to a ClearML Task:
|
||||
See [Explicit Reporting Tutorial](../guides/reporting/explicit_reporting.md).
|
||||
|
||||
## Remote Execution
|
||||
ClearML logs all the information required to reproduce a task on a different machine (installed packages,
|
||||
ClearML logs all the information required to reproduce a task run on a different machine (installed packages,
|
||||
uncommitted changes etc.). The [ClearML Agent](../clearml_agent.md) listens to designated queues and when a task is enqueued,
|
||||
the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the
|
||||
task manager.
|
||||
@ -118,7 +118,7 @@ Use the ClearML [Autoscalers](../cloud_autoscaling/autoscaling_overview.md), to
|
||||
cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents: the autoscaler automatically spins up
|
||||
and shuts down instances as needed, according to a resource budget that you set.
|
||||
|
||||
### Reproducing Tasks
|
||||
### Reproducing Task Runs
|
||||
|
||||
|
||||
|
||||
|
||||
@ -78,7 +78,7 @@ additional tools, like Matplotlib:
|
||||
|
||||
|
||||
## Remote Execution
|
||||
ClearML logs all the information required to reproduce a task on a different machine (installed packages,
|
||||
ClearML logs all the information required to reproduce a task run on a different machine (installed packages,
|
||||
uncommitted changes etc.). The [ClearML Agent](../clearml_agent.md) listens to designated queues and when a task is enqueued,
|
||||
the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the
|
||||
task manager.
|
||||
@ -94,7 +94,7 @@ Use the ClearML [Autoscalers](../cloud_autoscaling/autoscaling_overview.md) to h
|
||||
cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents: the autoscaler automatically spins up
|
||||
and shuts down instances as needed, according to a resource budget that you set.
|
||||
|
||||
### Reproducing Tasks
|
||||
### Reproducing Task Runs
|
||||
|
||||
|
||||
|
||||
|
||||
@ -94,7 +94,7 @@ You can view all of this captured information in the [ClearML Web UI](../webapp/
|
||||
|
||||
|
||||
## Remote Execution
|
||||
ClearML logs all the information required to reproduce a task on a different machine (installed packages,
|
||||
ClearML logs all the information required to reproduce a task run on a different machine (installed packages,
|
||||
uncommitted changes etc.). The [ClearML Agent](../clearml_agent.md) listens to designated queues and when a task is
|
||||
enqueued, the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the
|
||||
task manager.
|
||||
@ -111,7 +111,7 @@ cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents:
|
||||
and shuts down instances as needed, according to a resource budget that you set.
|
||||
|
||||
|
||||
### Reproducing Tasks
|
||||
### Reproducing Task Runs
|
||||
|
||||
|
||||
|
||||
|
||||
@ -89,7 +89,7 @@ TensorBoard scalars, histograms, images, and text, as well as all console output
|
||||
ClearML's automatic logging of parameters defined using `absl.flags`
|
||||
|
||||
## Remote Execution
|
||||
ClearML logs all the information required to reproduce a task on a different machine (installed packages,
|
||||
ClearML logs all the information required to reproduce a task run on a different machine (installed packages,
|
||||
uncommitted changes etc.). The [ClearML Agent](../clearml_agent.md) listens to designated queues and when a task is enqueued,
|
||||
the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the
|
||||
task manager.
|
||||
@ -105,7 +105,7 @@ Use the ClearML [Autoscalers](../cloud_autoscaling/autoscaling_overview.md) to h
|
||||
cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents: the autoscaler automatically spins up
|
||||
and shuts down instances as needed, according to a resource budget that you set.
|
||||
|
||||
### Reproducing Tasks
|
||||
### Reproducing Task Runs
|
||||
|
||||
|
||||
|
||||
|
||||
@ -60,7 +60,7 @@ You can also select multiple tasks and directly [compare](../webapp/webapp_exp_c
|
||||
See an example of Transformers and ClearML in action [here](../guides/frameworks/huggingface/transformers.md).
|
||||
|
||||
## Remote Execution
|
||||
ClearML logs all the information required to reproduce a task on a different machine (installed packages,
|
||||
ClearML logs all the information required to reproduce a task run on a different machine (installed packages,
|
||||
uncommitted changes etc.). The [ClearML Agent](../clearml_agent.md) listens to designated queues and when a task is
|
||||
enqueued, the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the
|
||||
task manager.
|
||||
|
||||
@ -102,7 +102,7 @@ additional tools, like Matplotlib and scikit-learn:
|
||||
* [XGBoost and scikit-learn](../guides/frameworks/xgboost/xgboost_sample.md) - Demonstrates ClearML automatic logging of XGBoost scalars and models
|
||||
|
||||
## Remote Execution
|
||||
ClearML logs all the information required to reproduce a task on a different machine (installed packages,
|
||||
ClearML logs all the information required to reproduce a task run on a different machine (installed packages,
|
||||
uncommitted changes etc.). The [ClearML Agent](../clearml_agent.md) listens to designated queues and when a task is enqueued,
|
||||
the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the
|
||||
task manager.
|
||||
@ -118,7 +118,7 @@ Use the ClearML [Autoscalers](../cloud_autoscaling/autoscaling_overview.md) to h
|
||||
cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents: the autoscaler automatically spins up
|
||||
and shuts down instances as needed, according to a resource budget that you set.
|
||||
|
||||
### Reproducing Tasks
|
||||
### Reproducing Task Runs
|
||||
|
||||
|
||||
|
||||
|
||||
@ -150,7 +150,7 @@ python train.py --img 640 --batch 16 --epochs 3 --data clearml://<your_dataset_i
|
||||
|
||||
|
||||
## Remote Execution
|
||||
ClearML logs all the information required to reproduce a task on a different machine (installed packages,
|
||||
ClearML logs all the information required to reproduce a task run on a different machine (installed packages,
|
||||
uncommitted changes etc.). The [ClearML Agent](../clearml_agent.md) listens to designated queues and when a task is
|
||||
enqueued, the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the
|
||||
task manager.
|
||||
@ -167,7 +167,7 @@ cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents:
|
||||
and shuts down instances as needed, according to a resource budget that you set.
|
||||
|
||||
|
||||
### Reproducing Tasks
|
||||
### Reproducing Task Runs
|
||||
|
||||
|
||||
|
||||
|
||||
@ -95,7 +95,7 @@ Add custom columns to the table, such as mAP values, so you can easily sort and
|
||||
You can also select multiple tasks and directly [compare](../webapp/webapp_exp_comparing.md) them.
|
||||
|
||||
## Remote Execution
|
||||
ClearML logs all the information required to reproduce a task on a different machine (installed packages,
|
||||
ClearML logs all the information required to reproduce a task run on a different machine (installed packages,
|
||||
uncommitted changes etc.). The [ClearML Agent](../clearml_agent.md) listens to designated queues and when a task is
|
||||
enqueued, the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the
|
||||
task manager.
|
||||
@ -112,9 +112,9 @@ cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents:
|
||||
shuts down instances as needed, according to a resource budget that you set.
|
||||
|
||||
|
||||
### Reproducing Tasks
|
||||
### Reproducing Task Runs
|
||||
|
||||
ClearML logs all the information required to reproduce a task, but you may also want to change a few parameters
|
||||
ClearML logs all the information required to reproduce a task run, but you may also want to change a few parameters
|
||||
and task details when you re-run it, which you can do through ClearML's UI.
|
||||
|
||||
In order to be able to override parameters via the UI,
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
---
|
||||
title: Reproducing Tasks
|
||||
title: Reproducing Task Runs
|
||||
---
|
||||
|
||||
Reproduce tasks on local or remote machines in one of the following ways:
|
||||
Reproducing task runs on local or remote machines in one of the following ways:
|
||||
* Cloning any task - Make an exact copy, while maintaining the original task
|
||||
* Resetting a task whose status is not *Published* - Delete the previous run's logs and output
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user