+
|Name | Description| Mandatory |
|---|----|---|
|`--id`| Build a worker environment for this Task ID.|

|
@@ -49,6 +51,8 @@ clearml-agent build [-h] --id TASK_ID [--target TARGET]
|`-O`| Compile optimized pyc code (see [python documentation](https://docs.python.org/3/using/cmdline.html#cmdoption-O)). Repeat for more optimization.|

|
|`--target`| The target folder for the virtual environment and source code that will be used at launch.|

|
+
+
## config
List your ClearML Agent configuration.
@@ -59,7 +63,7 @@ clearml-agent config [-h]
## daemon
Use the `daemon` command to spin up an agent on any machine: on-prem and/or cloud instance. When spinning up an agent,
-assign it a queue(s) to service, and when experiments are added to its queues, the agent will pull and execute them.
+assign it a queue(s) to service, and when tasks are added to its queues, the agent will pull and execute them.
With the `daemon` command you can configure your agent's behavior: allocate resources, prioritize queues, set it to run
in a Docker, and more.
@@ -80,6 +84,8 @@ clearml-agent daemon [-h] [--foreground] [--queue QUEUES [QUEUES ...]] [--order-
### Parameters
+
+
|Name | Description| Mandatory |
|---|----|---|
|`--child-report-tags`| List of tags to send with the status reports from the worker that executes a task.|

|
@@ -106,6 +112,8 @@ clearml-agent daemon [-h] [--foreground] [--queue QUEUES [QUEUES ...]] [--order-
|`--uptime`| Specify uptime for clearml-agent in `
` format. For example, use `17-20 TUE` to set Tuesday's uptime to 17-20.
NOTES:- This feature is available under the ClearML Enterprise plan
- Make sure to configure only `--uptime` or `--downtime`, but not both.
|
|
|`--use-owner-token`| Generate and use the task owner's token for the execution of the task.|
|
+
+
## execute
Use the `execute` command to set an agent to build and execute specific tasks directly without listening to a queue.
@@ -123,6 +131,8 @@ clearml-agent execute [-h] --id TASK_ID [--log-file LOG_FILE] [--disable-monitor
### Parameters
+
+
|Name | Description| Mandatory |
|---|----|---|
|`--id`| The ID of the Task to build|

|
@@ -141,6 +151,8 @@ clearml-agent execute [-h] --id TASK_ID [--log-file LOG_FILE] [--disable-monitor
|`--require-queue`| If the specified task is not queued, the execution will fail (used for 3rd party scheduler integration, e.g. K8s, SLURM, etc.)|

|
|`--standalone-mode`| Do not use any network connects, assume everything is pre-installed|

|
+
+
## list
List information about all active workers.
diff --git a/docs/clearml_agent/clearml_agent_setup.md b/docs/clearml_agent/clearml_agent_setup.md
index 2f460f19..2e744de0 100644
--- a/docs/clearml_agent/clearml_agent_setup.md
+++ b/docs/clearml_agent/clearml_agent_setup.md
@@ -9,7 +9,7 @@ If ClearML was previously configured, follow [this](#adding-clearml-agent-to-a-c
ClearML Agent specific configurations
:::
-To install ClearML Agent, execute
+To install [ClearML Agent](../clearml_agent.md), execute
```bash
pip install clearml-agent
```
@@ -27,7 +27,7 @@ it can't do that when running from a virtual environment.
clearml-agent init
```
- The setup wizard prompts for ClearML credentials (see [here](../webapp/settings/webapp_settings_profile.md#clearml-credentials) about obtaining credentials).
+ The setup wizard prompts for ClearML credentials (see [here](../webapp/settings/webapp_settings_profile.md#clearml-api-credentials) about obtaining credentials).
```
Please create new clearml credentials through the settings page in your `clearml-server` web app,
or create a free account at https://app.clear.ml/settings/webapp-configuration
@@ -146,7 +146,7 @@ In case a `clearml.conf` file already exists, add a few ClearML Agent specific c
worker_id: ""
}
```
- View a complete ClearML Agent configuration file sample including an `agent` section [here](https://github.com/allegroai/clearml-agent/blob/master/docs/clearml.conf).
+ View a complete ClearML Agent configuration file sample including an `agent` section [here](https://github.com/clearml/clearml-agent/blob/master/docs/clearml.conf).
1. Save the configuration.
diff --git a/docs/clearml_data/clearml_data.md b/docs/clearml_data/clearml_data.md
index cd7f3093..ca3bc274 100644
--- a/docs/clearml_data/clearml_data.md
+++ b/docs/clearml_data/clearml_data.md
@@ -11,10 +11,11 @@ In Machine Learning, you are very likely dealing with a gargantuan amount of dat
which you then need to be able to share, reproduce, and track.
ClearML Data Management solves two important challenges:
-- Accessibility - Making data easily accessible from every machine,
-- Versioning - Linking data and experiments for better **traceability**.
+- Accessibility - Making data easily accessible from every machine
+- Versioning - Linking data and tasks for better **traceability**.
-
+
+
**We believe Data is not code**. It should not be stored in a git tree, because progress on datasets is not always linear.
Moreover, it can be difficult and inefficient to find on a git tree the commit associated with a certain version of a dataset.
@@ -36,16 +37,16 @@ lineage and content information. See [dataset UI](../webapp/datasets/webapp_data
## Setup
-`clearml-data` comes built-in with the `clearml` python package! Check out the [Getting Started](../getting_started/ds/ds_first_steps.md)
+`clearml-data` comes built-in with the `clearml` Python package! Check out the [ClearML Setup](../clearml_sdk/clearml_sdk_setup)
guide for more info!
## Using ClearML Data
ClearML Data supports two interfaces:
- `clearml-data` - A CLI utility for creating, uploading, and managing datasets. See [CLI](clearml_data_cli.md) for a reference of `clearml-data` commands.
-- `clearml.Dataset` - A python interface for creating, retrieving, managing, and using datasets. See [SDK](clearml_data_sdk.md) for an overview of the basic methods of the `Dataset` module.
+- `clearml.Dataset` - A Python interface for creating, retrieving, managing, and using datasets. See [SDK](clearml_data_sdk.md) for an overview of the basic methods of the `Dataset` module.
-For an overview of recommendations for ClearML Data workflows and practices, see [Best Practices](best_practices.md).
+For an overview of recommendations for ClearML Data workflows and practices, see [Best Practices](../best_practices/data_best_practices.md).
## Dataset Version States
The following table displays the possible states for a dataset version.
diff --git a/docs/clearml_data/clearml_data_cli.md b/docs/clearml_data/clearml_data_cli.md
index bc205cf8..d1517e2f 100644
--- a/docs/clearml_data/clearml_data_cli.md
+++ b/docs/clearml_data/clearml_data_cli.md
@@ -7,7 +7,7 @@ This page covers `clearml-data`, ClearML's file-based data management solution.
See [Hyper-Datasets](../hyperdatasets/overview.md) for ClearML's advanced queryable dataset management solution.
:::
-`clearml-data` is a data management CLI tool that comes as part of the `clearml` python package. Use `clearml-data` to
+`clearml-data` is a data management CLI tool that comes as part of the `clearml` Python package. Use `clearml-data` to
create, modify, and manage your datasets. You can upload your dataset to any storage service of your choice (S3 / GS /
Azure / Network Storage) by setting the dataset's upload destination (see [`--storage`](#upload)). Once you have uploaded
your dataset, you can access it from any machine.
@@ -107,7 +107,7 @@ Upload the local dataset changes to the server. By default, it's uploaded to the
medium by entering an upload destination. For example:
* A shared folder: `/mnt/shared/folder`
* S3: `s3://bucket/folder`
-* Non-AWS S3-like services (such as MinIO): `s3://host_addr:port/bucket`
+* Non-AWS S3-like services (such as MinIO): `s3://host_addr:port/bucket`. **Note that port specification is required**.
* Google Cloud Storage: `gs://bucket-name/folder`
* Azure Storage: `azure://