Fix links (#165)

This commit is contained in:
pollfly 2022-01-19 14:26:14 +02:00 committed by GitHub
parent b48dfe1d50
commit 70595e4f84
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
22 changed files with 48 additions and 46 deletions

View File

@ -39,7 +39,7 @@ VS Code remote sessions use ports 8878 and 8898 respectively.
## How it Works
ClearML allows to leverage a resource (e.g. GPU or CPU machine) by utilizing the [ClearML Agent](../clearml_agent).
ClearML allows to leverage a resource (e.g. GPU or CPU machine) by utilizing the [ClearML Agent](../clearml_agent.md).
A ClearML Agent runs on a target machine, and ClearML Session instructs it to execute the Jupyter / VS Code
server to develop remotely.
After entering a `clearml-session` command with all specifications:

View File

@ -56,11 +56,11 @@ This adds the downloaded files to the current dataset.
dataset.upload()
```
This uploads the dataset to the ClearML Server by default. The dataset's destination can be changed by specifying the
target storage with the `output_url` parameter of the [`upload`](../../references/sdk/dataset#upload) method.
target storage with the `output_url` parameter of the [`upload`](../../references/sdk/dataset.md#upload) method.
### Finalizing the Dataset
Run the [`finalize`](../../references/sdk/dataset#finzalize) command to close the dataset and set that dataset's tasks
Run the [`finalize`](../../references/sdk/dataset.md#finalize) command to close the dataset and set that dataset's tasks
status to *completed*. The dataset can only be finalized if it doesn't have any pending uploads.
```python

View File

@ -21,7 +21,7 @@ In conjunction with the **ClearML Hosted Service** (or self-hosted **ClearML Ser
* [StorageManager](references/sdk/storage.md) - The `storage` module contains the `storage.manager.StorageManager` class which provides support for downloading and uploading from storage, including folders, S3, Google Cloud Storage, Azure Storage, and http(s).
* [Dataset](references/sdk/dataset) - The `dataset` module contains classes that helps manage Dataset. Users can create, modify and delete datasets as well as retrieve them for use in their code
* [Dataset](references/sdk/dataset.md) - The `dataset` module contains classes that helps manage Dataset. Users can create, modify and delete datasets as well as retrieve them for use in their code
## Examples

View File

@ -32,7 +32,7 @@ by setting [configuration options](../configs/clearml_conf.md).
CLEARML_CONFIG_FILE = MyOtherClearML.conf
For more information about running experiments inside Docker containers, see [ClearML Agent Execution](../clearml_agent#execution)
For more information about running experiments inside Docker containers, see [ClearML Agent Execution](../clearml_agent.md#execution)
and [ClearML Agent Reference](../references/clearml_agent_ref.md).
</div>

View File

@ -3,35 +3,37 @@ title: ClearML Server
---
## What is ClearML Server?
The **ClearML Server** is the backend service infrastructure for **ClearML**. It allows multiple users to collaborate and manage their experiments by working seamlessly with the **ClearML Python Package** and [**ClearML Agent**](clearml_agent.md). **ClearML Server** is composed of the following:
The **ClearML Server** is the backend service infrastructure for ClearML. It allows multiple users to collaborate and
manage their experiments by working seamlessly with the ClearML Python package and [ClearML Agent](../clearml_agent.md).
* Web server including the **ClearML Web UI**, which is the user interface for tracking, comparing, and managing experiments.
* API server which a RESTful API for:
ClearML Server is composed of the following:
* Web server including the [ClearML Web UI](../webapp/webapp_overview.md), which is the user interface for tracking, comparing, and managing experiments.
* API server which is a RESTful API for:
* Documenting and logging experiments, including information, statistics, and results.
* Querying experiments history, logs, and results.
* File server which stores media and models making them easily accessible using the **ClearML Web UI**.
* File server which stores media and models making them easily accessible using the ClearML Web UI.
The [**ClearML Hosted Service**](https://app.community.clear.ml) is essentially the **ClearML Server** maintained for you.
The [**ClearML Hosted Service**](https://app.community.clear.ml) is essentially the ClearML Server maintained for you.
![image](../img/ClearML_Server_Diagram.png)
**ClearML Web UI** is the **ClearML** user interface and is part of **ClearML Server**.
The ClearML Web UI is the ClearML user interface and is part of ClearML Server.
Use the **ClearML Web UI** to:
Use the ClearML Web UI to:
* Track experiments
* Compare experiments
* Manage experiments
For detailed information about the **ClearML Web UI**, see [User Interface](../webapp/webapp_home.md).
For detailed information about the ClearML Web UI, see [User Interface](../webapp/webapp_overview.md).
ClearML Server also comes with a [services agent](../clearml_agent.md#services-mode) preinstalled.
## Deployment
The **ClearML Server** can be deployed in any of the formats listed below. Once deployed, configure the server for web login
The ClearML Server can be deployed in any of the formats listed below. Once deployed, configure the server for web login
authentication, sub-domains, and load balancers, and use any of its many configuration settings.
**To deploy your own ClearML Server:**

View File

@ -157,7 +157,7 @@ task = Task.init(project_name='examples', task_name='storing model', output_uri=
```
To automatically store all created models from all experiments in a certain storage medium, edit the `clearml.conf` (see
[ClearML Configuration Reference](../configs/clearml_conf#sdkdevelopment)) and set `sdk.developmenmt.default_output_uri` to the desired
[ClearML Configuration Reference](../configs/clearml_conf.md#sdkdevelopment)) and set `sdk.developmenmt.default_output_uri` to the desired
storage (see [Storage](../integrations/storage.md)).
This is especially helpful when using [clearml-agent](../clearml_agent.md) to execute code.

View File

@ -111,7 +111,7 @@ Relying on environment variables makes an experiment not fully reproducible, sin
runtime.
:::
Environment variables can be logged by modifying the [clearml.conf](../configs/clearml_conf) file. Modify the *log_os_environments*
Environment variables can be logged by modifying the [clearml.conf](../configs/clearml_conf.md) file. Modify the *log_os_environments*
parameter specifying parameters to log.
`log_os_environments: ["AWS_*", "CUDA_VERSION"]`

View File

@ -25,7 +25,7 @@ another Task.
### How Do Pipelines Work?
Users can automate [Tasks](task) to run consecutively or according to some logic by putting the tasks into a pipeline.
Users can automate [Tasks](task.md) to run consecutively or according to some logic by putting the tasks into a pipeline.
Tasks in a pipeline can leverage other tasks' work products such as artifacts and parameters.
Before running a pipeline, we need to configure a Controller Task, in which the pipeline is defined. Pipelines are made
@ -47,7 +47,7 @@ create customized, step-specific callbacks.
### Simple DAG Pipelines
For a simple, DAG based logic, use the off-the-shelf [`PipelineController`](../references/sdk/automation_controller_pipelinecontroller.md) class to define the DAG (see an example
[here](../guides/pipeline/pipeline_controller)). Once the `PipelineController` object is populated and configured,
[here](../guides/pipeline/pipeline_controller.md)). Once the `PipelineController` object is populated and configured,
we can start the pipeline, which will begin executing the steps in succession, then it waits until the pipeline is completed.
The pipeline control logic is processed in a background thread.

View File

@ -21,7 +21,7 @@ clearml-init
## Auto-log Experiment
In ClearML, experiments are organized as [Tasks](../../fundamentals/task).
In ClearML, experiments are organized as [Tasks](../../fundamentals/task.md).
ClearML will automatically log your experiment and code, including outputs and parameters from popular ML frameworks,
once you integrate the ClearML [SDK](../../clearml_sdk.md) with your code. To control what ClearML automatically logs, see this [FAQ](../../faq.md#controlling_logging).
@ -50,7 +50,7 @@ The console should return the following output:
```
ClearML Task: created new task id=1ca59ef1f86d44bd81cb517d529d9e5a
2021-07-25 13:59:09
ClearML results page: https://community/projects/4043a1657f374e9298649c6ba72ad233/experiments/1ca59ef1f86d44bd81cb517d529d9e5a/output/log
ClearML results page: https://app.community.clear.ml/projects/4043a1657f374e9298649c6ba72ad233/experiments/1ca59ef1f86d44bd81cb517d529d9e5a/output/log
2021-07-25 13:59:16
```

View File

@ -139,7 +139,7 @@ It's possible to log everything, from time series data to confusion matrices to
![image](../../img/report_plotly.png)
Once everything is neatly logged and displayed, using the [comparison tool](../../webapp/webapp_exp_comparing) makes it easy to find the best configuration!
Once everything is neatly logged and displayed, using the [comparison tool](../../webapp/webapp_exp_comparing.md) makes it easy to find the best configuration!
## Track Experiments

View File

@ -7,7 +7,7 @@ From training models to data processing to deploying to production.
## Development - Preparing for Automation
Basically, track everything. There is nothing that is not worth having visibility to.
If you are afraid of clutter, use the archive option, and set up your own [cleanup service](../../guides/services/cleanup_service).
If you are afraid of clutter, use the archive option, and set up your own [cleanup service](../../guides/services/cleanup_service.md).
- Track the code base. There is no reason not to add metrics to any process in your workflow, even if it is not directly ML. Visibility is key to iterative improvement of your code / workflow.
- Create per-project [leaderboards](../../guides/ui/building_leader_board.md) based on custom columns

View File

@ -56,11 +56,11 @@ This adds the downloaded files to the current dataset.
dataset.upload()
```
This uploads the dataset to the ClearML Server by default. The dataset's destination can be changed by specifying the
target storage with the `output_url` parameter of the [`upload`](../../references/sdk/dataset#upload) method.
target storage with the `output_url` parameter of the [`upload`](../../references/sdk/dataset.md#upload) method.
### Finalizing the Dataset
Run the [`finalize`](../../references/sdk/dataset#finzalize) command to close the dataset and set that dataset's tasks
Run the [`finalize`](../../references/sdk/dataset.md#finalize) command to close the dataset and set that dataset's tasks
status to *completed*. The dataset can only be finalized if it doesn't have any pending uploads.
```python

View File

@ -14,7 +14,7 @@ dataset), and reports (uploads) the following to the main Task:
* Scalars - Loss reported as a scalar during training in each Task in a subprocess.
* Hyperparameters - Hyperparameters created in each Task are added to the hyperparameters in the main Task.
Each Task in a subprocess references the main Task by calling [Task.current_task](../../references/sdk/task#taskcurrent_task), which always returns
Each Task in a subprocess references the main Task by calling [Task.current_task](../../references/sdk/task.md#taskcurrent_task), which always returns
the main Task.
When the script runs, it creates an experiment named `test torch distributed`, which is associated with the `examples` project.
@ -38,7 +38,7 @@ All of these artifacts appear in the main Task under **ARTIFACTS** **>** **OTHER
## Scalars
Loss is reported to the main Task by calling the [Logger.report_scalar](../../references/sdk/logger#report_scalar)
Loss is reported to the main Task by calling the [Logger.report_scalar](../../references/sdk/logger.md#report_scalar)
method on `Task.current_task().get_logger`, which is the logger for the main Task. Since `Logger.report_scalar` is called
with the same title (`loss`), but a different series name (containing the subprocess' `rank`), all loss scalar series are
logged together.
@ -58,7 +58,7 @@ The single scalar plot for loss appears in **RESULTS** **>** **SCALARS**.
## Hyperparameters
**ClearML** automatically logs the argparse command line options. Since the [`Task.connect`](../../references/sdk/task#connect)
**ClearML** automatically logs the argparse command line options. Since the [`Task.connect`](../../references/sdk/task.md#connect)
method is called on [`Task.current_task`](../../references/sdk/task.md#taskcurrent_task), they are logged in the main Task. A different hyperparameter key is used in each
subprocess, so they do not overwrite each other in the main Task.

View File

@ -5,7 +5,7 @@ title: Subprocess
The [subprocess_example.py](https://github.com/allegroai/clearml/blob/master/examples/distributed/subprocess_example.py)
script demonstrates multiple subprocesses interacting and reporting to a main Task. The following happens in the script:
* This script initializes a main Task and spawns subprocesses, each for an instances of that Task.
* Each Task in a subprocess references the main Task by calling [Task.current_task](../../references/sdk/task#taskcurrent_task),
* Each Task in a subprocess references the main Task by calling [Task.current_task](../../references/sdk/task.md#taskcurrent_task),
which always returns the main Task.
* The Task in each subprocess reports the following to the main Task:
* Hyperparameters - Additional, different hyperparameters.
@ -15,7 +15,7 @@ which always returns the main Task.
## Hyperparameters
ClearML automatically logs the command line options defined with `argparse`. A parameter dictionary is logged by
connecting it to the Task using a call to the [Task.connect](../../references/sdk/task#connect) method.
connecting it to the Task using a call to the [Task.connect](../../references/sdk/task.md#connect) method.
```python
additional_parameters = {

View File

@ -6,7 +6,7 @@ The [html_reporting.py](https://github.com/allegroai/clearml/blob/master/example
demonstrates reporting local HTML files and HTML by URL, using the [Logger.report_media](../../references/sdk/logger.md#report_media)
method.
**ClearML** reports these HTML debug samples in the **ClearML Web UI** **>** experiment details **>** **RESULTS** tab **>**
ClearML reports these HTML debug samples in the **ClearML Web UI** **>** experiment details **>** **RESULTS** tab **>**
**DEBUG SAMPLES** sub-tab.
When the script runs, it creates an experiment named `html samples reporting`, which is associated with the `examples` project.
@ -18,7 +18,7 @@ When the script runs, it creates an experiment named `html samples reporting`, w
Report HTML by URL, using the [Logger.report_media](../../references/sdk/logger.md#report_media) method's `url` parameter.
See the example script's [report_html_url](https://github.com/allegroai/clearml/blob/master/examples/reporting/html_reporting.py#L16)
function, which reports the **ClearML** documentation's home page.
function, which reports the ClearML documentation's home page.
```python
Logger.current_logger().report_media("html", "url_html", iteration=iteration, url="https://clear.ml/docs")

View File

@ -10,7 +10,7 @@ method to upload from:
* URL of media already uploaded to some storage
**ClearML** uploads media to the bucket specified in the **ClearML** configuration file or **ClearML** can be configured for image storage, see [Logger.set_default_upload_destination](../../references/sdk/logger.md#set_default_upload_destination)
(storage for [artifacts](../../fundamentals/artifacts#setting-upload-destination) is different). Set credentials for storage in the **ClearML**
(storage for [artifacts](../../fundamentals/artifacts.md#setting-upload-destination) is different). Set credentials for storage in the **ClearML**
[configuration file](../../configs/clearml_conf.md).
**ClearML** reports media in the **ClearML Web UI** **>** experiment details **>** **RESULTS** tab **>** **DEBUG SAMPLES**

View File

@ -104,7 +104,7 @@ title: Version 0.10
* Add simple username / password login scheme, see [GitHub trains-server Issue 3](https://github.com/allegroai/trains-server/issues/3)
* Fixed User/Password configuration (see [instructions](../deploying_clearml/clearml_server_config#web-login-authentication)).
* Fixed User/Password configuration (see [instructions](../deploying_clearml/clearml_server_config.md#web-login-authentication)).
* Replace the Python web-server with NGINX.

View File

@ -202,7 +202,7 @@ title: Version 0.16
### Trains Server
:::important
Upgrading to this version requires a manual [data migration](../deploying_clearml/clearml_server_es7_migration/).
Upgrading to this version requires a manual [data migration](../deploying_clearml/clearml_server_es7_migration.md).
:::
**Bug Fixes**

View File

@ -9,7 +9,7 @@ to monitor experimentation, and more). The experiments table's auto-refresh allo
:::info
To assist in focusing on active experimentation, experiments and models can be archived, so they will not appear
in the active experiments and models tables. See [Archiving](webapp_archiving).
in the active experiments and models tables. See [Archiving](webapp_archiving.md).
:::
![Experiment table](../img/webapp_exp_table_01.png)
@ -23,7 +23,7 @@ The experiments table default and customizable columns are described in the foll
| **TYPE** | Type of experiment. **ClearML** supports multiple [task types](../fundamentals/task.md#task-types) for experimentation, and a variety of workflows and use cases. | Default |
| **NAME** | Experiment name. | Default |
| **TAGS** | Descriptive, user-defined, color-coded tags assigned to experiments. Use tags to classify experiments, and filter the list. See [tagging experiments](webapp_exp_track_visual#tagging-experiments). | Default |
| **STATUS** | Experiment state (status). See a list of the [task states and state transtions](../fundamentals/task.md#task-states). | Default |
| **STATUS** | Experiment state (status). See a list of the [task states and state transitions](../fundamentals/task.md#task-states). | Default |
| **PROJECT** | Name of experiment's project. | Default |
| **USER** | User who created or cloned the experiment. | Default (hidden) |
| **STARTED** | Elapsed time since the experiment started. To view the date and time of start, hover over the elapsed time. | Default |
@ -131,21 +131,21 @@ that allow each operation.
| Action | Description | States Valid for the Action | State Transition |
|---|---|---|---|
| View details | View experiment details in the experiments table, the [info panel](webapp_exp_track_visual#info-panel) (keep the experiments table in view), or the [full screen details view](webapp_exp_track_visual#full-screen-details-view). | Any state | None |
| Manage a queue | If an experiment is *Pending* in a queue, view the utilization of that queue, manage that queue (remove experiments and change the order of experiments), and view information about the worker(s) listening to the queue. See the [Workers and Queues](webapp_workers_queues) page. | *Enqueued* | None |
| View details | View experiment details in the experiments table, the [info panel](webapp_exp_track_visual.md#info-panel) (keep the experiments table in view), or the [full screen details view](webapp_exp_track_visual.md#full-screen-details-view). | Any state | None |
| Manage a queue | If an experiment is *Pending* in a queue, view the utilization of that queue, manage that queue (remove experiments and change the order of experiments), and view information about the worker(s) listening to the queue. See the [Workers and Queues](webapp_workers_queues.md) page. | *Enqueued* | None |
| View a worker | If an experiment is *Running*, view resource utilization, worker details, and queues to which a worker is listening. | *Running* | None |
| Share | For **ClearML Hosted Service** users only, [share](webapp_exp_sharing) an experiment and its model with a **ClearML Hosted Service** user in another workspace. | Any state | None |
| Archive | To more easily work with active experiments, move an experiment to the archive. See [Archiving](webapp_archiving). | Any state | None |
| Share | For **ClearML Hosted Service** users only, [share](webapp_exp_sharing.md) an experiment and its model with a **ClearML Hosted Service** user in another workspace. | Any state | None |
| Archive | To more easily work with active experiments, move an experiment to the archive. See [Archiving](webapp_archiving.md). | Any state | None |
| Enqueue | Add an experiment to a queue for a worker or workers (listening to the queue) to execute. | *Draft* | *Pending* |
| Dequeue | Remove an experiment from a queue. | *Pending* | *Draft* |
| Reset | Delete the log and output from a previous run of an experiment (for example, before rerunning it). | *Completed*, *Aborted*, or *Failed* | *Draft* |
| Abort | Manually terminate a *Running* experiment. | *Running* | *Aborted* |
| Abort All Children | Manually terminate all *Running* experiments which have this task as a parent | *Running* or *Aborted* | None for parent experiment, *Aborted* for child experiments |
| Publish | Publish an experiment to prevent changes to its tracking data, inputs, and outputs. Published experiments and their models are read-only. *Published* experiments cannot be enqueued, but they can be cloned, and their clones can be edited, tuned, and enqueued. | *Completed*, *Aborted*, or *Failed*. | *Published* |
| Tags | Tag experiments with color-coded labels to assist you in organizing your work. See [tagging experiments](webapp_exp_track_visual#tagging-experiments). | Any state | None |
| Tags | Tag experiments with color-coded labels to assist you in organizing your work. See [tagging experiments](webapp_exp_track_visual.md#tagging-experiments). | Any state | None |
| Clone | Make an exact, editable copy of an experiment (for example, to reproduce an experiment, but keep the original). | *Draft* | Newly Cloned Experiment is *Draft* |
| Move | Move an experiment to another project. | Any state | None |
| Custom action | The ClearML Enterprise Server provides a mechanism to define your own custom actions, which will appear in the context menu. See [Custom UI Context Menu Actions](../deploying_clearml/clearml_server_config#custom-ui-context-menu-actions). | Any State | None |
| Custom action | The ClearML Enterprise Server provides a mechanism to define your own custom actions, which will appear in the context menu. See [Custom UI Context Menu Actions](../deploying_clearml/clearml_server_config.md#custom-ui-context-menu-actions). | Any State | None |
These actions can be accessed with the context menu (when right-clicking an experiment or clicking the menu button <img src="/docs/latest/icons/ico-bars-menu.svg" alt="Menu" className="icon size-md space-sm" />
in an experiment's info panel).

View File

@ -77,7 +77,7 @@ Additionally, there is information about the ClearML Agent configuration. The Cl
Docker that ClearML Agent will use to remotely execute this experiment (see [Building Docker containers](../clearml_agent.md#exporting-a-task-into-a-standalone-docker-container)).
The output details include:
* The output destination used for storing model checkpoints (snapshots) and artifacts (see also, [default_output_uri](../configs/clearml_conf#config_default_output_uri)
* The output destination used for storing model checkpoints (snapshots) and artifacts (see also, [default_output_uri](../configs/clearml_conf.md#config_default_output_uri)
in the configuration file, and [output_uri](../references/sdk/task.md#taskinit)
in `Task.init` parameters).
@ -145,7 +145,7 @@ The **Args** parameter group shows automatically logged `argparse` arguments, an
#### Environment Variables
If the `CLEARML_LOG_ENVIRONMENT` variable was set, the **Environment** group will show environment variables (see [this FAQ](../faq#track-env-vars)).
If the `CLEARML_LOG_ENVIRONMENT` variable was set, the **Environment** group will show environment variables (see [this FAQ](../faq.md#track-env-vars)).
<details className="cml-expansion-panel screenshot">
<summary className="cml-expansion-panel-summary">View a screenshot</summary>

View File

@ -132,7 +132,7 @@ Add, change, or delete hyperparameters, which are organized in the **ClearML Web
* **General** - Parameter dictionaries (from code, connected to the Task by calling the [Task.connect](../references/sdk/task.md#connect)
method).
* Environment variables - Tracked if the `CLEARML_LOG_ENVIRONMENT` environment variable was set (see this [FAQ](../faq#track-env-vars)).
* Environment variables - Tracked if the `CLEARML_LOG_ENVIRONMENT` environment variable was set (see this [FAQ](../faq.md#track-env-vars)).
* Custom named parameter groups (see the `name` parameter in [Task.connect](../references/sdk/task.md#connect)).

View File

@ -60,7 +60,7 @@ allow each feature. Model states are *Draft* (editable) and *Published* (read-on
|---|---|--|
| View details | Model details include general information, the model configuration, and label enumeration. Click a model, and the info panel slides open. | Any state |
| Publish | Publish a model to prevent changes to it. *Published* models are read-only. If a model is Published, its experiment also becomes Published (read-only). | *Draft* |
| Archive | To more easily work with active models, move a model to the archive. See [Archiving](webapp_archiving). | Any state |
| Archive | To more easily work with active models, move a model to the archive. See [Archiving](webapp_archiving.md). | Any state |
| Tags | Tag models with color-coded labels to assist in organizing work. See [tagging models](#tagging-models). | Any state |
| Download | Download a model. The file format depends upon the framework. | *Published* |
| Move to project | To organize work and improve collaboration, move a model to another project. | Any state |