ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-04-26 01:09:38 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Split PipelineController and PipelineDecorator reference pages (#1063)
Some checks are pending
CI / build (push) Waiting to run
Details

Browse Source
This commit is contained in:
pollfly 2025-03-02 11:54:30 +02:00 committed by GitHub
parent 3829d1f4b7
commit 6f8d8601f0
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
6 changed files with 29 additions and 22 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
docs/guides/pipeline/pipeline_decorator.md
View File

@ -3,7 +3,7 @@ title: Pipeline from Decorators
--- ---
The [pipeline_from_decorator.py](https://github.com/clearml/clearml/blob/master/examples/pipeline/pipeline_from_decorator.py) The [pipeline_from_decorator.py](https://github.com/clearml/clearml/blob/master/examples/pipeline/pipeline_from_decorator.py)
example demonstrates the creation of a pipeline in ClearML using the [`PipelineDecorator`](../../references/sdk/automation_controller_pipelinecontroller.md#class-automationcontrollerpipelinedecorator) example demonstrates the creation of a pipeline in ClearML using the [`PipelineDecorator`](../../references/sdk/automation_controller_pipelinedecorator.md#class-automationcontrollerpipelinedecorator)
class. class.
This example creates a pipeline incorporating four tasks, each of which is created from a Python function using a custom decorator: This example creates a pipeline incorporating four tasks, each of which is created from a Python function using a custom decorator:
@ -14,11 +14,11 @@ This example creates a pipeline incorporating four tasks, each of which is creat
* `step_four` - Uses data from `step_two` and the model from `step_three` to make a prediction. * `step_four` - Uses data from `step_two` and the model from `step_three` to make a prediction.
The pipeline steps, defined in the `step_one`, `step_two`, `step_three`, and `step_four` functions, are each wrapped with the The pipeline steps, defined in the `step_one`, `step_two`, `step_three`, and `step_four` functions, are each wrapped with the
[`@PipelineDecorator.component`](../../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorcomponent) [`@PipelineDecorator.component`](../../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorcomponent)
decorator, which creates a ClearML pipeline step for each one when the pipeline is executed. decorator, which creates a ClearML pipeline step for each one when the pipeline is executed.
The logic that executes these steps and controls the interaction between them is implemented in the `executing_pipeline` The logic that executes these steps and controls the interaction between them is implemented in the `executing_pipeline`
function. This function is wrapped with the [`@PipelineDecorator.pipeline`](../../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorpipeline) function. This function is wrapped with the [`@PipelineDecorator.pipeline`](../../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorpipeline)
decorator which creates the ClearML pipeline task when it is executed. decorator which creates the ClearML pipeline task when it is executed.
The sections below describe in more detail what happens in the pipeline controller and steps. The sections below describe in more detail what happens in the pipeline controller and steps.
@ -28,7 +28,7 @@ The sections below describe in more detail what happens in the pipeline controll
In this example, the pipeline controller is implemented by the `executing_pipeline` function. In this example, the pipeline controller is implemented by the `executing_pipeline` function.
Using the `@PipelineDecorator.pipeline` decorator creates a ClearML Controller Task from the function when it is executed. Using the `@PipelineDecorator.pipeline` decorator creates a ClearML Controller Task from the function when it is executed.
For detailed information, see [`@PipelineDecorator.pipeline`](../../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorpipeline). For detailed information, see [`@PipelineDecorator.pipeline`](../../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorpipeline).
In the example script, the controller defines the interactions between the pipeline steps in the following way: In the example script, the controller defines the interactions between the pipeline steps in the following way:
1. The controller function passes its argument, `pickle_url`, to the pipeline's first step (`step_one`) 1. The controller function passes its argument, `pickle_url`, to the pipeline's first step (`step_one`)
@ -39,13 +39,13 @@ In the example script, the controller defines the interactions between the pipel
:::info Local Execution :::info Local Execution
In this example, the pipeline is set to run in local mode by using In this example, the pipeline is set to run in local mode by using
[`PipelineDecorator.run_locally()`](../../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorrun_locally) [`PipelineDecorator.run_locally()`](../../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorrun_locally)
before calling the pipeline function. See pipeline execution options [here](../../pipelines/pipelines_sdk_function_decorators.md#running-the-pipeline). before calling the pipeline function. See pipeline execution options [here](../../pipelines/pipelines_sdk_function_decorators.md#running-the-pipeline).
::: :::
## Pipeline Steps ## Pipeline Steps
Using the `@PipelineDecorator.component` decorator will make the function a pipeline component that can be called from the Using the `@PipelineDecorator.component` decorator will make the function a pipeline component that can be called from the
pipeline controller, which implements the pipeline's execution logic. For detailed information, see [`@PipelineDecorator.component`](../../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorcomponent). pipeline controller, which implements the pipeline's execution logic. For detailed information, see [`@PipelineDecorator.component`](../../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorcomponent).
When the pipeline controller calls a pipeline step, a corresponding ClearML task will be created. Notice that all package When the pipeline controller calls a pipeline step, a corresponding ClearML task will be created. Notice that all package
imports inside the function will be automatically logged as required packages for the pipeline execution step. imports inside the function will be automatically logged as required packages for the pipeline execution step.
@ -63,7 +63,7 @@ executing_pipeline(
``` ```
By default, the pipeline controller and the pipeline steps are launched through ClearML [queues](../../fundamentals/agents_and_queues.md#what-is-a-queue). By default, the pipeline controller and the pipeline steps are launched through ClearML [queues](../../fundamentals/agents_and_queues.md#what-is-a-queue).
Use the [`PipelineDecorator.set_default_execution_queue`](../../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorset_default_execution_queue) Use the [`PipelineDecorator.set_default_execution_queue`](../../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorset_default_execution_queue)
method to specify the execution queue of all pipeline steps. The `execution_queue` parameter of the `@PipelineDecorator.component` method to specify the execution queue of all pipeline steps. The `execution_queue` parameter of the `@PipelineDecorator.component`
decorator overrides the default queue value for the specific step for which it was specified. decorator overrides the default queue value for the specific step for which it was specified.

24
docs/pipelines/pipelines_sdk_function_decorators.md
View File

@ -4,14 +4,14 @@ title: PipelineDecorator
## Creating Pipelines Using Function Decorators ## Creating Pipelines Using Function Decorators
Use the [`PipelineDecorator`](../references/sdk/automation_controller_pipelinecontroller.md#class-automationcontrollerpipelinedecorator) Use the [`PipelineDecorator`](../references/sdk/automation_controller_pipelinedecorator.md#class-automationcontrollerpipelinedecorator)
class to create pipelines from your existing functions. Use [`@PipelineDecorator.component`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorcomponent) class to create pipelines from your existing functions. Use [`@PipelineDecorator.component`](../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorcomponent)
to denote functions that comprise the steps of your pipeline, and [`@PipelineDecorator.pipeline`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorpipeline) to denote functions that comprise the steps of your pipeline, and [`@PipelineDecorator.pipeline`](../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorpipeline)
for your main pipeline execution logic function. for your main pipeline execution logic function.
## @PipelineDecorator.pipeline ## @PipelineDecorator.pipeline
Using the [`@PipelineDecorator.pipeline`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorpipeline) Using the [`@PipelineDecorator.pipeline`](../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorpipeline)
decorator transforms the function which implements your pipeline's execution logic to a ClearML pipeline controller, decorator transforms the function which implements your pipeline's execution logic to a ClearML pipeline controller,
an independently executed task. an independently executed task.
@ -70,13 +70,13 @@ parameters. When launching a new pipeline run from the [UI](../webapp/pipelines/
![Pipeline new run](../img/pipelines_new_run.png) ![Pipeline new run](../img/pipelines_new_run.png)
## @PipelineDecorator.component ## @PipelineDecorator.component
Using the [`@PipelineDecorator.component`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorcomponent) Using the [`@PipelineDecorator.component`](../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorcomponent)
decorator transforms a function into a ClearML pipeline step when called from a pipeline controller. decorator transforms a function into a ClearML pipeline step when called from a pipeline controller.
When the pipeline controller calls a pipeline step, a corresponding ClearML task is created. When the pipeline controller calls a pipeline step, a corresponding ClearML task is created.
:::tip Package Imports :::tip Package Imports
In the case that the `skip_global_imports` parameter of [`@PipelineDecorator.pipeline`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorpipeline) In the case that the `skip_global_imports` parameter of [`@PipelineDecorator.pipeline`](../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorpipeline)
is set to `False`, all global imports will be automatically imported at the beginning of each step's execution. is set to `False`, all global imports will be automatically imported at the beginning of each step's execution.
Otherwise, if set to `True`, make sure that each function which makes up a pipeline step contains package imports, which Otherwise, if set to `True`, make sure that each function which makes up a pipeline step contains package imports, which
are automatically logged as required packages for the pipeline execution step. are automatically logged as required packages for the pipeline execution step.
@ -110,7 +110,7 @@ def step_one(pickle_data_url: str, extra: int = 43):
* `packages` - A list of required packages or a local requirements.txt file. Example: `["tqdm>=2.1", "scikit-learn"]` or * `packages` - A list of required packages or a local requirements.txt file. Example: `["tqdm>=2.1", "scikit-learn"]` or
`"./requirements.txt"`. If not provided, packages are automatically added based on the imports used inside the function. `"./requirements.txt"`. If not provided, packages are automatically added based on the imports used inside the function.
* `execution_queue` (optional) - Queue in which to enqueue the specific step. This overrides the queue set with the * `execution_queue` (optional) - Queue in which to enqueue the specific step. This overrides the queue set with the
[`PipelineDecorator.set_default_execution_queue method`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorset_default_execution_queue) [`PipelineDecorator.set_default_execution_queue method`](../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorset_default_execution_queue)
method. method.
* `continue_on_fail` - If `True`, a failed step does not cause the pipeline to stop (or marked as failed). Notice, that * `continue_on_fail` - If `True`, a failed step does not cause the pipeline to stop (or marked as failed). Notice, that
steps that are connected (or indirectly connected) to the failed step are skipped (default `False`) steps that are connected (or indirectly connected) to the failed step are skipped (default `False`)
@ -186,14 +186,14 @@ specify which frameworks to log. See `Task.init`'s [`auto_connect_framework` par
* `auto_connect_arg_parser` - Control automatic logging of argparse objects. See `Task.init`'s [`auto_connect_arg_parser` parameter](../references/sdk/task.md#taskinit) * `auto_connect_arg_parser` - Control automatic logging of argparse objects. See `Task.init`'s [`auto_connect_arg_parser` parameter](../references/sdk/task.md#taskinit)
You can also directly upload a model or an artifact from the step to the pipeline controller, using the You can also directly upload a model or an artifact from the step to the pipeline controller, using the
[`PipelineDecorator.upload_model`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorupload_model) [`PipelineDecorator.upload_model`](../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorupload_model)
and [`PipelineDecorator.upload_artifact`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorupload_artifact) and [`PipelineDecorator.upload_artifact`](../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorupload_artifact)
methods respectively. methods respectively.
## Controlling Pipeline Execution ## Controlling Pipeline Execution
### Default Execution Queue ### Default Execution Queue
The [`PipelineDecorator.set_default_execution_queue`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorset_default_execution_queue) The [`PipelineDecorator.set_default_execution_queue`](../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorset_default_execution_queue)
method lets you set a default queue through which all pipeline steps method lets you set a default queue through which all pipeline steps
will be executed. Once set, step-specific overrides can be specified through the `@PipelineDecorator.component` decorator. will be executed. Once set, step-specific overrides can be specified through the `@PipelineDecorator.component` decorator.
@ -226,7 +226,7 @@ You can run the pipeline logic locally, while keeping the pipeline components ex
#### Debugging Mode #### Debugging Mode
In debugging mode, the pipeline controller and all components are treated as regular Python functions, with components In debugging mode, the pipeline controller and all components are treated as regular Python functions, with components
called synchronously. This mode is great to debug the components and design the pipeline as the entire pipeline is called synchronously. This mode is great to debug the components and design the pipeline as the entire pipeline is
executed on the developer machine with full ability to debug each function call. Call [`PipelineDecorator.debug_pipeline`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratordebug_pipeline) executed on the developer machine with full ability to debug each function call. Call [`PipelineDecorator.debug_pipeline`](../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratordebug_pipeline)
before the main pipeline logic function call. before the main pipeline logic function call.
Example: Example:
@ -242,7 +242,7 @@ In local mode, the pipeline controller creates Tasks for each component, and com
into sub-processes running on the same machine. Notice that the data is passed between the components and the logic with into sub-processes running on the same machine. Notice that the data is passed between the components and the logic with
the exact same mechanism as in the remote mode (i.e. hyperparameters / artifacts), with the exception that the execution the exact same mechanism as in the remote mode (i.e. hyperparameters / artifacts), with the exception that the execution
itself is local. Notice that each subprocess is using the exact same Python environment as the main pipeline logic. Call itself is local. Notice that each subprocess is using the exact same Python environment as the main pipeline logic. Call
[`PipelineDecorator.run_locally`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorrun_locally) [`PipelineDecorator.run_locally`](../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorrun_locally)
before the main pipeline logic function. before the main pipeline logic function.
Example: Example:

5
docs/references/sdk/automation_controller_pipelinedecorator.md Normal file
View File

@ -0,0 +1,5 @@
---
title: PipelineDecorator
---
**AutoGenerated PlaceHolder**

2
docs/webapp/pipelines/webapp_pipeline_table.md
View File

@ -36,7 +36,7 @@ The pipeline run table contains the following columns:
| Column | Description | Type | | Column | Description | Type |
|---|---|---| |---|---|---|
| **RUN** | Pipeline run identifier | String | | **RUN** | Pipeline run identifier | String |
| **VERSION** | The pipeline version number. Corresponds to the [PipelineController](../../references/sdk/automation_controller_pipelinecontroller.md#class-pipelinecontroller)'s and [PipelineDecorator](../../references/sdk/automation_controller_pipelinecontroller.md#class-automationcontrollerpipelinedecorator)'s `version` parameter | Version string | | **VERSION** | The pipeline version number. Corresponds to the [PipelineController](../../references/sdk/automation_controller_pipelinecontroller.md#class-pipelinecontroller)'s and [PipelineDecorator](../../references/sdk/automation_controller_pipelinedecorator.md#class-automationcontrollerpipelinedecorator)'s `version` parameter | Version string |
| **TAGS** | Descriptive, user-defined, color-coded tags assigned to run. | Tag | | **TAGS** | Descriptive, user-defined, color-coded tags assigned to run. | Tag |
| **STATUS** | Pipeline run's status. See a list of the [task states and state transitions](../../fundamentals/task.md#task-states). For Running, Failed, and Aborted runs, you will also see a progress indicator next to the status. See [here](../../pipelines/pipelines.md#tracking-pipeline-progress). | String | | **STATUS** | Pipeline run's status. See a list of the [task states and state transitions](../../fundamentals/task.md#task-states). For Running, Failed, and Aborted runs, you will also see a progress indicator next to the status. See [here](../../pipelines/pipelines.md#tracking-pipeline-progress). | String |
| **USER** | User who created the run. | String | | **USER** | User who created the run. | String |

2
docs/webapp/pipelines/webapp_pipeline_viewing.md
View File

@ -108,7 +108,7 @@ The details panel includes three tabs:
![console](../../img/webapp_pipeline_step_console_dark.png#dark-mode-only) ![console](../../img/webapp_pipeline_step_console_dark.png#dark-mode-only)
* **Code** - For pipeline steps generated from functions using either [`PipelineController.add_function_step`](../../references/sdk/automation_controller_pipelinecontroller.md#add_function_step) * **Code** - For pipeline steps generated from functions using either [`PipelineController.add_function_step`](../../references/sdk/automation_controller_pipelinecontroller.md#add_function_step)
or [`PipelineDecorator.component`](../../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorcomponent), or [`PipelineDecorator.component`](../../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorcomponent),
you can view the selected step's code. you can view the selected step's code.
![code](../../img/webapp_pipeline_step_code.png#light-mode-only) ![code](../../img/webapp_pipeline_step_code.png#light-mode-only)

4
sidebars.js
View File

@ -399,8 +399,10 @@ module.exports = {
'references/sdk/dataset', 'references/sdk/dataset',
{'Pipeline': [ {'Pipeline': [
'references/sdk/automation_controller_pipelinecontroller', 'references/sdk/automation_controller_pipelinecontroller',
'references/sdk/automation_controller_pipelinedecorator',
'references/sdk/automation_job_clearmljob' 'references/sdk/automation_job_clearmljob'
]}, ]
},
'references/sdk/scheduler', 'references/sdk/scheduler',
'references/sdk/trigger', 'references/sdk/trigger',
{'HyperParameter Optimization': [ {'HyperParameter Optimization': [