ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-06-26 18:17:44 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Split PipelineController and PipelineDecorator reference pages (#1063)
Some checks are pending
CI / build (push) Waiting to run
Details

Browse Source
This commit is contained in:
pollfly
2025-03-02 11:54:30 +02:00
committed by GitHub
parent 3829d1f4b7
commit 6f8d8601f0
6 changed files with 29 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
docs/guides/pipeline/pipeline_decorator.md
View File

@@ -3,7 +3,7 @@ title: Pipeline from Decorators
---
The [pipeline_from_decorator.py](https://github.com/clearml/clearml/blob/master/examples/pipeline/pipeline_from_decorator.py)
example demonstrates the creation of a pipeline in ClearML using the [`PipelineDecorator`](../../references/sdk/automation_controller_pipelinecontroller.md#class-automationcontrollerpipelinedecorator)
example demonstrates the creation of a pipeline in ClearML using the [`PipelineDecorator`](../../references/sdk/automation_controller_pipelinedecorator.md#class-automationcontrollerpipelinedecorator)
class.
This example creates a pipeline incorporating four tasks, each of which is created from a Python function using a custom decorator:
@@ -14,11 +14,11 @@ This example creates a pipeline incorporating four tasks, each of which is creat
* `step_four` - Uses data from `step_two` and the model from `step_three` to make a prediction.
The pipeline steps, defined in the `step_one`, `step_two`, `step_three`, and `step_four` functions, are each wrapped with the
[`@PipelineDecorator.component`](../../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorcomponent)
[`@PipelineDecorator.component`](../../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorcomponent)
decorator, which creates a ClearML pipeline step for each one when the pipeline is executed.
The logic that executes these steps and controls the interaction between them is implemented in the `executing_pipeline`
function. This function is wrapped with the [`@PipelineDecorator.pipeline`](../../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorpipeline)
function. This function is wrapped with the [`@PipelineDecorator.pipeline`](../../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorpipeline)
decorator which creates the ClearML pipeline task when it is executed.
The sections below describe in more detail what happens in the pipeline controller and steps.
@@ -28,7 +28,7 @@ The sections below describe in more detail what happens in the pipeline controll
In this example, the pipeline controller is implemented by the `executing_pipeline` function.
Using the `@PipelineDecorator.pipeline` decorator creates a ClearML Controller Task from the function when it is executed.
For detailed information, see [`@PipelineDecorator.pipeline`](../../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorpipeline).
For detailed information, see [`@PipelineDecorator.pipeline`](../../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorpipeline).
In the example script, the controller defines the interactions between the pipeline steps in the following way:
1. The controller function passes its argument, `pickle_url`, to the pipeline's first step (`step_one`)
@@ -39,13 +39,13 @@ In the example script, the controller defines the interactions between the pipel
:::info Local Execution
In this example, the pipeline is set to run in local mode by using
[`PipelineDecorator.run_locally()`](../../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorrun_locally)
[`PipelineDecorator.run_locally()`](../../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorrun_locally)
before calling the pipeline function. See pipeline execution options [here](../../pipelines/pipelines_sdk_function_decorators.md#running-the-pipeline).
:::
## Pipeline Steps
Using the `@PipelineDecorator.component` decorator will make the function a pipeline component that can be called from the
pipeline controller, which implements the pipeline's execution logic. For detailed information, see [`@PipelineDecorator.component`](../../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorcomponent).
pipeline controller, which implements the pipeline's execution logic. For detailed information, see [`@PipelineDecorator.component`](../../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorcomponent).
When the pipeline controller calls a pipeline step, a corresponding ClearML task will be created. Notice that all package
imports inside the function will be automatically logged as required packages for the pipeline execution step.
@@ -63,7 +63,7 @@ executing_pipeline(
```
By default, the pipeline controller and the pipeline steps are launched through ClearML [queues](../../fundamentals/agents_and_queues.md#what-is-a-queue).
Use the [`PipelineDecorator.set_default_execution_queue`](../../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorset_default_execution_queue)
Use the [`PipelineDecorator.set_default_execution_queue`](../../references/sdk/automation_controller_pipelinedecorator.md#pipelinedecoratorset_default_execution_queue)
method to specify the execution queue of all pipeline steps. The `execution_queue` parameter of the `@PipelineDecorator.component`
decorator overrides the default queue value for the specific step for which it was specified.