ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-01-31 14:37:18 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Add pipeline step transformation info (#295)

Browse Source
This commit is contained in:
pollfly 2022-07-27 10:40:54 +03:00 committed by GitHub
parent 4e3f2dc7b8
commit 8afe79d3ee
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 8 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
docs/pipelines/pipelines.md
View File

@ -34,7 +34,8 @@ example of a pipeline with concurrent steps.
## Running Your Pipelines ## Running Your Pipelines
ClearML supports multiple modes for pipeline execution: ClearML supports multiple modes for pipeline execution:
* **Remote Mode** (default) - In this mode, the pipeline controller logic is executed through a designated queue, and all * **Remote Mode** (default) - In this mode, the pipeline controller logic is executed through a designated queue, and all
the pipeline steps are launched remotely through their respective queues. the pipeline steps are launched remotely through their respective queues. Since each task is executed independently,
it can have control over its git repository (if needed), required python packages and specific container to be used.
* **Local Mode** - In this mode, the pipeline is executed locally, and the steps are executed as sub-processes. Each * **Local Mode** - In this mode, the pipeline is executed locally, and the steps are executed as sub-processes. Each
subprocess uses the exact same Python environment as the main pipeline logic. subprocess uses the exact same Python environment as the main pipeline logic.
* **Debugging Mode** (for PipelineDecorator) - In this mode, the entire pipeline is executed locally, with the pipeline * **Debugging Mode** (for PipelineDecorator) - In this mode, the entire pipeline is executed locally, with the pipeline

5
docs/pipelines/pipelines_sdk_tasks.md
View File

@ -109,6 +109,11 @@ Examples:
Creating a pipeline step from a function means that when the function is called, it will be transformed into a ClearML task, Creating a pipeline step from a function means that when the function is called, it will be transformed into a ClearML task,
translating its arguments into parameters, and returning values into artifacts. translating its arguments into parameters, and returning values into artifacts.
:::info Function to ClearML Task conversion
As each function is transformed into an independently executed step, it needs to be self-contained. To facilitate this,
all package imports inside the function are automatically logged as required packages for the pipeline step.
:::
Function steps are added using the [`PipelineController.add_function_step`](../references/sdk/automation_controller_pipelinecontroller.md#add_function_step) Function steps are added using the [`PipelineController.add_function_step`](../references/sdk/automation_controller_pipelinecontroller.md#add_function_step)
method: method: