Update pipeline imports information ()

This commit is contained in:
pollfly 2024-01-28 11:37:53 +02:00 committed by GitHub
parent 9e975d944b
commit 1ab0083d01
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
4 changed files with 22 additions and 15 deletions

View File

@ -47,9 +47,8 @@ method before calling the pipeline function. See pipeline execution options [her
Using the `@PipelineDecorator.component` decorator will make the function a pipeline component that can be called from the Using the `@PipelineDecorator.component` decorator will make the function a pipeline component that can be called from the
pipeline controller, which implements the pipeline's execution logic. For detailed information, see [`@PipelineDecorator.component`](../../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorcomponent). pipeline controller, which implements the pipeline's execution logic. For detailed information, see [`@PipelineDecorator.component`](../../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorcomponent).
When the pipeline controller calls a pipeline step, a corresponding ClearML task will be created. For this reason, each When the pipeline controller calls a pipeline step, a corresponding ClearML task will be created. Notice that all package
function which makes up a pipeline step needs to be self-contained. Notice that all package imports inside the function imports inside the function will be automatically logged as required packages for the pipeline execution step.
will be automatically logged as required packages for the pipeline execution step.
## Pipeline Execution ## Pipeline Execution

View File

@ -15,9 +15,8 @@ The fourth task is the pipeline task, which is created when the pipeline is laun
The step functions will be registered as pipeline steps when they are added to the pipeline controller. The step functions will be registered as pipeline steps when they are added to the pipeline controller.
When the pipeline steps are executed, corresponding ClearML Tasks are created. For this reason, each function which makes When the pipeline steps are executed, corresponding ClearML Tasks are created. Notice that all package imports inside
up a pipeline step needs to be self-contained. Notice that all package imports inside the function will be automatically the function will be automatically logged as required packages for the pipeline execution step.
logged as required packages for the pipeline execution step.

View File

@ -12,9 +12,8 @@ for your main pipeline execution logic function.
## @PipelineDecorator.pipeline ## @PipelineDecorator.pipeline
Using the [`@PipelineDecorator.pipeline`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorpipeline) Using the [`@PipelineDecorator.pipeline`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorpipeline)
decorator transforms the function which implements your pipeline's execution logic to a ClearML pipeline controller. Since decorator transforms the function which implements your pipeline's execution logic to a ClearML pipeline controller,
the function is transformed into an independently executed task, it needs to be self-contained. To facilitate this, an independently executed task.
all package imports inside the function are automatically logged as required packages for the pipeline controller.
:::tip Multi-file Pipeline Implementation :::tip Multi-file Pipeline Implementation
In the case your pipeline is implemented across multiple files, make sure the pipeline step implementation (files containing In the case your pipeline is implemented across multiple files, make sure the pipeline step implementation (files containing
@ -61,6 +60,9 @@ def main(pickle_url, mock_parameter='mock'):
* `pipeline_execution_queue` - The queue in which to enqueue the pipeline controller task. The default value is the * `pipeline_execution_queue` - The queue in which to enqueue the pipeline controller task. The default value is the
`services` queue. To run the pipeline logic locally while the components are executed remotely, pass `services` queue. To run the pipeline logic locally while the components are executed remotely, pass
`pipeline_execution_queue=None` `pipeline_execution_queue=None`
* `skip_global_imports` If `True`, global imports will not be included in the steps execution. If `False` (default),
all global imports will be automatically imported at the beginning of each steps execution.
When the function is called, a corresponding ClearML Controller Task is created: its arguments are logged as the task's When the function is called, a corresponding ClearML Controller Task is created: its arguments are logged as the task's
parameters. When launching a new pipeline run from the [UI](../webapp/pipelines/webapp_pipeline_page.md), you can modify their values for the new run. parameters. When launching a new pipeline run from the [UI](../webapp/pipelines/webapp_pipeline_page.md), you can modify their values for the new run.
@ -71,9 +73,14 @@ parameters. When launching a new pipeline run from the [UI](../webapp/pipelines/
Using the [`@PipelineDecorator.component`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorcomponent) Using the [`@PipelineDecorator.component`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorcomponent)
decorator transforms a function into a ClearML pipeline step when called from a pipeline controller. decorator transforms a function into a ClearML pipeline step when called from a pipeline controller.
When the pipeline controller calls a pipeline step, a corresponding ClearML task is created. For this reason, each When the pipeline controller calls a pipeline step, a corresponding ClearML task is created.
function which makes up a pipeline step needs to be self-contained. All package imports inside the function are automatically
logged as required packages for the pipeline execution step. :::tip Package Imports
In the case that the `skip_global_imports` parameter of [`@PipelineDecorator.pipeline`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorpipeline)
is set to `False`, all global imports will be automatically imported at the beginning of each step's execution.
Otherwise, if set to `True`, make sure that each function which makes up a pipeline step contains package imports, which
are automatically logged as required packages for the pipeline execution step.
:::
```python ```python
from clearml.automation.controller import PipelineDecorator from clearml.automation.controller import PipelineDecorator

View File

@ -130,9 +130,11 @@ Examples:
Creating a pipeline step from a function means that when the function is called, it will be transformed into a ClearML task, Creating a pipeline step from a function means that when the function is called, it will be transformed into a ClearML task,
translating its arguments into parameters, and returning values into artifacts. translating its arguments into parameters, and returning values into artifacts.
:::info Function to ClearML Task conversion :::info Package Imports
As each function is transformed into an independently executed step, it needs to be self-contained. To facilitate this, In the case that the `skip_global_imports` parameter of [`PipelineController`](../references/sdk/automation_controller_pipelinecontroller.md)
all package imports inside the function are automatically logged as required packages for the pipeline step. is set to `False`, all global imports will be automatically imported at the beginning of each step's execution.
Otherwise, if set to `True`, make sure that each function which makes up a pipeline step contains package imports, which
are automatically logged as required packages for the pipeline execution step.
::: :::
Function steps are added using [`PipelineController.add_function_step()`](../references/sdk/automation_controller_pipelinecontroller.md#add_function_step): Function steps are added using [`PipelineController.add_function_step()`](../references/sdk/automation_controller_pipelinecontroller.md#add_function_step):