Update pipeline imports information (#765)

This commit is contained in:
pollfly 2024-01-28 11:37:53 +02:00 committed by GitHub
parent 9e975d944b
commit 1ab0083d01
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
4 changed files with 22 additions and 15 deletions

View File

@ -47,9 +47,8 @@ method before calling the pipeline function. See pipeline execution options [her
Using the `@PipelineDecorator.component` decorator will make the function a pipeline component that can be called from the
pipeline controller, which implements the pipeline's execution logic. For detailed information, see [`@PipelineDecorator.component`](../../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorcomponent).
When the pipeline controller calls a pipeline step, a corresponding ClearML task will be created. For this reason, each
function which makes up a pipeline step needs to be self-contained. Notice that all package imports inside the function
will be automatically logged as required packages for the pipeline execution step.
When the pipeline controller calls a pipeline step, a corresponding ClearML task will be created. Notice that all package
imports inside the function will be automatically logged as required packages for the pipeline execution step.
## Pipeline Execution

View File

@ -15,9 +15,8 @@ The fourth task is the pipeline task, which is created when the pipeline is laun
The step functions will be registered as pipeline steps when they are added to the pipeline controller.
When the pipeline steps are executed, corresponding ClearML Tasks are created. For this reason, each function which makes
up a pipeline step needs to be self-contained. Notice that all package imports inside the function will be automatically
logged as required packages for the pipeline execution step.
When the pipeline steps are executed, corresponding ClearML Tasks are created. Notice that all package imports inside
the function will be automatically logged as required packages for the pipeline execution step.

View File

@ -12,9 +12,8 @@ for your main pipeline execution logic function.
## @PipelineDecorator.pipeline
Using the [`@PipelineDecorator.pipeline`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorpipeline)
decorator transforms the function which implements your pipeline's execution logic to a ClearML pipeline controller. Since
the function is transformed into an independently executed task, it needs to be self-contained. To facilitate this,
all package imports inside the function are automatically logged as required packages for the pipeline controller.
decorator transforms the function which implements your pipeline's execution logic to a ClearML pipeline controller,
an independently executed task.
:::tip Multi-file Pipeline Implementation
In the case your pipeline is implemented across multiple files, make sure the pipeline step implementation (files containing
@ -61,6 +60,9 @@ def main(pickle_url, mock_parameter='mock'):
* `pipeline_execution_queue` - The queue in which to enqueue the pipeline controller task. The default value is the
`services` queue. To run the pipeline logic locally while the components are executed remotely, pass
`pipeline_execution_queue=None`
* `skip_global_imports` If `True`, global imports will not be included in the steps execution. If `False` (default),
all global imports will be automatically imported at the beginning of each steps execution.
When the function is called, a corresponding ClearML Controller Task is created: its arguments are logged as the task's
parameters. When launching a new pipeline run from the [UI](../webapp/pipelines/webapp_pipeline_page.md), you can modify their values for the new run.
@ -71,9 +73,14 @@ parameters. When launching a new pipeline run from the [UI](../webapp/pipelines/
Using the [`@PipelineDecorator.component`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorcomponent)
decorator transforms a function into a ClearML pipeline step when called from a pipeline controller.
When the pipeline controller calls a pipeline step, a corresponding ClearML task is created. For this reason, each
function which makes up a pipeline step needs to be self-contained. All package imports inside the function are automatically
logged as required packages for the pipeline execution step.
When the pipeline controller calls a pipeline step, a corresponding ClearML task is created.
:::tip Package Imports
In the case that the `skip_global_imports` parameter of [`@PipelineDecorator.pipeline`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorpipeline)
is set to `False`, all global imports will be automatically imported at the beginning of each step's execution.
Otherwise, if set to `True`, make sure that each function which makes up a pipeline step contains package imports, which
are automatically logged as required packages for the pipeline execution step.
:::
```python
from clearml.automation.controller import PipelineDecorator

View File

@ -130,9 +130,11 @@ Examples:
Creating a pipeline step from a function means that when the function is called, it will be transformed into a ClearML task,
translating its arguments into parameters, and returning values into artifacts.
:::info Function to ClearML Task conversion
As each function is transformed into an independently executed step, it needs to be self-contained. To facilitate this,
all package imports inside the function are automatically logged as required packages for the pipeline step.
:::info Package Imports
In the case that the `skip_global_imports` parameter of [`PipelineController`](../references/sdk/automation_controller_pipelinecontroller.md)
is set to `False`, all global imports will be automatically imported at the beginning of each step's execution.
Otherwise, if set to `True`, make sure that each function which makes up a pipeline step contains package imports, which
are automatically logged as required packages for the pipeline execution step.
:::
Function steps are added using [`PipelineController.add_function_step()`](../references/sdk/automation_controller_pipelinecontroller.md#add_function_step):