diff --git a/docs/guides/pipeline/pipeline_decorator.md b/docs/guides/pipeline/pipeline_decorator.md index d8ac0439..af0566f4 100644 --- a/docs/guides/pipeline/pipeline_decorator.md +++ b/docs/guides/pipeline/pipeline_decorator.md @@ -47,9 +47,8 @@ method before calling the pipeline function. See pipeline execution options [her Using the `@PipelineDecorator.component` decorator will make the function a pipeline component that can be called from the pipeline controller, which implements the pipeline's execution logic. For detailed information, see [`@PipelineDecorator.component`](../../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorcomponent). -When the pipeline controller calls a pipeline step, a corresponding ClearML task will be created. For this reason, each -function which makes up a pipeline step needs to be self-contained. Notice that all package imports inside the function -will be automatically logged as required packages for the pipeline execution step. +When the pipeline controller calls a pipeline step, a corresponding ClearML task will be created. Notice that all package +imports inside the function will be automatically logged as required packages for the pipeline execution step. ## Pipeline Execution diff --git a/docs/guides/pipeline/pipeline_functions.md b/docs/guides/pipeline/pipeline_functions.md index 72ebbba7..e34e27cb 100644 --- a/docs/guides/pipeline/pipeline_functions.md +++ b/docs/guides/pipeline/pipeline_functions.md @@ -15,9 +15,8 @@ The fourth task is the pipeline task, which is created when the pipeline is laun The step functions will be registered as pipeline steps when they are added to the pipeline controller. -When the pipeline steps are executed, corresponding ClearML Tasks are created. For this reason, each function which makes -up a pipeline step needs to be self-contained. Notice that all package imports inside the function will be automatically -logged as required packages for the pipeline execution step. +When the pipeline steps are executed, corresponding ClearML Tasks are created. Notice that all package imports inside +the function will be automatically logged as required packages for the pipeline execution step. diff --git a/docs/pipelines/pipelines_sdk_function_decorators.md b/docs/pipelines/pipelines_sdk_function_decorators.md index 34620151..478a318f 100644 --- a/docs/pipelines/pipelines_sdk_function_decorators.md +++ b/docs/pipelines/pipelines_sdk_function_decorators.md @@ -12,9 +12,8 @@ for your main pipeline execution logic function. ## @PipelineDecorator.pipeline Using the [`@PipelineDecorator.pipeline`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorpipeline) -decorator transforms the function which implements your pipeline's execution logic to a ClearML pipeline controller. Since -the function is transformed into an independently executed task, it needs to be self-contained. To facilitate this, -all package imports inside the function are automatically logged as required packages for the pipeline controller. +decorator transforms the function which implements your pipeline's execution logic to a ClearML pipeline controller, +an independently executed task. :::tip Multi-file Pipeline Implementation In the case your pipeline is implemented across multiple files, make sure the pipeline step implementation (files containing @@ -61,6 +60,9 @@ def main(pickle_url, mock_parameter='mock'): * `pipeline_execution_queue` - The queue in which to enqueue the pipeline controller task. The default value is the `services` queue. To run the pipeline logic locally while the components are executed remotely, pass `pipeline_execution_queue=None` +* `skip_global_imports` – If `True`, global imports will not be included in the steps’ execution. If `False` (default), + all global imports will be automatically imported at the beginning of each step’s execution. + When the function is called, a corresponding ClearML Controller Task is created: its arguments are logged as the task's parameters. When launching a new pipeline run from the [UI](../webapp/pipelines/webapp_pipeline_page.md), you can modify their values for the new run. @@ -71,9 +73,14 @@ parameters. When launching a new pipeline run from the [UI](../webapp/pipelines/ Using the [`@PipelineDecorator.component`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorcomponent) decorator transforms a function into a ClearML pipeline step when called from a pipeline controller. -When the pipeline controller calls a pipeline step, a corresponding ClearML task is created. For this reason, each -function which makes up a pipeline step needs to be self-contained. All package imports inside the function are automatically -logged as required packages for the pipeline execution step. +When the pipeline controller calls a pipeline step, a corresponding ClearML task is created. + +:::tip Package Imports +In the case that the `skip_global_imports` parameter of [`@PipelineDecorator.pipeline`](../references/sdk/automation_controller_pipelinecontroller.md#pipelinedecoratorpipeline) +is set to `False`, all global imports will be automatically imported at the beginning of each step's execution. +Otherwise, if set to `True`, make sure that each function which makes up a pipeline step contains package imports, which +are automatically logged as required packages for the pipeline execution step. +::: ```python from clearml.automation.controller import PipelineDecorator diff --git a/docs/pipelines/pipelines_sdk_tasks.md b/docs/pipelines/pipelines_sdk_tasks.md index 2ad0f102..e61ff1ac 100644 --- a/docs/pipelines/pipelines_sdk_tasks.md +++ b/docs/pipelines/pipelines_sdk_tasks.md @@ -130,9 +130,11 @@ Examples: Creating a pipeline step from a function means that when the function is called, it will be transformed into a ClearML task, translating its arguments into parameters, and returning values into artifacts. -:::info Function to ClearML Task conversion -As each function is transformed into an independently executed step, it needs to be self-contained. To facilitate this, -all package imports inside the function are automatically logged as required packages for the pipeline step. +:::info Package Imports +In the case that the `skip_global_imports` parameter of [`PipelineController`](../references/sdk/automation_controller_pipelinecontroller.md) +is set to `False`, all global imports will be automatically imported at the beginning of each step's execution. +Otherwise, if set to `True`, make sure that each function which makes up a pipeline step contains package imports, which +are automatically logged as required packages for the pipeline execution step. ::: Function steps are added using [`PipelineController.add_function_step()`](../references/sdk/automation_controller_pipelinecontroller.md#add_function_step):