ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-02-07 13:21:46 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Pipelines local imports (#203)

Browse Source
This commit is contained in:
Victor Sonck 2022-03-16 09:55:44 +01:00 committed by GitHub
parent 0e92230040
commit 31baed0a8b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 9 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
docs/fundamentals/pipelines.md
View File

@ -151,8 +151,8 @@ def main(pickle_url, mock_parameter='mock'):
data_frame = step_one(pickle_url) data_frame = step_one(pickle_url)
X_train, X_test, y_train, y_test = step_two(data_frame) X_train, X_test, y_train, y_test = step_two(data_frame)
model = step_three(X_train, y_train) model = step_three(X_train, y_train)
accuracy = 100 * step_four(model, X_data=X_test, Y_data=y_test) accuracy = 100 * step_four(model, X_data=X_test, Y_data=y_test)
print(f"Accuracy={accuracy}%") print(f"Accuracy={accuracy}%")
``` ```
Notice that the driver is the `main` function, calling ("launching") the different steps. Next we add the decorators over Notice that the driver is the `main` function, calling ("launching") the different steps. Next we add the decorators over
@ -218,8 +218,8 @@ def main(pickle_url, mock_parameter='mock'):
data_frame = step_one(pickle_url) data_frame = step_one(pickle_url)
X_train, X_test, y_train, y_test = step_two(data_frame) X_train, X_test, y_train, y_test = step_two(data_frame)
model = step_three(X_train, y_train) model = step_three(X_train, y_train)
accuracy = 100 * step_four(model, X_data=X_test, Y_data=y_test) accuracy = 100 * step_four(model, X_data=X_test, Y_data=y_test)
print(f"Accuracy={accuracy}%") print(f"Accuracy={accuracy}%")
``` ```
We wrap each pipeline component with `@PipelineDecorator.component`, and the main pipeline logic with We wrap each pipeline component with `@PipelineDecorator.component`, and the main pipeline logic with
@ -310,7 +310,11 @@ function) are not transferred into the standalone Task. This means that any pack
should be imported inside the function itself. should be imported inside the function itself.
All imports are automatically logged, and will appear as required packages for the remote Task. This means the All imports are automatically logged, and will appear as required packages for the remote Task. This means the
`clearml-agent` will install the requested environment automatically `clearml-agent` will install the requested environment automatically.
For local imports (e.g. `from my_helper_functions import foo`) you need to make sure these local files are available to
the standalone script at runtime. The only way to do this for now is to use the `repo`, `repo_branch` and `repo_commit`
arguments detailed above. This does mean the helper functions should be committed to that repository at runtime.
::: :::
### @PipelineDecorator.pipeline ### @PipelineDecorator.pipeline