diff --git a/docs/getting_started/ds/ds_first_steps.md b/docs/getting_started/ds/ds_first_steps.md index 9b715323..10e5de12 100644 --- a/docs/getting_started/ds/ds_first_steps.md +++ b/docs/getting_started/ds/ds_first_steps.md @@ -37,7 +37,7 @@ To ensure full automatic logging it is recommended to import the ClearML package Then initialize the Task object in your `main()` function, or the beginning of the script. ```python -task = Task.init(project_name=”great project”, task_name=”best experiment”) +task = Task.init(project_name='great project', task_name='best experiment') ``` Task name is not unique, it's possible to have multiple experiments with the same name. diff --git a/docs/getting_started/mlops/mlops_first_steps.md b/docs/getting_started/mlops/mlops_first_steps.md index 51cac3ec..611eaf5b 100644 --- a/docs/getting_started/mlops/mlops_first_steps.md +++ b/docs/getting_started/mlops/mlops_first_steps.md @@ -6,18 +6,21 @@ title: First Steps This tutorial assumes that you've already [signed up](https://app.community.clear.ml) to ClearML ::: -MLOps is all about automation! We'll discuss the need for automation and the Tools ClearML offers for automation, orchestration and tracking!
+MLOps is all about automation! We'll discuss the need for automation and the tools ClearML offers for automation, orchestration and tracking!
Effective MLOps relies on being able to scale work beyond one's own computer. Moving from your own machine can be inefficient, assuming that you have all the drivers and applications installed, you still need to manage multiple python environments -for different packages \ package versions, or worst - manage different docker for different package versions.
-Not to mention, when working on remote machines, executing experiments and tracking what's running where and making sure they are fully utilized at all times -becomes a daunting task.
+for different packages / package versions, or worse - manage different dockers for different package versions. + +Not to mention, when working on remote machines, executing experiments and tracking what's running where and making sure machines are fully utilized at all times +becomes a daunting task. + This can create overhead that derails you from the core work! -ClearML Agent was designed to deal with these and more! It is a module responsible executing experiments, -on remote machines, on premise or in the cloud!
-It will setup the environment for the specific Task (inside a docker, or bare-metal) install the required python packages and execute & monitor the process itself. +ClearML Agent was designed to deal with these and more! It is a module responsible for executing experiments +on remote machines, on premise, or in the cloud! + +It will set up the environment for the specific Task (inside a docker, or bare-metal) install the required python packages and execute & monitor the process itself. ## Spin up an Agent @@ -34,10 +37,10 @@ clearml-agent init ``` :::note -If you've already created credentials, you can copy-paste the default agent section from [here](https://github.com/allegroai/clearml-agent/blob/master/docs/clearml.conf#L15) (this is obviously optional if the section is not provided the default values will be used) +If you've already created credentials, you can copy-paste the default agent section from [here](https://github.com/allegroai/clearml-agent/blob/master/docs/clearml.conf#L15) (this is optional. If the section is not provided the default values will be used) ::: -Start the agent's daemon. The agent will start pulling Tasks from the assigned queue(default in our case), and execute them one after the other. +Start the agent's daemon. The agent will start pulling Tasks from the assigned queue (default in our case), and execute them one after the other. ```bash clearml-agent daemon --queue default @@ -45,8 +48,9 @@ clearml-agent daemon --queue default ## Clone an Experiment Creating a new "job" to be executed, is essentially cloning a Task in the system, then enqueueing the Task in one of the execution queues for the agent to execute it. -When cloning a Task we are creating another copy of the Task in a *draft* mode, allowing us to edit the Task's environment definitions.
-We can edit the git \ code references, control the python packages to be installed, specify docker container image to be used, or change the hyper-parameters and configuration files. +When cloning a Task we are creating another copy of the Task in a *draft* mode, allowing us to edit the Task's environment definitions. + +We can edit the git / code references, control the python packages to be installed, specify docker container image to be used, or change the hyper-parameters and configuration files. Once we are done, enqueuing the Task in one of the execution queues will put it in the execution queue. Multiple agents can listen to the same queue (or even multiple queues), but only a single agent will pick the Task to be executed. @@ -62,13 +66,15 @@ executed_task = Task.get_task(task_id='aabbcc') ``` ## Log Hyperparameters -Hyperparameters are an integral part of Machine Learning code as it lets you control the code without directly modifying it.
+Hyperparameters are an integral part of Machine Learning code as it lets you control the code without directly modifying it. + Hyperparameters can be added from anywhere in your code, and ClearML supports [multiple](../../fundamentals/hyperparameters.md) ways to obtain them! ClearML also allows users to change and track hyperparameter values without changing the code itself. When a cloned experiment is executed by an Agent, it will override the default values with new ones. -It's also possible to programatically change cloned experiments' parameters +It's also possible to programmatically change cloned experiments' parameters. + For example: ```python from clearml import Task @@ -82,7 +88,7 @@ Task.enqueue(cloned_task, queue_name='default') Artifacts are a great way to pass and reuse data between Tasks in the system. From anywhere in the code you can upload [multiple](../../fundamentals/artifacts.md#logging-artifacts) types of data, object and files. Artifacts are the base of ClearML's [Data Management](../../clearml_data.md) solution and as a way to communicate complex objects between different -stages of a [pipeline](../../fundamentals/pipelines.md) +stages of a [pipeline](../../fundamentals/pipelines.md). ```python import numpy as np @@ -93,7 +99,7 @@ Task.current_task().upload_artifact(name='numpy', artifact_object=np.ones(4,4)) ### Using Artifacts -Artifacts can be retrieved by [accessing](../../fundamentals/artifacts.md#uing-artifacts) the Task that created it. +Artifacts can be retrieved by [accessing](../../fundamentals/artifacts.md#using-artifacts) the Task that created it. ```python from clearml import Task executed_task = Task.get_task(task_id='aabbcc') @@ -104,9 +110,10 @@ a_numpy = executed_task.artifacts['numpy'].get() ``` ### Models -Model are a special type of artifact that's automatically logged. -Logging models into the model repository is the easiest way to integrate the development process directly with production.
-Any model stored by the supported frameworks (Keras \ TF \PyTorch \ Joblib) will be automatically logged into ClearML. +Models are a special type of artifact that's automatically logged. +Logging models into the model repository is the easiest way to integrate the development process directly with production. + +Any model stored by the supported frameworks (Keras / TF /PyTorch / Joblib) will be automatically logged into ClearML. Models can be automatically stored on a preferred storage medium (s3 bucket, google storage, etc...). ## Log Metrics @@ -138,9 +145,10 @@ tasks = Task.get_tasks(project_name='examples', task_name='partial_name_match', ## Manage Your Data Data is probably one of the biggest factors that determines the success of a project. Associating the data a model used to the model's configuration, code and results (such as accuracy) is key to deducing meaningful insights into how -models behave.
-[ClearML Data](../../clearml_data.md) allows you to version your data so it's never lost, fetch it from every machine with minimal code changes -and associate data to experiments results. -Logging data can be done via command line, or via code. If any preprocessing code is involved, ClearML logs it as well!
-Once data is logged, it can be used by other experiments. +models behave. + +[ClearML Data](../../clearml_data.md) allows you to version your data so it's never lost, fetch it from every machine with minimal code changes, +and associate data to experiment results. + +Logging data can be done via command line, or via code. If any preprocessing code is involved, ClearML logs it as well! Once data is logged, it can be used by other experiments.