version bump

Add docker_preprocess_bash_script to allow preprocessing bash to be added
Fix multiple python versions installed in the same docker by finding the highest installed python inside the docker
Fix conda_env_as_base_docker not set to False in docker mode

README.md

@@ -1,80 +1,107 @@
-# TRAINS Agent
-## Deep Learning DevOps For Everyone - Now supporting all platforms (Linux, macOS, and Windows)
+<div align="center">
 
-"All the Deep-Learning DevOps your research needs, and then some... Because ain't nobody got time for that"
+<img src="https://github.com/allegroai/clearml-agent/blob/master/docs/clearml_agent_logo.png?raw=true" width="250px">
+
+**ClearML Agent - ML-Ops made easy  
+ML-Ops scheduler & orchestration solution supporting Linux, macOS and Windows**
 
 [![GitHub license](https://img.shields.io/github/license/allegroai/trains-agent.svg)](https://img.shields.io/github/license/allegroai/trains-agent.svg)
-[![PyPI pyversions](https://img.shields.io/pypi/pyversions/trains-agent.svg)](https://img.shields.io/pypi/pyversions/trains-agent.svg)
-[![PyPI version shields.io](https://img.shields.io/pypi/v/trains-agent.svg)](https://img.shields.io/pypi/v/trains-agent.svg)
-[![PyPI status](https://img.shields.io/pypi/status/trains-agent.svg)](https://pypi.python.org/pypi/trains-agent/)
+[![PyPI pyversions](https://img.shields.io/pypi/pyversions/clearml-agent.svg)](https://img.shields.io/pypi/pyversions/clearml-agent.svg)
+[![PyPI version shields.io](https://img.shields.io/pypi/v/clearml-agent.svg)](https://img.shields.io/pypi/v/clearml-agent.svg)
+[![PyPI status](https://img.shields.io/pypi/status/clearml-agent.svg)](https://pypi.python.org/pypi/clearml-agent/)
 
-### Help improve Trains by filling our 2-min [user survey](https://allegro.ai/lp/trains-user-survey/)
+</div>
 
-**TRAINS Agent is an AI experiment cluster solution.**
+---
 
-It is a zero configuration fire-and-forget execution agent, which combined with trains-server provides a full AI cluster solution.
+### ClearML-Agent
+#### *Formerly known as Trains Agent*
 
-**Full AutoML in 5 steps** 
-1. Install the [TRAINS server](https://github.com/allegroai/trains-agent) (or use our [open server](https://demoapp.trains.allegro.ai))
-2. `pip install trains-agent` ([install](#installing-the-trains-agent) the TRAINS agent on any GPU machine: on-premises / cloud / ...)
-3. Add [TRAINS](https://github.com/allegroai/trains) to your code with just 2 lines & run it once (on your machine / laptop)
-4. Change the [parameters](#using-the-trains-agent) in the UI & schedule for [execution](#using-the-trains-agent) (or automate with an [AutoML pipeline](#automl-and-orchestration-pipelines-))
+
+* Run jobs (experiments) on any local or cloud based resource
+* Implement optimized resource utilization policies
+* Deploy execution environments with either virtualenv or fully docker containerized with zero effort
+* Launch-and-Forget service containers
+* [Cloud autoscaling](https://allegro.ai/clearml/docs/examples/services/aws_autoscaler/aws_autoscaler/)
+* [Customizable cleanup](https://allegro.ai/clearml/docs/examples/services/cleanup/cleanup_service/)
+* Advanced [pipeline building and execution](https://allegro.ai/clearml/docs/examples/frameworks/pytorch/notebooks/table/tabular_training_pipeline/)
+
+It is a zero configuration fire-and-forget execution agent, providing a full ML/DL cluster solution.
+
+**Full Automation in 5 steps**
+1. ClearML Server [self-hosted](https://github.com/allegroai/trains-server) or [free tier hosting](https://app.community.clear.ml)
+2. `pip install clearml-agent` ([install](#installing-the-clearml-agent) the ClearML Agent on any GPU machine: on-premises / cloud / ...)
+3. Create a [job](https://github.com/allegroai/clearml/docs/clearml-task.md) or Add [ClearML](https://github.com/allegroai/trains) to your code with just 2 lines
+4. Change the [parameters](#using-the-clearml-agent) in the UI & schedule for [execution](#using-the-clearml-agent) (or automate with an [AutoML pipeline](#automl-and-orchestration-pipelines-))
 5. :chart_with_downwards_trend: :chart_with_upwards_trend: :eyes:  :beer:
 
+"All the Deep/Machine-Learning DevOps your research needs, and then some... Because ain't nobody got time for that"
 
-**Using the TRAINS agent, you can now set up a dynamic cluster with \*epsilon DevOps**
+**Try ClearML now** [Self Hosted](https://github.com/allegroai/trains-server) or [Free tier Hosting](https://app.community.clear.ml)
+<a href="https://app.community.clear.ml"><img src="https://raw.githubusercontent.com/allegroai/trains-agent/9f1e86c1ca45c984ee13edc9353c7b10c55d7257/docs/screenshots.gif" width="100%"></a>
 
-*epsilon - Because we are scientists :triangular_ruler: and nothing is really zero work
-
-(Experience TRAINS live at [https://demoapp.trains.allegro.ai](https://demoapp.trains.allegro.ai))
-<a href="https://demoapp.trains.allegro.ai"><img src="https://raw.githubusercontent.com/allegroai/trains-agent/9f1e86c1ca45c984ee13edc9353c7b10c55d7257/docs/screenshots.gif" width="100%"></a>
-
-## Simple, Flexible Experiment Orchestration
-**The TRAINS Agent was built to address the DL/ML R&D DevOps needs:**
+### Simple, Flexible Experiment Orchestration
+**The ClearML Agent was built to address the DL/ML R&D DevOps needs:**
 
 * Easily add & remove machines from the cluster
 * Reuse machines without the need for any dedicated containers or images
 * **Combine GPU resources across any cloud and on-prem**
-* **No need for yaml/json/template configuration of any kind**
+* **No need for yaml / json / template configuration of any kind**
 * **User friendly UI**
 * Manageable resource allocation that can be used by researchers and engineers
 * Flexible and controllable scheduler with priority support
-* Automatic instance spinning in the cloud **(coming soon)**
+* Automatic instance spinning in the cloud
+
+**Using the ClearML Agent, you can now set up a dynamic cluster with \*epsilon DevOps**
+
+*epsilon - Because we are :triangular_ruler: and nothing is really zero work
 
 
-## But ... K8S?
-We think Kubernetes is awesome.  
-Combined with KubeFlow it is a robust solution for production-grade DevOps.  
-We've observed, however, that it can be a bit of an overkill as an R&D DL/ML solution.
-If you are considering K8S for your research, also consider that you will soon be managing **hundreds** of containers...
+### Kubernetes Integration (Optional)
+We think Kubernetes is awesome, but it should be a choice.
+We designed `clearml-agent` so you can run bare-metal or inside a pod with any mix that fits your environment.
+#### Benefits of integrating existing K8s with ClearML-Agent 
+- ClearML-Agent adds the missing scheduling capabilities to K8s
+- Allowing for more flexible automation from code
+- A programmatic interface for easier learning curve (and debugging)
+- Seamless integration with ML/DL experiment manager
+- Web UI for customization, scheduling & prioritization of jobs 
 
-In our experience, handling and building the environments, having to package every experiment in a docker, managing those hundreds (or more) containers and building pipelines on top of it all, is very complicated (also, it’s usually out of scope for the research team, and overwhelming even for the DevOps team).
+**Two K8s integration flavours** 
+- Spin ClearML-Agent as a long-lasting service pod
+    - use [clearml-agent](https://hub.docker.com/r/allegroai/trains-agent) docker image
+    - map docker socket into the pod (soon replaced by [podman](https://github.com/containers/podman))
+    - allow the clearml-agent to manage sibling dockers
+    - benefits: full use of the ClearML scheduling, no need to worry about wrong container images / lost pods etc.
+    - downside: Sibling containers
+- Kubernetes Glue, map ClearML jobs directly to K8s jobs
+    - Run the [clearml-k8s glue](https://github.com/allegroai/trains-agent/blob/master/examples/k8s_glue_example.py) on a K8s cpu node
+    - The clearml-k8s glue pulls jobs from the ClearML job execution queue and prepares a K8s job (based on provided yaml template)
+    - Inside the pod itself the clearml-agent will install the job (experiment) environment and spin and monitor the experiment's process
+    - benefits: Kubernetes full view of all running jobs in the system
+    - downside: No real scheduling (k8s scheduler), no docker image verification (post-mortem only) 
 
-We feel there has to be a better way, that can be just as powerful for R&D and at the same time allow integration with K8S **when the need arises**.  
-(If you already have a K8S cluster for AI, detailed instructions on how to integrate TRAINS into your K8S cluster are *coming soon*.)
-
-
-## Using the TRAINS Agent
+### Using the ClearML Agent
 **Full scale HPC with a click of a button**
 
-TRAINS Agent is a job scheduler that listens on job queue(s), pulls jobs, sets the job environments, executes the job and monitors its progress.
+The ClearML Agent is a job scheduler that listens on job queue(s), pulls jobs, sets the job environments, executes the job and monitors its progress.
 
-Any 'Draft' experiment can be scheduled for execution by a TRAINS agent.
+Any 'Draft' experiment can be scheduled for execution by a ClearML agent.
 
 A previously run experiment can be put into 'Draft' state by either of two methods:
 * Using the **'Reset'** action from the experiment right-click context menu in the
-  TRAINS UI - This will clear any results and artifacts the previous run had created.
+  ClearML UI - This will clear any results and artifacts the previous run had created.
 * Using the **'Clone'** action from the experiment right-click context menu in the
-  TRAINS UI - This will create a new 'Draft' experiment with the same configuration as the original experiment.
+  ClearML UI - This will create a new 'Draft' experiment with the same configuration as the original experiment.
 
 An experiment is scheduled for execution using the **'Enqueue'** action from the experiment
- right-click context menu in the TRAINS UI and selecting the execution queue.
+ right-click context menu in the ClearML UI and selecting the execution queue.
 
 See [creating an experiment and enqueuing it for execution](#from-scratch).
 
-Once an experiment is enqueued, it will be picked up and executed by a TRAINS agent monitoring this queue.
+Once an experiment is enqueued, it will be picked up and executed by a ClearML agent monitoring this queue.
 
-The TRAINS UI Workers & Queues page provides ongoing execution information:
+The ClearML UI Workers & Queues page provides ongoing execution information:
   - Workers Tab: Monitor you cluster
     - Review available resources
     - Monitor machines statistics (CPU / GPU / Disk / Network)
@@ -83,154 +110,129 @@ The TRAINS UI Workers & Queues page provides ongoing execution information:
     - Cancel or abort job execution
     - Move jobs between execution queues
 
-### What The TRAINS Agent Actually Does
-The TRAINS agent executes experiments using the following process:
+#### What The ClearML Agent Actually Does
+The ClearML Agent executes experiments using the following process:
   - Create a new virtual environment (or launch the selected docker image)
   - Clone the code into the virtual-environment (or inside the docker)
   - Install python packages based on the package requirements listed for the experiment
-    - Special note for PyTorch: The TRAINS agent will automatically select the
+    - Special note for PyTorch: The ClearML Agent will automatically select the
       torch packages based on the CUDA_VERSION environment variable of the machine
   - Execute the code, while monitoring the process
-  - Log all stdout/stderr in the TRAINS UI, including the cloning and installation process, for easy debugging
-  - Monitor the execution and allow you to manually abort the job using the TRAINS UI (or, in the unfortunate case of a code crash, catch the error and signal the experiment has failed)
+  - Log all stdout/stderr in the ClearML UI, including the cloning and installation process, for easy debugging
+  - Monitor the execution and allow you to manually abort the job using the ClearML UI (or, in the unfortunate case of a code crash, catch the error and signal the experiment has failed)
 
-### System Design & Flow
-```text
-                                                                              +-----------------+
-                                                                              |  GPU  Machine   |
-Development Machine                                                           |                 |
-+------------------------+                                                    | +-------------+ |
-|    Data Scientist's    |                            +--------------+        | |TRAINS Agent | |
-|      DL/ML Code        |                            |    WEB UI    |        | |             | |
-|                        |                            |              |        | | +---------+ | |
-|                        |                            |              |        | | |  DL/ML  | | |
-|                        |                            +--------------+        | | |  Code   | | |
-|                        |       User Clones Exp #1  / . . . . . . . /        | | |         | | |
-| +-------------------+  |           into Exp #2    / . . . . . . . /         | | +---------+ | |
-| |      TRAINS       |  |         +---------------/-_____________-/          | |             | |
-| +---------+---------+  |         |                                          | |      ^      | |
-+-----------|------------+         |                                          | +------|------+ |
-            |                      |                                          +--------|--------+
- Auto-Magically                    |                                                   |
- Creates Exp #1                    |                                      The TRAINS Agent
-             \          User Change Hyper-Parameters                      Pulls Exp #2, setup the
-             |                     |                                      environment & clone code.
-             |                     |                                      Start execution with the
-+------------|------------+        |            +--------------------+    new set of Hyper-Parameters.
-|  +---------v---------+  |        |            |   TRAINS-SERVER    |                 |
-|  | Experiment #1     |  |        |            |                    |                 |
-|  +-------------------+  |        |            |  Execution Queue   |                 |
-|            ||           |        |            |                    |                 |
-|  +-------------------+<----------+            |                    |                 |
-|  |                   |  |                     |                    |                 |
-|  | Experiment #2     |  |                     |                    |                 |
-|  +-------------------<------------\           |                    |                 |
-|                         |          ------------->---------------+  |                 |
-|                         |  User Send Exp #2   | |Execute Exp #2 +--------------------+
-|                         |  For Execution      | +---------------+  |
-|     TRAINS-SERVER       |                     |                    |
-+-------------------------+                     +--------------------+
-```
+#### System Design & Flow
 
-### Installing the TRAINS Agent
+<img src="https://allegro.ai/clearml/docs/img/ClearML_Architecture.png" width="100%" alt="clearml-architecture">
+
+
+#### Installing the ClearML Agent
 
 ```bash
-pip install trains-agent
+pip install clearml-agent
 ```
 
-### TRAINS Agent Usage Examples
+#### ClearML Agent Usage Examples
 
 Full Interface and capabilities are available with
 ```bash
-trains-agent --help
-trains-agent daemon --help
+clearml-agent --help
+clearml-agent daemon --help
 ```
 
-### Configuring the TRAINS Agent
+#### Configuring the ClearML Agent
 
 ```bash
-trains-agent init
+clearml-agent init
 ```
 
-Note: The TRAINS agent uses a cache folder to cache pip packages, apt packages and cloned repositories. The default TRAINS Agent cache folder is `~/.trains`
+Note: The ClearML Agent uses a cache folder to cache pip packages, apt packages and cloned repositories. The default ClearML Agent cache folder is `~/.clearml`
 
-See full details in your configuration file at `~/trains.conf`
+See full details in your configuration file at `~/clearml.conf`
 
-Note: The **TRAINS agent** extends the **TRAINS** configuration file `~/trains.conf`
-They are designed to share the same configuration file, see example [here](docs/trains.conf)
+Note: The **ClearML agent** extends the **ClearML** configuration file `~/clearml.conf`
+They are designed to share the same configuration file, see example [here](docs/clearml.conf)
 
-### Running the TRAINS Agent
+#### Running the ClearML Agent
 
-For debug and experimentation, start the TRAINS agent in `foreground` mode, where all the output is printed to screen
+For debug and experimentation, start the ClearML agent in `foreground` mode, where all the output is printed to screen
 ```bash
-trains-agent daemon --queue default --foreground
+clearml-agent daemon --queue default --foreground
 ```
 
 For actual service mode, all the stdout will be stored automatically into a temporary file (no need to pipe)
-Notice: with `--detached` flag, the *trains-agent* will be running in the background
+Notice: with `--detached` flag, the *clearml-agent* will be running in the background
 ```bash
-trains-agent daemon --detached --queue default
+clearml-agent daemon --detached --queue default
 ```
 
 GPU allocation is controlled via the standard OS environment `NVIDIA_VISIBLE_DEVICES` or `--gpus` flag (or disabled with `--cpu-only`).
 
-If no flag is set, and `NVIDIA_VISIBLE_DEVICES` variable doesn't exist, all GPU's will be allocated for the `trains-agent` <br>
-If `--cpu-only` flag is set, or `NVIDIA_VISIBLE_DEVICES` is an empty string (""), no gpu will be allocated for the `trains-agent`
+If no flag is set, and `NVIDIA_VISIBLE_DEVICES` variable doesn't exist, all GPU's will be allocated for the `clearml-agent` <br>
+If `--cpu-only` flag is set, or `NVIDIA_VISIBLE_DEVICES` is an empty string (""), no gpu will be allocated for the `clearml-agent`
 
 Example: spin two agents, one per gpu on the same machine:
-Notice: with `--detached` flag, the *trains-agent* will be running in the background
+Notice: with `--detached` flag, the *clearml-agent* will be running in the background
 ```bash
-trains-agent daemon --detached --gpus 0 --queue default
-trains-agent daemon --detached --gpus 1 --queue default
+clearml-agent daemon --detached --gpus 0 --queue default
+clearml-agent daemon --detached --gpus 1 --queue default
 ```
 
 Example: spin two agents, pulling from dedicated `dual_gpu` queue, two gpu's per agent
 ```bash
-trains-agent daemon --detached --gpus 0,1 --queue dual_gpu
-trains-agent daemon --detached --gpus 2,3 --queue dual_gpu
+clearml-agent daemon --detached --gpus 0,1 --queue dual_gpu
+clearml-agent daemon --detached --gpus 2,3 --queue dual_gpu
 ```
 
-#### Starting the TRAINS Agent in docker mode
+##### Starting the ClearML Agent in docker mode
 
-For debug and experimentation, start the TRAINS agent in `foreground` mode, where all the output is printed to screen
+For debug and experimentation, start the ClearML agent in `foreground` mode, where all the output is printed to screen
 ```bash
-trains-agent daemon --queue default --docker --foreground
+clearml-agent daemon --queue default --docker --foreground
 ```
 
 For actual service mode, all the stdout will be stored automatically into a file (no need to pipe)
-Notice: with `--detached` flag, the *trains-agent* will be running in the background
+Notice: with `--detached` flag, the *clearml-agent* will be running in the background
 ```bash
-trains-agent daemon --detached --queue default --docker
+clearml-agent daemon --detached --queue default --docker
 ```
 
 Example: spin two agents, one per gpu on the same machine, with default nvidia/cuda docker:
 ```bash
-trains-agent daemon --detached --gpus 0 --queue default --docker nvidia/cuda
-trains-agent daemon --detached --gpus 1 --queue default --docker nvidia/cuda
+clearml-agent daemon --detached --gpus 0 --queue default --docker nvidia/cuda
+clearml-agent daemon --detached --gpus 1 --queue default --docker nvidia/cuda
 ```
 
 Example: spin two agents, pulling from dedicated `dual_gpu` queue, two gpu's per agent, with default nvidia/cuda docker:
 ```bash
-trains-agent daemon --detached --gpus 0,1 --queue dual_gpu --docker nvidia/cuda
-trains-agent daemon --detached --gpus 2,3 --queue dual_gpu --docker nvidia/cuda
+clearml-agent daemon --detached --gpus 0,1 --queue dual_gpu --docker nvidia/cuda
+clearml-agent daemon --detached --gpus 2,3 --queue dual_gpu --docker nvidia/cuda
 ```
 
-#### Starting the TRAINS Agent - Priority Queues
+##### Starting the ClearML Agent - Priority Queues
 
 Priority Queues are also supported, example use case:
 
 High priority queue: `important_jobs`  Low priority queue: `default`
 ```bash
-trains-agent daemon --queue important_jobs default
+clearml-agent daemon --queue important_jobs default
 ```
-The **TRAINS agent** will first try to pull jobs from the `important_jobs` queue, only then it will fetch a job from the `default` queue.
+The **ClearML Agent** will first try to pull jobs from the `important_jobs` queue, only then it will fetch a job from the `default` queue.
 
-Adding queues, managing job order within a queue and moving jobs between queues, is available using the Web UI, see example on our [open server](https://demoapp.trains.allegro.ai/workers-and-queues/queues)
+Adding queues, managing job order within a queue and moving jobs between queues, is available using the Web UI, see example on our [free server](https://app.community.clear.ml/workers-and-queues/queues)
 
-# How do I create an experiment on the TRAINS server? <a name="from-scratch"></a>
-* Integrate [TRAINS](https://github.com/allegroai/trains) with your code
+##### Stopping the ClearML Agent
+
+To stop a **ClearML Agent** running in the background, run the same command line used to start the agent with `--stop` appended.
+For example, to stop the first of the above shown same machine, single gpu agents:
+```bash
+clearml-agent daemon --detached --gpus 0 --queue default --docker nvidia/cuda --stop
+```
+
+### How do I create an experiment on the ClearML Server? <a name="from-scratch"></a>
+* Integrate [ClearML](https://github.com/allegroai/trains) with your code
 * Execute the code on your machine (Manually / PyCharm / Jupyter Notebook)
-* As your code is running, **TRAINS** creates an experiment logging all the necessary execution information:
+* As your code is running, **ClearML** creates an experiment logging all the necessary execution information:
   - Git repository link and commit ID (or an entire jupyter notebook)
   - Git diff (we’re not saying you never commit and push, but still...)
   - Python packages used by your code (including specific versions used)
@@ -239,7 +241,7 @@ Adding queues, managing job order within a queue and moving jobs between queues,
 
   You now have a 'template' of your experiment with everything required for automated execution
 
-* In the TRAINS UI, Right click on the experiment and select 'clone'. A copy of your experiment will be created.
+* In the ClearML UI, Right click on the experiment and select 'clone'. A copy of your experiment will be created.
 * You now have a new draft experiment cloned from your original experiment, feel free to edit it
   - Change the Hyper-Parameters
   - Switch to the latest code base of the repository
@@ -248,23 +250,44 @@ Adding queues, managing job order within a queue and moving jobs between queues,
   - Or simply change nothing to run the same experiment again...
 * Schedule the newly created experiment for execution: Right-click the experiment and select 'enqueue'
 
-# AutoML and Orchestration Pipelines <a name="automl-pipes"></a>
-The TRAINS Agent can also be used to implement AutoML orchestration and Experiment Pipelines in conjunction with the TRAINS package.
+### ClearML-Agent Services Mode <a name="services"></a>
 
-Sample AutoML & Orchestration examples can be found in the TRAINS [example/automl](https://github.com/allegroai/trains/tree/master/examples/automl) folder.
+ClearML-Agent Services is a special mode of ClearML-Agent that provides the ability to launch long-lasting jobs
+that previously had to be executed on local / dedicated machines. It allows a single agent to
+launch multiple dockers (Tasks) for different use cases. To name a few use cases, auto-scaler service (spinning instances
+when the need arises and the budget allows), Controllers (Implementing pipelines and more sophisticated DevOps logic),
+Optimizer (such as Hyper-parameter Optimization or sweeping), and Application (such as interactive Bokeh apps for
+increased data transparency)
+
+ClearML-Agent Services mode will spin **any** task enqueued into the specified queue.
+Every task launched by ClearML-Agent Services will be registered as a new node in the system,
+providing tracking and transparency capabilities.
+Currently clearml-agent in services-mode supports cpu only configuration. ClearML-agent services mode can be launched alongside GPU agents.
+
+```bash
+clearml-agent daemon --services-mode --detached --queue services --create-queue --docker ubuntu:18.04 --cpu-only
+```
+
+**Note**: It is the user's responsibility to make sure the proper tasks are pushed into the specified queue.
+
+
+### AutoML and Orchestration Pipelines <a name="automl-pipes"></a>
+The ClearML Agent can also be used to implement AutoML orchestration and Experiment Pipelines in conjunction with the ClearML package.
+
+Sample AutoML & Orchestration examples can be found in the ClearML [example/automation](https://github.com/allegroai/trains/tree/master/examples/automation) folder.
 
 AutoML examples
-  - [Toy Keras training experiment](https://github.com/allegroai/trains/blob/master/examples/automl/automl_base_template_keras_simple.py)
+  - [Toy Keras training experiment](https://github.com/allegroai/trains/blob/master/examples/optimization/hyper-parameter-optimization/base_template_keras_simple.py)
     - In order to create an experiment-template in the system, this code must be executed once manually
-  - [Random Search over the above Keras experiment-template](https://github.com/allegroai/trains/blob/master/examples/automl/automl_random_search_example.py)
+  - [Random Search over the above Keras experiment-template](https://github.com/allegroai/trains/blob/master/examples/automation/manual_random_param_search_example.py)
     - This example will create multiple copies of the Keras experiment-template, with different hyper-parameter combinations
 
 Experiment Pipeline examples
-  - [First step experiment](https://github.com/allegroai/trains/blob/master/examples/automl/task_piping_example.py)
+  - [First step experiment](https://github.com/allegroai/trains/blob/master/examples/automation/task_piping_example.py)
     - This example will "process data", and once done, will launch a copy of the 'second step' experiment-template
-  - [Second step experiment](https://github.com/allegroai/trains/blob/master/examples/automl/toy_base_task.py)
+  - [Second step experiment](https://github.com/allegroai/trains/blob/master/examples/automation/toy_base_task.py)
     - In order to create an experiment-template in the system, this code must be executed once manually
 
-# License
+### License
 
 Apache License, Version 2.0 (see the [LICENSE](https://www.apache.org/licenses/LICENSE-2.0.html) for more information)

clearml_agent/__main__.py

@@ -4,13 +4,13 @@ import argparse
 import sys
 import warnings
 
-from trains_agent.backend_api.session.datamodel import UnusedKwargsWarning
+from clearml_agent.backend_api.session.datamodel import UnusedKwargsWarning
 
-import trains_agent
-from trains_agent.config import get_config
-from trains_agent.definitions import FileBuffering, CONFIG_FILE
-from trains_agent.helper.base import reverse_home_folder_expansion, chain_map, named_temporary_file
-from trains_agent.helper.process import ExitStatus
+import clearml_agent
+from clearml_agent.config import get_config
+from clearml_agent.definitions import FileBuffering, CONFIG_FILE
+from clearml_agent.helper.base import reverse_home_folder_expansion, chain_map, named_temporary_file
+from clearml_agent.helper.process import ExitStatus
 from . import interface, session, definitions, commands
 from .errors import ConfigFileNotFound, Sigterm, APIError
 from .helper.trace import PackageTrace
@@ -47,7 +47,7 @@ def run_command(parser, args, command_name):
     except ConfigFileNotFound:
         message = 'Cannot find configuration file in "{}".\n' \
                   'To create a configuration file, run:\n' \
-                  '$ trains_agent init'.format(reverse_home_folder_expansion(CONFIG_FILE))
+                  '$ clearml_agent init'.format(reverse_home_folder_expansion(CONFIG_FILE))
         command_class.exit(message)
     except APIError as api_error:
         if not debug:

clearml_agent/backend_api/config/default/agent.conf

@@ -0,0 +1,165 @@
+{
+    # unique name of this worker, if None, created based on hostname:process_id
+    # Override with os environment: CLEARML_WORKER_ID
+    # worker_id: "clearml-agent-machine1:gpu0"
+    worker_id: ""
+
+    # worker name, replaces the hostname when creating a unique name for this worker
+    # Override with os environment: CLEARML_WORKER_NAME
+    # worker_name: "clearml-agent-machine1"
+    worker_name: ""
+
+    # Set GIT user/pass credentials (if user/pass are set, GIT protocol will be set to https)
+    # leave blank for GIT SSH credentials (set force_git_ssh_protocol=true to force SSH protocol)
+    # git_user: ""
+    # git_pass: ""
+    # git_host: ""
+
+    # Force GIT protocol to use SSH regardless of the git url (Assumes GIT user/pass are blank)
+    force_git_ssh_protocol: false
+    # Force a specific SSH port when converting http to ssh links (the domain is kept the same)
+    # force_git_ssh_port: 0
+
+    # Set the python version to use when creating the virtual environment and launching the experiment
+    # Example values: "/usr/bin/python3" or "/usr/local/bin/python3.6"
+    # The default is the python executing the clearml_agent
+    python_binary: ""
+
+    # select python package manager:
+    # currently supported pip and conda
+    # poetry is used if pip selected and repository contains poetry.lock file
+    package_manager: {
+        # supported options: pip, conda, poetry
+        type: pip,
+
+        # specify pip version to use (examples "<20", "==19.3.1", "", empty string will install the latest version)
+        pip_version: "<20.2",
+
+        # virtual environment inheres packages from system
+        system_site_packages: false,
+
+        # install with --upgrade
+        force_upgrade: false,
+
+        # additional artifact repositories to use when installing python packages
+        # extra_index_url: ["https://allegroai.jfrog.io/clearmlai/api/pypi/public/simple"]
+
+        # additional conda channels to use when installing with conda package manager
+        conda_channels: ["defaults", "conda-forge", "pytorch", ]
+
+        # If set to true, Task's "installed packages" are ignored,
+        # and the repository's "requirements.txt" is used instead
+        # force_repo_requirements_txt: false
+
+        # set the priority packages to be installed before the rest of the required packages
+        # priority_packages: ["cython", "numpy", "setuptools", ]
+
+        # set the optional priority packages to be installed before the rest of the required packages,
+        # In case a package installation fails, the package will be ignored,
+        # and the virtual environment process will continue
+        # priority_optional_packages: ["pygobject", ]
+
+        # set the post packages to be installed after all the rest of the required packages
+        # post_packages: ["horovod", ]
+
+        # set the optional post packages to be installed after all the rest of the required packages,
+        # In case a package installation fails, the package will be ignored,
+        # and the virtual environment process will continue
+        # post_optional_packages: []
+
+        # set to True to support torch nightly build installation,
+        # notice: torch nightly builds are ephemeral and are deleted from time to time
+        torch_nightly: false,
+    },
+
+    # target folder for virtual environments builds, created when executing experiment
+    venvs_dir = ~/.clearml/venvs-builds
+
+    # cached git clone folder
+    vcs_cache: {
+        enabled: true,
+        path: ~/.clearml/vcs-cache
+    },
+
+    # use venv-update in order to accelerate python virtual environment building
+    # Still in beta, turned off by default
+    venv_update: {
+        enabled: false,
+    },
+
+    # cached folder for specific python package download (used for pytorch package caching)
+    pip_download_cache {
+        enabled: true,
+        path: ~/.clearml/pip-download-cache
+    },
+
+    translate_ssh: true,
+    # reload configuration file every daemon execution
+    reload_config: false,
+
+    # pip cache folder mapped into docker, used for python package caching
+    docker_pip_cache = ~/.clearml/pip-cache
+    # apt cache folder mapped into docker, used for ubuntu package caching
+    docker_apt_cache = ~/.clearml/apt-cache
+
+    # optional arguments to pass to docker image
+    # these are local for this agent and will not be updated in the experiment's docker_cmd section
+    # extra_docker_arguments: ["--ipc=host", ]
+
+    # optional shell script to run in docker when started before the experiment is started
+    # extra_docker_shell_script: ["apt-get install -y bindfs", ]
+
+    # optional uptime configuration, make sure to use only one of 'uptime/downtime' and not both.
+    # If uptime is specified, agent will actively poll (and execute) tasks in the time-spans defined here.
+    # Outside of the specified time-spans, the agent will be idle.
+    # Defined using a list of items of the format: "<hours> <days>".
+    # hours - use values 0-23, single values would count as start hour and end at midnight.
+    # days - use days in abbreviated format (SUN-SAT)
+    # use '-' for ranges and ',' to separate singular values.
+    # for example, to enable the workers every Sunday and Tuesday between 17:00-20:00 set uptime to:
+    # uptime: ["17-20 SUN,TUE"]
+
+    # optional downtime configuration, can be used only when uptime is not used.
+    # If downtime is specified, agent will be idle in the time-spans defined here.
+    # Outside of the specified time-spans, the agent will actively poll (and execute) tasks.
+    # Use the same format as described above for uptime
+    # downtime: []
+
+    # set to true in order to force "docker pull" before running an experiment using a docker image.
+    # This makes sure the docker image is updated.
+    docker_force_pull: false
+
+    default_docker: {
+        # default docker image to use when running in docker mode
+        image: "nvidia/cuda:10.1-runtime-ubuntu18.04"
+
+        # optional arguments to pass to docker image
+        # arguments: ["--ipc=host", ]
+    }
+
+    # set the initial bash script to execute at the startup of any docker.
+    # all lines will be executed regardless of their exit code.
+    # {python_single_digit} is translated to 'python3' or 'python2' according to requested python version
+    # docker_init_bash_script = [
+    #     "echo 'Binary::apt::APT::Keep-Downloaded-Packages \"true\";' > /etc/apt/apt.conf.d/docker-clean",
+    #     "chown -R root /root/.cache/pip",
+    #     "apt-get update",
+    #     "apt-get install -y git libsm6 libxext6 libxrender-dev libglib2.0-0",
+    #     "(which {python_single_digit} && {python_single_digit} -m pip --version) || apt-get install -y {python_single_digit}-pip",
+    # ]
+
+    # set the preprocessing bash script to execute at the startup of any docker.
+    # all lines will be executed regardless of their exit code.
+    # docker_preprocess_bash_script = [
+    #     "echo \"starting docker\"",
+    #]
+
+    # If False replace \r with \n and display full console output
+    # default is True, report a single \r line in a sequence of consecutive lines, per 5 seconds.
+    # suppress_carriage_return: true
+
+    # cuda versions used for solving pytorch wheel packages
+    # should be detected automatically. Override with os environment CUDA_VERSION / CUDNN_VERSION
+    # cuda_version: 10.1
+    # cudnn_version: 7.6
+}

clearml_agent/backend_api/config/default/sdk.conf

@@ -1,10 +1,10 @@
 {
-    # TRAINS - default SDK configuration
+    # ClearML - default SDK configuration
 
     storage {
         cache {
             # Defaults to system temp folder / cache
-            default_base_dir: "~/.trains/cache"
+            default_base_dir: "~/.clearml/cache"
             size {
                 # max_used_bytes = -1
                 min_free_bytes = 10GB
@@ -31,12 +31,18 @@
         # X images are stored in the upload destination for each matplotlib plot title.
         matplotlib_untitled_history_size: 100
 
+        # Limit the number of digits after the dot in plot reporting (reducing plot report size)
+        # plot_max_num_digits: 5
+
         # Settings for generated debug images
         images {
             format: JPEG
             quality: 87
             subsampling: 0
         }
+
+        # Support plot-per-graph fully matching Tensorboard behavior (i.e. if this is set to true, each series should have its own graph)
+        tensorboard_single_series_per_graph: false
     }
 
     network {
@@ -92,7 +98,7 @@
     google.storage {
         # # Default project and credentials file
         # # Will be used when no bucket configuration is found
-        # project: "trains"
+        # project: "clearml"
         # credentials_json: "/path/to/credentials.json"
 
         # # Specific credentials per bucket and sub directory
@@ -100,7 +106,7 @@
         #     {
         #         bucket: "my-bucket"
         #         subdir: "path/in/bucket" # Not required
-        #         project: "trains"
+        #         project: "clearml"
         #         credentials_json: "/path/to/credentials.json"
         #     },
         # ]
@@ -108,7 +114,7 @@
     azure.storage {
         # containers: [
         #     {
-        #         account_name: "trains"
+        #         account_name: "clearml"
         #         account_key: "secret"
         #         # container_name:
         #     }
@@ -117,11 +123,11 @@
 
     log {
         # debugging feature: set this to true to make null log propagate messages to root logger (so they appear in stdout)
-        null_log_propagate: False
+        null_log_propagate: false
         task_log_buffer_capacity: 66
 
         # disable urllib info and lower levels
-        disable_urllib3_info: True
+        disable_urllib3_info: true
     }
 
     development {
@@ -131,14 +137,30 @@
         task_reuse_time_window_in_hours: 72.0
 
         # Run VCS repository detection asynchronously
-        vcs_repo_detect_async: True
+        vcs_repo_detect_async: true
 
         # Store uncommitted git/hg source code diff in experiment manifest when training in development mode
         # This stores "git diff" or "hg diff" into the experiment's "script.requirements.diff" section
-        store_uncommitted_code_diff_on_train: True
+        store_uncommitted_code_diff: true
 
         # Support stopping an experiment in case it was externally stopped, status was changed or task was reset
-        support_stopping: True
+        support_stopping: true
+
+        # Default Task output_uri. if output_uri is not provided to Task.init, default_output_uri will be used instead.
+        default_output_uri: ""
+
+        # Default auto generated requirements optimize for smaller requirements
+        # If True, analyze the entire repository regardless of the entry point.
+        # If False, first analyze the entry point script, if it does not contain other to local files,
+        # do not analyze the entire repository.
+        force_analyze_entire_repo: false
+
+        # If set to true, *clearml* update message will not be printed to the console
+        # this value can be overwritten with os environment variable CLEARML_SUPPRESS_UPDATE_MESSAGE=1
+        suppress_update_message: false
+
+        # If this flag is true (default is false), instead of analyzing the code with Pigar, analyze with `pip freeze`
+        detect_with_pip_freeze: false
 
         # Development mode worker
         worker {
@@ -149,7 +171,11 @@
             ping_period_sec: 30
 
             # Log all stdout & stderr
-            log_stdout: True
+            log_stdout: true
+
+            # compatibility feature, report memory usage for the entire machine
+            # default (false), report only on the running process and its sub-processes
+            report_global_mem_used: false
         }
     }
-}
+}

clearml_agent/backend_api/session/client/client.py

@@ -106,15 +106,15 @@ class StrictSession(Session):
             init()
             return
 
-        original = os.environ.get(LOCAL_CONFIG_FILE_OVERRIDE_VAR, None)
+        original = LOCAL_CONFIG_FILE_OVERRIDE_VAR.get() or None
         try:
-            os.environ[LOCAL_CONFIG_FILE_OVERRIDE_VAR] = str(config_file)
+            LOCAL_CONFIG_FILE_OVERRIDE_VAR.set(str(config_file))
             init()
         finally:
             if original is None:
-                os.environ.pop(LOCAL_CONFIG_FILE_OVERRIDE_VAR, None)
+                LOCAL_CONFIG_FILE_OVERRIDE_VAR.pop()
             else:
-                os.environ[LOCAL_CONFIG_FILE_OVERRIDE_VAR] = original
+                LOCAL_CONFIG_FILE_OVERRIDE_VAR.set(original)
 
     def send(self, request, *args, **kwargs):
         result = super(StrictSession, self).send(request, *args, **kwargs)
@@ -222,7 +222,7 @@ class TableResponse(Response):
             return "" if result is None else result
 
         fields = fields or self.fields
-        from trains_agent.helper.base import create_table
+        from clearml_agent.helper.base import create_table
         return create_table(
             (dict((attr, getter(item, attr)) for attr in fields) for item in self),
             titles=fields, columns=fields, headers=True,

clearml_agent/backend_api/session/defs.py

@@ -0,0 +1,11 @@
+from ...backend_config.environment import EnvEntry
+
+
+ENV_HOST = EnvEntry("CLEARML_API_HOST", "TRAINS_API_HOST")
+ENV_WEB_HOST = EnvEntry("CLEARML_WEB_HOST", "TRAINS_WEB_HOST")
+ENV_FILES_HOST = EnvEntry("CLEARML_FILES_HOST", "TRAINS_FILES_HOST")
+ENV_ACCESS_KEY = EnvEntry("CLEARML_API_ACCESS_KEY", "TRAINS_API_ACCESS_KEY")
+ENV_SECRET_KEY = EnvEntry("CLEARML_API_SECRET_KEY", "TRAINS_API_SECRET_KEY")
+ENV_VERBOSE = EnvEntry("CLEARML_API_VERBOSE", "TRAINS_API_VERBOSE", type=bool, default=False)
+ENV_HOST_VERIFY_CERT = EnvEntry("CLEARML_API_HOST_VERIFY_CERT", "TRAINS_API_HOST_VERIFY_CERT", type=bool, default=True)
+ENV_CONDA_ENV_PACKAGE = EnvEntry("CLEARML_CONDA_ENV_PACKAGE", "TRAINS_CONDA_ENV_PACKAGE")

clearml_agent/backend_api/session/jsonmodels/__init__.py

@@ -0,0 +1,9 @@
+# coding: utf-8
+
+__author__ = 'Szczepan Cieślik'
+__email__ = 'szczepan.cieslik@gmail.com'
+__version__ = '2.4'
+
+from . import models
+from . import fields
+from . import errors

clearml_agent/backend_api/session/jsonmodels/builders.py

@@ -0,0 +1,230 @@
+"""Builders to generate in memory representation of model and fields tree."""
+
+from __future__ import absolute_import
+
+from collections import defaultdict
+
+import six
+
+from . import errors
+from .fields import NotSet
+
+
+class Builder(object):
+
+    def __init__(self, parent=None, nullable=False, default=NotSet):
+        self.parent = parent
+        self.types_builders = {}
+        self.types_count = defaultdict(int)
+        self.definitions = set()
+        self.nullable = nullable
+        self.default = default
+
+    @property
+    def has_default(self):
+        return self.default is not NotSet
+
+    def register_type(self, type, builder):
+        if self.parent:
+            return self.parent.register_type(type, builder)
+
+        self.types_count[type] += 1
+        if type not in self.types_builders:
+            self.types_builders[type] = builder
+
+    def get_builder(self, type):
+        if self.parent:
+            return self.parent.get_builder(type)
+
+        return self.types_builders[type]
+
+    def count_type(self, type):
+        if self.parent:
+            return self.parent.count_type(type)
+
+        return self.types_count[type]
+
+    @staticmethod
+    def maybe_build(value):
+        return value.build() if isinstance(value, Builder) else value
+
+    def add_definition(self, builder):
+        if self.parent:
+            return self.parent.add_definition(builder)
+
+        self.definitions.add(builder)
+
+
+class ObjectBuilder(Builder):
+
+    def __init__(self, model_type, *args, **kwargs):
+        super(ObjectBuilder, self).__init__(*args, **kwargs)
+        self.properties = {}
+        self.required = []
+        self.type = model_type
+
+        self.register_type(self.type, self)
+
+    def add_field(self, name, field, schema):
+        _apply_validators_modifications(schema, field)
+        self.properties[name] = schema
+        if field.required:
+            self.required.append(name)
+
+    def build(self):
+        builder = self.get_builder(self.type)
+        if self.is_definition and not self.is_root:
+            self.add_definition(builder)
+            [self.maybe_build(value) for _, value in self.properties.items()]
+            return '#/definitions/{name}'.format(name=self.type_name)
+        else:
+            return builder.build_definition(nullable=self.nullable)
+
+    @property
+    def type_name(self):
+        module_name = '{module}.{name}'.format(
+            module=self.type.__module__,
+            name=self.type.__name__,
+        )
+        return module_name.replace('.', '_').lower()
+
+    def build_definition(self, add_defintitions=True, nullable=False):
+        properties = dict(
+            (name, self.maybe_build(value))
+            for name, value
+            in self.properties.items()
+        )
+        schema = {
+            'type': 'object',
+            'additionalProperties': False,
+            'properties': properties,
+        }
+        if self.required:
+            schema['required'] = self.required
+        if self.definitions and add_defintitions:
+            schema['definitions'] = dict(
+                (builder.type_name, builder.build_definition(False, False))
+                for builder in self.definitions
+            )
+        return schema
+
+    @property
+    def is_definition(self):
+        if self.count_type(self.type) > 1:
+            return True
+        elif self.parent:
+            return self.parent.is_definition
+        else:
+            return False
+
+    @property
+    def is_root(self):
+        return not bool(self.parent)
+
+
+def _apply_validators_modifications(field_schema, field):
+    for validator in field.validators:
+        try:
+            validator.modify_schema(field_schema)
+        except AttributeError:
+            pass
+
+
+class PrimitiveBuilder(Builder):
+
+    def __init__(self, type, *args, **kwargs):
+        super(PrimitiveBuilder, self).__init__(*args, **kwargs)
+        self.type = type
+
+    def build(self):
+        schema = {}
+        if issubclass(self.type, six.string_types):
+            obj_type = 'string'
+        elif issubclass(self.type, bool):
+            obj_type = 'boolean'
+        elif issubclass(self.type, int):
+            obj_type = 'number'
+        elif issubclass(self.type, float):
+            obj_type = 'number'
+        else:
+            raise errors.FieldNotSupported(
+                "Can't specify value schema!", self.type
+            )
+
+        if self.nullable:
+            obj_type = [obj_type, 'null']
+        schema['type'] = obj_type
+
+        if self.has_default:
+            schema["default"] = self.default
+
+        return schema
+
+
+class ListBuilder(Builder):
+
+    def __init__(self, *args, **kwargs):
+        super(ListBuilder, self).__init__(*args, **kwargs)
+        self.schemas = []
+
+    def add_type_schema(self, schema):
+        self.schemas.append(schema)
+
+    def build(self):
+        schema = {'type': 'array'}
+        if self.nullable:
+            self.add_type_schema({'type': 'null'})
+
+        if self.has_default:
+            schema["default"] = [self.to_struct(i) for i in self.default]
+
+        schemas = [self.maybe_build(s) for s in self.schemas]
+        if len(schemas) == 1:
+            items = schemas[0]
+        else:
+            items = {'oneOf': schemas}
+
+        schema['items'] = items
+        return schema
+
+    @property
+    def is_definition(self):
+        return self.parent.is_definition
+
+    @staticmethod
+    def to_struct(item):
+        from .models import Base
+        if isinstance(item, Base):
+            return item.to_struct()
+        return item
+
+
+class EmbeddedBuilder(Builder):
+
+    def __init__(self, *args, **kwargs):
+        super(EmbeddedBuilder, self).__init__(*args, **kwargs)
+        self.schemas = []
+
+    def add_type_schema(self, schema):
+        self.schemas.append(schema)
+
+    def build(self):
+        if self.nullable:
+            self.add_type_schema({'type': 'null'})
+
+        schemas = [self.maybe_build(schema) for schema in self.schemas]
+        if len(schemas) == 1:
+            schema = schemas[0]
+        else:
+            schema = {'oneOf': schemas}
+
+        if self.has_default:
+            # The default value of EmbeddedField is expected to be an instance
+            # of a subclass of models.Base, thus have `to_struct`
+            schema["default"] = self.default.to_struct()
+
+        return schema
+
+    @property
+    def is_definition(self):
+        return self.parent.is_definition

clearml_agent/backend_api/session/jsonmodels/collections.py

@@ -0,0 +1,21 @@
+
+
+class ModelCollection(list):
+
+    """`ModelCollection` is list which validates stored values.
+
+    Validation is made with use of field passed to `__init__` at each point,
+    when new value is assigned.
+
+    """
+
+    def __init__(self, field):
+        self.field = field
+
+    def append(self, value):
+        self.field.validate_single_value(value)
+        super(ModelCollection, self).append(value)
+
+    def __setitem__(self, key, value):
+        self.field.validate_single_value(value)
+        super(ModelCollection, self).__setitem__(key, value)

clearml_agent/backend_api/session/jsonmodels/errors.py

@@ -0,0 +1,15 @@
+
+
+class ValidationError(RuntimeError):
+
+    pass
+
+
+class FieldNotFound(RuntimeError):
+
+    pass
+
+
+class FieldNotSupported(ValueError):
+
+    pass

clearml_agent/backend_api/session/jsonmodels/fields.py

@@ -0,0 +1,488 @@
+import datetime
+import re
+from weakref import WeakKeyDictionary
+
+import six
+from dateutil.parser import parse
+
+from .errors import ValidationError
+from .collections import ModelCollection
+
+
+# unique marker for "no default value specified". None is not good enough since
+# it is a completely valid default value.
+NotSet = object()
+
+
+class BaseField(object):
+
+    """Base class for all fields."""
+
+    types = None
+
+    def __init__(
+            self,
+            required=False,
+            nullable=False,
+            help_text=None,
+            validators=None,
+            default=NotSet,
+            name=None):
+        self.memory = WeakKeyDictionary()
+        self.required = required
+        self.help_text = help_text
+        self.nullable = nullable
+        self._assign_validators(validators)
+        self.name = name
+        self._validate_name()
+        if default is not NotSet:
+            self.validate(default)
+        self._default = default
+
+    @property
+    def has_default(self):
+        return self._default is not NotSet
+
+    def _assign_validators(self, validators):
+        if validators and not isinstance(validators, list):
+            validators = [validators]
+        self.validators = validators or []
+
+    def __set__(self, instance, value):
+        self._finish_initialization(type(instance))
+        value = self.parse_value(value)
+        self.validate(value)
+        self.memory[instance._cache_key] = value
+
+    def __get__(self, instance, owner=None):
+        if instance is None:
+            self._finish_initialization(owner)
+            return self
+
+        self._finish_initialization(type(instance))
+
+        self._check_value(instance)
+        return self.memory[instance._cache_key]
+
+    def _finish_initialization(self, owner):
+        pass
+
+    def _check_value(self, obj):
+        if obj._cache_key not in self.memory:
+            self.__set__(obj, self.get_default_value())
+
+    def validate_for_object(self, obj):
+        value = self.__get__(obj)
+        self.validate(value)
+
+    def validate(self, value):
+        self._check_types()
+        self._validate_against_types(value)
+        self._check_against_required(value)
+        self._validate_with_custom_validators(value)
+
+    def _check_against_required(self, value):
+        if value is None and self.required:
+            raise ValidationError('Field is required!')
+
+    def _validate_against_types(self, value):
+        if value is not None and not isinstance(value, self.types):
+            raise ValidationError(
+                'Value is wrong, expected type "{types}"'.format(
+                    types=', '.join([t.__name__ for t in self.types])
+                ),
+                value,
+            )
+
+    def _check_types(self):
+        if self.types is None:
+            raise ValidationError(
+                'Field "{type}" is not usable, try '
+                'different field type.'.format(type=type(self).__name__))
+
+    def to_struct(self, value):
+        """Cast value to Python structure."""
+        return value
+
+    def parse_value(self, value):
+        """Parse value from primitive to desired format.
+
+        Each field can parse value to form it wants it to be (like string or
+        int).
+
+        """
+        return value
+
+    def _validate_with_custom_validators(self, value):
+        if value is None and self.nullable:
+            return
+
+        for validator in self.validators:
+            try:
+                validator.validate(value)
+            except AttributeError:
+                validator(value)
+
+    def get_default_value(self):
+        """Get default value for field.
+
+        Each field can specify its default.
+
+        """
+        return self._default if self.has_default else None
+
+    def _validate_name(self):
+        if self.name is None:
+            return
+        if not re.match('^[A-Za-z_](([\w\-]*)?\w+)?$', self.name):
+            raise ValueError('Wrong name', self.name)
+
+    def structue_name(self, default):
+        return self.name if self.name is not None else default
+
+
+class StringField(BaseField):
+
+    """String field."""
+
+    types = six.string_types
+
+
+class IntField(BaseField):
+
+    """Integer field."""
+
+    types = (int,)
+
+    def parse_value(self, value):
+        """Cast value to `int`, e.g. from string or long"""
+        parsed = super(IntField, self).parse_value(value)
+        if parsed is None:
+            return parsed
+        return int(parsed)
+
+
+class FloatField(BaseField):
+
+    """Float field."""
+
+    types = (float, int)
+
+
+class BoolField(BaseField):
+
+    """Bool field."""
+
+    types = (bool,)
+
+    def parse_value(self, value):
+        """Cast value to `bool`."""
+        parsed = super(BoolField, self).parse_value(value)
+        return bool(parsed) if parsed is not None else None
+
+
+class ListField(BaseField):
+
+    """List field."""
+
+    types = (list,)
+
+    def __init__(self, items_types=None, *args, **kwargs):
+        """Init.
+
+        `ListField` is **always not required**. If you want to control number
+        of items use validators.
+
+        """
+        self._assign_types(items_types)
+        super(ListField, self).__init__(*args, **kwargs)
+        self.required = False
+
+    def get_default_value(self):
+        default = super(ListField, self).get_default_value()
+        if default is None:
+            return ModelCollection(self)
+        return default
+
+    def _assign_types(self, items_types):
+        if items_types:
+            try:
+                self.items_types = tuple(items_types)
+            except TypeError:
+                self.items_types = items_types,
+        else:
+            self.items_types = tuple()
+
+        types = []
+        for type_ in self.items_types:
+            if isinstance(type_, six.string_types):
+                types.append(_LazyType(type_))
+            else:
+                types.append(type_)
+        self.items_types = tuple(types)
+
+    def validate(self, value):
+        super(ListField, self).validate(value)
+
+        if len(self.items_types) == 0:
+            return
+
+        for item in value:
+            self.validate_single_value(item)
+
+    def validate_single_value(self, item):
+        if len(self.items_types) == 0:
+            return
+
+        if not isinstance(item, self.items_types):
+            raise ValidationError(
+                'All items must be instances '
+                'of "{types}", and not "{type}".'.format(
+                    types=', '.join([t.__name__ for t in self.items_types]),
+                    type=type(item).__name__,
+                ))
+
+    def parse_value(self, values):
+        """Cast value to proper collection."""
+        result = self.get_default_value()
+
+        if not values:
+            return result
+
+        if not isinstance(values, list):
+            return values
+
+        return [self._cast_value(value) for value in values]
+
+    def _cast_value(self, value):
+        if isinstance(value, self.items_types):
+            return value
+        else:
+            if len(self.items_types) != 1:
+                tpl = 'Cannot decide which type to choose from "{types}".'
+                raise ValidationError(
+                    tpl.format(
+                        types=', '.join([t.__name__ for t in self.items_types])
+                    )
+                )
+            return self.items_types[0](**value)
+
+    def _finish_initialization(self, owner):
+        super(ListField, self)._finish_initialization(owner)
+
+        types = []
+        for type in self.items_types:
+            if isinstance(type, _LazyType):
+                types.append(type.evaluate(owner))
+            else:
+                types.append(type)
+        self.items_types = tuple(types)
+
+    def _elem_to_struct(self, value):
+        try:
+            return value.to_struct()
+        except AttributeError:
+            return value
+
+    def to_struct(self, values):
+        return [self._elem_to_struct(v) for v in values]
+
+
+class EmbeddedField(BaseField):
+
+    """Field for embedded models."""
+
+    def __init__(self, model_types, *args, **kwargs):
+        self._assign_model_types(model_types)
+        super(EmbeddedField, self).__init__(*args, **kwargs)
+
+    def _assign_model_types(self, model_types):
+        if not isinstance(model_types, (list, tuple)):
+            model_types = (model_types,)
+
+        types = []
+        for type_ in model_types:
+            if isinstance(type_, six.string_types):
+                types.append(_LazyType(type_))
+            else:
+                types.append(type_)
+        self.types = tuple(types)
+
+    def _finish_initialization(self, owner):
+        super(EmbeddedField, self)._finish_initialization(owner)
+
+        types = []
+        for type in self.types:
+            if isinstance(type, _LazyType):
+                types.append(type.evaluate(owner))
+            else:
+                types.append(type)
+        self.types = tuple(types)
+
+    def validate(self, value):
+        super(EmbeddedField, self).validate(value)
+        try:
+            value.validate()
+        except AttributeError:
+            pass
+
+    def parse_value(self, value):
+        """Parse value to proper model type."""
+        if not isinstance(value, dict):
+            return value
+
+        embed_type = self._get_embed_type()
+        return embed_type(**value)
+
+    def _get_embed_type(self):
+        if len(self.types) != 1:
+            raise ValidationError(
+                'Cannot decide which type to choose from "{types}".'.format(
+                    types=', '.join([t.__name__ for t in self.types])
+                )
+            )
+        return self.types[0]
+
+    def to_struct(self, value):
+        return value.to_struct()
+
+
+class _LazyType(object):
+
+    def __init__(self, path):
+        self.path = path
+
+    def evaluate(self, base_cls):
+        module, type_name = _evaluate_path(self.path, base_cls)
+        return _import(module, type_name)
+
+
+def _evaluate_path(relative_path, base_cls):
+    base_module = base_cls.__module__
+
+    modules = _get_modules(relative_path, base_module)
+
+    type_name = modules.pop()
+    module = '.'.join(modules)
+    if not module:
+        module = base_module
+    return module, type_name
+
+
+def _get_modules(relative_path, base_module):
+    canonical_path = relative_path.lstrip('.')
+    canonical_modules = canonical_path.split('.')
+
+    if not relative_path.startswith('.'):
+        return canonical_modules
+
+    parents_amount = len(relative_path) - len(canonical_path)
+    parent_modules = base_module.split('.')
+    parents_amount = max(0, parents_amount - 1)
+    if parents_amount > len(parent_modules):
+        raise ValueError("Can't evaluate path '{}'".format(relative_path))
+    return parent_modules[:parents_amount * -1] + canonical_modules
+
+
+def _import(module_name, type_name):
+    module = __import__(module_name, fromlist=[type_name])
+    try:
+        return getattr(module, type_name)
+    except AttributeError:
+        raise ValueError(
+            "Can't find type '{}.{}'.".format(module_name, type_name))
+
+
+class TimeField(StringField):
+
+    """Time field."""
+
+    types = (datetime.time,)
+
+    def __init__(self, str_format=None, *args, **kwargs):
+        """Init.
+
+        :param str str_format: Format to cast time to (if `None` - casting to
+            ISO 8601 format).
+
+        """
+        self.str_format = str_format
+        super(TimeField, self).__init__(*args, **kwargs)
+
+    def to_struct(self, value):
+        """Cast `time` object to string."""
+        if self.str_format:
+            return value.strftime(self.str_format)
+        return value.isoformat()
+
+    def parse_value(self, value):
+        """Parse string into instance of `time`."""
+        if value is None:
+            return value
+        if isinstance(value, datetime.time):
+            return value
+        return parse(value).timetz()
+
+
+class DateField(StringField):
+
+    """Date field."""
+
+    types = (datetime.date,)
+    default_format = '%Y-%m-%d'
+
+    def __init__(self, str_format=None, *args, **kwargs):
+        """Init.
+
+        :param str str_format: Format to cast date to (if `None` - casting to
+            %Y-%m-%d format).
+
+        """
+        self.str_format = str_format
+        super(DateField, self).__init__(*args, **kwargs)
+
+    def to_struct(self, value):
+        """Cast `date` object to string."""
+        if self.str_format:
+            return value.strftime(self.str_format)
+        return value.strftime(self.default_format)
+
+    def parse_value(self, value):
+        """Parse string into instance of `date`."""
+        if value is None:
+            return value
+        if isinstance(value, datetime.date):
+            return value
+        return parse(value).date()
+
+
+class DateTimeField(StringField):
+
+    """Datetime field."""
+
+    types = (datetime.datetime,)
+
+    def __init__(self, str_format=None, *args, **kwargs):
+        """Init.
+
+        :param str str_format: Format to cast datetime to (if `None` - casting
+            to ISO 8601 format).
+
+        """
+        self.str_format = str_format
+        super(DateTimeField, self).__init__(*args, **kwargs)
+
+    def to_struct(self, value):
+        """Cast `datetime` object to string."""
+        if self.str_format:
+            return value.strftime(self.str_format)
+        return value.isoformat()
+
+    def parse_value(self, value):
+        """Parse string into instance of `datetime`."""
+        if isinstance(value, datetime.datetime):
+            return value
+        if value:
+            return parse(value)
+        else:
+            return None

clearml_agent/backend_api/session/jsonmodels/models.py

@@ -0,0 +1,154 @@
+import six
+
+from . import parsers, errors
+from .fields import BaseField
+from .errors import ValidationError
+
+
+class JsonmodelMeta(type):
+
+    def __new__(cls, name, bases, attributes):
+        cls.validate_fields(attributes)
+        return super(cls, cls).__new__(cls, name, bases, attributes)
+
+    @staticmethod
+    def validate_fields(attributes):
+        fields = {
+            key: value for key, value in attributes.items()
+            if isinstance(value, BaseField)
+        }
+        taken_names = set()
+        for name, field in fields.items():
+            structue_name = field.structue_name(name)
+            if structue_name in taken_names:
+                raise ValueError('Name taken', structue_name, name)
+            taken_names.add(structue_name)
+
+
+class Base(six.with_metaclass(JsonmodelMeta, object)):
+
+    """Base class for all models."""
+
+    def __init__(self, **kwargs):
+        self._cache_key = _CacheKey()
+        self.populate(**kwargs)
+
+    def populate(self, **values):
+        """Populate values to fields. Skip non-existing."""
+        values = values.copy()
+        fields = list(self.iterate_with_name())
+        for _, structure_name, field in fields:
+            if structure_name in values:
+                field.__set__(self, values.pop(structure_name))
+        for name, _, field in fields:
+            if name in values:
+                field.__set__(self, values.pop(name))
+
+    def get_field(self, field_name):
+        """Get field associated with given attribute."""
+        for attr_name, field in self:
+            if field_name == attr_name:
+                return field
+
+        raise errors.FieldNotFound('Field not found', field_name)
+
+    def __iter__(self):
+        """Iterate through fields and values."""
+        for name, field in self.iterate_over_fields():
+            yield name, field
+
+    def validate(self):
+        """Explicitly validate all the fields."""
+        for name, field in self:
+            try:
+                field.validate_for_object(self)
+            except ValidationError as error:
+                raise ValidationError(
+                    "Error for field '{name}'.".format(name=name),
+                    error,
+                )
+
+    @classmethod
+    def iterate_over_fields(cls):
+        """Iterate through fields as `(attribute_name, field_instance)`."""
+        for attr in dir(cls):
+            clsattr = getattr(cls, attr)
+            if isinstance(clsattr, BaseField):
+                yield attr, clsattr
+
+    @classmethod
+    def iterate_with_name(cls):
+        """Iterate over fields, but also give `structure_name`.
+
+        Format is `(attribute_name, structue_name, field_instance)`.
+        Structure name is name under which value is seen in structure and
+        schema (in primitives) and only there.
+        """
+        for attr_name, field in cls.iterate_over_fields():
+            structure_name = field.structue_name(attr_name)
+            yield attr_name, structure_name, field
+
+    def to_struct(self):
+        """Cast model to Python structure."""
+        return parsers.to_struct(self)
+
+    @classmethod
+    def to_json_schema(cls):
+        """Generate JSON schema for model."""
+        return parsers.to_json_schema(cls)
+
+    def __repr__(self):
+        attrs = {}
+        for name, _ in self:
+            try:
+                attr = getattr(self, name)
+                if attr is not None:
+                    attrs[name] = repr(attr)
+            except ValidationError:
+                pass
+
+        return '{class_name}({fields})'.format(
+            class_name=self.__class__.__name__,
+            fields=', '.join(
+                '{0[0]}={0[1]}'.format(x) for x in sorted(attrs.items())
+            ),
+        )
+
+    def __str__(self):
+        return '{name} object'.format(name=self.__class__.__name__)
+
+    def __setattr__(self, name, value):
+        try:
+            return super(Base, self).__setattr__(name, value)
+        except ValidationError as error:
+            raise ValidationError(
+                "Error for field '{name}'.".format(name=name),
+                error
+            )
+
+    def __eq__(self, other):
+        if type(other) is not type(self):
+            return False
+
+        for name, _ in self.iterate_over_fields():
+            try:
+                our = getattr(self, name)
+            except errors.ValidationError:
+                our = None
+
+            try:
+                their = getattr(other, name)
+            except errors.ValidationError:
+                their = None
+
+            if our != their:
+                return False
+
+        return True
+
+    def __ne__(self, other):
+        return not (self == other)
+
+
+class _CacheKey(object):
+    """Object to identify model in memory."""

clearml_agent/backend_api/session/jsonmodels/parsers.py

@@ -0,0 +1,106 @@
+"""Parsers to change model structure into different ones."""
+import inspect
+
+from . import fields, builders, errors
+
+
+def to_struct(model):
+    """Cast instance of model to python structure.
+
+    :param model: Model to be casted.
+    :rtype: ``dict``
+
+    """
+    model.validate()
+
+    resp = {}
+    for _, name, field in model.iterate_with_name():
+        value = field.__get__(model)
+        if value is None:
+            continue
+
+        value = field.to_struct(value)
+        resp[name] = value
+    return resp
+
+
+def to_json_schema(cls):
+    """Generate JSON schema for given class.
+
+    :param cls: Class to be casted.
+    :rtype: ``dict``
+
+    """
+    builder = build_json_schema(cls)
+    return builder.build()
+
+
+def build_json_schema(value, parent_builder=None):
+    from .models import Base
+
+    cls = value if inspect.isclass(value) else value.__class__
+    if issubclass(cls, Base):
+        return build_json_schema_object(cls, parent_builder)
+    else:
+        return build_json_schema_primitive(cls, parent_builder)
+
+
+def build_json_schema_object(cls, parent_builder=None):
+    builder = builders.ObjectBuilder(cls, parent_builder)
+    if builder.count_type(builder.type) > 1:
+        return builder
+    for _, name, field in cls.iterate_with_name():
+        if isinstance(field, fields.EmbeddedField):
+            builder.add_field(name, field, _parse_embedded(field, builder))
+        elif isinstance(field, fields.ListField):
+            builder.add_field(name, field, _parse_list(field, builder))
+        else:
+            builder.add_field(
+                name, field, _create_primitive_field_schema(field))
+    return builder
+
+
+def _parse_list(field, parent_builder):
+    builder = builders.ListBuilder(
+        parent_builder, field.nullable, default=field._default)
+    for type in field.items_types:
+        builder.add_type_schema(build_json_schema(type, builder))
+    return builder
+
+
+def _parse_embedded(field, parent_builder):
+    builder = builders.EmbeddedBuilder(
+        parent_builder, field.nullable, default=field._default)
+    for type in field.types:
+        builder.add_type_schema(build_json_schema(type, builder))
+    return builder
+
+
+def build_json_schema_primitive(cls, parent_builder):
+    builder = builders.PrimitiveBuilder(cls, parent_builder)
+    return builder
+
+
+def _create_primitive_field_schema(field):
+    if isinstance(field, fields.StringField):
+        obj_type = 'string'
+    elif isinstance(field, fields.IntField):
+        obj_type = 'number'
+    elif isinstance(field, fields.FloatField):
+        obj_type = 'float'
+    elif isinstance(field, fields.BoolField):
+        obj_type = 'boolean'
+    else:
+        raise errors.FieldNotSupported(
+            'Field {field} is not supported!'.format(
+                field=type(field).__class__.__name__))
+
+    if field.nullable:
+        obj_type = [obj_type, 'null']
+
+    schema = {'type': obj_type}
+
+    if field.has_default:
+        schema["default"] = field._default
+
+    return schema

clearml_agent/backend_api/session/jsonmodels/utilities.py

@@ -0,0 +1,156 @@
+from __future__ import absolute_import
+
+import six
+import re
+from collections import namedtuple
+
+SCALAR_TYPES = tuple(list(six.string_types) + [int, float, bool])
+
+ECMA_TO_PYTHON_FLAGS = {
+    'i': re.I,
+    'm': re.M,
+}
+
+PYTHON_TO_ECMA_FLAGS = dict(
+    (value, key) for key, value in ECMA_TO_PYTHON_FLAGS.items()
+)
+
+PythonRegex = namedtuple('PythonRegex', ['regex', 'flags'])
+
+
+def _normalize_string_type(value):
+    if isinstance(value, six.string_types):
+        return six.text_type(value)
+    else:
+        return value
+
+
+def _compare_dicts(one, two):
+    if len(one) != len(two):
+        return False
+
+    for key, value in one.items():
+        if key not in one or key not in two:
+            return False
+
+        if not compare_schemas(one[key], two[key]):
+            return False
+    return True
+
+
+def _compare_lists(one, two):
+    if len(one) != len(two):
+        return False
+
+    they_match = False
+    for first_item in one:
+        for second_item in two:
+            if they_match:
+                continue
+            they_match = compare_schemas(first_item, second_item)
+    return they_match
+
+
+def _assert_same_types(one, two):
+    if not isinstance(one, type(two)) or not isinstance(two, type(one)):
+        raise RuntimeError('Types mismatch! "{type1}" and "{type2}".'.format(
+            type1=type(one).__name__, type2=type(two).__name__))
+
+
+def compare_schemas(one, two):
+    """Compare two structures that represents JSON schemas.
+
+    For comparison you can't use normal comparison, because in JSON schema
+    lists DO NOT keep order (and Python lists do), so this must be taken into
+    account during comparison.
+
+    Note this wont check all configurations, only first one that seems to
+    match, which can lead to wrong results.
+
+    :param one: First schema to compare.
+    :param two: Second schema to compare.
+    :rtype: `bool`
+
+    """
+    one = _normalize_string_type(one)
+    two = _normalize_string_type(two)
+
+    _assert_same_types(one, two)
+
+    if isinstance(one, list):
+        return _compare_lists(one, two)
+    elif isinstance(one, dict):
+        return _compare_dicts(one, two)
+    elif isinstance(one, SCALAR_TYPES):
+        return one == two
+    elif one is None:
+        return one is two
+    else:
+        raise RuntimeError('Not allowed type "{type}"'.format(
+            type=type(one).__name__))
+
+
+def is_ecma_regex(regex):
+    """Check if given regex is of type ECMA 262 or not.
+
+    :rtype: bool
+
+    """
+    parts = regex.split('/')
+
+    if len(parts) == 1:
+        return False
+
+    if len(parts) < 3:
+        raise ValueError('Given regex isn\'t ECMA regex nor Python regex.')
+    parts.pop()
+    parts.append('')
+
+    raw_regex = '/'.join(parts)
+    if raw_regex.startswith('/') and raw_regex.endswith('/'):
+        return True
+    return False
+
+
+def convert_ecma_regex_to_python(value):
+    """Convert ECMA 262 regex to Python tuple with regex and flags.
+
+    If given value is already Python regex it will be returned unchanged.
+
+    :param string value: ECMA regex.
+    :return: 2-tuple with `regex` and `flags`
+    :rtype: namedtuple
+
+    """
+    if not is_ecma_regex(value):
+        return PythonRegex(value, [])
+
+    parts = value.split('/')
+    flags = parts.pop()
+
+    try:
+        result_flags = [ECMA_TO_PYTHON_FLAGS[f] for f in flags]
+    except KeyError:
+        raise ValueError('Wrong flags "{}".'.format(flags))
+
+    return PythonRegex('/'.join(parts[1:]), result_flags)
+
+
+def convert_python_regex_to_ecma(value, flags=[]):
+    """Convert Python regex to ECMA 262 regex.
+
+    If given value is already ECMA regex it will be returned unchanged.
+
+    :param string value: Python regex.
+    :param list flags: List of flags (allowed flags: `re.I`, `re.M`)
+    :return: ECMA 262 regex
+    :rtype: str
+
+    """
+    if is_ecma_regex(value):
+        return value
+
+    result_flags = [PYTHON_TO_ECMA_FLAGS[f] for f in flags]
+    result_flags = ''.join(result_flags)
+
+    return '/{value}/{flags}'.format(value=value, flags=result_flags)

clearml_agent/backend_api/session/jsonmodels/validators.py

@@ -0,0 +1,202 @@
+"""Predefined validators."""
+import re
+
+from six.moves import reduce
+
+from .errors import ValidationError
+from . import utilities
+
+
+class Min(object):
+
+    """Validator for minimum value."""
+
+    def __init__(self, minimum_value, exclusive=False):
+        """Init.
+
+        :param minimum_value: Minimum value for validator.
+        :param bool exclusive: If `True`, then validated value must be strongly
+            lower than given threshold.
+
+        """
+        self.minimum_value = minimum_value
+        self.exclusive = exclusive
+
+    def validate(self, value):
+        """Validate value."""
+        if self.exclusive:
+            if value <= self.minimum_value:
+                tpl = "'{value}' is lower or equal than minimum ('{min}')."
+                raise ValidationError(
+                    tpl.format(value=value, min=self.minimum_value))
+        else:
+            if value < self.minimum_value:
+                raise ValidationError(
+                    "'{value}' is lower than minimum ('{min}').".format(
+                        value=value, min=self.minimum_value))
+
+    def modify_schema(self, field_schema):
+        """Modify field schema."""
+        field_schema['minimum'] = self.minimum_value
+        if self.exclusive:
+            field_schema['exclusiveMinimum'] = True
+
+
+class Max(object):
+
+    """Validator for maximum value."""
+
+    def __init__(self, maximum_value, exclusive=False):
+        """Init.
+
+        :param maximum_value: Maximum value for validator.
+        :param bool exclusive: If `True`, then validated value must be strongly
+            bigger than given threshold.
+
+        """
+        self.maximum_value = maximum_value
+        self.exclusive = exclusive
+
+    def validate(self, value):
+        """Validate value."""
+        if self.exclusive:
+            if value >= self.maximum_value:
+                tpl = "'{val}' is bigger or equal than maximum ('{max}')."
+                raise ValidationError(
+                    tpl.format(val=value, max=self.maximum_value))
+        else:
+            if value > self.maximum_value:
+                raise ValidationError(
+                    "'{value}' is bigger than maximum ('{max}').".format(
+                        value=value, max=self.maximum_value))
+
+    def modify_schema(self, field_schema):
+        """Modify field schema."""
+        field_schema['maximum'] = self.maximum_value
+        if self.exclusive:
+            field_schema['exclusiveMaximum'] = True
+
+
+class Regex(object):
+
+    """Validator for regular expressions."""
+
+    FLAGS = {
+        'ignorecase': re.I,
+        'multiline': re.M,
+    }
+
+    def __init__(self, pattern, **flags):
+        """Init.
+
+        Note, that if given pattern is ECMA regex, given flags will be
+        **completely ignored** and taken from given regex.
+
+
+        :param string pattern: Pattern of regex.
+        :param bool flags: Flags used for the regex matching.
+            Allowed flag names are in the `FLAGS` attribute. The flag value
+            does not matter as long as it evaluates to True.
+            Flags with False values will be ignored.
+            Invalid flags will be ignored.
+
+        """
+        if utilities.is_ecma_regex(pattern):
+            result = utilities.convert_ecma_regex_to_python(pattern)
+            self.pattern, self.flags = result
+        else:
+            self.pattern = pattern
+            self.flags = [self.FLAGS[key] for key, value in flags.items()
+                          if key in self.FLAGS and value]
+
+    def validate(self, value):
+        """Validate value."""
+        flags = self._calculate_flags()
+
+        try:
+            result = re.search(self.pattern, value, flags)
+        except TypeError as te:
+            raise ValidationError(*te.args)
+
+        if not result:
+            raise ValidationError(
+                'Value "{value}" did not match pattern "{pattern}".'.format(
+                    value=value, pattern=self.pattern
+                ))
+
+    def _calculate_flags(self):
+        return reduce(lambda x, y: x | y, self.flags, 0)
+
+    def modify_schema(self, field_schema):
+        """Modify field schema."""
+        field_schema['pattern'] = utilities.convert_python_regex_to_ecma(
+            self.pattern, self.flags)
+
+
+class Length(object):
+
+    """Validator for length."""
+
+    def __init__(self, minimum_value=None, maximum_value=None):
+        """Init.
+
+        Note that if no `minimum_value` neither `maximum_value` will be
+        specified, `ValueError` will be raised.
+
+        :param int minimum_value: Minimum value (optional).
+        :param int maximum_value: Maximum value (optional).
+
+        """
+        if minimum_value is None and maximum_value is None:
+            raise ValueError(
+                "Either 'minimum_value' or 'maximum_value' must be specified.")
+
+        self.minimum_value = minimum_value
+        self.maximum_value = maximum_value
+
+    def validate(self, value):
+        """Validate value."""
+        len_ = len(value)
+
+        if self.minimum_value is not None and len_ < self.minimum_value:
+            tpl = "Value '{val}' length is lower than allowed minimum '{min}'."
+            raise ValidationError(tpl.format(
+                val=value, min=self.minimum_value
+            ))
+
+        if self.maximum_value is not None and len_ > self.maximum_value:
+            raise ValidationError(
+                "Value '{val}' length is bigger than "
+                "allowed maximum '{max}'.".format(
+                    val=value,
+                    max=self.maximum_value,
+                ))
+
+    def modify_schema(self, field_schema):
+        """Modify field schema."""
+        if self.minimum_value:
+            field_schema['minLength'] = self.minimum_value
+
+        if self.maximum_value:
+            field_schema['maxLength'] = self.maximum_value
+
+
+class Enum(object):
+
+    """Validator for enums."""
+
+    def __init__(self, *choices):
+        """Init.
+
+        :param [] choices: Valid choices for the field.
+        """
+
+        self.choices = list(choices)
+
+    def validate(self, value):
+        if value not in self.choices:
+            tpl = "Value '{val}' is not a valid choice."
+            raise ValidationError(tpl.format(val=value))
+
+    def modify_schema(self, field_schema):
+        field_schema['enum'] = self.choices

clearml_agent/backend_api/session/response.py

@@ -1,10 +1,8 @@
 import requests
 
 import six
-import jsonmodels.models
-import jsonmodels.fields
-import jsonmodels.errors
 
+from . import jsonmodels
 from .apimodel import ApiModel
 from .datamodel import NonStrictDataModelMixin
 

clearml_agent/backend_api/session/session.py

@@ -29,24 +29,25 @@ class MaxRequestSizeError(Exception):
 
 
 class Session(TokenManager):
-    """ TRAINS API Session class. """
+    """ ClearML API Session class. """
 
     _AUTHORIZATION_HEADER = "Authorization"
-    _WORKER_HEADER = "X-Trains-Worker"
-    _ASYNC_HEADER = "X-Trains-Async"
-    _CLIENT_HEADER = "X-Trains-Agent"
+    _WORKER_HEADER = ("X-ClearML-Worker", "X-Trains-Worker", )
+    _ASYNC_HEADER = ("X-ClearML-Async", "X-Trains-Async", )
+    _CLIENT_HEADER = ("X-ClearML-Agent", "X-Trains-Agent", )
 
     _async_status_code = 202
     _session_requests = 0
     _session_initial_timeout = (3.0, 10.)
     _session_timeout = (10.0, 30.)
+    _session_initial_connect_retry = 4
     _write_session_data_size = 15000
     _write_session_timeout = (30.0, 30.)
 
     api_version = '2.1'
-    default_host = "https://demoapi.trains.allegro.ai"
-    default_web = "https://demoapp.trains.allegro.ai"
-    default_files = "https://demofiles.trains.allegro.ai"
+    default_host = "https://demoapi.demo.clear.ml"
+    default_web = "https://demoapp.demo.clear.ml"
+    default_files = "https://demofiles.demo.clear.ml"
     default_key = "EGRTCO8JMSIGI6S39GTP43NFWXDQOW"
     default_secret = "x!XTov_G-#vspE*Y(h$Anm&DIc5Ou-F)jsl$PdOyj5wG1&E!Z8"
 
@@ -85,6 +86,7 @@ class Session(TokenManager):
         initialize_logging=True,
         client=None,
         config=None,
+        http_retries_config=None,
         **kwargs
     ):
         # add backward compatibility support for old environment variables
@@ -95,7 +97,7 @@ class Session(TokenManager):
         else:
             self.config = load()
             if initialize_logging:
-                self.config.initialize_logging()
+                self.config.initialize_logging(debug=kwargs.get('debug', False))
 
         token_expiration_threshold_sec = self.config.get(
             "auth.token_expiration_threshold_sec", 60
@@ -129,11 +131,10 @@ class Session(TokenManager):
             raise ValueError("host is required in init or config")
 
         self.__host = host.strip("/")
-        http_retries_config = self.config.get(
+        http_retries_config = http_retries_config or self.config.get(
             "api.http.retries", ConfigTree()
         ).as_plain_ordered_dict()
         http_retries_config["status_forcelist"] = self._retry_codes
-        self.__http_session = get_http_session_with_retry(**http_retries_config)
 
         self.__worker = worker or gethostname()
 
@@ -143,7 +144,14 @@ class Session(TokenManager):
 
         self.client = client or "api-{}".format(__version__)
 
+        # limit the reconnect retries, so we get an error if we are starting the session
+        http_no_retries_config = dict(**http_retries_config)
+        http_no_retries_config['connect'] = self._session_initial_connect_retry
+        self.__http_session = get_http_session_with_retry(**http_no_retries_config)
+        # try to connect with the server
         self.refresh_token()
+        # create the default session with many retries
+        self.__http_session = get_http_session_with_retry(**http_retries_config)
 
         # update api version from server response
         try:
@@ -184,8 +192,10 @@ class Session(TokenManager):
         """
         host = self.host
         headers = headers.copy() if headers else {}
-        headers[self._WORKER_HEADER] = self.worker
-        headers[self._CLIENT_HEADER] = self.client
+        for h in self._WORKER_HEADER:
+            headers[h] = self.worker
+        for h in self._CLIENT_HEADER:
+            headers[h] = self.client
 
         token_refreshed_on_error = False
         url = (
@@ -260,7 +270,8 @@ class Session(TokenManager):
             headers.copy() if headers else {}
         )
         if async_enable:
-            headers[self._ASYNC_HEADER] = "1"
+            for h in self._ASYNC_HEADER:
+                headers[h] = "1"
         return self._send_request(
             service=service,
             action=action,
@@ -426,16 +437,15 @@ class Session(TokenManager):
     @classmethod
     def get_api_server_host(cls, config=None):
         if not config:
-            from ...config import config_obj
-            config = config_obj
+            return None
+
         return ENV_HOST.get(default=(config.get("api.api_server", None) or
                                      config.get("api.host", None) or cls.default_host))
 
     @classmethod
     def get_app_server_host(cls, config=None):
         if not config:
-            from ...config import config_obj
-            config = config_obj
+            return None
 
         # get from config/environment
         web_host = ENV_WEB_HOST.get(default=config.get("api.web_server", None))
@@ -457,13 +467,13 @@ class Session(TokenManager):
         if parsed.port == 8008:
             return host.replace(':8008', ':8080', 1)
 
-        raise ValueError('Could not detect TRAINS web application server')
+        raise ValueError('Could not detect ClearML web application server')
 
     @classmethod
     def get_files_server_host(cls, config=None):
         if not config:
-            from ...config import config_obj
-            config = config_obj
+            return None
+
         # get from config/environment
         files_host = ENV_FILES_HOST.get(default=(config.get("api.files_server", None)))
         if files_host:
@@ -541,10 +551,13 @@ class Session(TokenManager):
             # check if this is a misconfigured api server (getting 200 without the data section)
             if res and res.status_code == 200:
                 raise ValueError('It seems *api_server* is misconfigured. '
-                                 'Is this the TRAINS API server {} ?'.format(self.get_api_server_host()))
+                                 'Is this the ClearML API server {} ?'.format(self.get_api_server_host()))
             else:
                 raise LoginError("Response data mismatch: No 'token' in 'data' value from res, receive : {}, "
                                  "exception: {}".format(res, ex))
+        except requests.ConnectionError as ex:
+            raise ValueError('Connection Error: it seems *api_server* is misconfigured. '
+                             'Is this the ClearML API server {} ?'.format('/'.join(ex.request.url.split('/')[:3])))
         except Exception as ex:
             raise LoginError('Unrecognized Authentication Error: {} {}'.format(type(ex), ex))
 

clearml_agent/backend_api/utils.py

@@ -107,7 +107,7 @@ def get_http_session_with_retry(
     if not session.verify and __disable_certificate_verification_warning < 2:
         # show warning
         __disable_certificate_verification_warning += 1
-        logging.getLogger('TRAINS').warning(
+        logging.getLogger('ClearML').warning(
             msg='InsecureRequestWarning: Certificate verification is disabled! Adding '
                 'certificate verification is strongly advised. See: '
                 'https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings')

clearml_agent/backend_config/__init__.py

@@ -1,4 +1,3 @@
 from .defs import Environment
 from .config import Config, ConfigEntry
 from .errors import ConfigurationError
-from .environment import EnvEntry

clearml_agent/backend_config/config.py

@@ -138,7 +138,7 @@ class Config(object):
         else:
             env_config_paths = []
 
-        env_config_path_override = os.environ.get(ENV_CONFIG_PATH_OVERRIDE_VAR)
+        env_config_path_override = ENV_CONFIG_PATH_OVERRIDE_VAR.get()
         if env_config_path_override:
             env_config_paths = [expanduser(env_config_path_override)]
 
@@ -165,7 +165,7 @@ class Config(object):
             )
 
         local_config_files = LOCAL_CONFIG_FILES
-        local_config_override = os.environ.get(LOCAL_CONFIG_FILE_OVERRIDE_VAR)
+        local_config_override = LOCAL_CONFIG_FILE_OVERRIDE_VAR.get()
         if local_config_override:
             local_config_files = [expanduser(local_config_override)]
 
@@ -190,7 +190,7 @@ class Config(object):
     def reload(self):
         self.replace(self._reload())
 
-    def initialize_logging(self):
+    def initialize_logging(self, debug=False):
         logging_config = self._config.get("logging", None)
         if not logging_config:
             return False
@@ -217,6 +217,8 @@ class Config(object):
         )
         for logger in loggers:
             handlers = logger.get("handlers", None)
+            if debug:
+                logger['level'] = 'DEBUG'
             if not handlers:
                 continue
             logger["handlers"] = [h for h in handlers if h not in deleted]

clearml_agent/backend_config/defs.py

@@ -1,6 +1,8 @@
 from os.path import expanduser
 from pathlib2 import Path
 
+from ..backend_config.environment import EnvEntry
+
 ENV_VAR = 'TRAINS_ENV'
 """ Name of system environment variable that can be used to specify the config environment name """
 
@@ -17,23 +19,24 @@ ENV_CONFIG_PATHS = [
 
 
 LOCAL_CONFIG_PATHS = [
-    # '/etc/opt/trains',               # used by servers for docker-generated configuration
-    # expanduser('~/.trains/config'),
+    # '/etc/opt/clearml',               # used by servers for docker-generated configuration
+    # expanduser('~/.clearml/config'),
 ]
 """ Local config paths, not related to environment """
 
 
 LOCAL_CONFIG_FILES = [
     expanduser('~/trains.conf'),    # used for workstation configuration (end-users, workers)
+    expanduser('~/clearml.conf'),    # used for workstation configuration (end-users, workers)
 ]
 """ Local config files (not paths) """
 
 
-LOCAL_CONFIG_FILE_OVERRIDE_VAR = 'TRAINS_CONFIG_FILE'
+LOCAL_CONFIG_FILE_OVERRIDE_VAR = EnvEntry('CLEARML_CONFIG_FILE', 'TRAINS_CONFIG_FILE', )
 """ Local config file override environment variable. If this is set, no other local config files will be used. """
 
 
-ENV_CONFIG_PATH_OVERRIDE_VAR = 'TRAINS_CONFIG_PATH'
+ENV_CONFIG_PATH_OVERRIDE_VAR = EnvEntry('CLEARML_CONFIG_PATH', 'TRAINS_CONFIG_PATH', )
 """ 
 Environment-related config path override environment variable. If this is set, no other env config path will be used. 
 """
@@ -46,6 +49,15 @@ class Environment(object):
     local = 'local'
 
 
+class UptimeConf(object):
+    min_api_version = "2.10"
+    queue_tag_on = "force_workers:on"
+    queue_tag_off = "force_workers:off"
+    worker_key = "force"
+    worker_value_off = ["off"]
+    worker_value_on = ["on"]
+
+
 CONFIG_FILE_EXTENSION = '.conf'
 
 

clearml_agent/backend_config/entry.py

@@ -85,8 +85,9 @@ class Entry(object):
 
     def set(self, value):
         # type: (Any, Any) -> (Text, Any)
-        key, _ = self.get_pair(default=None, converter=None)
-        self._set(key, str(value))
+        # key, _ = self.get_pair(default=None, converter=None)
+        for k in self.keys:
+            self._set(k, str(value))
 
     def _set(self, key, value):
         # type: (Text, Text) -> None

clearml_agent/backend_config/environment.py

@@ -0,0 +1,64 @@
+from os import getenv, environ
+
+from .converters import text_to_bool
+from .entry import Entry, NotSet
+
+
+class EnvEntry(Entry):
+    @classmethod
+    def default_conversions(cls):
+        conversions = super(EnvEntry, cls).default_conversions().copy()
+        conversions[bool] = text_to_bool
+        return conversions
+
+    def pop(self):
+        for k in self.keys:
+            environ.pop(k, None)
+
+    def _get(self, key):
+        value = getenv(key, "").strip()
+        return value or NotSet
+
+    def _set(self, key, value):
+        environ[key] = value
+
+    def __str__(self):
+        return "env:{}".format(super(EnvEntry, self).__str__())
+
+    def error(self, message):
+        print("Environment configuration: {}".format(message))
+
+
+def backward_compatibility_support():
+    from ..definitions import ENVIRONMENT_CONFIG, ENVIRONMENT_SDK_PARAMS, ENVIRONMENT_BACKWARD_COMPATIBLE
+    if ENVIRONMENT_BACKWARD_COMPATIBLE.get():
+        # Add TRAINS_ prefix on every CLEARML_ os environment we support
+        for k, v in ENVIRONMENT_CONFIG.items():
+            try:
+                trains_vars = [var for var in v.vars if var.startswith('CLEARML_')]
+                if not trains_vars:
+                    continue
+                alg_var = trains_vars[0].replace('CLEARML_', 'TRAINS_', 1)
+                if alg_var not in v.vars:
+                    v.vars = tuple(list(v.vars) + [alg_var])
+            except:
+                continue
+        for k, v in ENVIRONMENT_SDK_PARAMS.items():
+            try:
+                trains_vars = [var for var in v if var.startswith('CLEARML_')]
+                if not trains_vars:
+                    continue
+                alg_var = trains_vars[0].replace('CLEARML_', 'TRAINS_', 1)
+                if alg_var not in v:
+                    ENVIRONMENT_SDK_PARAMS[k] = tuple(list(v) + [alg_var])
+            except:
+                continue
+
+    # set OS environ:
+    keys = environ.keys()
+    for k in keys:
+        if not k.startswith('CLEARML_'):
+            continue
+        backwards_k = k.replace('CLEARML_', 'TRAINS_', 1)
+        if backwards_k not in keys:
+            environ[backwards_k] = environ[k]

clearml_agent/backend_config/log.py

@@ -4,11 +4,11 @@ from pathlib2 import Path
 
 
 def logger(path=None):
-    name = "trains"
+    name = "clearml"
     if path:
         p = Path(path)
         module = (p.parent if p.stem.startswith('_') else p).stem
-        name = "trains.%s" % module
+        name = "clearml.%s" % module
     return logging.getLogger(name)
 
 

clearml_agent/commands/base.py

@@ -9,16 +9,16 @@ from operator import attrgetter
 from traceback import print_exc
 from typing import Text
 
-from trains_agent.helper.console import ListFormatter, print_text
-from trains_agent.helper.dicts import filter_keys
+from clearml_agent.helper.console import ListFormatter, print_text
+from clearml_agent.helper.dicts import filter_keys
 
 import six
-from trains_agent.backend_api import services
+from clearml_agent.backend_api import services
 
-from trains_agent.errors import APIError, CommandFailedError
-from trains_agent.helper.base import Singleton, return_list, print_parameters, dump_yaml, load_yaml, error, warning
-from trains_agent.interface.base import ObjectID
-from trains_agent.session import Session
+from clearml_agent.errors import APIError, CommandFailedError
+from clearml_agent.helper.base import Singleton, return_list, print_parameters, dump_yaml, load_yaml, error, warning
+from clearml_agent.interface.base import ObjectID
+from clearml_agent.session import Session
 
 
 class NameResolutionError(CommandFailedError):
@@ -74,7 +74,7 @@ class BaseCommandSection(object):
 
     @staticmethod
     def log(message, *args):
-        print("trains-agent: {}".format(message % args))
+        print("clearml-agent: {}".format(message % args))
 
     @classmethod
     def exit(cls, message, code=1):  # type: (Text, int) -> ()

clearml_agent/commands/check_config.py

@@ -1,4 +1,4 @@
-from trains_agent.commands.base import ServiceCommandSection
+from clearml_agent.commands.base import ServiceCommandSection
 
 
 class Config(ServiceCommandSection):

clearml_agent/commands/config.py

@@ -5,13 +5,15 @@ from pyhocon import ConfigFactory, ConfigMissingException
 from pathlib2 import Path
 from six.moves.urllib.parse import urlparse
 
-from trains_agent.backend_api.session import Session
-from trains_agent.backend_api.session.defs import ENV_HOST
-from trains_agent.backend_config.defs import LOCAL_CONFIG_FILES
+from clearml_agent.backend_api.session import Session
+from clearml_agent.backend_api.session.defs import ENV_HOST
+from clearml_agent.backend_config.defs import LOCAL_CONFIG_FILES
 
 
 description = """
-Please create new trains credentials through the profile page in your trains web app (e.g. https://demoapp.trains.allegro.ai/profile)
+Please create new clearml credentials through the profile page in your clearml web app (e.g. https://demoapp.demo.clear.ml/profile)
+Or with the free hosted service at https://app.community.clear.ml/profile
+    
 In the profile page, press "Create new credentials", then press "Copy to clipboard".
 
 Paste copied configuration here: 
@@ -25,7 +27,7 @@ except Exception:
 
 host_description = """
 Editing configuration file: {CONFIG_FILE}
-Enter the url of the trains-server's Web service, for example: {HOST}
+Enter the url of the clearml-server's Web service, for example: {HOST}
 """.format(
     CONFIG_FILE=LOCAL_CONFIG_FILES[0],
     HOST=def_host,
@@ -33,8 +35,12 @@ Enter the url of the trains-server's Web service, for example: {HOST}
 
 
 def main():
-    print('TRAINS-AGENT setup process')
-    conf_file = Path(LOCAL_CONFIG_FILES[0]).absolute()
+    print('CLEARML-AGENT setup process')
+    for f in LOCAL_CONFIG_FILES:
+        conf_file = Path(f).absolute()
+        if conf_file.exists():
+            break
+
     if conf_file.exists() and conf_file.is_file() and conf_file.stat().st_size > 0:
         print('Configuration file already exists: {}'.format(str(conf_file)))
         print('Leaving setup, feel free to edit the configuration file.')
@@ -42,7 +48,12 @@ def main():
 
     print(description, end='')
     sentinel = ''
-    parse_input = '\n'.join(iter(input, sentinel))
+    parse_input = ''
+    for line in iter(input, sentinel):
+        parse_input += line+'\n'
+        if line.rstrip() == '}':
+            break
+
     credentials = None
     api_server = None
     web_server = None
@@ -86,7 +97,7 @@ def main():
 
     files_host = input_url('File Store Host', files_host)
 
-    print('\nTRAINS Hosts configuration:\nWeb App: {}\nAPI: {}\nFile Store: {}\n'.format(
+    print('\nClearML Hosts configuration:\nWeb App: {}\nAPI: {}\nFile Store: {}\n'.format(
         web_host, api_host, files_host))
 
     retry = 1
@@ -140,13 +151,14 @@ def main():
     # noinspection PyBroadException
     try:
         with open(str(conf_file), 'wt') as f:
-            header = '# TRAINS-AGENT configuration file\n' \
+            header = '# CLEARML-AGENT configuration file\n' \
                      'api {\n' \
+                     '    # Notice: \'host\' is the api server (default port 8008), not the web server.\n' \
                      '    api_server: %s\n' \
                      '    web_server: %s\n' \
                      '    files_server: %s\n' \
                      '    # Credentials are generated using the webapp, %s/profile\n' \
-                     '    # Override with os environment: TRAINS_API_ACCESS_KEY / TRAINS_API_SECRET_KEY\n' \
+                     '    # Override with os environment: CLEARML_API_ACCESS_KEY / CLEARML_API_SECRET_KEY\n' \
                      '    credentials {"access_key": "%s", "secret_key": "%s"}\n' \
                      '}\n\n' % (api_host, web_host, files_host,
                                 web_host, credentials['access_key'], credentials['secret_key'])
@@ -157,7 +169,7 @@ def main():
                               'agent.git_pass=\"{}\"\n' \
                               '\n'.format(git_user or '', git_pass or '')
             f.write(git_credentials)
-            extra_index_str = '# extra_index_url: ["https://allegroai.jfrog.io/trainsai/api/pypi/public/simple"]\n' \
+            extra_index_str = '# extra_index_url: ["https://allegroai.jfrog.io/clearml/api/pypi/public/simple"]\n' \
                               'agent.package_manager.extra_index_url= ' \
                               '[\n{}\n]\n\n'.format("\n".join(map("\"{}\"".format, extra_index_urls)))
             f.write(extra_index_str)
@@ -167,7 +179,7 @@ def main():
         return
 
     print('\nNew configuration stored in {}'.format(str(conf_file)))
-    print('TRAINS-AGENT setup completed successfully.')
+    print('CLEARML-AGENT setup completed successfully.')
 
 
 def parse_host(parsed_host, allow_input=True):
@@ -233,7 +245,8 @@ def verify_credentials(api_host, credentials):
     try:
         print('Verifying credentials ...')
         if api_host:
-            Session(api_key=credentials['access_key'], secret_key=credentials['secret_key'], host=api_host)
+            Session(api_key=credentials['access_key'], secret_key=credentials['secret_key'], host=api_host,
+                    http_retries_config={"total": 2})
             print('Credentials verified!')
             return True
         else:
@@ -275,7 +288,7 @@ def read_manual_credentials():
 
 def input_url(host_type, host=None):
     while True:
-        print('{} configured to: [{}] '.format(host_type, host), end='')
+        print('{} configured to: {}'.format(host_type, '[{}] '.format(host) if host else ''), end='')
         parse_input = input()
         if host and (not parse_input or parse_input.lower() == 'yes' or parse_input.lower() == 'y'):
             break
@@ -289,11 +302,12 @@ def input_url(host_type, host=None):
 def input_host_port(host_type, parsed_host):
     print('Enter port for {} host '.format(host_type), end='')
     replace_port = input().lower()
-    return parsed_host.scheme + "://" + parsed_host.netloc + (':{}'.format(replace_port) if replace_port else '') + \
-           parsed_host.path
+    return parsed_host.scheme + "://" + parsed_host.netloc + (
+        ':{}'.format(replace_port) if replace_port else '') + parsed_host.path
 
 
 def verify_url(parse_input):
+    # noinspection PyBroadException
     try:
         if not parse_input.startswith('http://') and not parse_input.startswith('https://'):
             # if we have a specific port, use http prefix, otherwise assume https
@@ -306,7 +320,7 @@ def verify_url(parse_input):
             parsed_host = None
     except Exception:
         parsed_host = None
-        print('Could not parse url {}\nEnter your trains-server host: '.format(parse_input), end='')
+        print('Could not parse url {}\nEnter your clearml-server host: '.format(parse_input), end='')
     return parsed_host
 
 

clearml_agent/commands/events.py

@@ -5,8 +5,8 @@ import time
 
 from future.builtins import super
 
-from trains_agent.commands.base import ServiceCommandSection
-from trains_agent.helper.base import return_list
+from clearml_agent.commands.base import ServiceCommandSection
+from clearml_agent.helper.base import return_list
 
 
 class Events(ServiceCommandSection):

clearml_agent/complete.py

@@ -1,8 +1,8 @@
 """
 Script for generating command-line completion.
-Called by trains_agent/utilities/complete.sh (or a copy of it) like so:
+Called by clearml_agent/utilities/complete.sh (or a copy of it) like so:
 
-python -m trains_agent.complete "current command line"
+python -m clearml_agent.complete "current command line"
 
 And writes line-separated completion targets to stdout.
 Results are line-separated in order to enable other whitespace in results.
@@ -13,7 +13,7 @@ from __future__ import print_function
 import argparse
 import sys
 
-from trains_agent.interface import get_parser
+from clearml_agent.interface import get_parser
 
 
 def is_argument_required(action):

clearml_agent/config.py

@@ -1,7 +1,7 @@
 from pyhocon import ConfigTree
 
 import six
-from trains_agent.helper.base import Singleton
+from clearml_agent.helper.base import Singleton
 
 
 @six.add_metaclass(Singleton)

clearml_agent/definitions.py

@@ -1,22 +1,22 @@
 from datetime import timedelta
 from distutils.util import strtobool
 from enum import IntEnum
-from os import getenv
+from os import getenv, environ
 from typing import Text, Optional, Union, Tuple, Any
 
 from furl import furl
 from pathlib2 import Path
 
 import six
-from trains_agent.helper.base import normalize_path
+from clearml_agent.helper.base import normalize_path
 
-PROGRAM_NAME = "trains-agent"
+PROGRAM_NAME = "clearml-agent"
 FROM_FILE_PREFIX_CHARS = "@"
 
-CONFIG_DIR = normalize_path("~/.trains")
-TOKEN_CACHE_FILE = normalize_path("~/.trains.trains_agent.tmp")
+CONFIG_DIR = normalize_path("~/.clearml")
+TOKEN_CACHE_FILE = normalize_path("~/.clearml.clearml_agent.tmp")
 
-CONFIG_FILE_CANDIDATES = ["~/trains.conf"]
+CONFIG_FILE_CANDIDATES = ["~/clearml.conf"]
 
 
 def find_config_path():
@@ -40,6 +40,14 @@ class EnvironmentConfig(object):
         self.vars = names
         self.type = kwargs.pop("type", six.text_type)
 
+    def pop(self):
+        for k in self.vars:
+            environ.pop(k, None)
+
+    def set(self, value):
+        for k in self.vars:
+            environ[k] = str(value)
+
     def convert(self, value):
         return self.conversions.get(self.type, self.type)(value)
 
@@ -55,23 +63,23 @@ class EnvironmentConfig(object):
 
 
 ENVIRONMENT_CONFIG = {
-    "api.api_server": EnvironmentConfig("TRAINS_API_HOST", ),
+    "api.api_server": EnvironmentConfig("CLEARML_API_HOST", "TRAINS_API_HOST", ),
     "api.credentials.access_key": EnvironmentConfig(
-        "TRAINS_API_ACCESS_KEY",
+        "CLEARML_API_ACCESS_KEY", "TRAINS_API_ACCESS_KEY",
     ),
     "api.credentials.secret_key": EnvironmentConfig(
-        "TRAINS_API_SECRET_KEY",
+        "CLEARML_API_SECRET_KEY", "TRAINS_API_SECRET_KEY",
     ),
-    "agent.worker_name": EnvironmentConfig("TRAINS_WORKER_NAME", ),
-    "agent.worker_id": EnvironmentConfig("TRAINS_WORKER_ID", ),
+    "agent.worker_name": EnvironmentConfig("CLEARML_WORKER_NAME", "TRAINS_WORKER_NAME", ),
+    "agent.worker_id": EnvironmentConfig("CLEARML_WORKER_ID", "TRAINS_WORKER_ID", ),
     "agent.cuda_version": EnvironmentConfig(
-        "TRAINS_CUDA_VERSION", "CUDA_VERSION"
+        "CLEARML_CUDA_VERSION", "TRAINS_CUDA_VERSION", "CUDA_VERSION"
     ),
     "agent.cudnn_version": EnvironmentConfig(
-        "TRAINS_CUDNN_VERSION", "CUDNN_VERSION"
+        "CLEARML_CUDNN_VERSION", "TRAINS_CUDNN_VERSION", "CUDNN_VERSION"
     ),
     "agent.cpu_only": EnvironmentConfig(
-        "TRAINS_CPU_ONLY", "CPU_ONLY", type=bool
+        names=("CLEARML_CPU_ONLY", "TRAINS_CPU_ONLY", "CPU_ONLY"), type=bool
     ),
     "sdk.aws.s3.key": EnvironmentConfig("AWS_ACCESS_KEY_ID"),
     "sdk.aws.s3.secret": EnvironmentConfig("AWS_SECRET_ACCESS_KEY"),
@@ -82,13 +90,14 @@ ENVIRONMENT_CONFIG = {
 }
 
 ENVIRONMENT_SDK_PARAMS = {
-    "task_id": ("TRAINS_TASK_ID", ),
-    "config_file": ("TRAINS_CONFIG_FILE", ),
-    "log_level": ("TRAINS_LOG_LEVEL", ),
-    "log_to_backend": ("TRAINS_LOG_TASK_TO_BACKEND", ),
+    "task_id": ("CLEARML_TASK_ID", "TRAINS_TASK_ID", ),
+    "config_file": ("CLEARML_CONFIG_FILE", "TRAINS_CONFIG_FILE", ),
+    "log_level": ("CLEARML_LOG_LEVEL", "TRAINS_LOG_LEVEL", ),
+    "log_to_backend": ("CLEARML_LOG_TASK_TO_BACKEND", "TRAINS_LOG_TASK_TO_BACKEND", ),
 }
 
-ENVIRONMENT_BACKWARD_COMPATIBLE = EnvironmentConfig("TRAINS_AGENT_ALG_ENV", type=bool)
+ENVIRONMENT_BACKWARD_COMPATIBLE = EnvironmentConfig(
+    names=("CLEARML_AGENT_ALG_ENV", "TRAINS_AGENT_ALG_ENV"), type=bool)
 
 VIRTUAL_ENVIRONMENT_PATH = {
     "python2": normalize_path(CONFIG_DIR, "py2venv"),
@@ -96,7 +105,7 @@ VIRTUAL_ENVIRONMENT_PATH = {
 }
 
 DEFAULT_BASE_DIR = normalize_path(CONFIG_DIR, "data_cache")
-DEFAULT_HOST = "https://demoapi.trains.allegro.ai"
+DEFAULT_HOST = "https://demoapi.demo.clear.ml"
 MAX_DATASET_SOURCES_COUNT = 50000
 
 INVALID_WORKER_ID = (400, 1001)
@@ -105,11 +114,6 @@ WORKER_ALREADY_REGISTERED = (400, 1003)
 API_VERSION = "v1.5"
 TOKEN_EXPIRATION_SECONDS = int(timedelta(days=2).total_seconds())
 
-HTTP_HEADERS = {
-    "worker": "X-Trains-Worker",
-    "act-as": "X-Trains-Act-As",
-    "client": "X-Trains-Agent",
-}
 METADATA_EXTENSION = ".json"
 
 DEFAULT_VENV_UPDATE_URL = (
@@ -120,11 +124,16 @@ DEFAULT_VCS_CACHE = normalize_path(CONFIG_DIR, "vcs-cache")
 PIP_EXTRA_INDICES = [
 ]
 DEFAULT_PIP_DOWNLOAD_CACHE = normalize_path(CONFIG_DIR, "pip-download-cache")
-ENV_AGENT_GIT_USER = EnvironmentConfig('TRAINS_AGENT_GIT_USER')
-ENV_AGENT_GIT_PASS = EnvironmentConfig('TRAINS_AGENT_GIT_PASS')
-ENV_TASK_EXECUTE_AS_USER = 'TRAINS_AGENT_EXEC_USER'
-ENV_TASK_EXTRA_PYTHON_PATH = 'TRAINS_AGENT_EXTRA_PYTHON_PATH'
-ENV_DOCKER_HOST_MOUNT = EnvironmentConfig('TRAINS_AGENT_K8S_HOST_MOUNT', 'TRAINS_AGENT_DOCKER_HOST_MOUNT')
+ENV_DOCKER_IMAGE = EnvironmentConfig('CLEARML_DOCKER_IMAGE', 'TRAINS_DOCKER_IMAGE')
+ENV_WORKER_ID = EnvironmentConfig('CLEARML_WORKER_ID', 'TRAINS_WORKER_ID')
+ENV_DOCKER_SKIP_GPUS_FLAG = EnvironmentConfig('CLEARML_DOCKER_SKIP_GPUS_FLAG', 'TRAINS_DOCKER_SKIP_GPUS_FLAG')
+ENV_AGENT_GIT_USER = EnvironmentConfig('CLEARML_AGENT_GIT_USER', 'TRAINS_AGENT_GIT_USER')
+ENV_AGENT_GIT_PASS = EnvironmentConfig('CLEARML_AGENT_GIT_PASS', 'TRAINS_AGENT_GIT_PASS')
+ENV_AGENT_GIT_HOST = EnvironmentConfig('CLEARML_AGENT_GIT_HOST', 'TRAINS_AGENT_GIT_HOST')
+ENV_TASK_EXECUTE_AS_USER = EnvironmentConfig('CLEARML_AGENT_EXEC_USER', 'TRAINS_AGENT_EXEC_USER')
+ENV_TASK_EXTRA_PYTHON_PATH = EnvironmentConfig('CLEARML_AGENT_EXTRA_PYTHON_PATH', 'TRAINS_AGENT_EXTRA_PYTHON_PATH')
+ENV_DOCKER_HOST_MOUNT = EnvironmentConfig('CLEARML_AGENT_K8S_HOST_MOUNT', 'CLEARML_AGENT_DOCKER_HOST_MOUNT',
+                                          'TRAINS_AGENT_K8S_HOST_MOUNT', 'TRAINS_AGENT_DOCKER_HOST_MOUNT')
 
 
 class FileBuffering(IntEnum):

clearml_agent/external/requirements_parser/__init__.py

@@ -0,0 +1,22 @@
+from .parser import parse   # noqa
+
+_MAJOR = 0
+_MINOR = 2
+_PATCH = 0
+
+
+def version_tuple():
+    '''
+    Returns a 3-tuple of ints that represent the version
+    '''
+    return (_MAJOR, _MINOR, _PATCH)
+
+
+def version():
+    '''
+    Returns a string representation of the version
+    '''
+    return '%d.%d.%d' % (version_tuple())
+
+
+__version__ = version()

clearml_agent/external/requirements_parser/fragment.py

@@ -0,0 +1,44 @@
+import re
+
+# Copied from pip
+# https://github.com/pypa/pip/blob/281eb61b09d87765d7c2b92f6982b3fe76ccb0af/pip/index.py#L947
+HASH_ALGORITHMS = set(['sha1', 'sha224', 'sha384', 'sha256', 'sha512', 'md5'])
+
+extras_require_search = re.compile(
+    r'(?P<name>.+)\[(?P<extras>[^\]]+)\]').search
+
+
+def parse_fragment(fragment_string):
+    """Takes a fragment string nd returns a dict of the components"""
+    fragment_string = fragment_string.lstrip('#')
+
+    try:
+        return dict(
+            key_value_string.split('=')
+            for key_value_string in fragment_string.split('&')
+        )
+    except ValueError:
+        raise ValueError(
+            'Invalid fragment string {fragment_string}'.format(
+                fragment_string=fragment_string
+            )
+        )
+
+
+def get_hash_info(d):
+    """Returns the first matching hashlib name and value from a dict"""
+    for key in d.keys():
+        if key.lower() in HASH_ALGORITHMS:
+            return key, d[key]
+
+    return None, None
+
+
+def parse_extras_require(egg):
+    if egg is not None:
+        match = extras_require_search(egg)
+        if match is not None:
+            name = match.group('name')
+            extras = match.group('extras')
+            return name, [extra.strip() for extra in extras.split(',')]
+    return egg, []

clearml_agent/external/requirements_parser/parser.py

@@ -0,0 +1,50 @@
+import os
+import warnings
+
+from .requirement import Requirement
+
+
+def parse(reqstr):
+    """
+    Parse a requirements file into a list of Requirements
+
+    See: pip/req.py:parse_requirements()
+
+    :param reqstr: a string or file like object containing requirements
+    :returns: a *generator* of Requirement objects
+    """
+    filename = getattr(reqstr, 'name', None)
+    try:
+        # Python 2.x compatibility
+        if not isinstance(reqstr, basestring):
+            reqstr = reqstr.read()
+    except NameError:
+        # Python 3.x only
+        if not isinstance(reqstr, str):
+            reqstr = reqstr.read()
+
+    for line in reqstr.splitlines():
+        line = line.strip()
+        if line == '':
+            continue
+        elif not line or line.startswith('#'):
+            # comments are lines that start with # only
+            continue
+        elif line.startswith('-r') or line.startswith('--requirement'):
+            _, new_filename = line.split()
+            new_file_path = os.path.join(os.path.dirname(filename or '.'),
+                                         new_filename)
+            with open(new_file_path) as f:
+                for requirement in parse(f):
+                    yield requirement
+        elif line.startswith('-f') or line.startswith('--find-links') or \
+                line.startswith('-i') or line.startswith('--index-url') or \
+                line.startswith('--extra-index-url') or \
+                line.startswith('--no-index'):
+            warnings.warn('Private repos not supported. Skipping.')
+            continue
+        elif line.startswith('-Z') or line.startswith('--always-unzip'):
+            warnings.warn('Unused option --always-unzip. Skipping.')
+            continue
+        else:
+            yield Requirement.parse(line)

clearml_agent/external/requirements_parser/requirement.py

@@ -0,0 +1,241 @@
+from __future__ import unicode_literals
+import re
+from pkg_resources import Requirement as Req
+
+from .fragment import get_hash_info, parse_fragment, parse_extras_require
+from .vcs import VCS, VCS_SCHEMES
+
+
+URI_REGEX = re.compile(
+    r'^(?P<scheme>https?|file|ftps?)://(?P<path>[^#]+)'
+    r'(#(?P<fragment>\S+))?'
+)
+
+VCS_REGEX = re.compile(
+    r'^(?P<scheme>{0})://'.format(r'|'.join(
+        [scheme.replace('+', r'\+') for scheme in VCS_SCHEMES])) +
+    r'((?P<login>[^/@]+)@)?'
+    r'(?P<path>[^#@]+)'
+    r'(@(?P<revision>[^#]+))?'
+    r'(#(?P<fragment>\S+))?'
+)
+
+# This matches just about everyting
+LOCAL_REGEX = re.compile(
+    r'^((?P<scheme>file)://)?'
+    r'(?P<path>[^#]+)' +
+    r'(#(?P<fragment>\S+))?'
+)
+
+
+class Requirement(object):
+    """
+    Represents a single requirementfrom clearml_agent.external.requirements_parser.requirement import Requirement
+
+    Typically instances of this class are created with ``Requirement.parse``.
+    For local file requirements, there's no verification that the file
+    exists. This class attempts to be *dict-like*.
+
+    See: http://www.pip-installer.org/en/latest/logic.html
+
+    **Members**:
+
+    * ``line`` - the actual requirement line being parsed
+    * ``editable`` - a boolean whether this requirement is "editable"
+    * ``local_file`` - a boolean whether this requirement is a local file/path
+    * ``specifier`` - a boolean whether this requirement used a requirement
+      specifier (eg. "django>=1.5" or "requirements")
+    * ``vcs`` - a string specifying the version control system
+    * ``revision`` - a version control system specifier
+    * ``name`` - the name of the requirement
+    * ``uri`` - the URI if this requirement was specified by URI
+    * ``subdirectory`` - the subdirectory fragment of the URI
+    * ``path`` - the local path to the requirement
+    * ``hash_name`` - the type of hashing algorithm indicated in the line
+    * ``hash`` - the hash value indicated by the requirement line
+    * ``extras`` - a list of extras for this requirement
+      (eg. "mymodule[extra1, extra2]")
+    * ``specs`` - a list of specs for this requirement
+      (eg. "mymodule>1.5,<1.6" => [('>', '1.5'), ('<', '1.6')])
+    """
+
+    def __init__(self, line):
+        # Do not call this private method
+        self.line = line
+        self.editable = False
+        self.local_file = False
+        self.specifier = False
+        self.vcs = None
+        self.name = None
+        self.subdirectory = None
+        self.uri = None
+        self.path = None
+        self.revision = None
+        self.hash_name = None
+        self.hash = None
+        self.extras = []
+        self.specs = []
+
+    def __repr__(self):
+        return '<Requirement: "{0}">'.format(self.line)
+
+    def __getitem__(self, key):
+        return getattr(self, key)
+
+    def keys(self):
+        return self.__dict__.keys()
+
+    @classmethod
+    def parse_editable(cls, line):
+        """
+        Parses a Requirement from an "editable" requirement which is either
+        a local project path or a VCS project URI.
+
+        See: pip/req.py:from_editable()
+
+        :param line: an "editable" requirement
+        :returns: a Requirement instance for the given line
+        :raises: ValueError on an invalid requirement
+        """
+
+        req = cls('-e {0}'.format(line))
+        req.editable = True
+        vcs_match = VCS_REGEX.match(line)
+        local_match = LOCAL_REGEX.match(line)
+
+        if vcs_match is not None:
+            groups = vcs_match.groupdict()
+            if groups.get('login'):
+                req.uri = '{scheme}://{login}@{path}'.format(**groups)
+            else:
+                req.uri = '{scheme}://{path}'.format(**groups)
+            req.revision = groups['revision']
+            if groups['fragment']:
+                fragment = parse_fragment(groups['fragment'])
+                egg = fragment.get('egg')
+                req.name, req.extras = parse_extras_require(egg)
+                req.hash_name, req.hash = get_hash_info(fragment)
+                req.subdirectory = fragment.get('subdirectory')
+            for vcs in VCS:
+                if req.uri.startswith(vcs):
+                    req.vcs = vcs
+        else:
+            assert local_match is not None, 'This should match everything'
+            groups = local_match.groupdict()
+            req.local_file = True
+            if groups['fragment']:
+                fragment = parse_fragment(groups['fragment'])
+                egg = fragment.get('egg')
+                req.name, req.extras = parse_extras_require(egg)
+                req.hash_name, req.hash = get_hash_info(fragment)
+                req.subdirectory = fragment.get('subdirectory')
+            req.path = groups['path']
+
+        return req
+
+    @classmethod
+    def parse_line(cls, line):
+        """
+        Parses a Requirement from a non-editable requirement.
+
+        See: pip/req.py:from_line()
+
+        :param line: a "non-editable" requirement
+        :returns: a Requirement instance for the given line
+        :raises: ValueError on an invalid requirement
+        """
+
+        req = cls(line)
+
+        vcs_match = VCS_REGEX.match(line)
+        uri_match = URI_REGEX.match(line)
+        local_match = LOCAL_REGEX.match(line)
+
+        if vcs_match is not None:
+            groups = vcs_match.groupdict()
+            if groups.get('login'):
+                req.uri = '{scheme}://{login}@{path}'.format(**groups)
+            else:
+                req.uri = '{scheme}://{path}'.format(**groups)
+            req.revision = groups['revision']
+            if groups['fragment']:
+                fragment = parse_fragment(groups['fragment'])
+                egg = fragment.get('egg')
+                req.name, req.extras = parse_extras_require(egg)
+                req.hash_name, req.hash = get_hash_info(fragment)
+                req.subdirectory = fragment.get('subdirectory')
+            for vcs in VCS:
+                if req.uri.startswith(vcs):
+                    req.vcs = vcs
+        elif uri_match is not None:
+            groups = uri_match.groupdict()
+            req.uri = '{scheme}://{path}'.format(**groups)
+            if groups['fragment']:
+                fragment = parse_fragment(groups['fragment'])
+                egg = fragment.get('egg')
+                req.name, req.extras = parse_extras_require(egg)
+                req.hash_name, req.hash = get_hash_info(fragment)
+                req.subdirectory = fragment.get('subdirectory')
+            if groups['scheme'] == 'file':
+                req.local_file = True
+        elif '#egg=' in line:
+            # Assume a local file match
+            assert local_match is not None, 'This should match everything'
+            groups = local_match.groupdict()
+            req.local_file = True
+            if groups['fragment']:
+                fragment = parse_fragment(groups['fragment'])
+                egg = fragment.get('egg')
+                name, extras = parse_extras_require(egg)
+                req.name = fragment.get('egg')
+                req.hash_name, req.hash = get_hash_info(fragment)
+                req.subdirectory = fragment.get('subdirectory')
+            req.path = groups['path']
+        else:
+            # This is a requirement specifier.
+            # Delegate to pkg_resources and hope for the best
+            req.specifier = True
+            pkg_req = Req.parse(line)
+            req.name = pkg_req.unsafe_name
+            req.extras = list(pkg_req.extras)
+            req.specs = pkg_req.specs
+        return req
+
+    @classmethod
+    def parse(cls, line):
+        """
+        Parses a Requirement from a line of a requirement file.
+
+        :param line: a line of a requirement file
+        :returns: a Requirement instance for the given line
+        :raises: ValueError on an invalid requirement
+        """
+        line = line.lstrip()
+        if line.startswith('-e') or line.startswith('--editable'):
+            # Editable installs are either a local project path
+            # or a VCS project URI
+            return cls.parse_editable(
+                re.sub(r'^(-e|--editable=?)\s*', '', line))
+        elif '@' in line and ('#' not in line or line.index('#') > line.index('@')):
+            # Allegro bug fix: support 'name @ git+' entries
+            name, uri = line.split('@', 1)
+            name = name.strip()
+            uri = uri.strip()
+            # noinspection PyBroadException
+            try:
+                # check if the name is valid & parsed
+                Req.parse(name)
+                # if we are here, name is a valid package name, check if the vcs part is valid
+                if VCS_REGEX.match(uri):
+                    req = cls.parse_line(uri)
+                    req.name = name
+                    return req
+                elif URI_REGEX.match(uri):
+                    req = cls.parse_line(uri)
+                    req.name = name
+                    req.line = line
+                    return req
+            except Exception:
+                pass
+
+        return cls.parse_line(line)

clearml_agent/external/requirements_parser/vcs.py

@@ -0,0 +1,30 @@
+from __future__ import unicode_literals
+
+VCS = [
+    'git',
+    'hg',
+    'svn',
+    'bzr',
+]
+
+VCS_SCHEMES = [
+    'git',
+    'git+https',
+    'git+ssh',
+    'git+git',
+    'hg+http',
+    'hg+https',
+    'hg+static-http',
+    'hg+ssh',
+    'svn',
+    'svn+svn',
+    'svn+http',
+    'svn+https',
+    'svn+ssh',
+    'bzr+http',
+    'bzr+https',
+    'bzr+ssh',
+    'bzr+sftp',
+    'bzr+ftp',
+    'bzr+lp',
+]

clearml_agent/glue/k8s.py

@@ -0,0 +1,518 @@
+from __future__ import print_function, division, unicode_literals
+
+import base64
+import logging
+import os
+import re
+import subprocess
+import tempfile
+from copy import deepcopy
+
+import yaml
+import json
+from time import sleep
+from typing import Text, List
+
+from clearml_agent.commands.events import Events
+from clearml_agent.commands.worker import Worker
+from clearml_agent.definitions import ENV_DOCKER_IMAGE
+from clearml_agent.errors import APIError
+from clearml_agent.helper.base import safe_remove_file
+from clearml_agent.helper.dicts import merge_dicts
+from clearml_agent.helper.process import get_bash_output
+from clearml_agent.helper.resource_monitor import ResourceMonitor
+from clearml_agent.interface.base import ObjectID
+
+
+class K8sIntegration(Worker):
+    K8S_PENDING_QUEUE = "k8s_scheduler"
+
+    KUBECTL_APPLY_CMD = "kubectl apply -f"
+
+    KUBECTL_RUN_CMD = "kubectl run clearml-{queue_name}-id-{task_id} " \
+                      "--image {docker_image} " \
+                      "--restart=Never --replicas=1 " \
+                      "--generator=run-pod/v1 " \
+                      "--namespace=clearml"
+
+    KUBECTL_DELETE_CMD = "kubectl delete pods " \
+                         "--selector=TRAINS=agent " \
+                         "--field-selector=status.phase!=Pending,status.phase!=Running " \
+                         "--namespace=clearml"
+
+    BASH_INSTALL_SSH_CMD = [
+        "apt-get install -y openssh-server",
+        "mkdir -p /var/run/sshd",
+        "echo 'root:training' | chpasswd",
+        "echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config",
+        "sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config",
+        r"sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd",
+        "echo 'AcceptEnv TRAINS_API_ACCESS_KEY TRAINS_API_SECRET_KEY CLEARML_API_ACCESS_KEY CLEARML_API_SECRET_KEY' "
+        ">> /etc/ssh/sshd_config",
+        'echo "export VISIBLE=now" >> /etc/profile',
+        'echo "export PATH=$PATH" >> /etc/profile',
+        'echo "ldconfig" >> /etc/profile',
+        "/usr/sbin/sshd -p {port}"]
+
+    CONTAINER_BASH_SCRIPT = [
+        "export DEBIAN_FRONTEND='noninteractive'",
+        "echo 'Binary::apt::APT::Keep-Downloaded-Packages \"true\";' > /etc/apt/apt.conf.d/docker-clean",
+        "chown -R root /root/.cache/pip",
+        "apt-get update",
+        "apt-get install -y git libsm6 libxext6 libxrender-dev libglib2.0-0",
+        "declare LOCAL_PYTHON",
+        "for i in {{10..5}}; do which python3.$i && python3.$i -m pip --version && "
+        "export LOCAL_PYTHON=$(which python3.$i) && break ; done",
+        "[ ! -z $LOCAL_PYTHON ] || apt-get install -y python3-pip",
+        "[ ! -z $LOCAL_PYTHON ] || export LOCAL_PYTHON=python3",
+        "$LOCAL_PYTHON -m pip install clearml-agent",
+        "{extra_bash_init_cmd}",
+        "$LOCAL_PYTHON -m clearml_agent execute --full-monitoring --require-queue --id {task_id}"
+    ]
+
+    AGENT_LABEL = "TRAINS=agent"
+    LIMIT_POD_LABEL = "ai.allegro.agent.serial=pod-{pod_number}"
+
+    _edit_hyperparams_version = "2.9"
+
+    def __init__(
+            self,
+            k8s_pending_queue_name=None,
+            kubectl_cmd=None,
+            container_bash_script=None,
+            debug=False,
+            ports_mode=False,
+            num_of_services=20,
+            user_props_cb=None,
+            overrides_yaml=None,
+            template_yaml=None,
+            trains_conf_file=None,
+            extra_bash_init_script=None,
+    ):
+        """
+        Initialize the k8s integration glue layer daemon
+
+        :param str k8s_pending_queue_name: queue name to use when task is pending in the k8s scheduler
+        :param str|callable kubectl_cmd: kubectl command line str, supports formatting (default: KUBECTL_RUN_CMD)
+            example: "task={task_id} image={docker_image} queue_id={queue_id}"
+            or a callable function: kubectl_cmd(task_id, docker_image, queue_id, task_data)
+        :param str container_bash_script: container bash script to be executed in k8s (default: CONTAINER_BASH_SCRIPT)
+            Notice this string will use format() call, if you have curly brackets they should be doubled { -> {{
+            Format arguments passed: {task_id} and {extra_bash_init_cmd}
+        :param bool debug: Switch logging on
+        :param bool ports_mode: Adds a label to each pod which can be used in services in order to expose ports.
+            Requires the `num_of_services` parameter.
+        :param int num_of_services: Number of k8s services configured in the cluster. Required if `port_mode` is True.
+            (default: 20)
+        :param callable user_props_cb: An Optional callable allowing additional user properties to be specified
+            when scheduling a task to run in a pod. Callable can receive an optional pod number and should return
+            a dictionary of user properties (name and value). Signature is [[Optional[int]], Dict[str,str]]
+        :param str overrides_yaml: YAML file containing the overrides for the pod (optional)
+        :param str template_yaml: YAML file containing the template  for the pod (optional).
+            If provided the pod is scheduled with kubectl apply and overrides are ignored, otherwise with kubectl run.
+        :param str trains_conf_file: clearml.conf file to be use by the pod itself (optional)
+        :param str extra_bash_init_script: Additional bash script to run before starting the Task inside the container
+        """
+        super(K8sIntegration, self).__init__()
+        self.k8s_pending_queue_name = k8s_pending_queue_name or self.K8S_PENDING_QUEUE
+        self.kubectl_cmd = kubectl_cmd or self.KUBECTL_RUN_CMD
+        self.container_bash_script = container_bash_script or self.CONTAINER_BASH_SCRIPT
+        # Always do system packages, because by we will be running inside a docker
+        self._session.config.put("agent.package_manager.system_site_packages", True)
+        # Add debug logging
+        if debug:
+            self.log.logger.disabled = False
+            self.log.logger.setLevel(logging.INFO)
+        self.ports_mode = ports_mode
+        self.num_of_services = num_of_services
+        self._edit_hyperparams_support = None
+        self._user_props_cb = user_props_cb
+        self.trains_conf_file = None
+        self.overrides_json_string = None
+        self.template_dict = None
+        self.extra_bash_init_script = extra_bash_init_script or None
+        if self.extra_bash_init_script and not isinstance(self.extra_bash_init_script, str):
+            self.extra_bash_init_script = ' ; '.join(self.extra_bash_init_script)  # noqa
+        self.pod_limits = []
+        self.pod_requests = []
+        if overrides_yaml:
+            with open(os.path.expandvars(os.path.expanduser(str(overrides_yaml))), 'rt') as f:
+                overrides = yaml.load(f, Loader=getattr(yaml, 'FullLoader', None))
+            if overrides:
+                containers = overrides.get('spec', {}).get('containers', [])
+                for c in containers:
+                    resources = {str(k).lower(): v for k, v in c.get('resources', {}).items()}
+                    if not resources:
+                        continue
+                    if resources.get('limits'):
+                        self.pod_limits += ['{}={}'.format(k, v) for k, v in resources['limits'].items()]
+                    if resources.get('requests'):
+                        self.pod_requests += ['{}={}'.format(k, v) for k, v in resources['requests'].items()]
+                # remove double entries
+                self.pod_limits = list(set(self.pod_limits))
+                self.pod_requests = list(set(self.pod_requests))
+                if self.pod_limits or self.pod_requests:
+                    self.log.warning('Found pod container requests={} limits={}'.format(
+                        self.pod_limits, self.pod_requests))
+                if containers:
+                    self.log.warning('Removing containers section: {}'.format(overrides['spec'].pop('containers')))
+                self.overrides_json_string = json.dumps(overrides)
+        if template_yaml:
+            with open(os.path.expandvars(os.path.expanduser(str(template_yaml))), 'rt') as f:
+                self.template_dict = yaml.load(f, Loader=getattr(yaml, 'FullLoader', None))
+
+        if trains_conf_file:
+            with open(os.path.expandvars(os.path.expanduser(str(trains_conf_file))), 'rt') as f:
+                self.trains_conf_file = f.read()
+            # make sure we use system packages!
+            self.trains_conf_file += '\nagent.package_manager.system_site_packages=true\n'
+
+    def _set_task_user_properties(self, task_id: str, **properties: str):
+        if self._edit_hyperparams_support is not True:
+            # either not supported or never tested
+            if self._edit_hyperparams_support == self._session.api_version:
+                # tested against latest api_version, not supported
+                return
+            if not self._session.check_min_api_version(self._edit_hyperparams_version):
+                # not supported due to insufficient api_version
+                self._edit_hyperparams_support = self._session.api_version
+                return
+        try:
+            self._session.get(
+                service="tasks",
+                action="edit_hyper_params",
+                task=task_id,
+                hyperparams=[
+                    {
+                        "section": "properties",
+                        "name": k,
+                        "value": str(v),
+                    }
+                    for k, v in properties.items()
+                ],
+            )
+            # definitely supported
+            self._runtime_props_support = True
+        except APIError as error:
+            if error.code == 404:
+                self._edit_hyperparams_support = self._session.api_version
+
+    def run_one_task(self, queue: Text, task_id: Text, worker_args=None, **_):
+        print('Pulling task {} launching on kubernetes cluster'.format(task_id))
+        task_data = self._session.api_client.tasks.get_all(id=[task_id])[0]
+
+        # push task into the k8s queue, so we have visibility on pending tasks in the k8s scheduler
+        try:
+            print('Pushing task {} into temporary pending queue'.format(task_id))
+            self._session.api_client.tasks.reset(task_id)
+            self._session.api_client.tasks.enqueue(task_id, queue=self.k8s_pending_queue_name,
+                                                   status_reason='k8s pending scheduler')
+        except Exception as e:
+            self.log.error("ERROR: Could not push back task [{}] to k8s pending queue [{}], error: {}".format(
+                task_id, self.k8s_pending_queue_name, e))
+            return
+
+        if task_data.execution.docker_cmd:
+            docker_parts = task_data.execution.docker_cmd
+        else:
+            docker_parts = str(ENV_DOCKER_IMAGE.get() or
+                               self._session.config.get("agent.default_docker.image", "nvidia/cuda"))
+
+        # take the first part, this is the docker image name (not arguments)
+        docker_parts = docker_parts.split()
+        docker_image = docker_parts[0]
+        docker_args = docker_parts[1:] if len(docker_parts) > 1 else []
+
+        # get the clearml.conf encoded file
+        # noinspection PyProtectedMember
+        hocon_config_encoded = (self.trains_conf_file or self._session._config_file).encode('ascii')
+        create_trains_conf = "echo '{}' | base64 --decode >> ~/clearml.conf".format(
+            base64.b64encode(
+                hocon_config_encoded
+            ).decode('ascii')
+        )
+
+        if self.ports_mode:
+            print("Kubernetes looking for available pod to use")
+
+        # noinspection PyBroadException
+        try:
+            queue_name = self._session.api_client.queues.get_by_id(queue=queue).name
+        except Exception:
+            queue_name = 'k8s'
+
+        # conform queue name to k8s standards
+        safe_queue_name = queue_name.lower().strip()
+        safe_queue_name = re.sub(r'\W+', '', safe_queue_name).replace('_', '').replace('-', '')
+
+        # Search for a free pod number
+        pod_number = 1
+        while self.ports_mode:
+            kubectl_cmd_new = "kubectl get pods -l {pod_label},{agent_label} -n clearml".format(
+                pod_label=self.LIMIT_POD_LABEL.format(pod_number=pod_number),
+                agent_label=self.AGENT_LABEL
+            )
+            process = subprocess.Popen(kubectl_cmd_new.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+            output, error = process.communicate()
+            output = '' if not output else output if isinstance(output, str) else output.decode('utf-8')
+            error = '' if not error else error if isinstance(error, str) else error.decode('utf-8')
+
+            if not output:
+                # No such pod exist so we can use the pod_number we found
+                break
+            if pod_number >= self.num_of_services:
+                # All pod numbers are taken, exit
+                self.log.warning(
+                    "kubectl last result: {}\n{}\nAll k8s services are in use, task '{}' "
+                    "will be enqueued back to queue '{}'".format(
+                        error, output, task_id, queue
+                    )
+                )
+                self._session.api_client.tasks.reset(task_id)
+                self._session.api_client.tasks.enqueue(
+                    task_id, queue=queue, status_reason='k8s max pod limit (no free k8s service)')
+                return
+            pod_number += 1
+
+        labels = ([self.LIMIT_POD_LABEL.format(pod_number=pod_number)] if self.ports_mode else []) + [self.AGENT_LABEL]
+
+        if self.ports_mode:
+            print("Kubernetes scheduling task id={} on pod={}".format(task_id, pod_number))
+        else:
+            print("Kubernetes scheduling task id={}".format(task_id))
+
+        if self.template_dict:
+            output, error = self._kubectl_apply(
+                create_trains_conf=create_trains_conf,
+                labels=labels, docker_image=docker_image, docker_args=docker_args,
+                task_id=task_id, queue=queue, queue_name=safe_queue_name)
+        else:
+            output, error = self._kubectl_run(
+                create_trains_conf=create_trains_conf,
+                labels=labels, docker_image=docker_image,
+                task_data=task_data,
+                task_id=task_id, queue=queue, queue_name=safe_queue_name)
+
+        error = '' if not error else (error if isinstance(error, str) else error.decode('utf-8'))
+        output = '' if not output else (output if isinstance(output, str) else output.decode('utf-8'))
+        print('kubectl output:\n{}\n{}'.format(error, output))
+        if error:
+            self.log.error("Running kubectl encountered an error: {}".format(error))
+
+        user_props = {"k8s-queue": str(queue_name)}
+        if self.ports_mode:
+            user_props.update({"k8s-pod-number": pod_number, "k8s-pod-label": labels[0]})
+
+        if self._user_props_cb:
+            # noinspection PyBroadException
+            try:
+                custom_props = self._user_props_cb(pod_number) if self.ports_mode else self._user_props_cb()
+                user_props.update(custom_props)
+            except Exception:
+                pass
+
+        if user_props:
+            self._set_task_user_properties(
+                task_id=task_id,
+                **user_props
+            )
+
+    def _parse_docker_args(self, docker_args):
+        # type: (list) -> dict
+        kube_args = {'env': []}
+        while docker_args:
+            cmd = docker_args.pop().strip()
+            if cmd in ('-e', '--env',):
+                env = docker_args.pop().strip()
+                key, value = env.split('=', 1)
+                kube_args[key] += {key: value}
+            else:
+                self.log.warning('skipping docker argument {} (only -e --env supported)'.format(cmd))
+        return kube_args
+
+    def _kubectl_apply(self, create_trains_conf, docker_image, docker_args, labels, queue, task_id, queue_name):
+        template = deepcopy(self.template_dict)
+        template.setdefault('apiVersion', 'v1')
+        template['kind'] = 'Pod'
+        template.setdefault('metadata', {})
+        name = 'clearml-{queue}-id-{task_id}'.format(queue=queue_name, task_id=task_id)
+        template['metadata']['name'] = name
+        template.setdefault('spec', {})
+        template['spec'].setdefault('containers', [])
+        if labels:
+            labels_dict = dict(pair.split('=', 1) for pair in labels)
+            template['metadata'].setdefault('labels', {})
+            template['metadata']['labels'].update(labels_dict)
+        container = self._parse_docker_args(docker_args)
+
+        container_bash_script = [self.container_bash_script] if isinstance(self.container_bash_script, str) \
+            else self.container_bash_script
+
+        script_encoded = '\n'.join(
+            ['#!/bin/bash', ] +
+            [line.format(extra_bash_init_cmd=self.extra_bash_init_script or '', task_id=task_id)
+             for line in container_bash_script])
+
+        create_init_script = \
+            "echo '{}' | base64 --decode >> ~/__start_agent__.sh ; " \
+            "/bin/bash ~/__start_agent__.sh".format(
+                base64.b64encode(
+                    script_encoded.encode('ascii')
+                ).decode('ascii'))
+
+        container = merge_dicts(
+            container,
+            dict(name=name, image=docker_image,
+                 command=['/bin/bash'],
+                 args=['-c', '{} ; {}'.format(create_trains_conf, create_init_script)])
+        )
+
+        if template['spec']['containers']:
+            template['spec']['containers'][0] = merge_dicts(template['spec']['containers'][0], container)
+        else:
+            template['spec']['containers'].append(container)
+
+        fp, yaml_file = tempfile.mkstemp(prefix='clearml_k8stmpl_', suffix='.yml')
+        os.close(fp)
+        with open(yaml_file, 'wt') as f:
+            yaml.dump(template, f)
+
+        kubectl_cmd = self.KUBECTL_APPLY_CMD.format(
+            task_id=task_id,
+            docker_image=docker_image,
+            queue_id=queue,
+        )
+        # make sure we provide a list
+        if isinstance(kubectl_cmd, str):
+            kubectl_cmd = kubectl_cmd.split()
+
+        # add the template file at the end
+        kubectl_cmd += [yaml_file]
+        try:
+            process = subprocess.Popen(kubectl_cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+            output, error = process.communicate()
+        except Exception as ex:
+            return None, str(ex)
+        finally:
+            safe_remove_file(yaml_file)
+
+        return output, error
+
+    def _kubectl_run(self, create_trains_conf, docker_image, labels, queue, task_data, task_id, queue_name):
+        if callable(self.kubectl_cmd):
+            kubectl_cmd = self.kubectl_cmd(task_id, docker_image, queue, task_data, queue_name)
+        else:
+            kubectl_cmd = self.kubectl_cmd.format(
+                queue_name=queue_name,
+                task_id=task_id,
+                docker_image=docker_image,
+                queue_id=queue
+            )
+        # make sure we provide a list
+        if isinstance(kubectl_cmd, str):
+            kubectl_cmd = kubectl_cmd.split()
+
+        if self.overrides_json_string:
+            kubectl_cmd += ['--overrides=' + self.overrides_json_string]
+
+        if self.pod_limits:
+            kubectl_cmd += ['--limits', ",".join(self.pod_limits)]
+        if self.pod_requests:
+            kubectl_cmd += ['--requests', ",".join(self.pod_requests)]
+
+        container_bash_script = [self.container_bash_script] if isinstance(self.container_bash_script, str) \
+            else self.container_bash_script
+        container_bash_script = ' ; '.join(container_bash_script)
+
+        kubectl_cmd += [
+            "--labels=" + ",".join(labels),
+            "--command",
+            "--",
+            "/bin/sh",
+            "-c",
+            "{} ; {}".format(create_trains_conf, container_bash_script.format(
+                extra_bash_init_cmd=self.extra_bash_init_script, task_id=task_id)),
+        ]
+        process = subprocess.Popen(kubectl_cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        output, error = process.communicate()
+        return output, error
+
+    def run_tasks_loop(self, queues: List[Text], worker_params, **kwargs):
+        """
+        :summary: Pull and run tasks from queues.
+        :description: 1. Go through ``queues`` by order.
+                      2. Try getting the next task for each and run the first one that returns.
+                      3. Go to step 1
+        :param queues: IDs of queues to pull tasks from
+        :type queues: list of ``Text``
+        :param worker_params: Worker command line arguments
+        :type worker_params: ``clearml_agent.helper.process.WorkerParams``
+        """
+        events_service = self.get_service(Events)
+
+        # make sure we have a k8s pending queue
+        # noinspection PyBroadException
+        try:
+            self._session.api_client.queues.create(self.k8s_pending_queue_name)
+        except Exception:
+            pass
+        # get queue id
+        self.k8s_pending_queue_name = self._resolve_name(self.k8s_pending_queue_name, "queues")
+
+        _last_machine_update_ts = 0
+        while True:
+            # iterate over queues (priority style, queues[0] is highest)
+            for queue in queues:
+                # delete old completed / failed pods
+                get_bash_output(self.KUBECTL_DELETE_CMD)
+
+                # get next task in queue
+                try:
+                    response = self._session.api_client.queues.get_next_task(queue=queue)
+                except Exception as e:
+                    print("Warning: Could not access task queue [{}], error: {}".format(queue, e))
+                    continue
+                else:
+                    try:
+                        task_id = response.entry.task
+                    except AttributeError:
+                        print("No tasks in queue {}".format(queue))
+                        continue
+                    events_service.send_log_events(
+                        self.worker_id,
+                        task_id=task_id,
+                        lines="task {} pulled from {} by worker {}".format(
+                            task_id, queue, self.worker_id
+                        ),
+                        level="INFO",
+                    )
+
+                    self.report_monitor(ResourceMonitor.StatusReport(queues=queues, queue=queue, task=task_id))
+                    self.run_one_task(queue, task_id, worker_params)
+                    self.report_monitor(ResourceMonitor.StatusReport(queues=self.queues))
+                    break
+            else:
+                # sleep and retry polling
+                print("No tasks in Queues, sleeping for {:.1f} seconds".format(self._polling_interval))
+                sleep(self._polling_interval)
+
+            if self._session.config["agent.reload_config"]:
+                self.reload_config()
+
+    def k8s_daemon(self, queue):
+        """
+        Start the k8s Glue service.
+        This service will be pulling tasks from *queue* and scheduling them for execution using kubectl.
+        Notice all scheduled tasks are pushed back into K8S_PENDING_QUEUE,
+        and popped when execution actually starts. This creates full visibility into the k8s scheduler.
+        Manually popping a task from the K8S_PENDING_QUEUE,
+        will cause the k8s scheduler to skip the execution once the scheduled tasks needs to be executed
+
+        :param list(str) queue: queue name to pull from
+        """
+        return self.daemon(queues=[ObjectID(name=queue)] if queue else None,
+                           log_level=logging.INFO, foreground=True, docker=False)
+
+    @classmethod
+    def get_ssh_server_bash(cls, ssh_port_number):
+        return ' ; '.join(line.format(port=ssh_port_number) for line in cls.BASH_INSTALL_SSH_CMD)

clearml_agent/helper/base.py

@@ -1,4 +1,4 @@
-""" TRAINS-AGENT Stdout Helper Functions  """
+""" CLEARML-AGENT Stdout Helper Functions  """
 from __future__ import print_function, unicode_literals
 
 import io
@@ -28,8 +28,8 @@ from tqdm import tqdm
 
 import six
 from six.moves import reduce
-from trains_agent.errors import CommandFailedError
-from trains_agent.helper.dicts import filter_keys
+from clearml_agent.errors import CommandFailedError
+from clearml_agent.helper.dicts import filter_keys
 
 pretty_lines = False
 
@@ -173,14 +173,32 @@ def normalize_path(*paths):
 
 
 def safe_remove_file(filename, error_message=None):
+    # noinspection PyBroadException
     try:
-        os.remove(filename)
+        if filename:
+            os.remove(filename)
     except Exception:
         if error_message:
             print(error_message)
 
 
-def get_python_path(script_dir, entry_point, package_api):
+def safe_remove_tree(filename):
+    if not filename:
+        return
+    # noinspection PyBroadException
+    try:
+        shutil.rmtree(filename, ignore_errors=True)
+    except Exception:
+        pass
+    # noinspection PyBroadException
+    try:
+        os.remove(filename)
+    except Exception:
+        pass
+
+
+def get_python_path(script_dir, entry_point, package_api, is_conda_env=False):
+    # noinspection PyBroadException
     try:
         python_path_sep = ';' if is_windows_platform() else ':'
         python_path_cmd = package_api.get_python_command(
@@ -192,9 +210,9 @@ def get_python_path(script_dir, entry_point, package_api):
             (Path(script_dir) / Path(entry_point)).parent.absolute().as_posix(),
             python_path_sep=python_path_sep)
         if is_windows_platform():
-            return python_path.replace('/', '\\') + org_python_path
+            python_path = python_path.replace('/', '\\')
 
-        return python_path + org_python_path
+        return python_path if is_conda_env else (python_path + org_python_path)
     except Exception:
         return None
 
@@ -362,11 +380,11 @@ AllDumper.add_multi_representer(object, lambda dumper, data: dumper.represent_st
 
 
 def error(message):
-    print('\ntrains_agent: ERROR: {}\n'.format(message))
+    print('\nclearml_agent: ERROR: {}\n'.format(message))
 
 
 def warning(message):
-    print('trains_agent: Warning: {}'.format(message))
+    print('clearml_agent: Warning: {}'.format(message))
 
 
 class TqdmStream(object):
@@ -442,9 +460,9 @@ def chain_map(*args):
     return reduce(lambda x, y: x.update(y) or x, args, {})
 
 
-def check_directory_path(path):
+def check_directory_path(path, check_whitespace_in_path=True):
     message = 'Could not create directory "{}": {}'
-    if not is_windows_platform():
+    if not is_windows_platform() and check_whitespace_in_path:
         match = re.search(r'\s', path)
         if match:
             raise CommandFailedError(
@@ -537,6 +555,7 @@ class ExecutionInfo(NonStrictAttrs):
     branch = nullable_string
     version_num = nullable_string
     tag = nullable_string
+    docker_cmd = nullable_string
 
     @classmethod
     def from_task(cls, task_info):
@@ -554,4 +573,24 @@ class ExecutionInfo(NonStrictAttrs):
             execution.entry_point = entry_point
             execution.working_dir = working_dir or ""
 
+        # noinspection PyBroadException
+        try:
+            execution.docker_cmd = task_info.execution.docker_cmd
+        except Exception:
+            pass
+
         return execution
+
+
+class safe_furl(furl.furl):
+
+    @property
+    def port(self):
+        return self._port
+
+    @port.setter
+    def port(self, port):
+        """
+        Any port value is valid
+        """
+        self._port = port

clearml_agent/helper/check_update.py

@@ -21,14 +21,14 @@ def start_check_update_daemon():
 
 def _check_new_version_available():
     cur_version = __version__
-    update_server_releases = requests.get('https://updates.trains.allegro.ai/updates',
-                                          data=json.dumps({"versions": {"trains-agent": str(cur_version)}}),
+    update_server_releases = requests.get('https://updates.clear.ml/updates',
+                                          data=json.dumps({"versions": {"clearml-agent": str(cur_version)}}),
                                           timeout=3.0)
     if update_server_releases.ok:
         update_server_releases = update_server_releases.json()
     else:
         return None
-    trains_answer = update_server_releases.get("trains-agent", {})
+    trains_answer = update_server_releases.get("clearml-agent", {})
     latest_version = trains_answer.get("version")
     cur_version = cur_version
     latest_version = latest_version or ''
@@ -48,7 +48,7 @@ def _check_update_daemon():
             if latest_version:
                 if latest_version[1]:
                     sep = os.linesep
-                    print('TRAINS-AGENT new package available: UPGRADE to v{} is recommended!\nRelease Notes:\n{}'.format(
+                    print('CLEARML-AGENT new package available: UPGRADE to v{} is recommended!\nRelease Notes:\n{}'.format(
                         latest_version[0], sep.join(latest_version[2])))
                 else:
                     print('TRAINS-SERVER new version available: upgrade to v{} is recommended!'.format(

clearml_agent/helper/console.py

@@ -9,7 +9,7 @@ from attr import attrs, attrib
 
 import six
 from six import binary_type, text_type
-from trains_agent.helper.base import nonstrict_in_place_sort, create_tree
+from clearml_agent.helper.base import nonstrict_in_place_sort
 
 
 def print_text(text, newline=True):
@@ -22,15 +22,21 @@ def print_text(text, newline=True):
         sys.stdout.write(data)
 
 
-def decode_binary_lines(binary_lines, encoding='utf-8'):
+def decode_binary_lines(binary_lines, encoding='utf-8', replace_cr=False, overwrite_cr=False):
     # decode per line, if we failed decoding skip the line
     lines = []
     for b in binary_lines:
+        # noinspection PyBroadException
         try:
-            l = b.decode(encoding=encoding, errors='replace').replace('\r', '\n')
-        except:
-            l = ''
-        lines.append(l + '\n' if l and l[-1] != '\n' else l)
+            line = b.decode(encoding=encoding, errors='replace')
+            if replace_cr:
+                line = line.replace('\r', '\n')
+            elif overwrite_cr:
+                cr_lines = line.split('\r')
+                line = cr_lines[-1] if cr_lines[-1] or len(cr_lines) < 2 else cr_lines[-2]
+        except Exception:
+            line = ''
+        lines.append(line + '\n' if not line or line[-1] != '\n' else line)
     return lines
 
 

clearml_agent/helper/dicts.py

@@ -0,0 +1,17 @@
+from typing import Callable, Dict, Any
+
+
+def filter_keys(filter_, dct):  # type: (Callable[[Any], bool], Dict) -> Dict
+    return {key: value for key, value in dct.items() if filter_(key)}
+
+
+def merge_dicts(dict1, dict2):
+    """ Recursively merges dict2 into dict1 """
+    if not isinstance(dict1, dict) or not isinstance(dict2, dict):
+        return dict2
+    for k in dict2:
+        if k in dict1:
+            dict1[k] = merge_dicts(dict1[k], dict2[k])
+        else:
+            dict1[k] = dict2[k]
+    return dict1

clearml_agent/helper/gpu/gpustat.py

@@ -200,24 +200,30 @@ class GPUStatCollection(object):
                     GPUStatCollection.global_processes[nv_process.pid] = \
                         psutil.Process(pid=nv_process.pid)
                 ps_process = GPUStatCollection.global_processes[nv_process.pid]
-                process['username'] = ps_process.username()
-                # cmdline returns full path;
-                # as in `ps -o comm`, get short cmdnames.
-                _cmdline = ps_process.cmdline()
-                if not _cmdline:
-                    # sometimes, zombie or unknown (e.g. [kworker/8:2H])
-                    process['command'] = '?'
-                    process['full_command'] = ['?']
-                else:
-                    process['command'] = os.path.basename(_cmdline[0])
-                    process['full_command'] = _cmdline
-                # Bytes to MBytes
-                process['gpu_memory_usage'] = nv_process.usedGpuMemory // MB
-                process['cpu_percent'] = ps_process.cpu_percent()
-                process['cpu_memory_usage'] = \
-                    round((ps_process.memory_percent() / 100.0) *
-                          psutil.virtual_memory().total)
                 process['pid'] = nv_process.pid
+                # noinspection PyBroadException
+                try:
+                    # we do not actually use these, so no point in collecting them
+                    # process['username'] = ps_process.username()
+                    # # cmdline returns full path;
+                    # # as in `ps -o comm`, get short cmdnames.
+                    # _cmdline = ps_process.cmdline()
+                    # if not _cmdline:
+                    #     # sometimes, zombie or unknown (e.g. [kworker/8:2H])
+                    #     process['command'] = '?'
+                    #     process['full_command'] = ['?']
+                    # else:
+                    #     process['command'] = os.path.basename(_cmdline[0])
+                    #     process['full_command'] = _cmdline
+                    # process['cpu_percent'] = ps_process.cpu_percent()
+                    # process['cpu_memory_usage'] = \
+                    #     round((ps_process.memory_percent() / 100.0) *
+                    #           psutil.virtual_memory().total)
+                    # Bytes to MBytes
+                    process['gpu_memory_usage'] = nv_process.usedGpuMemory // MB
+                except Exception:
+                    # insufficient permissions
+                    pass
                 return process
 
             if not GPUStatCollection._gpu_device_info.get(index):
@@ -285,12 +291,13 @@ class GPUStatCollection(object):
                         # e.g. nvidia-smi reset  or  reboot the system
                         pass
 
-                # TODO: Do not block if full process info is not requested
-                time.sleep(0.1)
-                for process in processes:
-                    pid = process['pid']
-                    cache_process = GPUStatCollection.global_processes[pid]
-                    process['cpu_percent'] = cache_process.cpu_percent()
+                # we do not actually use these, so no point in collecting them
+                # # TODO: Do not block if full process info is not requested
+                # time.sleep(0.1)
+                # for process in processes:
+                #     pid = process['pid']
+                #     cache_process = GPUStatCollection.global_processes[pid]
+                #     process['cpu_percent'] = cache_process.cpu_percent()
 
             index = N.nvmlDeviceGetIndex(handle)
             gpu_info = {

clearml_agent/helper/package/base.py

@@ -5,8 +5,8 @@ from contextlib import contextmanager
 from typing import Text, Iterable, Union
 
 import six
-from trains_agent.helper.base import mkstemp, safe_remove_file, join_lines
-from trains_agent.helper.process import Executable, Argv, PathLike
+from clearml_agent.helper.base import mkstemp, safe_remove_file, join_lines, select_for_platform
+from clearml_agent.helper.process import Executable, Argv, PathLike
 
 
 @six.add_metaclass(abc.ABCMeta)
@@ -66,7 +66,20 @@ class PackageManager(object):
         pass
 
     def upgrade_pip(self):
-        return self._install("pip"+self.get_pip_version(), "--upgrade")
+        result = self._install(
+            select_for_platform(windows='"pip{}"', linux='pip{}').format(self.get_pip_version()), "--upgrade")
+        packages = self.run_with_env(('list',), output=True).splitlines()
+        # p.split is ('pip', 'x.y.z')
+        pip = [p.split() for p in packages if len(p.split()) == 2 and p.split()[0] == 'pip']
+        if pip:
+            # noinspection PyBroadException
+            try:
+                from .requirements import MarkerRequirement
+                pip = pip[0][1].split('.')
+                MarkerRequirement.pip_new_version = bool(int(pip[0]) >= 20)
+            except Exception:
+                pass
+        return result
 
     def get_python_command(self, extra=()):
         # type: (...) -> Executable
@@ -111,10 +124,12 @@ class PackageManager(object):
     def out_of_scope_install_package(cls, package_name, *args):
         if PackageManager._selected_manager is not None:
             try:
-                return PackageManager._selected_manager._install(package_name, *args)
+                result = PackageManager._selected_manager._install(package_name, *args)
+                if result not in (0, None, True):
+                    return False
             except Exception:
-                pass
-        return
+                return False
+        return True
 
     @classmethod
     def out_of_scope_freeze(cls):

clearml_agent/helper/package/conda_api.py

@@ -2,8 +2,9 @@ from __future__ import unicode_literals
 
 import json
 import re
-import shutil
+import os
 import subprocess
+from collections import OrderedDict
 from distutils.spawn import find_executable
 from functools import partial
 from itertools import chain
@@ -14,17 +15,18 @@ import yaml
 from time import time
 from attr import attrs, attrib, Factory
 from pathlib2 import Path
-from requirements import parse
-from requirements.requirement import Requirement
+from clearml_agent.external.requirements_parser import parse
+from clearml_agent.external.requirements_parser.requirement import Requirement
 
-from trains_agent.errors import CommandFailedError
-from trains_agent.helper.base import rm_tree, NonStrictAttrs, select_for_platform, is_windows_platform
-from trains_agent.helper.process import Argv, Executable, DEVNULL, CommandSequence, PathLike
-from trains_agent.helper.package.requirements import SimpleVersion
-from trains_agent.session import Session
+from clearml_agent.errors import CommandFailedError
+from clearml_agent.helper.base import rm_tree, NonStrictAttrs, select_for_platform, is_windows_platform, ExecutionInfo
+from clearml_agent.helper.process import Argv, Executable, DEVNULL, CommandSequence, PathLike
+from clearml_agent.helper.package.requirements import SimpleVersion
+from clearml_agent.session import Session
 from .base import PackageManager
 from .pip_api.venv import VirtualenvPip
 from .requirements import RequirementsManager, MarkerRequirement
+from ...backend_api.session.defs import ENV_CONDA_ENV_PACKAGE
 
 package_normalize = partial(re.compile(r"""\[version=['"](.*)['"]\]""").sub, r"\1")
 
@@ -40,8 +42,8 @@ def _package_diff(path, packages):
 
 class CondaPip(VirtualenvPip):
     def __init__(self, source=None, *args, **kwargs):
-        super(CondaPip, self).__init__(*args, interpreter=Path(kwargs.get('path'), "python.exe") \
-            if is_windows_platform() and kwargs.get('path') else None, **kwargs)
+        super(CondaPip, self).__init__(*args, interpreter=Path(kwargs.get('path'), "python.exe")
+                                       if is_windows_platform() and kwargs.get('path') else None, **kwargs)
         self.source = source
 
     def run_with_env(self, command, output=False, **kwargs):
@@ -61,8 +63,8 @@ class CondaAPI(PackageManager):
 
     MINIMUM_VERSION = "4.3.30"
 
-    def __init__(self, session, path, python, requirements_manager):
-        # type: (Session, PathLike, float, RequirementsManager) -> None
+    def __init__(self, session, path, python, requirements_manager, execution_info=None, **kwargs):
+        # type: (Session, PathLike, float, RequirementsManager, ExecutionInfo, Any) -> None
         """
         :param python: base python version to use (e.g python3.6)
         :param path: path of env
@@ -72,7 +74,15 @@ class CondaAPI(PackageManager):
         self.source = None
         self.requirements_manager = requirements_manager
         self.path = path
+        self.env_read_only = False
         self.extra_channels = self.session.config.get('agent.package_manager.conda_channels', [])
+        self.conda_env_as_base_docker = \
+            self.session.config.get('agent.package_manager.conda_env_as_base_docker', None) or \
+            bool(ENV_CONDA_ENV_PACKAGE.get())
+        if ENV_CONDA_ENV_PACKAGE.get():
+            self.conda_pre_build_env_path = ENV_CONDA_ENV_PACKAGE.get()
+        else:
+            self.conda_pre_build_env_path = execution_info.docker_cmd if execution_info else None
         self.pip = CondaPip(
             session=self.session,
             source=self.source,
@@ -80,10 +90,15 @@ class CondaAPI(PackageManager):
             requirements_manager=self.requirements_manager,
             path=self.path,
         )
-        self.conda = (
-            find_executable("conda")
-            or Argv(select_for_platform(windows="where", linux="which"), "conda").get_output(shell=True).strip()
-        )
+        try:
+            self.conda = (
+                find_executable("conda") or
+                Argv(select_for_platform(windows="where", linux="which"), "conda").get_output(
+                    shell=select_for_platform(windows=True, linux=False)).strip()
+            )
+        except Exception:
+            raise ValueError("ERROR: package manager \"conda\" selected, "
+                             "but \'conda\' executable could not be located")
         try:
             output = Argv(self.conda, "--version").get_output(stderr=subprocess.STDOUT)
         except subprocess.CalledProcessError as ex:
@@ -111,13 +126,58 @@ class CondaAPI(PackageManager):
     def bin(self):
         return self.pip.bin
 
+    # noinspection SpellCheckingInspection
     def upgrade_pip(self):
-        return self._install("pip" + self.pip.get_pip_version())
+        # do not change pip version if pre built environement is used
+        if self.env_read_only:
+            print('Conda environment in read-only mode, skipping pip upgrade.')
+            return ''
+        return self._install(select_for_platform(windows='"pip{}"', linux='pip{}').format(self.pip.get_pip_version()))
 
     def create(self):
         """
         Create a new environment
         """
+        if self.conda_env_as_base_docker and self.conda_pre_build_env_path:
+            if Path(self.conda_pre_build_env_path).is_dir():
+                print("Using pre-existing Conda environment from {}".format(self.conda_pre_build_env_path))
+                self.path = Path(self.conda_pre_build_env_path)
+                self.source = ("conda", "activate", self.path.as_posix())
+                self.pip = CondaPip(
+                    session=self.session,
+                    source=self.source,
+                    python=self.python,
+                    requirements_manager=self.requirements_manager,
+                    path=self.path,
+                )
+                conda_env = self._get_conda_sh()
+                self.source = self.pip.source = CommandSequence(('source', conda_env.as_posix()), self.source)
+                self.env_read_only = True
+                return self
+            elif Path(self.conda_pre_build_env_path).is_file():
+                print("Restoring Conda environment from {}".format(self.conda_pre_build_env_path))
+                tar_path = find_executable("tar")
+                self.path.mkdir(parents=True, exist_ok=True)
+                output = Argv(
+                    tar_path,
+                    "-xzf",
+                    self.conda_pre_build_env_path,
+                    "-C",
+                    self.path,
+                ).get_output()
+
+                self.source = self.pip.source = ("conda", "activate", self.path.as_posix())
+                conda_env = self._get_conda_sh()
+                self.source = self.pip.source = CommandSequence(('source', conda_env.as_posix()), self.source)
+                # unpack cleanup
+                print("Fixing prefix in Conda environment {}".format(self.path))
+                CommandSequence(('source', conda_env.as_posix()),
+                                ((self.path / 'bin' / 'conda-unpack').as_posix(), )).get_output()
+                return self
+            else:
+                raise ValueError("Could not restore Conda environment, cannot find {}".format(
+                    self.conda_pre_build_env_path))
+
         output = Argv(
             self.conda,
             "create",
@@ -133,13 +193,15 @@ class CondaAPI(PackageManager):
         self.source = self.pip.source = (
             tuple(match.group(1).split()) + (match.group(2),)
             if match
-            else ("activate", self.path)
+            else ("conda", "activate", self.path.as_posix())
         )
-        conda_env = Path(self.conda).parent.parent / 'etc' / 'profile.d' / 'conda.sh'
+
+        conda_env = self._get_conda_sh()
         if conda_env.is_file() and not is_windows_platform():
             self.source = self.pip.source = CommandSequence(('source', conda_env.as_posix()), self.source)
 
         # install cuda toolkit
+        # noinspection PyBroadException
         try:
             cuda_version = float(int(self.session.config['agent.cuda_version'])) / 10.0
             if cuda_version > 0:
@@ -181,6 +243,10 @@ class CondaAPI(PackageManager):
 
     def _install(self, *args):
         # type: (*PathLike) -> ()
+        # if we are in read only mode, do not install anything
+        if self.env_read_only:
+            print('Conda environment in read-only mode, skipping package installing: {}'.format(args))
+            return
         channels_args = tuple(
             chain.from_iterable(("-c", channel) for channel in self.extra_channels)
         )
@@ -208,6 +274,10 @@ class CondaAPI(PackageManager):
         return self._install(*packages)
 
     def uninstall_packages(self, *packages):
+        # if we are in read only mode, do not uninstall anything
+        if self.env_read_only:
+            print('Conda environment in read-only mode, skipping package uninstalling: {}'.format(packages))
+            return ''
         return self._run_command(("uninstall", "-p", self.path))
 
     def install_from_file(self, path):
@@ -226,23 +296,158 @@ class CondaAPI(PackageManager):
         with self.temp_file("pip_reqs", pip_packages) as reqs:
             self.pip.install_from_file(reqs)
 
-    def freeze(self):
+    def freeze(self, freeze_full_environment=False):
         requirements = self.pip.freeze()
+        req_lines = []
+        conda_lines = []
+
+        # noinspection PyBroadException
         try:
-            conda_packages = json.loads(self._run_command((self.conda, "list", "--json", "-p", self.path), raw=True))
-            conda_packages_txt = []
-            requirements_pip = [r.split('==')[0].strip().lower() for r in requirements['pip']]
-            for pkg in conda_packages:
-                # skip if this is a pypi package or it is not a python package at all
-                if pkg['channel'] == 'pypi' or pkg['name'].lower() not in requirements_pip:
+            pip_lines = requirements['pip']
+            conda_packages_json = json.loads(
+                self._run_command((self.conda, "list", "--json", "-p", self.path), raw=True))
+            for r in conda_packages_json:
+                # check if this is a pypi package, if it is, leave it outside
+                if not r.get('channel') or r.get('channel') == 'pypi':
+                    name = (r['name'].replace('-', '_'), r['name'])
+                    pip_req_line = [l for l in pip_lines
+                                    if l.split('==', 1)[0].strip() in name or l.split('@', 1)[0].strip() in name]
+                    if pip_req_line and \
+                            ('@' not in pip_req_line[0] or
+                             not pip_req_line[0].split('@', 1)[1].strip().startswith('file://')):
+                        req_lines.append(pip_req_line[0])
+                        continue
+
+                    req_lines.append(
+                        '{}=={}'.format(name[1], r['version']) if r.get('version') else '{}'.format(name[1]))
                     continue
-                conda_packages_txt.append('{0}{1}{2}'.format(pkg['name'], '==', pkg['version']))
-            requirements['conda'] = conda_packages_txt
-        except:
+
+                # check if we have it in our required packages
+                name = r['name']
+                # hack support pytorch/torch different naming convention
+                if name == 'pytorch':
+                    name = 'torch'
+                # skip over packages with _
+                if name.startswith('_'):
+                    continue
+                conda_lines.append('{}=={}'.format(name, r['version']) if r.get('version') else '{}'.format(name))
+            # make sure we see the conda packages, put them into the pip as well
+            if conda_lines:
+                req_lines = ['# Conda Packages', ''] + conda_lines + ['', '# pip Packages', ''] + req_lines
+
+            requirements['pip'] = req_lines
+            requirements['conda'] = conda_lines
+        except Exception:
             pass
+
+        if freeze_full_environment:
+            # noinspection PyBroadException
+            try:
+                conda_env_json = json.loads(
+                    self._run_command((self.conda, "env", "export", "--json", "-p", self.path), raw=True))
+                conda_env_json.pop('name', None)
+                conda_env_json.pop('prefix', None)
+                conda_env_json.pop('channels', None)
+                requirements['conda_env_json'] = json.dumps(conda_env_json)
+            except Exception:
+                pass
+
         return requirements
 
+    def _load_conda_full_env(self, conda_env_dict, requirements):
+        # noinspection PyBroadException
+        try:
+            cuda_version = int(self.session.config.get('agent.cuda_version', 0))
+        except Exception:
+            cuda_version = 0
+
+        conda_env_dict['channels'] = self.extra_channels
+        if 'dependencies' not in conda_env_dict:
+            conda_env_dict['dependencies'] = []
+        new_dependencies = OrderedDict()
+        pip_requirements = None
+        for line in conda_env_dict['dependencies']:
+            if isinstance(line, dict):
+                pip_requirements = line.pop('pip', None)
+                continue
+            name = line.strip().split('=', 1)[0].lower()
+            if name == 'pip':
+                continue
+            elif name == 'python':
+                line = 'python={}'.format('.'.join(line.split('=')[1].split('.')[:2]))
+            elif name == 'tensorflow-gpu' and cuda_version == 0:
+                line = 'tensorflow={}'.format(line.split('=')[1])
+            elif name == 'tensorflow' and cuda_version > 0:
+                line = 'tensorflow-gpu={}'.format(line.split('=')[1])
+            elif name in ('cupti', 'cudnn'):
+                # cudatoolkit should pull them based on the cudatoolkit version
+                continue
+            elif name.startswith('_'):
+                continue
+            new_dependencies[line.split('=', 1)[0].strip()] = line
+
+        # fix packages:
+        conda_env_dict['dependencies'] = list(new_dependencies.values())
+
+        with self.temp_file("conda_env", yaml.dump(conda_env_dict), suffix=".yml") as name:
+            print('Conda: Trying to install requirements:\n{}'.format(conda_env_dict['dependencies']))
+            result = self._run_command(
+                ("env", "update", "-p", self.path, "--file", name)
+            )
+
+        # check if we need to remove specific packages
+        bad_req = self._parse_conda_result_bad_packges(result)
+        if bad_req:
+            print('failed installing the following conda packages: {}'.format(bad_req))
+            return False
+
+        if pip_requirements:
+            # create a list of vcs packages that we need to replace in the pip section
+            vcs_reqs = {}
+            if 'pip' in requirements:
+                pip_lines = requirements['pip'].splitlines() \
+                    if isinstance(requirements['pip'], six.string_types) else requirements['pip']
+                for line in pip_lines:
+                    try:
+                        marker = list(parse(line))
+                    except Exception:
+                        marker = None
+                    if not marker:
+                        continue
+
+                    m = MarkerRequirement(marker[0])
+                    if m.vcs:
+                        vcs_reqs[m.name] = m
+            try:
+                pip_req_str = [str(vcs_reqs.get(r.split('=', 1)[0], r)) for r in pip_requirements
+                               if not r.startswith('pip=') and not r.startswith('virtualenv=')]
+                print('Conda: Installing requirements: step 2 - using pip:\n{}'.format(pip_req_str))
+                PackageManager._selected_manager = self.pip
+                self.pip.load_requirements({'pip': '\n'.join(pip_req_str)})
+            except Exception as e:
+                print(e)
+                raise e
+            finally:
+                PackageManager._selected_manager = self
+
+        self.requirements_manager.post_install(self.session)
+
     def load_requirements(self, requirements):
+        # if we are in read only mode, do not uninstall anything
+        if self.env_read_only:
+            print('Conda environment in read-only mode, skipping requirements installation.')
+            return None
+
+        # if we have a full conda environment, use it and pass the pip to pip
+        if requirements.get('conda_env_json'):
+            # noinspection PyBroadException
+            try:
+                conda_env_json = json.loads(requirements.get('conda_env_json'))
+                print('Conda restoring full yaml environment')
+                return self._load_conda_full_env(conda_env_json, requirements)
+            except Exception:
+                print('Could not load fully stored conda environment, falling back to requirements')
+
         # create new environment file
         conda_env = dict()
         conda_env['channels'] = self.extra_channels
@@ -276,6 +481,15 @@ class CondaAPI(PackageManager):
             if m.vcs:
                 pip_requirements.append(m)
                 continue
+            # Skip over pip
+            if m.name in ('pip', 'virtualenv', ):
+                continue
+            # python version, only major.minor
+            if m.name == 'python' and m.specs:
+                m.specs = [(m.specs[0][0], '.'.join(m.specs[0][1].split('.')[:2])), ]
+                if '.' not in m.specs[0][1]:
+                    continue
+
             conda_supported_req_names.append(m.name.lower())
             if m.req.name.lower() == 'matplotlib':
                 has_matplotlib = True
@@ -303,15 +517,20 @@ class CondaAPI(PackageManager):
                     continue
 
                 m = MarkerRequirement(marker[0])
+                # skip over local files (we cannot change the version to a local file)
+                if m.local_file:
+                    continue
                 m_name = m.name.lower()
                 if m_name in conda_supported_req_names:
                     # this package is in the conda list,
                     # make sure that if we changed version and we match it in conda
-                    conda_supported_req_names.remove(m_name)
+                    ## conda_supported_req_names.remove(m_name)
                     for cr in reqs:
-                        if m_name == cr.name.lower():
+                        if m_name.lower().replace('_', '-') == cr.name.lower().replace('_', '-'):
                             # match versions
                             cr.specs = m.specs
+                            # # conda always likes "-" not "_" but only on pypi packages
+                            # cr.name = cr.name.lower().replace('_', '-')
                             break
                 else:
                     # not in conda, it is a pip package
@@ -319,29 +538,39 @@ class CondaAPI(PackageManager):
                     if m_name == 'matplotlib':
                         has_matplotlib = True
 
-            # remove any leftover conda packages (they were removed from the pip list)
-            if conda_supported_req_names:
-                reqs = [r for r in reqs if r.name.lower() not in conda_supported_req_names]
-
         # Conda requirements Hacks:
         if has_matplotlib:
             reqs.append(MarkerRequirement(Requirement.parse('graphviz')))
             reqs.append(MarkerRequirement(Requirement.parse('python-graphviz')))
             reqs.append(MarkerRequirement(Requirement.parse('kiwisolver')))
+
+        # remove specific cudatoolkit, it should have being preinstalled.
+        # allow to override default cudatoolkit, but not the derivative packages, cudatoolkit should pull them
+        reqs = [r for r in reqs if r.name not in ('cudnn', 'cupti')]
+
         if has_torch and cuda_version == 0:
             reqs.append(MarkerRequirement(Requirement.parse('cpuonly')))
 
+        # make sure we have no double entries
+        reqs = list(OrderedDict((r.name, r) for r in reqs).values())
+
         # conform conda packages (version/name)
         for r in reqs:
+            # change _ to - in name but not the prefix _ (as this is conda prefix)
+            if not r.name.startswith('_') and not requirements.get('conda', None):
+                r.name = r.name.replace('_', '-')
             # remove .post from version numbers, it fails ~= version, and change == to ~=
             if r.specs and r.specs[0]:
                 r.specs = [(r.specs[0][0].replace('==', '~='), r.specs[0][1].split('.post')[0])]
-            # conda always likes "-" not "_"
-            r.req.name = r.req.name.replace('_', '-')
 
         while reqs:
             # notice, we give conda more freedom in version selection, to help it choose best combination
-            conda_env['dependencies'] = [r.tostr() for r in reqs]
+            def clean_ver(ar):
+                if not ar.specs:
+                    return ar.tostr()
+                ar.specs = [(ar.specs[0][0], ar.specs[0][1] + '.0' if '.' not in ar.specs[0][1] else ar.specs[0][1])]
+                return ar.tostr()
+            conda_env['dependencies'] = [clean_ver(r) for r in reqs]
             with self.temp_file("conda_env", yaml.dump(conda_env), suffix=".yml") as name:
                 print('Conda: Trying to install requirements:\n{}'.format(conda_env['dependencies']))
                 result = self._run_command(
@@ -371,14 +600,17 @@ class CondaAPI(PackageManager):
 
         if pip_requirements:
             try:
-                pip_req_str = [r.tostr() for r in pip_requirements]
+                pip_req_str = [r.tostr() for r in pip_requirements if r.name not in ('pip', 'virtualenv', )]
                 print('Conda: Installing requirements: step 2 - using pip:\n{}'.format(pip_req_str))
-                self.pip.load_requirements('\n'.join(pip_req_str))
+                PackageManager._selected_manager = self.pip
+                self.pip.load_requirements({'pip': '\n'.join(pip_req_str)})
             except Exception as e:
                 print(e)
                 raise e
+            finally:
+                PackageManager._selected_manager = self
 
-        self.requirements_manager.post_install()
+        self.requirements_manager.post_install(self.session)
         return True
 
     def _parse_conda_result_bad_packges(self, result_dict):
@@ -441,8 +673,22 @@ class CondaAPI(PackageManager):
     def get_python_command(self, extra=()):
         return CommandSequence(self.source, self.pip.get_python_command(extra=extra))
 
+    def _get_conda_sh(self):
+        # type () -> Path
+        base_conda_env = Path(self.conda).parent.parent / 'etc' / 'profile.d' / 'conda.sh'
+        if base_conda_env.is_file():
+            return base_conda_env
+        for path in os.environ.get('PATH', '').split(select_for_platform(windows=';', linux=':')):
+            conda = find_executable("conda", path=path)
+            if not conda:
+                continue
+            conda_env = Path(conda).parent.parent / 'etc' / 'profile.d' / 'conda.sh'
+            if conda_env.is_file():
+                return conda_env
+        return base_conda_env
 
-# enable hashing with cmp=False because pdb fails on unhashable exceptions
+
+# enable hashing with cmp=False because pdb fails on un-hashable exceptions
 exception = attrs(str=True, cmp=False)
 
 

clearml_agent/helper/package/external_req.py

@@ -0,0 +1,106 @@
+import re
+from collections import OrderedDict
+from typing import Text
+
+from .base import PackageManager
+from .requirements import SimpleSubstitution
+from ..base import safe_furl as furl
+
+
+class ExternalRequirements(SimpleSubstitution):
+
+    name = "external_link"
+
+    def __init__(self, *args, **kwargs):
+        super(ExternalRequirements, self).__init__(*args, **kwargs)
+        self.post_install_req = []
+        self.post_install_req_lookup = OrderedDict()
+
+    def match(self, req):
+        # match both editable or code or unparsed
+        if not (not req.name or req.req and (req.req.editable or req.req.vcs)):
+            return False
+        if not req.req or not req.req.line or not req.req.line.strip() or req.req.line.strip().startswith('#'):
+            return False
+        if req.pip_new_version and not (req.req.editable or req.req.vcs):
+            return False
+        return True
+
+    def post_install(self, session):
+        post_install_req = self.post_install_req
+        self.post_install_req = []
+        for req in post_install_req:
+            try:
+                freeze_base = PackageManager.out_of_scope_freeze() or ''
+            except:
+                freeze_base = ''
+
+            req_line = req.tostr(markers=False)
+            if req_line.strip().startswith('-e ') or req_line.strip().startswith('--editable'):
+                req_line = re.sub(r'^(-e|--editable=?)\s*', '', req_line, count=1)
+
+            if req.req.vcs and req_line.startswith('git+'):
+                try:
+                    url_no_frag = furl(req_line)
+                    url_no_frag.set(fragment=None)
+                    # reverse replace
+                    fragment = req_line[::-1].replace(url_no_frag.url[::-1], '', 1)[::-1]
+                    vcs_url = req_line[4:]
+                    # reverse replace
+                    vcs_url = vcs_url[::-1].replace(fragment[::-1], '', 1)[::-1]
+                    from ..repo import Git
+                    vcs = Git(session=session, url=vcs_url, location=None, revision=None)
+                    vcs._set_ssh_url()
+                    new_req_line = 'git+{}{}'.format(vcs.url_with_auth, fragment)
+                    if new_req_line != req_line:
+                        furl_line = furl(new_req_line)
+                        print('Replacing original pip vcs \'{}\' with \'{}\''.format(
+                            req_line,
+                            furl_line.set(password='xxxxxx').tostr() if furl_line.password else new_req_line))
+                        req_line = new_req_line
+                except Exception:
+                    print('WARNING: Failed parsing pip git install, using original line {}'.format(req_line))
+
+            # if we have older pip version we have to make sure we replace back the package name with the
+            # git repository link. In new versions this is supported and we get "package @ git+https://..."
+            if not req.pip_new_version:
+                PackageManager.out_of_scope_install_package(req_line, "--no-deps")
+                # noinspection PyBroadException
+                try:
+                    freeze_post = PackageManager.out_of_scope_freeze() or ''
+                    package_name = list(set(freeze_post['pip']) - set(freeze_base['pip']))
+                    if package_name and package_name[0] not in self.post_install_req_lookup:
+                        self.post_install_req_lookup[package_name[0]] = req.req.line
+                except Exception:
+                    pass
+
+            # no need to force reinstall, pip will always rebuilt if the package comes from git
+            # and make sure the required packages are installed (if they are not it will install them)
+            if not PackageManager.out_of_scope_install_package(req_line):
+                raise ValueError("Failed installing GIT/HTTPs package \'{}\'".format(req_line))
+
+    def replace(self, req):
+        """
+        Replace a requirement
+        :raises: ValueError if version is pre-release
+        """
+        # Store in post req install, and return nothing
+        self.post_install_req.append(req)
+        # mark skip package, we will install it in post install hook
+        return Text('')
+
+    def replace_back(self, list_of_requirements):
+        if not list_of_requirements:
+            return list_of_requirements
+
+        for k in list_of_requirements:
+            # k is either pip/conda
+            if k not in ('pip', 'conda'):
+                continue
+                
+            original_requirements = list_of_requirements[k]
+            list_of_requirements[k] = [r for r in original_requirements
+                                       if r not in self.post_install_req_lookup]
+            list_of_requirements[k] += [self.post_install_req_lookup.get(r, '')
+                                        for r in self.post_install_req_lookup.keys() if r in original_requirements]
+        return list_of_requirements

clearml_agent/helper/package/pip_api/system.py

@@ -2,10 +2,10 @@ import sys
 from itertools import chain
 from typing import Text, Optional
 
-from trains_agent.definitions import PIP_EXTRA_INDICES, PROGRAM_NAME
-from trains_agent.helper.package.base import PackageManager
-from trains_agent.helper.process import Argv, DEVNULL
-from trains_agent.session import Session
+from clearml_agent.definitions import PIP_EXTRA_INDICES, PROGRAM_NAME
+from clearml_agent.helper.package.base import PackageManager
+from clearml_agent.helper.process import Argv, DEVNULL
+from clearml_agent.session import Session
 
 
 class SystemPip(PackageManager):

clearml_agent/helper/package/pip_api/venv.py

@@ -1,16 +1,18 @@
+from typing import Any
+
 from pathlib2 import Path
 
-from trains_agent.helper.base import select_for_platform, rm_tree
-from trains_agent.helper.package.base import PackageManager
-from trains_agent.helper.process import Argv, PathLike
-from trains_agent.session import Session
+from clearml_agent.helper.base import select_for_platform, rm_tree, ExecutionInfo
+from clearml_agent.helper.package.base import PackageManager
+from clearml_agent.helper.process import Argv, PathLike
+from clearml_agent.session import Session
 from ..pip_api.system import SystemPip
 from ..requirements import RequirementsManager
 
 
 class VirtualenvPip(SystemPip, PackageManager):
-    def __init__(self, session, python, requirements_manager, path, interpreter=None):
-        # type: (Session, float, RequirementsManager, PathLike, PathLike) -> ()
+    def __init__(self, session, python, requirements_manager, path, interpreter=None, execution_info=None, **kwargs):
+        # type: (Session, float, RequirementsManager, PathLike, PathLike, ExecutionInfo, Any) -> ()
         """
         Program interface to virtualenv pip.
         Must be given either path to virtualenv or source command.
@@ -37,7 +39,7 @@ class VirtualenvPip(SystemPip, PackageManager):
         if isinstance(requirements, dict) and requirements.get("pip"):
             requirements["pip"] = self.requirements_manager.replace(requirements["pip"])
         super(VirtualenvPip, self).load_requirements(requirements)
-        self.requirements_manager.post_install()
+        self.requirements_manager.post_install(self.session)
 
     def create_flags(self):
         """

clearml_agent/helper/package/poetry_api.py

@@ -5,8 +5,8 @@ import attr
 import sys
 import os
 from pathlib2 import Path
-from trains_agent.helper.process import Argv, DEVNULL, check_if_command_exists
-from trains_agent.session import Session, POETRY
+from clearml_agent.helper.process import Argv, DEVNULL, check_if_command_exists
+from clearml_agent.session import Session, POETRY
 
 
 def prop_guard(prop, log_prop=None):

clearml_agent/helper/package/post_req.py

@@ -0,0 +1,48 @@
+from typing import Text
+
+from .base import PackageManager
+from .requirements import SimpleSubstitution
+
+
+class PostRequirement(SimpleSubstitution):
+
+    name = ("horovod", )
+    optional_package_names = tuple()
+
+    def __init__(self, *args, **kwargs):
+        super(PostRequirement, self).__init__(*args, **kwargs)
+        self.post_install_req = []
+        # check if we need to replace the packages:
+        post_packages = self.config.get('agent.package_manager.post_packages', None)
+        if post_packages:
+            self.__class__.name = post_packages
+        post_optional_packages = self.config.get('agent.package_manager.post_optional_packages', None)
+        if post_optional_packages:
+            self.__class__.optional_package_names = post_optional_packages
+
+    def match(self, req):
+        # match both horovod
+        return req.name and (req.name.lower() in self.name or req.name.lower() in self.optional_package_names)
+
+    def post_install(self, session):
+        for req in self.post_install_req:
+            if req.name in self.optional_package_names:
+                # noinspection PyBroadException
+                try:
+                    PackageManager.out_of_scope_install_package(req.tostr(markers=False))
+                except Exception:
+                    pass
+            else:
+                PackageManager.out_of_scope_install_package(req.tostr(markers=False))
+
+        self.post_install_req = []
+
+    def replace(self, req):
+        """
+        Replace a requirement
+        :raises: ValueError if version is pre-release
+        """
+        # Store in post req install, and return nothing
+        self.post_install_req.append(req)
+        # mark skip package, we will install it in post install hook
+        return Text('')

clearml_agent/helper/package/priority_req.py

@@ -0,0 +1,75 @@
+from typing import Text
+
+from .base import PackageManager
+from .requirements import SimpleSubstitution
+
+
+class PriorityPackageRequirement(SimpleSubstitution):
+
+    name = ("cython", "numpy", "setuptools", )
+    optional_package_names = tuple()
+
+    def __init__(self, *args, **kwargs):
+        super(PriorityPackageRequirement, self).__init__(*args, **kwargs)
+        # check if we need to replace the packages:
+        priority_packages = self.config.get('agent.package_manager.priority_packages', None)
+        if priority_packages:
+            self.__class__.name = priority_packages
+        priority_optional_packages = self.config.get('agent.package_manager.priority_optional_packages', None)
+        if priority_optional_packages:
+            self.__class__.optional_package_names = priority_optional_packages
+
+    def match(self, req):
+        # match both Cython & cython
+        return req.name and (req.name.lower() in self.name or req.name.lower() in self.optional_package_names)
+
+    def replace(self, req):
+        """
+        Replace a requirement
+        :raises: ValueError if version is pre-release
+        """
+        if req.name in self.optional_package_names:
+            # noinspection PyBroadException
+            try:
+                if PackageManager.out_of_scope_install_package(str(req)):
+                    return Text(req)
+            except Exception:
+                pass
+            return Text('')
+        PackageManager.out_of_scope_install_package(str(req))
+        return Text(req)
+
+
+class PackageCollectorRequirement(SimpleSubstitution):
+    """
+    This RequirementSubstitution class will allow you to have multiple instances of the same
+    package, it will output the last one (by order) to be actually used.
+    """
+    name = tuple()
+
+    def __init__(self, session, collect_package):
+        super(PackageCollectorRequirement, self).__init__(session)
+        self._collect_packages = collect_package or tuple()
+        self._last_req = None
+
+    def match(self, req):
+        # match package names
+        return req.name and req.name.lower() in self._collect_packages
+
+    def replace(self, req):
+        """
+        Replace a requirement
+        :raises: ValueError if version is pre-release
+        """
+        self._last_req = req.clone()
+        return ''
+
+    def post_scan_add_req(self):
+        """
+        Allows the RequirementSubstitution to add an extra line/requirements after
+        the initial requirements scan is completed.
+        Called only once per requirements.txt object
+        """
+        last_req = self._last_req
+        self._last_req = None
+        return last_req

clearml_agent/helper/package/pytorch.py

@@ -82,6 +82,8 @@ class SimplePytorchRequirement(SimpleSubstitution):
         92: 'https://download.pytorch.org/whl/cu92/torch_stable.html',
         100: 'https://download.pytorch.org/whl/cu100/torch_stable.html',
         101: 'https://download.pytorch.org/whl/cu101/torch_stable.html',
+        102: 'https://download.pytorch.org/whl/cu102/torch_stable.html',
+        110: 'https://download.pytorch.org/whl/cu110/torch_stable.html',
     }
 
     def __init__(self, *args, **kwargs):
@@ -117,20 +119,24 @@ class SimplePytorchRequirement(SimpleSubstitution):
 
     @classmethod
     def get_torch_page(cls, cuda_version, nightly=False):
+        # noinspection PyBroadException
         try:
             cuda = int(cuda_version)
-        except:
+        except Exception:
             cuda = 0
 
         if nightly:
-            # then try the nightly builds, it might be there...
-            torch_url = cls.nightly_page_lookup_template.format(cuda)
-            try:
-                if requests.get(torch_url, timeout=10).ok:
-                    cls.torch_page_lookup[cuda] = torch_url
-                    return cls.torch_page_lookup[cuda], cuda
-            except Exception:
-                pass
+            for c in range(cuda, max(-1, cuda-15), -1):
+                # then try the nightly builds, it might be there...
+                torch_url = cls.nightly_page_lookup_template.format(c)
+                # noinspection PyBroadException
+                try:
+                    if requests.get(torch_url, timeout=10).ok:
+                        print('Torch nightly CUDA {} download page found'.format(c))
+                        cls.torch_page_lookup[c] = torch_url
+                        return cls.torch_page_lookup[c], c
+                except Exception:
+                    pass
             return
 
         # first check if key is valid
@@ -138,13 +144,16 @@ class SimplePytorchRequirement(SimpleSubstitution):
             return cls.torch_page_lookup[cuda], cuda
 
         # then try a new cuda version page
-        torch_url = cls.page_lookup_template.format(cuda)
-        try:
-            if requests.get(torch_url, timeout=10).ok:
-                cls.torch_page_lookup[cuda] = torch_url
-                return cls.torch_page_lookup[cuda], cuda
-        except Exception:
-            pass
+        for c in range(cuda, max(-1, cuda-15), -1):
+            torch_url = cls.page_lookup_template.format(c)
+            # noinspection PyBroadException
+            try:
+                if requests.get(torch_url, timeout=10).ok:
+                    print('Torch CUDA {} download page found'.format(c))
+                    cls.torch_page_lookup[c] = torch_url
+                    return cls.torch_page_lookup[c], c
+            except Exception:
+                pass
 
         keys = sorted(cls.torch_page_lookup.keys(), reverse=True)
         for k in keys:
@@ -157,7 +166,7 @@ class SimplePytorchRequirement(SimpleSubstitution):
 class PytorchRequirement(SimpleSubstitution):
 
     name = "torch"
-    packages = ("torch", "torchvision", "torchaudio")
+    packages = ("torch", "torchvision", "torchaudio", "torchcsprng", "torchtext")
 
     def __init__(self, *args, **kwargs):
         os_name = kwargs.pop("os_override", None)
@@ -235,6 +244,7 @@ class PytorchRequirement(SimpleSubstitution):
         py_ver = self.python_major_minor_str.replace('.', '')
         url = None
         last_v = None
+        closest_v = None
         # search for our package
         for l in links_parser.links:
             parts = l.split('/')[-1].split('-')
@@ -244,73 +254,94 @@ class PytorchRequirement(SimpleSubstitution):
                 continue
             # version (ignore +cpu +cu92 etc. + is %2B in the file link)
             # version ignore .postX suffix (treat as regular version)
+            # noinspection PyBroadException
             try:
                 v = str(parts[1].split('%')[0].split('+')[0])
             except Exception:
                 continue
+            if len(parts) < 3 or not parts[2].endswith(py_ver):
+                continue
+            if len(parts) < 5 or platform_wheel not in parts[4]:
+                continue
+            # update the closest matched version (from above)
+            if not closest_v:
+                closest_v = v
+            elif SimpleVersion.compare_versions(
+                    version_a=closest_v, op='>=', version_b=v, num_parts=3) and \
+                    SimpleVersion.compare_versions(
+                        version_a=v, op='>=', version_b=req.specs[0][1], num_parts=3):
+                closest_v = v
+            # check if this an actual match
             if not req.compare_version(v) or \
                     (last_v and SimpleVersion.compare_versions(last_v, '>', v, ignore_sub_versions=False)):
                 continue
-            if not parts[2].endswith(py_ver):
-                continue
-            if platform_wheel not in parts[4]:
-                continue
+
             url = '/'.join(torch_url.split('/')[:-1] + l.split('/'))
             last_v = v
             # if we found an exact match, use it
+            # noinspection PyBroadException
             try:
                 if req.specs[0][0] == '==' and \
                         SimpleVersion.compare_versions(req.specs[0][1], '==', v, ignore_sub_versions=False):
                     break
-            except:
+            except Exception:
                 pass
 
-        return url
+        return url, last_v or closest_v
 
     def get_url_for_platform(self, req):
         # check if package is already installed with system packages
+        # noinspection PyBroadException
         try:
-            if self.config.get("agent.package_manager.system_site_packages"):
+            if self.config.get("agent.package_manager.system_site_packages", None):
                 from pip._internal.commands.show import search_packages_info
                 installed_torch = list(search_packages_info([req.name]))
-                # notice the comparision order, the first part will make sure we have a valid installed package
-                if installed_torch[0]['version'] and req.compare_version(installed_torch[0]['version']):
+                # notice the comparison order, the first part will make sure we have a valid installed package
+                if installed_torch and installed_torch[0]['version'] and \
+                        req.compare_version(installed_torch[0]['version']):
                     print('PyTorch: requested "{}" version {}, using pre-installed version {}'.format(
                         req.name, req.specs[0] if req.specs else 'unspecified', installed_torch[0]['version']))
                     # package already installed, do nothing
-                    return str(req), True
-        except:
+                    req.specs = [('==', str(installed_torch[0]['version']))]
+                    return '{} {} {}'.format(req.name, req.specs[0][0], req.specs[0][1]), True
+        except Exception:
             pass
 
         # make sure we have a specific version to retrieve
         if not req.specs:
             req.specs = [('>', '0')]
 
+        # noinspection PyBroadException
         try:
             req.specs[0] = (req.specs[0][0], req.specs[0][1].split('+')[0])
-        except:
+        except Exception:
             pass
         op, version = req.specs[0]
         # assert op == "=="
 
         torch_url, torch_url_key = SimplePytorchRequirement.get_torch_page(self.cuda_version)
-        url = self._get_link_from_torch_page(req, torch_url)
-        if not url and self.config.get("agent.package_manager.torch_nightly"):
+        url, closest_matched_version = self._get_link_from_torch_page(req, torch_url)
+        if not url and self.config.get("agent.package_manager.torch_nightly", None):
             torch_url, torch_url_key = SimplePytorchRequirement.get_torch_page(self.cuda_version, nightly=True)
-            url = self._get_link_from_torch_page(req, torch_url)
+            url, closest_matched_version = self._get_link_from_torch_page(req, torch_url)
         # try one more time, with a lower cuda version (never fallback to CPU):
         while not url and torch_url_key > 0:
             previous_cuda_key = torch_url_key
+            print('Warning, could not locate PyTorch {} matching CUDA version {}, best candidate {}\n'.format(
+                    req, previous_cuda_key, closest_matched_version))
+            url, closest_matched_version = self._get_link_from_torch_page(req, torch_url)
+            if url:
+                break
             torch_url, torch_url_key = SimplePytorchRequirement.get_torch_page(int(torch_url_key)-1)
             # never fallback to CPU
             if torch_url_key < 1:
-                print('Warning! Could not locate PyTorch version {} matching CUDA version {}'.format(
-                    req, previous_cuda_key))
-                raise ValueError('Could not locate PyTorch version {} matching CUDA version {}'.format(
-                    req, self.cuda_version))
-            print('Warning! Could not locate PyTorch version {} matching CUDA version {}, trying CUDA version {}'.format(
-                req, previous_cuda_key, torch_url_key))
-            url = self._get_link_from_torch_page(req, torch_url)
+                print(
+                    'Error! Could not locate PyTorch version {} matching CUDA version {}'.format(
+                        req, previous_cuda_key))
+                raise ValueError(
+                    'Could not locate PyTorch version {} matching CUDA version {}'.format(req, self.cuda_version))
+            else:
+                print('Trying PyTorch CUDA version {} support'.format(torch_url_key))
 
         if not url:
             url = PytorchWheel(
@@ -322,6 +353,8 @@ class PytorchRequirement(SimpleSubstitution):
         if url:
             # normalize url (sometimes we will get ../ which we should not...
             url = '/'.join(url.split('/')[:3]) + urllib.parse.quote(str(furl(url).path.normalize()))
+            # print found
+            print('Found PyTorch version {} matching CUDA version {}'.format(req, torch_url_key))
 
         self.log.debug("checking url: %s", url)
         return url, requests.head(url, timeout=10).ok
@@ -457,7 +490,13 @@ class PytorchRequirement(SimpleSubstitution):
                         if req.req.name == parts[0]:
                             # support for pip >= 20.1
                             if '@' in line:
-                                lines[i] = '{} # {}'.format(str(req), str(new_req))
+                                # skip if we have nothing to add
+                                if str(req).strip() != str(new_req).strip():
+                                    # if this is local file and use the version detection
+                                    if req.local_file:
+                                        lines[i] = '{}'.format(str(new_req))
+                                    else:
+                                        lines[i] = '{} # {}'.format(str(req), str(new_req))
                             else:
                                 lines[i] = '{} # {}'.format(line, str(new_req))
                             break

clearml_agent/helper/package/requirements.py

@@ -4,7 +4,7 @@ import operator
 import os
 import re
 from abc import ABCMeta, abstractmethod
-from copy import deepcopy
+from copy import deepcopy, copy
 from itertools import chain, starmap
 from operator import itemgetter
 from os import path
@@ -12,15 +12,15 @@ from typing import Text, List, Type, Optional, Tuple, Dict
 
 from pathlib2 import Path
 from pyhocon import ConfigTree
-from requirements import parse
-# noinspection PyPackageRequirements
-from requirements.requirement import Requirement
 
 import six
-from trains_agent.definitions import PIP_EXTRA_INDICES
-from trains_agent.helper.base import warning, is_conda, which, join_lines, is_windows_platform
-from trains_agent.helper.process import Argv, PathLike
-from trains_agent.session import Session, normalize_cuda_version
+from clearml_agent.definitions import PIP_EXTRA_INDICES
+from clearml_agent.helper.base import warning, is_conda, which, join_lines, is_windows_platform
+from clearml_agent.helper.process import Argv, PathLike
+from clearml_agent.session import Session, normalize_cuda_version
+from clearml_agent.external.requirements_parser import parse
+from clearml_agent.external.requirements_parser.requirement import Requirement
+
 from .translator import RequirementsTranslator
 
 
@@ -35,6 +35,10 @@ class FatalSpecsResolutionError(Exception):
 @six.python_2_unicode_compatible
 class MarkerRequirement(object):
 
+    # if True pip version above 20.x and with support for "package @ scheme://link"
+    # default is True
+    pip_new_version = True
+
     def __init__(self, req):  # type: (Requirement) -> None
         self.req = req
 
@@ -54,7 +58,21 @@ class MarkerRequirement(object):
 
             if self.specifier:
                 parts.append(self.format_specs())
-
+        elif self.vcs:
+            # leave the line as is, let pip handle it
+            if self.line:
+                return self.line
+            else:
+                # let's build the line manually
+                parts = [
+                    self.uri,
+                    '@{}'.format(self.revision) if self.revision else '',
+                    '#subdirectory={}'.format(self.subdirectory) if self.subdirectory else ''
+                ]
+        elif self.pip_new_version and self.uri and self.name and self.line and self.local_file:
+            # package @ file:///example.com/somewheel.whl
+            # leave the line as is, let pip handle it
+            return self.line
         else:
             parts = [self.uri]
 
@@ -63,6 +81,9 @@ class MarkerRequirement(object):
 
         return ''.join(parts)
 
+    def clone(self):
+        return MarkerRequirement(copy(self.req))
+
     __str__ = tostr
 
     def __repr__(self):
@@ -128,7 +149,8 @@ class MarkerRequirement(object):
         version = self.specs[0][1]
         op = (op or self.specs[0][0]).strip()
 
-        return SimpleVersion.compare_versions(requested_version, op, version)
+        return SimpleVersion.compare_versions(
+            version_a=requested_version, op=op, version_b=version, num_parts=num_parts)
 
 
 class SimpleVersion:
@@ -167,7 +189,7 @@ class SimpleVersion:
     _regex = re.compile(r"^\s*" + VERSION_PATTERN + r"\s*$", re.VERBOSE | re.IGNORECASE)
 
     @classmethod
-    def compare_versions(cls, version_a, op, version_b, ignore_sub_versions=True):
+    def compare_versions(cls, version_a, op, version_b, ignore_sub_versions=True, num_parts=3):
         """
         Compare two versions based on the op operator
         returns bool(version_a op version_b)
@@ -178,12 +200,12 @@ class SimpleVersion:
         :param str version_b:
         :param bool ignore_sub_versions: if true compare only major.minor.patch
             (ignore a/b/rc/post/dev in the comparison)
+        :param int num_parts: number of parts to compare, split by . (dot)
         :return bool: version_a op version_b
         """
 
         if not version_b:
             return True
-        num_parts = 3
 
         if op == '~=':
             num_parts = max(num_parts, 2)
@@ -316,7 +338,15 @@ class RequirementSubstitution(object):
         """
         pass
 
-    def post_install(self):
+    def post_scan_add_req(self):  # type: () -> Optional[MarkerRequirement]
+        """
+        Allows the RequirementSubstitution to add an extra line/requirements after
+        the initial requirements scan is completed.
+        Called only once per requirements.txt object
+        """
+        return None
+
+    def post_install(self, session):
         pass
 
     @classmethod
@@ -470,16 +500,28 @@ class RequirementsManager(object):
         )
         if not conda:
             result = map(self.translator.translate, result)
+
+        result = list(result)
+        # add post scan add requirements call back
+        for h in self.handlers:
+            req = h.post_scan_add_req()
+            if req:
+                result.append(req.tostr())
+
         return join_lines(result)
 
-    def post_install(self):
+    def post_install(self, session):
         for h in self.handlers:
             try:
-                h.post_install()
+                h.post_install(session)
             except Exception as ex:
                 print('RequirementsManager handler {} raised exception: {}'.format(h, ex))
+                raise
 
     def replace_back(self, requirements):
+        if self.translator:
+            requirements = self.translator.replace_back(requirements)
+
         for h in self.handlers:
             try:
                 requirements = h.replace_back(requirements)