Fix docker force pull in k8s glue _kubectl_apply()

Fix k8s glue not passing required extra_docker_bash_script to string format

README.md

@@ -1,80 +1,106 @@
-# Allegro Trains Agent
-## Deep Learning DevOps For Everyone - Now supporting all platforms (Linux, macOS, and Windows)
+<div align="center">
 
-"All the Deep-Learning DevOps your research needs, and then some... Because ain't nobody got time for that"
+<img src="https://github.com/allegroai/clearml-agent/blob/master/docs/clearml_agent_logo.png?raw=true" width="250px">
 
-[![GitHub license](https://img.shields.io/github/license/allegroai/trains-agent.svg)](https://img.shields.io/github/license/allegroai/trains-agent.svg)
-[![PyPI pyversions](https://img.shields.io/pypi/pyversions/trains-agent.svg)](https://img.shields.io/pypi/pyversions/trains-agent.svg)
-[![PyPI version shields.io](https://img.shields.io/pypi/v/trains-agent.svg)](https://img.shields.io/pypi/v/trains-agent.svg)
-[![PyPI status](https://img.shields.io/pypi/status/trains-agent.svg)](https://pypi.python.org/pypi/trains-agent/)
+**ClearML Agent - ML-Ops made easy  
+ML-Ops scheduler & orchestration solution supporting Linux, macOS and Windows**
 
-### Help improve Trains by filling our 2-min [user survey](https://allegro.ai/lp/trains-user-survey/)
+[![GitHub license](https://img.shields.io/github/license/allegroai/clearml-agent.svg)](https://img.shields.io/github/license/allegroai/clearml-agent.svg)
+[![PyPI pyversions](https://img.shields.io/pypi/pyversions/clearml-agent.svg)](https://img.shields.io/pypi/pyversions/clearml-agent.svg)
+[![PyPI version shields.io](https://img.shields.io/pypi/v/clearml-agent.svg)](https://img.shields.io/pypi/v/clearml-agent.svg)
 
-**Trains Agent is an AI experiment cluster solution.**
+</div>
 
-It is a zero configuration fire-and-forget execution agent, which combined with trains-server provides a full AI cluster solution.
+---
 
-**Full AutoML in 5 steps** 
-1. Install the [Trains Server](https://github.com/allegroai/trains-agent) (or use our [open server](https://demoapp.trains.allegro.ai))
-2. `pip install trains-agent` ([install](#installing-the-trains-agent) the Trains Agent on any GPU machine: on-premises / cloud / ...)
-3. Add [Trains](https://github.com/allegroai/trains) to your code with just 2 lines & run it once (on your machine / laptop)
-4. Change the [parameters](#using-the-trains-agent) in the UI & schedule for [execution](#using-the-trains-agent) (or automate with an [AutoML pipeline](#automl-and-orchestration-pipelines-))
+### ClearML-Agent
+#### *Formerly known as Trains Agent*
+
+
+* Run jobs (experiments) on any local or cloud based resource
+* Implement optimized resource utilization policies
+* Deploy execution environments with either virtualenv or fully docker containerized with zero effort
+* Launch-and-Forget service containers
+* [Cloud autoscaling](https://allegro.ai/clearml/docs/docs/examples/services/aws_autoscaler/aws_autoscaler.html)
+* [Customizable cleanup](https://allegro.ai/clearml/docs/docs/examples/services/cleanup/cleanup_service.html)
+* Advanced [pipeline building and execution](https://allegro.ai/clearml/docs/docs/examples/frameworks/pytorch/notebooks/table/tabular_training_pipeline.html)
+
+It is a zero configuration fire-and-forget execution agent, providing a full ML/DL cluster solution.
+
+**Full Automation in 5 steps**
+1. ClearML Server [self-hosted](https://github.com/allegroai/clearml-server) or [free tier hosting](https://app.community.clear.ml)
+2. `pip install clearml-agent` ([install](#installing-the-clearml-agent) the ClearML Agent on any GPU machine: on-premises / cloud / ...)
+3. Create a [job](https://github.com/allegroai/clearml/docs/clearml-task.md) or Add [ClearML](https://github.com/allegroai/clearml) to your code with just 2 lines
+4. Change the [parameters](#using-the-clearml-agent) in the UI & schedule for [execution](#using-the-clearml-agent) (or automate with an [AutoML pipeline](#automl-and-orchestration-pipelines-))
 5. :chart_with_downwards_trend: :chart_with_upwards_trend: :eyes:  :beer:
 
+"All the Deep/Machine-Learning DevOps your research needs, and then some... Because ain't nobody got time for that"
 
-**Using the Trains Agent, you can now set up a dynamic cluster with \*epsilon DevOps**
+**Try ClearML now** [Self Hosted](https://github.com/allegroai/clearml-server) or [Free tier Hosting](https://app.community.clear.ml)
+<a href="https://app.community.clear.ml"><img src="https://github.com/allegroai/clearml/blob/master/docs/webapp_screenshots.gif?raw=true" width="100%"></a>
 
-*epsilon - Because we are scientists :triangular_ruler: and nothing is really zero work
-
-(Experience Trains live at [https://demoapp.trains.allegro.ai](https://demoapp.trains.allegro.ai))
-<a href="https://demoapp.trains.allegro.ai"><img src="https://raw.githubusercontent.com/allegroai/trains-agent/9f1e86c1ca45c984ee13edc9353c7b10c55d7257/docs/screenshots.gif" width="100%"></a>
-
-## Simple, Flexible Experiment Orchestration
-**The Trains Agent was built to address the DL/ML R&D DevOps needs:**
+### Simple, Flexible Experiment Orchestration
+**The ClearML Agent was built to address the DL/ML R&D DevOps needs:**
 
 * Easily add & remove machines from the cluster
 * Reuse machines without the need for any dedicated containers or images
 * **Combine GPU resources across any cloud and on-prem**
-* **No need for yaml/json/template configuration of any kind**
+* **No need for yaml / json / template configuration of any kind**
 * **User friendly UI**
 * Manageable resource allocation that can be used by researchers and engineers
 * Flexible and controllable scheduler with priority support
-* Automatic instance spinning in the cloud **(coming soon)**
+* Automatic instance spinning in the cloud
+
+**Using the ClearML Agent, you can now set up a dynamic cluster with \*epsilon DevOps**
+
+*epsilon - Because we are :triangular_ruler: and nothing is really zero work
 
 
-## But ... K8S?
-We think Kubernetes is awesome.  
-Combined with KubeFlow it is a robust solution for production-grade DevOps.  
-We've observed, however, that it can be a bit of an overkill as an R&D DL/ML solution.
-If you are considering K8S for your research, also consider that you will soon be managing **hundreds** of containers...
+### Kubernetes Integration (Optional)
+We think Kubernetes is awesome, but it should be a choice.
+We designed `clearml-agent` so you can run bare-metal or inside a pod with any mix that fits your environment.
+#### Benefits of integrating existing K8s with ClearML-Agent 
+- ClearML-Agent adds the missing scheduling capabilities to K8s
+- Allowing for more flexible automation from code
+- A programmatic interface for easier learning curve (and debugging)
+- Seamless integration with ML/DL experiment manager
+- Web UI for customization, scheduling & prioritization of jobs 
 
-In our experience, handling and building the environments, having to package every experiment in a docker, managing those hundreds (or more) containers and building pipelines on top of it all, is very complicated (also, it’s usually out of scope for the research team, and overwhelming even for the DevOps team).
+**Two K8s integration flavours** 
+- Spin ClearML-Agent as a long-lasting service pod
+    - use [clearml-agent](https://hub.docker.com/r/allegroai/clearml-agent) docker image
+    - map docker socket into the pod (soon replaced by [podman](https://github.com/containers/podman))
+    - allow the clearml-agent to manage sibling dockers
+    - benefits: full use of the ClearML scheduling, no need to worry about wrong container images / lost pods etc.
+    - downside: Sibling containers
+- Kubernetes Glue, map ClearML jobs directly to K8s jobs
+    - Run the [clearml-k8s glue](https://github.com/allegroai/clearml-agent/blob/master/examples/k8s_glue_example.py) on a K8s cpu node
+    - The clearml-k8s glue pulls jobs from the ClearML job execution queue and prepares a K8s job (based on provided yaml template)
+    - Inside the pod itself the clearml-agent will install the job (experiment) environment and spin and monitor the experiment's process
+    - benefits: Kubernetes full view of all running jobs in the system
+    - downside: No real scheduling (k8s scheduler), no docker image verification (post-mortem only) 
 
-We feel there has to be a better way, that can be just as powerful for R&D and at the same time allow integration with K8S **when the need arises**.  
-(If you already have a K8S cluster for AI, detailed instructions on how to integrate Trains into your K8S cluster are [here](https://github.com/allegroai/trains-server-k8s/tree/master/trains-server-chart) with included [helm chart](https://github.com/allegroai/trains-server-helm))
-
-
-## Using the Trains Agent
+### Using the ClearML Agent
 **Full scale HPC with a click of a button**
 
-The Trains Agent is a job scheduler that listens on job queue(s), pulls jobs, sets the job environments, executes the job and monitors its progress.
+The ClearML Agent is a job scheduler that listens on job queue(s), pulls jobs, sets the job environments, executes the job and monitors its progress.
 
-Any 'Draft' experiment can be scheduled for execution by a Trains agent.
+Any 'Draft' experiment can be scheduled for execution by a ClearML agent.
 
 A previously run experiment can be put into 'Draft' state by either of two methods:
 * Using the **'Reset'** action from the experiment right-click context menu in the
-  Trains UI - This will clear any results and artifacts the previous run had created.
+  ClearML UI - This will clear any results and artifacts the previous run had created.
 * Using the **'Clone'** action from the experiment right-click context menu in the
-  Trains UI - This will create a new 'Draft' experiment with the same configuration as the original experiment.
+  ClearML UI - This will create a new 'Draft' experiment with the same configuration as the original experiment.
 
 An experiment is scheduled for execution using the **'Enqueue'** action from the experiment
- right-click context menu in the Trains UI and selecting the execution queue.
+ right-click context menu in the ClearML UI and selecting the execution queue.
 
 See [creating an experiment and enqueuing it for execution](#from-scratch).
 
-Once an experiment is enqueued, it will be picked up and executed by a Trains agent monitoring this queue.
+Once an experiment is enqueued, it will be picked up and executed by a ClearML agent monitoring this queue.
 
-The Trains UI Workers & Queues page provides ongoing execution information:
+The ClearML UI Workers & Queues page provides ongoing execution information:
   - Workers Tab: Monitor you cluster
     - Review available resources
     - Monitor machines statistics (CPU / GPU / Disk / Network)
@@ -83,162 +109,129 @@ The Trains UI Workers & Queues page provides ongoing execution information:
     - Cancel or abort job execution
     - Move jobs between execution queues
 
-### What The Trains Agent Actually Does
-The Trains Agent executes experiments using the following process:
+#### What The ClearML Agent Actually Does
+The ClearML Agent executes experiments using the following process:
   - Create a new virtual environment (or launch the selected docker image)
   - Clone the code into the virtual-environment (or inside the docker)
   - Install python packages based on the package requirements listed for the experiment
-    - Special note for PyTorch: The Trains Agent will automatically select the
+    - Special note for PyTorch: The ClearML Agent will automatically select the
       torch packages based on the CUDA_VERSION environment variable of the machine
   - Execute the code, while monitoring the process
-  - Log all stdout/stderr in the Trains UI, including the cloning and installation process, for easy debugging
-  - Monitor the execution and allow you to manually abort the job using the Trains UI (or, in the unfortunate case of a code crash, catch the error and signal the experiment has failed)
+  - Log all stdout/stderr in the ClearML UI, including the cloning and installation process, for easy debugging
+  - Monitor the execution and allow you to manually abort the job using the ClearML UI (or, in the unfortunate case of a code crash, catch the error and signal the experiment has failed)
 
-### System Design & Flow
-```text
-                                                                              +-----------------+
-                                                                              |  GPU  Machine   |
-Development Machine                                                           |                 |
-+------------------------+                                                    | +-------------+ |
-|    Data Scientist's    |                            +--------------+        | |Trains Agent | |
-|      DL/ML Code        |                            |    WEB UI    |        | |             | |
-|                        |                            |              |        | | +---------+ | |
-|                        |                            |              |        | | |  DL/ML  | | |
-|                        |                            +--------------+        | | |  Code   | | |
-|                        |       User Clones Exp #1  / . . . . . . . /        | | |         | | |
-| +-------------------+  |           into Exp #2    / . . . . . . . /         | | +---------+ | |
-| |      Trains       |  |         +---------------/-_____________-/          | |             | |
-| +---------+---------+  |         |                                          | |      ^      | |
-+-----------|------------+         |                                          | +------|------+ |
-            |                      |                                          +--------|--------+
- Auto-Magically                    |                                                   |
- Creates Exp #1                    |                                      The Trains Agent
-             \          User Change Hyper-Parameters                      Pulls Exp #2, setup the
-             |                     |                                      environment & clone code.
-             |                     |                                      Start execution with the
-+------------|------------+        |            +--------------------+    new set of Hyper-Parameters.
-|  +---------v---------+  |        |            |   Trains Server    |                 |
-|  | Experiment #1     |  |        |            |                    |                 |
-|  +-------------------+  |        |            |  Execution Queue   |                 |
-|            ||           |        |            |                    |                 |
-|  +-------------------+<----------+            |                    |                 |
-|  |                   |  |                     |                    |                 |
-|  | Experiment #2     |  |                     |                    |                 |
-|  +-------------------<------------\           |                    |                 |
-|                         |          ------------->---------------+  |                 |
-|                         |  User Send Exp #2   | |Execute Exp #2 +--------------------+
-|                         |  For Execution      | +---------------+  |
-|     Trains Server       |                     |                    |
-+-------------------------+                     +--------------------+
-```
+#### System Design & Flow
 
-### Installing the Trains Agent
+<img src="https://allegro.ai/clearml/docs/_images/ClearML_Architecture.png" width="100%" alt="clearml-architecture">
+
+
+#### Installing the ClearML Agent
 
 ```bash
-pip install trains-agent
+pip install clearml-agent
 ```
 
-### Trains Agent Usage Examples
+#### ClearML Agent Usage Examples
 
 Full Interface and capabilities are available with
 ```bash
-trains-agent --help
-trains-agent daemon --help
+clearml-agent --help
+clearml-agent daemon --help
 ```
 
-### Configuring the Trains Agent
+#### Configuring the ClearML Agent
 
 ```bash
-trains-agent init
+clearml-agent init
 ```
 
-Note: The Trains Agent uses a cache folder to cache pip packages, apt packages and cloned repositories. The default Trains Agent cache folder is `~/.trains`
+Note: The ClearML Agent uses a cache folder to cache pip packages, apt packages and cloned repositories. The default ClearML Agent cache folder is `~/.clearml`
 
-See full details in your configuration file at `~/trains.conf`
+See full details in your configuration file at `~/clearml.conf`
 
-Note: The **Trains agent** extends the **Trains** configuration file `~/trains.conf`
-They are designed to share the same configuration file, see example [here](docs/trains.conf)
+Note: The **ClearML agent** extends the **ClearML** configuration file `~/clearml.conf`
+They are designed to share the same configuration file, see example [here](docs/clearml.conf)
 
-### Running the Trains Agent
+#### Running the ClearML Agent
 
-For debug and experimentation, start the Trains agent in `foreground` mode, where all the output is printed to screen
+For debug and experimentation, start the ClearML agent in `foreground` mode, where all the output is printed to screen
 ```bash
-trains-agent daemon --queue default --foreground
+clearml-agent daemon --queue default --foreground
 ```
 
 For actual service mode, all the stdout will be stored automatically into a temporary file (no need to pipe)
-Notice: with `--detached` flag, the *trains-agent* will be running in the background
+Notice: with `--detached` flag, the *clearml-agent* will be running in the background
 ```bash
-trains-agent daemon --detached --queue default
+clearml-agent daemon --detached --queue default
 ```
 
 GPU allocation is controlled via the standard OS environment `NVIDIA_VISIBLE_DEVICES` or `--gpus` flag (or disabled with `--cpu-only`).
 
-If no flag is set, and `NVIDIA_VISIBLE_DEVICES` variable doesn't exist, all GPU's will be allocated for the `trains-agent` <br>
-If `--cpu-only` flag is set, or `NVIDIA_VISIBLE_DEVICES` is an empty string (""), no gpu will be allocated for the `trains-agent`
+If no flag is set, and `NVIDIA_VISIBLE_DEVICES` variable doesn't exist, all GPU's will be allocated for the `clearml-agent` <br>
+If `--cpu-only` flag is set, or `NVIDIA_VISIBLE_DEVICES` is an empty string (""), no gpu will be allocated for the `clearml-agent`
 
 Example: spin two agents, one per gpu on the same machine:
-Notice: with `--detached` flag, the *trains-agent* will be running in the background
+Notice: with `--detached` flag, the *clearml-agent* will be running in the background
 ```bash
-trains-agent daemon --detached --gpus 0 --queue default
-trains-agent daemon --detached --gpus 1 --queue default
+clearml-agent daemon --detached --gpus 0 --queue default
+clearml-agent daemon --detached --gpus 1 --queue default
 ```
 
 Example: spin two agents, pulling from dedicated `dual_gpu` queue, two gpu's per agent
 ```bash
-trains-agent daemon --detached --gpus 0,1 --queue dual_gpu
-trains-agent daemon --detached --gpus 2,3 --queue dual_gpu
+clearml-agent daemon --detached --gpus 0,1 --queue dual_gpu
+clearml-agent daemon --detached --gpus 2,3 --queue dual_gpu
 ```
 
-#### Starting the Trains Agent in docker mode
+##### Starting the ClearML Agent in docker mode
 
-For debug and experimentation, start the Trains agent in `foreground` mode, where all the output is printed to screen
+For debug and experimentation, start the ClearML agent in `foreground` mode, where all the output is printed to screen
 ```bash
-trains-agent daemon --queue default --docker --foreground
+clearml-agent daemon --queue default --docker --foreground
 ```
 
 For actual service mode, all the stdout will be stored automatically into a file (no need to pipe)
-Notice: with `--detached` flag, the *trains-agent* will be running in the background
+Notice: with `--detached` flag, the *clearml-agent* will be running in the background
 ```bash
-trains-agent daemon --detached --queue default --docker
+clearml-agent daemon --detached --queue default --docker
 ```
 
-Example: spin two agents, one per gpu on the same machine, with default nvidia/cuda docker:
+Example: spin two agents, one per gpu on the same machine, with default nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04 docker:
 ```bash
-trains-agent daemon --detached --gpus 0 --queue default --docker nvidia/cuda
-trains-agent daemon --detached --gpus 1 --queue default --docker nvidia/cuda
+clearml-agent daemon --detached --gpus 0 --queue default --docker nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04
+clearml-agent daemon --detached --gpus 1 --queue default --docker nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04
 ```
 
-Example: spin two agents, pulling from dedicated `dual_gpu` queue, two gpu's per agent, with default nvidia/cuda docker:
+Example: spin two agents, pulling from dedicated `dual_gpu` queue, two gpu's per agent, with default nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04 docker:
 ```bash
-trains-agent daemon --detached --gpus 0,1 --queue dual_gpu --docker nvidia/cuda
-trains-agent daemon --detached --gpus 2,3 --queue dual_gpu --docker nvidia/cuda
+clearml-agent daemon --detached --gpus 0,1 --queue dual_gpu --docker nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04
+clearml-agent daemon --detached --gpus 2,3 --queue dual_gpu --docker nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04
 ```
 
-#### Starting the Trains Agent - Priority Queues
+##### Starting the ClearML Agent - Priority Queues
 
 Priority Queues are also supported, example use case:
 
 High priority queue: `important_jobs`  Low priority queue: `default`
 ```bash
-trains-agent daemon --queue important_jobs default
+clearml-agent daemon --queue important_jobs default
 ```
-The **Trains Agent** will first try to pull jobs from the `important_jobs` queue, only then it will fetch a job from the `default` queue.
+The **ClearML Agent** will first try to pull jobs from the `important_jobs` queue, only then it will fetch a job from the `default` queue.
 
-Adding queues, managing job order within a queue and moving jobs between queues, is available using the Web UI, see example on our [open server](https://demoapp.trains.allegro.ai/workers-and-queues/queues)
+Adding queues, managing job order within a queue and moving jobs between queues, is available using the Web UI, see example on our [free server](https://app.community.clear.ml/workers-and-queues/queues)
 
-#### Stopping the Trains Agent
+##### Stopping the ClearML Agent
 
-To stop a **Trains Agent** running in the background, run the same command line used to start the agent with `--stop` appended.  
+To stop a **ClearML Agent** running in the background, run the same command line used to start the agent with `--stop` appended.
 For example, to stop the first of the above shown same machine, single gpu agents:
 ```bash
-trains-agent daemon --detached --gpus 0 --queue default --docker nvidia/cuda --stop
+clearml-agent daemon --detached --gpus 0 --queue default --docker nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04 --stop
 ```
 
-## How do I create an experiment on the Trains Server? <a name="from-scratch"></a>
-* Integrate [Trains](https://github.com/allegroai/trains) with your code
+### How do I create an experiment on the ClearML Server? <a name="from-scratch"></a>
+* Integrate [ClearML](https://github.com/allegroai/clearml) with your code
 * Execute the code on your machine (Manually / PyCharm / Jupyter Notebook)
-* As your code is running, **Trains** creates an experiment logging all the necessary execution information:
+* As your code is running, **ClearML** creates an experiment logging all the necessary execution information:
   - Git repository link and commit ID (or an entire jupyter notebook)
   - Git diff (we’re not saying you never commit and push, but still...)
   - Python packages used by your code (including specific versions used)
@@ -247,7 +240,7 @@ trains-agent daemon --detached --gpus 0 --queue default --docker nvidia/cuda --s
 
   You now have a 'template' of your experiment with everything required for automated execution
 
-* In the Trains UI, Right click on the experiment and select 'clone'. A copy of your experiment will be created.
+* In the ClearML UI, Right click on the experiment and select 'clone'. A copy of your experiment will be created.
 * You now have a new draft experiment cloned from your original experiment, feel free to edit it
   - Change the Hyper-Parameters
   - Switch to the latest code base of the repository
@@ -256,44 +249,44 @@ trains-agent daemon --detached --gpus 0 --queue default --docker nvidia/cuda --s
   - Or simply change nothing to run the same experiment again...
 * Schedule the newly created experiment for execution: Right-click the experiment and select 'enqueue'
 
-## Trains-Agent Services Mode <a name="services"></a> 
+### ClearML-Agent Services Mode <a name="services"></a>
 
-Trains-Agent Services is a special mode of Trains-Agent that provides the ability to launch long-lasting jobs 
-that previously had to be executed on local / dedicated machines. It allows a single agent to 
-launch multiple dockers (Tasks) for different use cases. To name a few use cases, auto-scaler service (spinning instances 
+ClearML-Agent Services is a special mode of ClearML-Agent that provides the ability to launch long-lasting jobs
+that previously had to be executed on local / dedicated machines. It allows a single agent to
+launch multiple dockers (Tasks) for different use cases. To name a few use cases, auto-scaler service (spinning instances
 when the need arises and the budget allows), Controllers (Implementing pipelines and more sophisticated DevOps logic),
-Optimizer (such as Hyper-parameter Optimization or sweeping), and Application (such as interactive Bokeh apps for 
+Optimizer (such as Hyper-parameter Optimization or sweeping), and Application (such as interactive Bokeh apps for
 increased data transparency)
 
-Trains-Agent Services mode will spin **any** task enqueued into the specified queue. 
-Every task launched by Trains-Agent Services will be registered as a new node in the system, 
-providing tracking and transparency capabilities. 
-Currently trains-agent in services-mode supports cpu only configuration. Trains-agent services mode can be launched alongside GPU agents.
+ClearML-Agent Services mode will spin **any** task enqueued into the specified queue.
+Every task launched by ClearML-Agent Services will be registered as a new node in the system,
+providing tracking and transparency capabilities.
+Currently clearml-agent in services-mode supports cpu only configuration. ClearML-agent services mode can be launched alongside GPU agents.
 
 ```bash
-trains-agent daemon --services-mode --detached --queue services --create-queue --docker ubuntu:18.04 --cpu-only
+clearml-agent daemon --services-mode --detached --queue services --create-queue --docker ubuntu:18.04 --cpu-only
 ```
 
-**Note**: It is the user's responsibility to make sure the proper tasks are pushed into the specified queue. 
+**Note**: It is the user's responsibility to make sure the proper tasks are pushed into the specified queue.
 
 
-## AutoML and Orchestration Pipelines <a name="automl-pipes"></a>
-The Trains Agent can also be used to implement AutoML orchestration and Experiment Pipelines in conjunction with the Trains package.
+### AutoML and Orchestration Pipelines <a name="automl-pipes"></a>
+The ClearML Agent can also be used to implement AutoML orchestration and Experiment Pipelines in conjunction with the ClearML package.
 
-Sample AutoML & Orchestration examples can be found in the Trains [example/automation](https://github.com/allegroai/trains/tree/master/examples/automation) folder.
+Sample AutoML & Orchestration examples can be found in the ClearML [example/automation](https://github.com/allegroai/clearml/tree/master/examples/automation) folder.
 
 AutoML examples
-  - [Toy Keras training experiment](https://github.com/allegroai/trains/blob/master/examples/optimization/hyper-parameter-optimization/base_template_keras_simple.py)
+  - [Toy Keras training experiment](https://github.com/allegroai/clearml/blob/master/examples/optimization/hyper-parameter-optimization/base_template_keras_simple.py)
     - In order to create an experiment-template in the system, this code must be executed once manually
-  - [Random Search over the above Keras experiment-template](https://github.com/allegroai/trains/blob/master/examples/automation/manual_random_param_search_example.py)
+  - [Random Search over the above Keras experiment-template](https://github.com/allegroai/clearml/blob/master/examples/automation/manual_random_param_search_example.py)
     - This example will create multiple copies of the Keras experiment-template, with different hyper-parameter combinations
 
 Experiment Pipeline examples
-  - [First step experiment](https://github.com/allegroai/trains/blob/master/examples/automation/task_piping_example.py)
+  - [First step experiment](https://github.com/allegroai/clearml/blob/master/examples/automation/task_piping_example.py)
     - This example will "process data", and once done, will launch a copy of the 'second step' experiment-template
-  - [Second step experiment](https://github.com/allegroai/trains/blob/master/examples/automation/toy_base_task.py)
+  - [Second step experiment](https://github.com/allegroai/clearml/blob/master/examples/automation/toy_base_task.py)
     - In order to create an experiment-template in the system, this code must be executed once manually
 
-## License
+### License
 
 Apache License, Version 2.0 (see the [LICENSE](https://www.apache.org/licenses/LICENSE-2.0.html) for more information)

clearml_agent/__main__.py

@@ -4,13 +4,13 @@ import argparse
 import sys
 import warnings
 
-from trains_agent.backend_api.session.datamodel import UnusedKwargsWarning
+from clearml_agent.backend_api.session.datamodel import UnusedKwargsWarning
 
-import trains_agent
-from trains_agent.config import get_config
-from trains_agent.definitions import FileBuffering, CONFIG_FILE
-from trains_agent.helper.base import reverse_home_folder_expansion, chain_map, named_temporary_file
-from trains_agent.helper.process import ExitStatus
+import clearml_agent
+from clearml_agent.config import get_config
+from clearml_agent.definitions import FileBuffering, CONFIG_FILE
+from clearml_agent.helper.base import reverse_home_folder_expansion, chain_map, named_temporary_file
+from clearml_agent.helper.process import ExitStatus
 from . import interface, session, definitions, commands
 from .errors import ConfigFileNotFound, Sigterm, APIError
 from .helper.trace import PackageTrace
@@ -47,7 +47,7 @@ def run_command(parser, args, command_name):
     except ConfigFileNotFound:
         message = 'Cannot find configuration file in "{}".\n' \
                   'To create a configuration file, run:\n' \
-                  '$ trains_agent init'.format(reverse_home_folder_expansion(CONFIG_FILE))
+                  '$ clearml_agent init'.format(reverse_home_folder_expansion(CONFIG_FILE))
         command_class.exit(message)
     except APIError as api_error:
         if not debug:

clearml_agent/backend_api/config/default/agent.conf

@@ -1,12 +1,12 @@
 {
     # unique name of this worker, if None, created based on hostname:process_id
-    # Override with os environment: TRAINS_WORKER_ID
-    # worker_id: "trains-agent-machine1:gpu0"
+    # Override with os environment: CLEARML_WORKER_ID
+    # worker_id: "clearml-agent-machine1:gpu0"
     worker_id: ""
 
     # worker name, replaces the hostname when creating a unique name for this worker
-    # Override with os environment: TRAINS_WORKER_NAME
-    # worker_name: "trains-agent-machine1"
+    # Override with os environment: CLEARML_WORKER_NAME
+    # worker_name: "clearml-agent-machine1"
     worker_name: ""
 
     # Set GIT user/pass credentials (if user/pass are set, GIT protocol will be set to https)
@@ -19,11 +19,16 @@
     force_git_ssh_protocol: false
     # Force a specific SSH port when converting http to ssh links (the domain is kept the same)
     # force_git_ssh_port: 0
+    # Force a specific SSH username when converting http to ssh links (the default username is 'git')
+    # force_git_ssh_user: git
 
     # Set the python version to use when creating the virtual environment and launching the experiment
     # Example values: "/usr/bin/python3" or "/usr/local/bin/python3.6"
-    # The default is the python executing the trains_agent
+    # The default is the python executing the clearml_agent
     python_binary: ""
+    # ignore any requested python version (Default: False, if a Task was using a
+    # specific python version and the system supports multiple python the agent will use the requested python version)
+    # ignore_requested_python_version: true
 
     # select python package manager:
     # currently supported pip and conda
@@ -42,10 +47,10 @@
         force_upgrade: false,
 
         # additional artifact repositories to use when installing python packages
-        # extra_index_url: ["https://allegroai.jfrog.io/trainsai/api/pypi/public/simple"]
+        # extra_index_url: ["https://allegroai.jfrog.io/clearmlai/api/pypi/public/simple"]
 
         # additional conda channels to use when installing with conda package manager
-        conda_channels: ["defaults", "conda-forge", "pytorch", ]
+        conda_channels: ["pytorch", "conda-forge", "defaults", ]
 
         # If set to true, Task's "installed packages" are ignored,
         # and the repository's "requirements.txt" is used instead
@@ -73,12 +78,22 @@
     },
 
     # target folder for virtual environments builds, created when executing experiment
-    venvs_dir = ~/.trains/venvs-builds
+    venvs_dir = ~/.clearml/venvs-builds
+
+    # cached virtual environment folder
+    venvs_cache: {
+        # maximum number of cached venvs
+        max_entries: 10
+        # minimum required free space to allow for cache entry, disable by passing 0 or negative value
+        free_space_threshold_gb: 2.0
+        # unmark to enable virtual environment caching
+        # path: ~/.clearml/venvs-cache
+    },
 
     # cached git clone folder
     vcs_cache: {
         enabled: true,
-        path: ~/.trains/vcs-cache
+        path: ~/.clearml/vcs-cache
     },
 
     # use venv-update in order to accelerate python virtual environment building
@@ -90,7 +105,7 @@
     # cached folder for specific python package download (used for pytorch package caching)
     pip_download_cache {
         enabled: true,
-        path: ~/.trains/pip-download-cache
+        path: ~/.clearml/pip-download-cache
     },
 
     translate_ssh: true,
@@ -98,9 +113,9 @@
     reload_config: false,
 
     # pip cache folder mapped into docker, used for python package caching
-    docker_pip_cache = ~/.trains/pip-cache
+    docker_pip_cache = ~/.clearml/pip-cache
     # apt cache folder mapped into docker, used for ubuntu package caching
-    docker_apt_cache = ~/.trains/apt-cache
+    docker_apt_cache = ~/.clearml/apt-cache
 
     # optional arguments to pass to docker image
     # these are local for this agent and will not be updated in the experiment's docker_cmd section
@@ -109,6 +124,11 @@
     # optional shell script to run in docker when started before the experiment is started
     # extra_docker_shell_script: ["apt-get install -y bindfs", ]
 
+    # Install the required packages for opencv libraries (libsm6 libxext6 libxrender-dev libglib2.0-0),
+    # for backwards compatibility reasons, true as default,
+    # change to false to skip installation and decrease docker spin up time
+    # docker_install_opencv_libs: true
+
     # optional uptime configuration, make sure to use only one of 'uptime/downtime' and not both.
     # If uptime is specified, agent will actively poll (and execute) tasks in the time-spans defined here.
     # Outside of the specified time-spans, the agent will be idle.
@@ -131,12 +151,15 @@
 
     default_docker: {
         # default docker image to use when running in docker mode
-        image: "nvidia/cuda:10.1-runtime-ubuntu18.04"
+        image: "nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04"
 
         # optional arguments to pass to docker image
         # arguments: ["--ipc=host", ]
     }
 
+    # set the OS environments based on the Task's Environment section before launching the Task process.
+    enable_task_env: false
+
     # set the initial bash script to execute at the startup of any docker.
     # all lines will be executed regardless of their exit code.
     # {python_single_digit} is translated to 'python3' or 'python2' according to requested python version
@@ -162,4 +185,16 @@
     # should be detected automatically. Override with os environment CUDA_VERSION / CUDNN_VERSION
     # cuda_version: 10.1
     # cudnn_version: 7.6
+
+    # Hide docker environment variables containing secrets when printing out the docker command by replacing their
+    # values with "********". Turning this feature on will hide the following environment variables values:
+    #   CLEARML_API_SECRET_KEY, CLEARML_AGENT_GIT_PASS, AWS_SECRET_ACCESS_KEY, AZURE_STORAGE_KEY
+    # To include more environment variables, add their keys to the "extra_keys" list. E.g. to make sure the value of
+    # your custom environment variable named MY_SPECIAL_PASSWORD will not show in the logs when included in the
+    # docker command, set:
+    #   extra_keys: ["MY_SPECIAL_PASSWORD"]
+    hide_docker_command_env_vars {
+        enabled: true
+        extra_keys: []
+    }
 }

clearml_agent/backend_api/config/default/sdk.conf

@@ -1,10 +1,10 @@
 {
-    # TRAINS - default SDK configuration
+    # ClearML - default SDK configuration
 
     storage {
         cache {
             # Defaults to system temp folder / cache
-            default_base_dir: "~/.trains/cache"
+            default_base_dir: "~/.clearml/cache"
             size {
                 # max_used_bytes = -1
                 min_free_bytes = 10GB
@@ -98,7 +98,7 @@
     google.storage {
         # # Default project and credentials file
         # # Will be used when no bucket configuration is found
-        # project: "trains"
+        # project: "clearml"
         # credentials_json: "/path/to/credentials.json"
 
         # # Specific credentials per bucket and sub directory
@@ -106,7 +106,7 @@
         #     {
         #         bucket: "my-bucket"
         #         subdir: "path/in/bucket" # Not required
-        #         project: "trains"
+        #         project: "clearml"
         #         credentials_json: "/path/to/credentials.json"
         #     },
         # ]
@@ -114,7 +114,7 @@
     azure.storage {
         # containers: [
         #     {
-        #         account_name: "trains"
+        #         account_name: "clearml"
         #         account_key: "secret"
         #         # container_name:
         #     }
@@ -155,8 +155,8 @@
         # do not analyze the entire repository.
         force_analyze_entire_repo: false
 
-        # If set to true, *trains* update message will not be printed to the console
-        # this value can be overwritten with os environment variable TRAINS_SUPPRESS_UPDATE_MESSAGE=1
+        # If set to true, *clearml* update message will not be printed to the console
+        # this value can be overwritten with os environment variable CLEARML_SUPPRESS_UPDATE_MESSAGE=1
         suppress_update_message: false
 
         # If this flag is true (default is false), instead of analyzing the code with Pigar, analyze with `pip freeze`

clearml_agent/backend_api/session/client/client.py

@@ -106,15 +106,15 @@ class StrictSession(Session):
             init()
             return
 
-        original = os.environ.get(LOCAL_CONFIG_FILE_OVERRIDE_VAR, None)
+        original = LOCAL_CONFIG_FILE_OVERRIDE_VAR.get() or None
         try:
-            os.environ[LOCAL_CONFIG_FILE_OVERRIDE_VAR] = str(config_file)
+            LOCAL_CONFIG_FILE_OVERRIDE_VAR.set(str(config_file))
             init()
         finally:
             if original is None:
-                os.environ.pop(LOCAL_CONFIG_FILE_OVERRIDE_VAR, None)
+                LOCAL_CONFIG_FILE_OVERRIDE_VAR.pop()
             else:
-                os.environ[LOCAL_CONFIG_FILE_OVERRIDE_VAR] = original
+                LOCAL_CONFIG_FILE_OVERRIDE_VAR.set(original)
 
     def send(self, request, *args, **kwargs):
         result = super(StrictSession, self).send(request, *args, **kwargs)
@@ -222,7 +222,7 @@ class TableResponse(Response):
             return "" if result is None else result
 
         fields = fields or self.fields
-        from trains_agent.helper.base import create_table
+        from clearml_agent.helper.base import create_table
         return create_table(
             (dict((attr, getter(item, attr)) for attr in fields) for item in self),
             titles=fields, columns=fields, headers=True,

clearml_agent/backend_api/session/defs.py

@@ -0,0 +1,11 @@
+from ...backend_config.environment import EnvEntry
+
+
+ENV_HOST = EnvEntry("CLEARML_API_HOST", "TRAINS_API_HOST")
+ENV_WEB_HOST = EnvEntry("CLEARML_WEB_HOST", "TRAINS_WEB_HOST")
+ENV_FILES_HOST = EnvEntry("CLEARML_FILES_HOST", "TRAINS_FILES_HOST")
+ENV_ACCESS_KEY = EnvEntry("CLEARML_API_ACCESS_KEY", "TRAINS_API_ACCESS_KEY")
+ENV_SECRET_KEY = EnvEntry("CLEARML_API_SECRET_KEY", "TRAINS_API_SECRET_KEY")
+ENV_VERBOSE = EnvEntry("CLEARML_API_VERBOSE", "TRAINS_API_VERBOSE", type=bool, default=False)
+ENV_HOST_VERIFY_CERT = EnvEntry("CLEARML_API_HOST_VERIFY_CERT", "TRAINS_API_HOST_VERIFY_CERT", type=bool, default=True)
+ENV_CONDA_ENV_PACKAGE = EnvEntry("CLEARML_CONDA_ENV_PACKAGE", "TRAINS_CONDA_ENV_PACKAGE")

clearml_agent/backend_api/session/session.py

@@ -29,12 +29,12 @@ class MaxRequestSizeError(Exception):
 
 
 class Session(TokenManager):
-    """ TRAINS API Session class. """
+    """ ClearML API Session class. """
 
     _AUTHORIZATION_HEADER = "Authorization"
-    _WORKER_HEADER = "X-Trains-Worker"
-    _ASYNC_HEADER = "X-Trains-Async"
-    _CLIENT_HEADER = "X-Trains-Agent"
+    _WORKER_HEADER = ("X-ClearML-Worker", "X-Trains-Worker", )
+    _ASYNC_HEADER = ("X-ClearML-Async", "X-Trains-Async", )
+    _CLIENT_HEADER = ("X-ClearML-Agent", "X-Trains-Agent", )
 
     _async_status_code = 202
     _session_requests = 0
@@ -45,9 +45,9 @@ class Session(TokenManager):
     _write_session_timeout = (30.0, 30.)
 
     api_version = '2.1'
-    default_host = "https://demoapi.trains.allegro.ai"
-    default_web = "https://demoapp.trains.allegro.ai"
-    default_files = "https://demofiles.trains.allegro.ai"
+    default_host = "https://demoapi.demo.clear.ml"
+    default_web = "https://demoapp.demo.clear.ml"
+    default_files = "https://demofiles.demo.clear.ml"
     default_key = "EGRTCO8JMSIGI6S39GTP43NFWXDQOW"
     default_secret = "x!XTov_G-#vspE*Y(h$Anm&DIc5Ou-F)jsl$PdOyj5wG1&E!Z8"
 
@@ -111,7 +111,8 @@ class Session(TokenManager):
         self._logger = logger
 
         self.__access_key = api_key or ENV_ACCESS_KEY.get(
-            default=(self.config.get("api.credentials.access_key", None) or self.default_key)
+            default=(self.config.get("api.credentials.access_key", None) or self.default_key),
+            value_cb=lambda key, value: print("Using environment access key {}={}".format(key, value))
         )
         if not self.access_key:
             raise ValueError(
@@ -119,7 +120,8 @@ class Session(TokenManager):
             )
 
         self.__secret_key = secret_key or ENV_SECRET_KEY.get(
-            default=(self.config.get("api.credentials.secret_key", None) or self.default_secret)
+            default=(self.config.get("api.credentials.secret_key", None) or self.default_secret),
+            value_cb=lambda key, value: print("Using environment secret key {}=********".format(key))
         )
         if not self.secret_key:
             raise ValueError(
@@ -155,7 +157,7 @@ class Session(TokenManager):
 
         # update api version from server response
         try:
-            token_dict = jwt.decode(self.token, verify=False)
+            token_dict = TokenManager.get_decoded_token(self.token, verify=False)
             api_version = token_dict.get('api_version')
             if not api_version:
                 api_version = '2.2' if token_dict.get('env', '') == 'prod' else Session.api_version
@@ -192,8 +194,10 @@ class Session(TokenManager):
         """
         host = self.host
         headers = headers.copy() if headers else {}
-        headers[self._WORKER_HEADER] = self.worker
-        headers[self._CLIENT_HEADER] = self.client
+        for h in self._WORKER_HEADER:
+            headers[h] = self.worker
+        for h in self._CLIENT_HEADER:
+            headers[h] = self.client
 
         token_refreshed_on_error = False
         url = (
@@ -268,7 +272,8 @@ class Session(TokenManager):
             headers.copy() if headers else {}
         )
         if async_enable:
-            headers[self._ASYNC_HEADER] = "1"
+            for h in self._ASYNC_HEADER:
+                headers[h] = "1"
         return self._send_request(
             service=service,
             action=action,
@@ -464,7 +469,7 @@ class Session(TokenManager):
         if parsed.port == 8008:
             return host.replace(':8008', ':8080', 1)
 
-        raise ValueError('Could not detect TRAINS web application server')
+        raise ValueError('Could not detect ClearML web application server')
 
     @classmethod
     def get_files_server_host(cls, config=None):
@@ -548,13 +553,13 @@ class Session(TokenManager):
             # check if this is a misconfigured api server (getting 200 without the data section)
             if res and res.status_code == 200:
                 raise ValueError('It seems *api_server* is misconfigured. '
-                                 'Is this the TRAINS API server {} ?'.format(self.get_api_server_host()))
+                                 'Is this the ClearML API server {} ?'.format(self.get_api_server_host()))
             else:
                 raise LoginError("Response data mismatch: No 'token' in 'data' value from res, receive : {}, "
                                  "exception: {}".format(res, ex))
         except requests.ConnectionError as ex:
             raise ValueError('Connection Error: it seems *api_server* is misconfigured. '
-                             'Is this the TRAINS API server {} ?'.format('/'.join(ex.request.url.split('/')[:3])))
+                             'Is this the ClearML API server {} ?'.format('/'.join(ex.request.url.split('/')[:3])))
         except Exception as ex:
             raise LoginError('Unrecognized Authentication Error: {} {}'.format(type(ex), ex))
 

clearml_agent/backend_api/session/token_manager.py

@@ -3,6 +3,7 @@ from abc import ABCMeta, abstractmethod
 from time import time
 
 import jwt
+from jwt.algorithms import get_default_algorithms
 import six
 
 
@@ -66,10 +67,18 @@ class TokenManager(object):
                 pass
         return 0
 
+    @classmethod
+    def get_decoded_token(cls, token, verify=False):
+        """ Get token expiration time. If not present, assume forever """
+        return jwt.decode(
+            token, verify=verify,
+            options=dict(verify_signature=False),
+            algorithms=get_default_algorithms())
+
     @classmethod
     def _get_token_exp(cls, token):
         """ Get token expiration time. If not present, assume forever """
-        return jwt.decode(token, verify=False).get('exp', sys.maxsize)
+        return cls.get_decoded_token(token).get('exp', sys.maxsize)
 
     def _set_token(self, token):
         if token:

clearml_agent/backend_api/utils.py

@@ -6,16 +6,9 @@ import requests
 from requests.adapters import HTTPAdapter
 from urllib3.util import Retry
 from urllib3 import PoolManager
-import six
 
 from .session.defs import ENV_HOST_VERIFY_CERT
 
-if six.PY3:
-    from functools import lru_cache
-elif six.PY2:
-    # python 2 support
-    from backports.functools_lru_cache import lru_cache
-
 
 __disable_certificate_verification_warning = 0
 
@@ -107,7 +100,7 @@ def get_http_session_with_retry(
     if not session.verify and __disable_certificate_verification_warning < 2:
         # show warning
         __disable_certificate_verification_warning += 1
-        logging.getLogger('TRAINS').warning(
+        logging.getLogger('ClearML').warning(
             msg='InsecureRequestWarning: Certificate verification is disabled! Adding '
                 'certificate verification is strongly advised. See: '
                 'https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings')

clearml_agent/backend_config/__init__.py

@@ -1,4 +1,3 @@
 from .defs import Environment
 from .config import Config, ConfigEntry
 from .errors import ConfigurationError
-from .environment import EnvEntry

clearml_agent/backend_config/config.py

@@ -138,7 +138,7 @@ class Config(object):
         else:
             env_config_paths = []
 
-        env_config_path_override = os.environ.get(ENV_CONFIG_PATH_OVERRIDE_VAR)
+        env_config_path_override = ENV_CONFIG_PATH_OVERRIDE_VAR.get()
         if env_config_path_override:
             env_config_paths = [expanduser(env_config_path_override)]
 
@@ -165,7 +165,7 @@ class Config(object):
             )
 
         local_config_files = LOCAL_CONFIG_FILES
-        local_config_override = os.environ.get(LOCAL_CONFIG_FILE_OVERRIDE_VAR)
+        local_config_override = LOCAL_CONFIG_FILE_OVERRIDE_VAR.get()
         if local_config_override:
             local_config_files = [expanduser(local_config_override)]
 

clearml_agent/backend_config/defs.py

@@ -1,6 +1,8 @@
 from os.path import expanduser
 from pathlib2 import Path
 
+from ..backend_config.environment import EnvEntry
+
 ENV_VAR = 'TRAINS_ENV'
 """ Name of system environment variable that can be used to specify the config environment name """
 
@@ -17,23 +19,24 @@ ENV_CONFIG_PATHS = [
 
 
 LOCAL_CONFIG_PATHS = [
-    # '/etc/opt/trains',               # used by servers for docker-generated configuration
-    # expanduser('~/.trains/config'),
+    # '/etc/opt/clearml',               # used by servers for docker-generated configuration
+    # expanduser('~/.clearml/config'),
 ]
 """ Local config paths, not related to environment """
 
 
 LOCAL_CONFIG_FILES = [
     expanduser('~/trains.conf'),    # used for workstation configuration (end-users, workers)
+    expanduser('~/clearml.conf'),    # used for workstation configuration (end-users, workers)
 ]
 """ Local config files (not paths) """
 
 
-LOCAL_CONFIG_FILE_OVERRIDE_VAR = 'TRAINS_CONFIG_FILE'
+LOCAL_CONFIG_FILE_OVERRIDE_VAR = EnvEntry('CLEARML_CONFIG_FILE', 'TRAINS_CONFIG_FILE', )
 """ Local config file override environment variable. If this is set, no other local config files will be used. """
 
 
-ENV_CONFIG_PATH_OVERRIDE_VAR = 'TRAINS_CONFIG_PATH'
+ENV_CONFIG_PATH_OVERRIDE_VAR = EnvEntry('CLEARML_CONFIG_PATH', 'TRAINS_CONFIG_PATH', )
 """ 
 Environment-related config path override environment variable. If this is set, no other env config path will be used. 
 """

clearml_agent/backend_config/entry.py

@@ -64,8 +64,8 @@ class Entry(object):
             converter = self.default_conversions().get(self.type, self.type)
         return converter(value)
 
-    def get_pair(self, default=NotSet, converter=None):
-        # type: (Any, Converter) -> Optional[Tuple[Text, Any]]
+    def get_pair(self, default=NotSet, converter=None, value_cb=None):
+        # type: (Any, Converter, Callable[[str, Any], None]) -> Optional[Tuple[Text, Any]]
         for key in self.keys:
             value = self._get(key)
             if value is NotSet:
@@ -75,18 +75,26 @@ class Entry(object):
             except Exception as ex:
                 self.error("invalid value {key}={value}: {ex}".format(**locals()))
                 break
+            # noinspection PyBroadException
+            try:
+                if value_cb:
+                    value_cb(key, value)
+            except Exception:
+                pass
             return key, value
+
         result = self.default if default is NotSet else default
         return self.key, result
 
-    def get(self, default=NotSet, converter=None):
-        # type: (Any, Converter) -> Optional[Any]
-        return self.get_pair(default=default, converter=converter)[1]
+    def get(self, default=NotSet, converter=None, value_cb=None):
+        # type: (Any, Converter, Callable[[str, Any], None]) -> Optional[Any]
+        return self.get_pair(default=default, converter=converter, value_cb=value_cb)[1]
 
     def set(self, value):
         # type: (Any, Any) -> (Text, Any)
-        key, _ = self.get_pair(default=None, converter=None)
-        self._set(key, str(value))
+        # key, _ = self.get_pair(default=None, converter=None)
+        for k in self.keys:
+            self._set(k, str(value))
 
     def _set(self, key, value):
         # type: (Text, Text) -> None

clearml_agent/backend_config/environment.py

@@ -0,0 +1,64 @@
+from os import getenv, environ
+
+from .converters import text_to_bool
+from .entry import Entry, NotSet
+
+
+class EnvEntry(Entry):
+    @classmethod
+    def default_conversions(cls):
+        conversions = super(EnvEntry, cls).default_conversions().copy()
+        conversions[bool] = text_to_bool
+        return conversions
+
+    def pop(self):
+        for k in self.keys:
+            environ.pop(k, None)
+
+    def _get(self, key):
+        value = getenv(key, "").strip()
+        return value or NotSet
+
+    def _set(self, key, value):
+        environ[key] = value
+
+    def __str__(self):
+        return "env:{}".format(super(EnvEntry, self).__str__())
+
+    def error(self, message):
+        print("Environment configuration: {}".format(message))
+
+
+def backward_compatibility_support():
+    from ..definitions import ENVIRONMENT_CONFIG, ENVIRONMENT_SDK_PARAMS, ENVIRONMENT_BACKWARD_COMPATIBLE
+    if ENVIRONMENT_BACKWARD_COMPATIBLE.get():
+        # Add TRAINS_ prefix on every CLEARML_ os environment we support
+        for k, v in ENVIRONMENT_CONFIG.items():
+            try:
+                trains_vars = [var for var in v.vars if var.startswith('CLEARML_')]
+                if not trains_vars:
+                    continue
+                alg_var = trains_vars[0].replace('CLEARML_', 'TRAINS_', 1)
+                if alg_var not in v.vars:
+                    v.vars = tuple(list(v.vars) + [alg_var])
+            except:
+                continue
+        for k, v in ENVIRONMENT_SDK_PARAMS.items():
+            try:
+                trains_vars = [var for var in v if var.startswith('CLEARML_')]
+                if not trains_vars:
+                    continue
+                alg_var = trains_vars[0].replace('CLEARML_', 'TRAINS_', 1)
+                if alg_var not in v:
+                    ENVIRONMENT_SDK_PARAMS[k] = tuple(list(v) + [alg_var])
+            except:
+                continue
+
+    # set OS environ:
+    keys = list(environ.keys())
+    for k in keys:
+        if not k.startswith('CLEARML_'):
+            continue
+        backwards_k = k.replace('CLEARML_', 'TRAINS_', 1)
+        if backwards_k not in keys:
+            environ[backwards_k] = environ[k]

clearml_agent/backend_config/log.py

@@ -4,11 +4,11 @@ from pathlib2 import Path
 
 
 def logger(path=None):
-    name = "trains"
+    name = "clearml"
     if path:
         p = Path(path)
         module = (p.parent if p.stem.startswith('_') else p).stem
-        name = "trains.%s" % module
+        name = "clearml.%s" % module
     return logging.getLogger(name)
 
 

clearml_agent/commands/base.py

@@ -9,16 +9,16 @@ from operator import attrgetter
 from traceback import print_exc
 from typing import Text
 
-from trains_agent.helper.console import ListFormatter, print_text
-from trains_agent.helper.dicts import filter_keys
+from clearml_agent.helper.console import ListFormatter, print_text
+from clearml_agent.helper.dicts import filter_keys
 
 import six
-from trains_agent.backend_api import services
+from clearml_agent.backend_api import services
 
-from trains_agent.errors import APIError, CommandFailedError
-from trains_agent.helper.base import Singleton, return_list, print_parameters, dump_yaml, load_yaml, error, warning
-from trains_agent.interface.base import ObjectID
-from trains_agent.session import Session
+from clearml_agent.errors import APIError, CommandFailedError
+from clearml_agent.helper.base import Singleton, return_list, print_parameters, dump_yaml, load_yaml, error, warning
+from clearml_agent.interface.base import ObjectID
+from clearml_agent.session import Session
 
 
 class NameResolutionError(CommandFailedError):
@@ -74,7 +74,7 @@ class BaseCommandSection(object):
 
     @staticmethod
     def log(message, *args):
-        print("trains-agent: {}".format(message % args))
+        print("clearml-agent: {}".format(message % args))
 
     @classmethod
     def exit(cls, message, code=1):  # type: (Text, int) -> ()

clearml_agent/commands/check_config.py

@@ -1,4 +1,4 @@
-from trains_agent.commands.base import ServiceCommandSection
+from clearml_agent.commands.base import ServiceCommandSection
 
 
 class Config(ServiceCommandSection):

clearml_agent/commands/config.py

@@ -5,13 +5,15 @@ from pyhocon import ConfigFactory, ConfigMissingException
 from pathlib2 import Path
 from six.moves.urllib.parse import urlparse
 
-from trains_agent.backend_api.session import Session
-from trains_agent.backend_api.session.defs import ENV_HOST
-from trains_agent.backend_config.defs import LOCAL_CONFIG_FILES
+from clearml_agent.backend_api.session import Session
+from clearml_agent.backend_api.session.defs import ENV_HOST
+from clearml_agent.backend_config.defs import LOCAL_CONFIG_FILES
 
 
 description = """
-Please create new trains credentials through the profile page in your trains web app (e.g. https://demoapp.trains.allegro.ai/profile)
+Please create new clearml credentials through the profile page in your clearml web app (e.g. https://demoapp.demo.clear.ml/profile)
+Or with the free hosted service at https://app.community.clear.ml/profile
+    
 In the profile page, press "Create new credentials", then press "Copy to clipboard".
 
 Paste copied configuration here: 
@@ -25,7 +27,7 @@ except Exception:
 
 host_description = """
 Editing configuration file: {CONFIG_FILE}
-Enter the url of the trains-server's Web service, for example: {HOST}
+Enter the url of the clearml-server's Web service, for example: {HOST}
 """.format(
     CONFIG_FILE=LOCAL_CONFIG_FILES[0],
     HOST=def_host,
@@ -33,8 +35,12 @@ Enter the url of the trains-server's Web service, for example: {HOST}
 
 
 def main():
-    print('TRAINS-AGENT setup process')
-    conf_file = Path(LOCAL_CONFIG_FILES[0]).absolute()
+    print('CLEARML-AGENT setup process')
+    for f in LOCAL_CONFIG_FILES:
+        conf_file = Path(f).absolute()
+        if conf_file.exists():
+            break
+
     if conf_file.exists() and conf_file.is_file() and conf_file.stat().st_size > 0:
         print('Configuration file already exists: {}'.format(str(conf_file)))
         print('Leaving setup, feel free to edit the configuration file.')
@@ -42,7 +48,12 @@ def main():
 
     print(description, end='')
     sentinel = ''
-    parse_input = '\n'.join(iter(input, sentinel))
+    parse_input = ''
+    for line in iter(input, sentinel):
+        parse_input += line+'\n'
+        if line.rstrip() == '}':
+            break
+
     credentials = None
     api_server = None
     web_server = None
@@ -86,7 +97,7 @@ def main():
 
     files_host = input_url('File Store Host', files_host)
 
-    print('\nTRAINS Hosts configuration:\nWeb App: {}\nAPI: {}\nFile Store: {}\n'.format(
+    print('\nClearML Hosts configuration:\nWeb App: {}\nAPI: {}\nFile Store: {}\n'.format(
         web_host, api_host, files_host))
 
     retry = 1
@@ -140,14 +151,14 @@ def main():
     # noinspection PyBroadException
     try:
         with open(str(conf_file), 'wt') as f:
-            header = '# TRAINS-AGENT configuration file\n' \
+            header = '# CLEARML-AGENT configuration file\n' \
                      'api {\n' \
                      '    # Notice: \'host\' is the api server (default port 8008), not the web server.\n' \
                      '    api_server: %s\n' \
                      '    web_server: %s\n' \
                      '    files_server: %s\n' \
                      '    # Credentials are generated using the webapp, %s/profile\n' \
-                     '    # Override with os environment: TRAINS_API_ACCESS_KEY / TRAINS_API_SECRET_KEY\n' \
+                     '    # Override with os environment: CLEARML_API_ACCESS_KEY / CLEARML_API_SECRET_KEY\n' \
                      '    credentials {"access_key": "%s", "secret_key": "%s"}\n' \
                      '}\n\n' % (api_host, web_host, files_host,
                                 web_host, credentials['access_key'], credentials['secret_key'])
@@ -158,7 +169,7 @@ def main():
                               'agent.git_pass=\"{}\"\n' \
                               '\n'.format(git_user or '', git_pass or '')
             f.write(git_credentials)
-            extra_index_str = '# extra_index_url: ["https://allegroai.jfrog.io/trainsai/api/pypi/public/simple"]\n' \
+            extra_index_str = '# extra_index_url: ["https://allegroai.jfrog.io/clearml/api/pypi/public/simple"]\n' \
                               'agent.package_manager.extra_index_url= ' \
                               '[\n{}\n]\n\n'.format("\n".join(map("\"{}\"".format, extra_index_urls)))
             f.write(extra_index_str)
@@ -168,7 +179,7 @@ def main():
         return
 
     print('\nNew configuration stored in {}'.format(str(conf_file)))
-    print('TRAINS-AGENT setup completed successfully.')
+    print('CLEARML-AGENT setup completed successfully.')
 
 
 def parse_host(parsed_host, allow_input=True):
@@ -309,7 +320,7 @@ def verify_url(parse_input):
             parsed_host = None
     except Exception:
         parsed_host = None
-        print('Could not parse url {}\nEnter your trains-server host: '.format(parse_input), end='')
+        print('Could not parse url {}\nEnter your clearml-server host: '.format(parse_input), end='')
     return parsed_host
 
 

clearml_agent/commands/events.py

@@ -5,8 +5,8 @@ import time
 
 from future.builtins import super
 
-from trains_agent.commands.base import ServiceCommandSection
-from trains_agent.helper.base import return_list
+from clearml_agent.commands.base import ServiceCommandSection
+from clearml_agent.helper.base import return_list
 
 
 class Events(ServiceCommandSection):

clearml_agent/complete.py

@@ -1,8 +1,8 @@
 """
 Script for generating command-line completion.
-Called by trains_agent/utilities/complete.sh (or a copy of it) like so:
+Called by clearml_agent/utilities/complete.sh (or a copy of it) like so:
 
-python -m trains_agent.complete "current command line"
+python -m clearml_agent.complete "current command line"
 
 And writes line-separated completion targets to stdout.
 Results are line-separated in order to enable other whitespace in results.
@@ -13,7 +13,7 @@ from __future__ import print_function
 import argparse
 import sys
 
-from trains_agent.interface import get_parser
+from clearml_agent.interface import get_parser
 
 
 def is_argument_required(action):

clearml_agent/config.py

@@ -1,7 +1,7 @@
 from pyhocon import ConfigTree
 
 import six
-from trains_agent.helper.base import Singleton
+from clearml_agent.helper.base import Singleton
 
 
 @six.add_metaclass(Singleton)

clearml_agent/definitions.py

@@ -1,22 +1,22 @@
 from datetime import timedelta
 from distutils.util import strtobool
 from enum import IntEnum
-from os import getenv
+from os import getenv, environ
 from typing import Text, Optional, Union, Tuple, Any
 
 from furl import furl
 from pathlib2 import Path
 
 import six
-from trains_agent.helper.base import normalize_path
+from clearml_agent.helper.base import normalize_path
 
-PROGRAM_NAME = "trains-agent"
+PROGRAM_NAME = "clearml-agent"
 FROM_FILE_PREFIX_CHARS = "@"
 
-CONFIG_DIR = normalize_path("~/.trains")
-TOKEN_CACHE_FILE = normalize_path("~/.trains.trains_agent.tmp")
+CONFIG_DIR = normalize_path("~/.clearml")
+TOKEN_CACHE_FILE = normalize_path("~/.clearml.clearml_agent.tmp")
 
-CONFIG_FILE_CANDIDATES = ["~/trains.conf"]
+CONFIG_FILE_CANDIDATES = ["~/clearml.conf"]
 
 
 def find_config_path():
@@ -40,6 +40,14 @@ class EnvironmentConfig(object):
         self.vars = names
         self.type = kwargs.pop("type", six.text_type)
 
+    def pop(self):
+        for k in self.vars:
+            environ.pop(k, None)
+
+    def set(self, value):
+        for k in self.vars:
+            environ[k] = str(value)
+
     def convert(self, value):
         return self.conversions.get(self.type, self.type)(value)
 
@@ -54,41 +62,46 @@ class EnvironmentConfig(object):
         return None
 
 
+ENV_AGENT_SECRET_KEY = EnvironmentConfig("CLEARML_API_SECRET_KEY", "TRAINS_API_SECRET_KEY")
+ENV_AWS_SECRET_KEY = EnvironmentConfig("AWS_SECRET_ACCESS_KEY")
+ENV_AZURE_ACCOUNT_KEY = EnvironmentConfig("AZURE_STORAGE_KEY")
+
 ENVIRONMENT_CONFIG = {
-    "api.api_server": EnvironmentConfig("TRAINS_API_HOST", ),
+    "api.api_server": EnvironmentConfig("CLEARML_API_HOST", "TRAINS_API_HOST", ),
+    "api.files_server": EnvironmentConfig("CLEARML_FILES_HOST", "TRAINS_FILES_HOST", ),
+    "api.web_server": EnvironmentConfig("CLEARML_WEB_HOST", "TRAINS_WEB_HOST", ),
     "api.credentials.access_key": EnvironmentConfig(
-        "TRAINS_API_ACCESS_KEY",
+        "CLEARML_API_ACCESS_KEY", "TRAINS_API_ACCESS_KEY",
     ),
-    "api.credentials.secret_key": EnvironmentConfig(
-        "TRAINS_API_SECRET_KEY",
-    ),
-    "agent.worker_name": EnvironmentConfig("TRAINS_WORKER_NAME", ),
-    "agent.worker_id": EnvironmentConfig("TRAINS_WORKER_ID", ),
+    "api.credentials.secret_key": ENV_AGENT_SECRET_KEY,
+    "agent.worker_name": EnvironmentConfig("CLEARML_WORKER_NAME", "TRAINS_WORKER_NAME", ),
+    "agent.worker_id": EnvironmentConfig("CLEARML_WORKER_ID", "TRAINS_WORKER_ID", ),
     "agent.cuda_version": EnvironmentConfig(
-        "TRAINS_CUDA_VERSION", "CUDA_VERSION"
+        "CLEARML_CUDA_VERSION", "TRAINS_CUDA_VERSION", "CUDA_VERSION"
     ),
     "agent.cudnn_version": EnvironmentConfig(
-        "TRAINS_CUDNN_VERSION", "CUDNN_VERSION"
+        "CLEARML_CUDNN_VERSION", "TRAINS_CUDNN_VERSION", "CUDNN_VERSION"
     ),
     "agent.cpu_only": EnvironmentConfig(
-        "TRAINS_CPU_ONLY", "CPU_ONLY", type=bool
+        names=("CLEARML_CPU_ONLY", "TRAINS_CPU_ONLY", "CPU_ONLY"), type=bool
     ),
     "sdk.aws.s3.key": EnvironmentConfig("AWS_ACCESS_KEY_ID"),
-    "sdk.aws.s3.secret": EnvironmentConfig("AWS_SECRET_ACCESS_KEY"),
+    "sdk.aws.s3.secret": ENV_AWS_SECRET_KEY,
     "sdk.aws.s3.region": EnvironmentConfig("AWS_DEFAULT_REGION"),
     "sdk.azure.storage.containers.0": {'account_name': EnvironmentConfig("AZURE_STORAGE_ACCOUNT"),
-                                       'account_key': EnvironmentConfig("AZURE_STORAGE_KEY")},
+                                       'account_key': ENV_AZURE_ACCOUNT_KEY},
     "sdk.google.storage.credentials_json": EnvironmentConfig("GOOGLE_APPLICATION_CREDENTIALS"),
 }
 
 ENVIRONMENT_SDK_PARAMS = {
-    "task_id": ("TRAINS_TASK_ID", ),
-    "config_file": ("TRAINS_CONFIG_FILE", ),
-    "log_level": ("TRAINS_LOG_LEVEL", ),
-    "log_to_backend": ("TRAINS_LOG_TASK_TO_BACKEND", ),
+    "task_id": ("CLEARML_TASK_ID", "TRAINS_TASK_ID", ),
+    "config_file": ("CLEARML_CONFIG_FILE", "TRAINS_CONFIG_FILE", ),
+    "log_level": ("CLEARML_LOG_LEVEL", "TRAINS_LOG_LEVEL", ),
+    "log_to_backend": ("CLEARML_LOG_TASK_TO_BACKEND", "TRAINS_LOG_TASK_TO_BACKEND", ),
 }
 
-ENVIRONMENT_BACKWARD_COMPATIBLE = EnvironmentConfig("TRAINS_AGENT_ALG_ENV", type=bool)
+ENVIRONMENT_BACKWARD_COMPATIBLE = EnvironmentConfig(
+    names=("CLEARML_AGENT_ALG_ENV", "TRAINS_AGENT_ALG_ENV"), type=bool)
 
 VIRTUAL_ENVIRONMENT_PATH = {
     "python2": normalize_path(CONFIG_DIR, "py2venv"),
@@ -96,7 +109,7 @@ VIRTUAL_ENVIRONMENT_PATH = {
 }
 
 DEFAULT_BASE_DIR = normalize_path(CONFIG_DIR, "data_cache")
-DEFAULT_HOST = "https://demoapi.trains.allegro.ai"
+DEFAULT_HOST = "https://demoapi.demo.clear.ml"
 MAX_DATASET_SOURCES_COUNT = 50000
 
 INVALID_WORKER_ID = (400, 1001)
@@ -105,11 +118,6 @@ WORKER_ALREADY_REGISTERED = (400, 1003)
 API_VERSION = "v1.5"
 TOKEN_EXPIRATION_SECONDS = int(timedelta(days=2).total_seconds())
 
-HTTP_HEADERS = {
-    "worker": "X-Trains-Worker",
-    "act-as": "X-Trains-Act-As",
-    "client": "X-Trains-Agent",
-}
 METADATA_EXTENSION = ".json"
 
 DEFAULT_VENV_UPDATE_URL = (
@@ -120,12 +128,17 @@ DEFAULT_VCS_CACHE = normalize_path(CONFIG_DIR, "vcs-cache")
 PIP_EXTRA_INDICES = [
 ]
 DEFAULT_PIP_DOWNLOAD_CACHE = normalize_path(CONFIG_DIR, "pip-download-cache")
-ENV_AGENT_GIT_USER = EnvironmentConfig('TRAINS_AGENT_GIT_USER')
-ENV_AGENT_GIT_PASS = EnvironmentConfig('TRAINS_AGENT_GIT_PASS')
-ENV_AGENT_GIT_HOST = EnvironmentConfig('TRAINS_AGENT_GIT_HOST')
-ENV_TASK_EXECUTE_AS_USER = 'TRAINS_AGENT_EXEC_USER'
-ENV_TASK_EXTRA_PYTHON_PATH = 'TRAINS_AGENT_EXTRA_PYTHON_PATH'
-ENV_DOCKER_HOST_MOUNT = EnvironmentConfig('TRAINS_AGENT_K8S_HOST_MOUNT', 'TRAINS_AGENT_DOCKER_HOST_MOUNT')
+ENV_DOCKER_IMAGE = EnvironmentConfig('CLEARML_DOCKER_IMAGE', 'TRAINS_DOCKER_IMAGE')
+ENV_WORKER_ID = EnvironmentConfig('CLEARML_WORKER_ID', 'TRAINS_WORKER_ID')
+ENV_DOCKER_SKIP_GPUS_FLAG = EnvironmentConfig('CLEARML_DOCKER_SKIP_GPUS_FLAG', 'TRAINS_DOCKER_SKIP_GPUS_FLAG')
+ENV_AGENT_GIT_USER = EnvironmentConfig('CLEARML_AGENT_GIT_USER', 'TRAINS_AGENT_GIT_USER')
+ENV_AGENT_GIT_PASS = EnvironmentConfig('CLEARML_AGENT_GIT_PASS', 'TRAINS_AGENT_GIT_PASS')
+ENV_AGENT_GIT_HOST = EnvironmentConfig('CLEARML_AGENT_GIT_HOST', 'TRAINS_AGENT_GIT_HOST')
+ENV_AGENT_DISABLE_SSH_MOUNT = EnvironmentConfig('CLEARML_AGENT_DISABLE_SSH_MOUNT', type=bool)
+ENV_TASK_EXECUTE_AS_USER = EnvironmentConfig('CLEARML_AGENT_EXEC_USER', 'TRAINS_AGENT_EXEC_USER')
+ENV_TASK_EXTRA_PYTHON_PATH = EnvironmentConfig('CLEARML_AGENT_EXTRA_PYTHON_PATH', 'TRAINS_AGENT_EXTRA_PYTHON_PATH')
+ENV_DOCKER_HOST_MOUNT = EnvironmentConfig('CLEARML_AGENT_K8S_HOST_MOUNT', 'CLEARML_AGENT_DOCKER_HOST_MOUNT',
+                                          'TRAINS_AGENT_K8S_HOST_MOUNT', 'TRAINS_AGENT_DOCKER_HOST_MOUNT')
 
 
 class FileBuffering(IntEnum):

clearml_agent/external/requirements_parser/parser.py

@@ -4,19 +4,20 @@ import warnings
 from .requirement import Requirement
 
 
-def parse(reqstr):
+def parse(reqstr, cwd=None):
     """
     Parse a requirements file into a list of Requirements
 
     See: pip/req.py:parse_requirements()
 
     :param reqstr: a string or file like object containing requirements
+    :param cwd: Optional current working dir for -r file.txt loading
     :returns: a *generator* of Requirement objects
     """
     filename = getattr(reqstr, 'name', None)
     try:
         # Python 2.x compatibility
-        if not isinstance(reqstr, basestring):
+        if not isinstance(reqstr, basestring):  # noqa
             reqstr = reqstr.read()
     except NameError:
         # Python 3.x only
@@ -32,8 +33,8 @@ def parse(reqstr):
             continue
         elif line.startswith('-r') or line.startswith('--requirement'):
             _, new_filename = line.split()
-            new_file_path = os.path.join(os.path.dirname(filename or '.'),
-                                         new_filename)
+            new_file_path = os.path.join(
+                os.path.dirname(filename or '.') if filename or not cwd else cwd, new_filename)
             with open(new_file_path) as f:
                 for requirement in parse(f):
                     yield requirement

clearml_agent/external/requirements_parser/requirement.py

@@ -20,6 +20,15 @@ VCS_REGEX = re.compile(
     r'(#(?P<fragment>\S+))?'
 )
 
+VCS_EXT_REGEX = re.compile(
+    r'^(?P<scheme>{0})(@)'.format(r'|'.join(
+        [scheme.replace('+', r'\+') for scheme in ['git+git']])) +
+    r'((?P<login>[^/@]+)@)?'
+    r'(?P<path>[^#@]+)'
+    r'(@(?P<revision>[^#]+))?'
+    r'(#(?P<fragment>\S+))?'
+)
+
 # This matches just about everyting
 LOCAL_REGEX = re.compile(
     r'^((?P<scheme>file)://)?'
@@ -30,7 +39,7 @@ LOCAL_REGEX = re.compile(
 
 class Requirement(object):
     """
-    Represents a single requirementfrom trains_agent.external.requirements_parser.requirement import Requirement
+    Represents a single requirementfrom clearml_agent.external.requirements_parser.requirement import Requirement
 
     Typically instances of this class are created with ``Requirement.parse``.
     For local file requirements, there's no verification that the file
@@ -100,7 +109,7 @@ class Requirement(object):
 
         req = cls('-e {0}'.format(line))
         req.editable = True
-        vcs_match = VCS_REGEX.match(line)
+        vcs_match = VCS_REGEX.match(line) or VCS_EXT_REGEX.match(line)
         local_match = LOCAL_REGEX.match(line)
 
         if vcs_match is not None:
@@ -147,7 +156,7 @@ class Requirement(object):
 
         req = cls(line)
 
-        vcs_match = VCS_REGEX.match(line)
+        vcs_match = VCS_REGEX.match(line) or VCS_EXT_REGEX.match(line)
         uri_match = URI_REGEX.match(line)
         local_match = LOCAL_REGEX.match(line)
 
@@ -226,7 +235,7 @@ class Requirement(object):
                 # check if the name is valid & parsed
                 Req.parse(name)
                 # if we are here, name is a valid package name, check if the vcs part is valid
-                if VCS_REGEX.match(uri):
+                if VCS_REGEX.match(uri) or VCS_EXT_REGEX.match(uri):
                     req = cls.parse_line(uri)
                     req.name = name
                     return req

clearml_agent/glue/k8s.py

@@ -1,52 +1,61 @@
 from __future__ import print_function, division, unicode_literals
 
 import base64
+import functools
+import json
 import logging
 import os
 import re
 import subprocess
 import tempfile
 from copy import deepcopy
+from pathlib import Path
+from threading import Thread
+from time import sleep
+from typing import Text, List, Callable, Any, Collection, Optional, Union
 
 import yaml
-import json
-from time import sleep
-from typing import Text, List
 
-from trains_agent.commands.events import Events
-from trains_agent.commands.worker import Worker
-from trains_agent.errors import APIError
-from trains_agent.helper.base import safe_remove_file
-from trains_agent.helper.dicts import merge_dicts
-from trains_agent.helper.process import get_bash_output
-from trains_agent.helper.resource_monitor import ResourceMonitor
-from trains_agent.interface.base import ObjectID
+from clearml_agent.commands.events import Events
+from clearml_agent.commands.worker import Worker, get_task_container
+from clearml_agent.definitions import ENV_DOCKER_IMAGE
+from clearml_agent.errors import APIError
+from clearml_agent.helper.base import safe_remove_file
+from clearml_agent.helper.dicts import merge_dicts
+from clearml_agent.helper.process import get_bash_output
+from clearml_agent.helper.resource_monitor import ResourceMonitor
+from clearml_agent.interface.base import ObjectID
 
 
 class K8sIntegration(Worker):
     K8S_PENDING_QUEUE = "k8s_scheduler"
 
-    KUBECTL_APPLY_CMD = "kubectl apply -f"
+    K8S_DEFAULT_NAMESPACE = "clearml"
+    AGENT_LABEL = "CLEARML=agent"
+    LIMIT_POD_LABEL = "ai.allegro.agent.serial=pod-{pod_number}"
 
-    KUBECTL_RUN_CMD = "kubectl run trains-{queue_name}-id-{task_id} " \
-                      "--image {docker_image} " \
-                      "--restart=Never --replicas=1 " \
-                      "--generator=run-pod/v1 " \
-                      "--namespace=trains"
+    KUBECTL_APPLY_CMD = "kubectl apply --namespace={namespace} -f"
+
+    KUBECTL_RUN_CMD = "kubectl run clearml-id-{task_id} " \
+                      "--image {docker_image} {docker_args} " \
+                      "--restart=Never " \
+                      "--namespace={namespace}"
 
     KUBECTL_DELETE_CMD = "kubectl delete pods " \
-                         "--selector=TRAINS=agent " \
+                         "--selector={selector} " \
                          "--field-selector=status.phase!=Pending,status.phase!=Running " \
-                         "--namespace=trains"
+                         "--namespace={namespace}"
 
     BASH_INSTALL_SSH_CMD = [
+        "apt-get update",
         "apt-get install -y openssh-server",
         "mkdir -p /var/run/sshd",
         "echo 'root:training' | chpasswd",
         "echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config",
         "sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config",
         r"sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd",
-        "echo 'AcceptEnv TRAINS_API_ACCESS_KEY TRAINS_API_SECRET_KEY' >> /etc/ssh/sshd_config",
+        "echo 'AcceptEnv TRAINS_API_ACCESS_KEY TRAINS_API_SECRET_KEY CLEARML_API_ACCESS_KEY CLEARML_API_SECRET_KEY' "
+        ">> /etc/ssh/sshd_config",
         'echo "export VISIBLE=now" >> /etc/profile',
         'echo "export PATH=$PATH" >> /etc/profile',
         'echo "ldconfig" >> /etc/profile',
@@ -63,14 +72,12 @@ class K8sIntegration(Worker):
         "export LOCAL_PYTHON=$(which python3.$i) && break ; done",
         "[ ! -z $LOCAL_PYTHON ] || apt-get install -y python3-pip",
         "[ ! -z $LOCAL_PYTHON ] || export LOCAL_PYTHON=python3",
-        "$LOCAL_PYTHON -m pip install trains-agent",
+        "$LOCAL_PYTHON -m pip install clearml-agent",
         "{extra_bash_init_cmd}",
-        "$LOCAL_PYTHON -m trains_agent execute --full-monitoring --require-queue --id {task_id}"
+        "{extra_docker_bash_script}",
+        "$LOCAL_PYTHON -m clearml_agent execute --full-monitoring --require-queue --id {task_id}"
     ]
 
-    AGENT_LABEL = "TRAINS=agent"
-    LIMIT_POD_LABEL = "ai.allegro.agent.serial=pod-{pod_number}"
-
     _edit_hyperparams_version = "2.9"
 
     def __init__(
@@ -81,11 +88,15 @@ class K8sIntegration(Worker):
             debug=False,
             ports_mode=False,
             num_of_services=20,
+            base_pod_num=1,
             user_props_cb=None,
             overrides_yaml=None,
             template_yaml=None,
-            trains_conf_file=None,
+            clearml_conf_file=None,
             extra_bash_init_script=None,
+            namespace=None,
+            max_pods_limit=None,
+            **kwargs
     ):
         """
         Initialize the k8s integration glue layer daemon
@@ -93,7 +104,7 @@ class K8sIntegration(Worker):
         :param str k8s_pending_queue_name: queue name to use when task is pending in the k8s scheduler
         :param str|callable kubectl_cmd: kubectl command line str, supports formatting (default: KUBECTL_RUN_CMD)
             example: "task={task_id} image={docker_image} queue_id={queue_id}"
-            or a callable function: kubectl_cmd(task_id, docker_image, queue_id, task_data)
+            or a callable function: kubectl_cmd(task_id, docker_image, docker_args, queue_id, task_data)
         :param str container_bash_script: container bash script to be executed in k8s (default: CONTAINER_BASH_SCRIPT)
             Notice this string will use format() call, if you have curly brackets they should be doubled { -> {{
             Format arguments passed: {task_id} and {extra_bash_init_cmd}
@@ -102,14 +113,17 @@ class K8sIntegration(Worker):
             Requires the `num_of_services` parameter.
         :param int num_of_services: Number of k8s services configured in the cluster. Required if `port_mode` is True.
             (default: 20)
+        :param int base_pod_num: Used when `ports_mode` is True, sets the base pod number to a given value (default: 1)
         :param callable user_props_cb: An Optional callable allowing additional user properties to be specified
             when scheduling a task to run in a pod. Callable can receive an optional pod number and should return
             a dictionary of user properties (name and value). Signature is [[Optional[int]], Dict[str,str]]
         :param str overrides_yaml: YAML file containing the overrides for the pod (optional)
         :param str template_yaml: YAML file containing the template  for the pod (optional).
             If provided the pod is scheduled with kubectl apply and overrides are ignored, otherwise with kubectl run.
-        :param str trains_conf_file: trains.conf file to be use by the pod itself (optional)
+        :param str clearml_conf_file: clearml.conf file to be use by the pod itself (optional)
         :param str extra_bash_init_script: Additional bash script to run before starting the Task inside the container
+        :param str namespace: K8S namespace to be used when creating the new pods (default: clearml)
+        :param int max_pods_limit: Maximum number of pods that K8S glue can run at the same time
         """
         super(K8sIntegration, self).__init__()
         self.k8s_pending_queue_name = k8s_pending_queue_name or self.K8S_PENDING_QUEUE
@@ -123,16 +137,19 @@ class K8sIntegration(Worker):
             self.log.logger.setLevel(logging.INFO)
         self.ports_mode = ports_mode
         self.num_of_services = num_of_services
+        self.base_pod_num = base_pod_num
         self._edit_hyperparams_support = None
         self._user_props_cb = user_props_cb
-        self.trains_conf_file = None
+        self.conf_file_content = None
         self.overrides_json_string = None
         self.template_dict = None
         self.extra_bash_init_script = extra_bash_init_script or None
         if self.extra_bash_init_script and not isinstance(self.extra_bash_init_script, str):
             self.extra_bash_init_script = ' ; '.join(self.extra_bash_init_script)  # noqa
+        self.namespace = namespace or self.K8S_DEFAULT_NAMESPACE
         self.pod_limits = []
         self.pod_requests = []
+        self.max_pods_limit = max_pods_limit if not self.ports_mode else None
         if overrides_yaml:
             with open(os.path.expandvars(os.path.expanduser(str(overrides_yaml))), 'rt') as f:
                 overrides = yaml.load(f, Loader=getattr(yaml, 'FullLoader', None))
@@ -159,11 +176,59 @@ class K8sIntegration(Worker):
             with open(os.path.expandvars(os.path.expanduser(str(template_yaml))), 'rt') as f:
                 self.template_dict = yaml.load(f, Loader=getattr(yaml, 'FullLoader', None))
 
-        if trains_conf_file:
-            with open(os.path.expandvars(os.path.expanduser(str(trains_conf_file))), 'rt') as f:
-                self.trains_conf_file = f.read()
+        clearml_conf_file = clearml_conf_file or kwargs.get('trains_conf_file')
+
+        if clearml_conf_file:
+            with open(os.path.expandvars(os.path.expanduser(str(clearml_conf_file))), 'rt') as f:
+                self.conf_file_content = f.read()
             # make sure we use system packages!
-            self.trains_conf_file += '\nagent.package_manager.system_site_packages=true\n'
+            self.conf_file_content += '\nagent.package_manager.system_site_packages=true\n'
+
+        self._monitor_hanging_pods()
+
+    def _monitor_hanging_pods(self):
+        _check_pod_thread = Thread(target=self._monitor_hanging_pods_daemon)
+        _check_pod_thread.daemon = True
+        _check_pod_thread.start()
+
+    def _monitor_hanging_pods_daemon(self):
+        while True:
+            output = get_bash_output('kubectl get pods -n {namespace} -o=JSON'.format(
+                namespace=self.namespace
+            ))
+            output = '' if not output else output if isinstance(output, str) else output.decode('utf-8')
+            try:
+                output_config = json.loads(output)
+            except Exception as ex:
+                self.log.warning('K8S Glue pods monitor: Failed parsing kubectl output:\n{}\nEx: {}'.format(output, ex))
+                sleep(self._polling_interval)
+                continue
+            pods = output_config.get('items', [])
+            for pod in pods:
+                try:
+                    reason = functools.reduce(
+                        lambda a, b: a[b], ('status', 'containerStatuses', 0, 'state', 'waiting', 'reason'), pod
+                    )
+                except (IndexError, KeyError):
+                    continue
+                if reason == 'ImagePullBackOff':
+                    pod_name = pod.get('metadata', {}).get('name', None)
+                    if pod_name:
+                        task_id = pod_name.rpartition('-')[-1]
+                        delete_pod_cmd = 'kubectl delete pods {} -n {}'.format(pod_name, self.namespace)
+                        get_bash_output(delete_pod_cmd)
+                        try:
+                            self._session.api_client.tasks.failed(
+                                task=task_id,
+                                status_reason="K8S glue error due to ImagePullBackOff",
+                                status_message="Changed by K8S glue",
+                                force=True
+                            )
+                        except Exception as ex:
+                            self.log.warning(
+                                'K8S Glue pods monitor: Failed deleting task "{}"\nEX: {}'.format(task_id, ex)
+                            )
+            sleep(self._polling_interval)
 
     def _set_task_user_properties(self, task_id: str, **properties: str):
         if self._edit_hyperparams_support is not True:
@@ -210,21 +275,20 @@ class K8sIntegration(Worker):
                 task_id, self.k8s_pending_queue_name, e))
             return
 
-        if task_data.execution.docker_cmd:
-            docker_parts = task_data.execution.docker_cmd
-        else:
-            docker_parts = str(os.environ.get("TRAINS_DOCKER_IMAGE") or
-                               self._session.config.get("agent.default_docker.image", "nvidia/cuda"))
+        container = get_task_container(self._session, task_id)
+        if not container.get('image'):
+            container['image'] = str(
+                ENV_DOCKER_IMAGE.get() or self._session.config.get("agent.default_docker.image", "nvidia/cuda")
+            )
+            container['arguments'] = self._session.config.get("agent.default_docker.arguments", None)
 
-        # take the first part, this is the docker image name (not arguments)
-        docker_parts = docker_parts.split()
-        docker_image = docker_parts[0]
-        docker_args = docker_parts[1:] if len(docker_parts) > 1 else []
-
-        # get the trains.conf encoded file
+        # get the clearml.conf encoded file
         # noinspection PyProtectedMember
-        hocon_config_encoded = (self.trains_conf_file or self._session._config_file).encode('ascii')
-        create_trains_conf = "echo '{}' | base64 --decode >> ~/trains.conf".format(
+        hocon_config_encoded = (
+            self.conf_file_content
+            or Path(self._session._config_file).read_text()
+        ).encode("ascii")
+        create_clearml_conf = "echo '{}' | base64 --decode >> ~/clearml.conf".format(
             base64.b64encode(
                 hocon_config_encoded
             ).decode('ascii')
@@ -239,17 +303,22 @@ class K8sIntegration(Worker):
         except Exception:
             queue_name = 'k8s'
 
-        # conform queue name to k8s standards
-        safe_queue_name = queue_name.lower().strip()
-        safe_queue_name = re.sub(r'\W+', '', safe_queue_name).replace('_', '').replace('-', '')
-
         # Search for a free pod number
-        pod_number = 1
-        while self.ports_mode:
-            kubectl_cmd_new = "kubectl get pods -l {pod_label},{agent_label} -n trains".format(
-                pod_label=self.LIMIT_POD_LABEL.format(pod_number=pod_number),
-                agent_label=self.AGENT_LABEL
-            )
+        pod_count = 0
+        pod_number = self.base_pod_num
+        while self.ports_mode or self.max_pods_limit:
+            pod_number = self.base_pod_num + pod_count
+            if self.ports_mode:
+                kubectl_cmd_new = "kubectl get pods -l {pod_label},{agent_label} -n {namespace}".format(
+                    pod_label=self.LIMIT_POD_LABEL.format(pod_number=pod_number),
+                    agent_label=self.AGENT_LABEL,
+                    namespace=self.namespace,
+                )
+            else:
+                kubectl_cmd_new = "kubectl get pods -l {agent_label} -n {namespace} -o json".format(
+                    agent_label=self.AGENT_LABEL,
+                    namespace=self.namespace,
+                )
             process = subprocess.Popen(kubectl_cmd_new.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
             output, error = process.communicate()
             output = '' if not output else output if isinstance(output, str) else output.decode('utf-8')
@@ -258,48 +327,85 @@ class K8sIntegration(Worker):
             if not output:
                 # No such pod exist so we can use the pod_number we found
                 break
-            if pod_number >= self.num_of_services:
-                # All pod numbers are taken, exit
+
+            if self.max_pods_limit:
+                try:
+                    current_pod_count = len(json.loads(output).get("items", []))
+                except (ValueError, TypeError) as ex:
+                    self.log.warning(
+                        "K8S Glue pods monitor: Failed parsing kubectl output:\n{}\ntask '{}' "
+                        "will be enqueued back to queue '{}'\nEx: {}".format(
+                            output, task_id, queue, ex
+                        )
+                    )
+                    self._session.api_client.tasks.reset(task_id)
+                    self._session.api_client.tasks.enqueue(task_id, queue=queue, status_reason='kubectl parsing error')
+                    return
+                max_count = self.max_pods_limit
+            else:
+                current_pod_count = pod_count
+                max_count = self.num_of_services - 1
+
+            if current_pod_count >= max_count:
+                # All pods are taken, exit
+                self.log.debug(
+                    "kubectl last result: {}\n{}".format(error, output))
                 self.log.warning(
-                    "kubectl last result: {}\n{}\nAll k8s services are in use, task '{}' "
+                    "All k8s services are in use, task '{}' "
                     "will be enqueued back to queue '{}'".format(
-                        error, output, task_id, queue
+                        task_id, queue
                     )
                 )
                 self._session.api_client.tasks.reset(task_id)
                 self._session.api_client.tasks.enqueue(
                     task_id, queue=queue, status_reason='k8s max pod limit (no free k8s service)')
                 return
-            pod_number += 1
+            elif self.max_pods_limit:
+                # max pods limit hasn't reached yet, so we can create the pod
+                break
+            pod_count += 1
 
         labels = ([self.LIMIT_POD_LABEL.format(pod_number=pod_number)] if self.ports_mode else []) + [self.AGENT_LABEL]
+        labels.append("clearml-agent-queue={}".format(self._safe_k8s_label_value(queue)))
+        labels.append("clearml-agent-queue-name={}".format(self._safe_k8s_label_value(queue_name)))
 
         if self.ports_mode:
-            print("Kubernetes scheduling task id={} on pod={}".format(task_id, pod_number))
+            print("Kubernetes scheduling task id={} on pod={} (pod_count={})".format(task_id, pod_number, pod_count))
         else:
             print("Kubernetes scheduling task id={}".format(task_id))
 
+        kubectl_kwargs = dict(
+            create_clearml_conf=create_clearml_conf,
+            labels=labels,
+            docker_image=container['image'],
+            docker_args=container['arguments'],
+            docker_bash=container.get('setup_shell_script'),
+            task_id=task_id,
+            queue=queue
+        )
+
         if self.template_dict:
-            output, error = self._kubectl_apply(
-                create_trains_conf=create_trains_conf,
-                labels=labels, docker_image=docker_image, docker_args=docker_args,
-                task_id=task_id, queue=queue, queue_name=safe_queue_name)
+            output, error = self._kubectl_apply(**kubectl_kwargs)
         else:
-            output, error = self._kubectl_run(
-                create_trains_conf=create_trains_conf,
-                labels=labels, docker_image=docker_image,
-                task_data=task_data,
-                task_id=task_id, queue=queue, queue_name=safe_queue_name)
+            output, error = self._kubectl_run(task_data=task_data, **kubectl_kwargs)
 
         error = '' if not error else (error if isinstance(error, str) else error.decode('utf-8'))
         output = '' if not output else (output if isinstance(output, str) else output.decode('utf-8'))
         print('kubectl output:\n{}\n{}'.format(error, output))
         if error:
-            self.log.error("Running kubectl encountered an error: {}".format(error))
+            send_log = "Running kubectl encountered an error: {}".format(error)
+            self.log.error(send_log)
+            self.send_logs(task_id, send_log.splitlines())
 
         user_props = {"k8s-queue": str(queue_name)}
         if self.ports_mode:
-            user_props.update({"k8s-pod-number": pod_number, "k8s-pod-label": labels[0]})
+            user_props.update(
+                {
+                    "k8s-pod-number": pod_number,
+                    "k8s-pod-label": labels[0],
+                    "k8s-internal-pod-count": pod_count,
+                }
+            )
 
         if self._user_props_cb:
             # noinspection PyBroadException
@@ -315,40 +421,66 @@ class K8sIntegration(Worker):
                 **user_props
             )
 
-    def _parse_docker_args(self, docker_args):
-        # type: (list) -> dict
-        kube_args = {'env': []}
-        while docker_args:
-            cmd = docker_args.pop().strip()
-            if cmd in ('-e', '--env',):
-                env = docker_args.pop().strip()
-                key, value = env.split('=', 1)
-                kube_args[key] += {key: value}
+    def _get_docker_args(self, docker_args, flags, target=None, convert=None):
+        # type: (List[str], Collection[str], Optional[str], Callable[[str], Any]) -> Union[dict, List[str]]
+        """
+        Get docker args matching specific flags.
+
+        :argument docker_args: List of docker argument strings (flags and values)
+        :argument flags: List of flags/names to intercept (e.g. "--env" etc.)
+        :argument target: Controls return format. If provided, returns a dict with a target field containing a list
+         of result strings, otherwise returns a list of result strings
+        :argument convert: Optional conversion function for each result string
+        """
+        args = docker_args[:] if docker_args else []
+        results = []
+        while args:
+            cmd = args.pop(0).strip()
+            if cmd in flags:
+                env = args.pop(0).strip()
+                if convert:
+                    env = convert(env)
+                results.append(env)
             else:
                 self.log.warning('skipping docker argument {} (only -e --env supported)'.format(cmd))
-        return kube_args
+        if target:
+            return {target: results} if results else {}
+        return results
 
-    def _kubectl_apply(self, create_trains_conf, docker_image, docker_args, labels, queue, task_id, queue_name):
+    def _kubectl_apply(self, create_clearml_conf, docker_image, docker_args, docker_bash, labels, queue, task_id):
         template = deepcopy(self.template_dict)
         template.setdefault('apiVersion', 'v1')
         template['kind'] = 'Pod'
         template.setdefault('metadata', {})
-        name = 'trains-{queue}-id-{task_id}'.format(queue=queue_name, task_id=task_id)
+        name = 'clearml-id-{task_id}'.format(task_id=task_id)
         template['metadata']['name'] = name
         template.setdefault('spec', {})
         template['spec'].setdefault('containers', [])
+        template['spec'].setdefault('restartPolicy', 'Never')
         if labels:
             labels_dict = dict(pair.split('=', 1) for pair in labels)
             template['metadata'].setdefault('labels', {})
             template['metadata']['labels'].update(labels_dict)
-        container = self._parse_docker_args(docker_args)
+
+        container = self._get_docker_args(
+            docker_args,
+            target="env",
+            flags={"-e", "--env"},
+            convert=lambda env: {'name': env.partition("=")[0], 'value': env.partition("=")[2]},
+        )
 
         container_bash_script = [self.container_bash_script] if isinstance(self.container_bash_script, str) \
             else self.container_bash_script
 
+        extra_docker_bash_script = '\n'.join(self._session.config.get("agent.extra_docker_shell_script", None) or [])
+        if docker_bash:
+            extra_docker_bash_script += '\n' + str(docker_bash) + '\n'
+
         script_encoded = '\n'.join(
             ['#!/bin/bash', ] +
-            [line.format(extra_bash_init_cmd=self.extra_bash_init_script or '', task_id=task_id)
+            [line.format(extra_bash_init_cmd=self.extra_bash_init_script or '',
+                         task_id=task_id,
+                         extra_docker_bash_script=extra_docker_bash_script)
              for line in container_bash_script])
 
         create_init_script = \
@@ -358,19 +490,24 @@ class K8sIntegration(Worker):
                     script_encoded.encode('ascii')
                 ).decode('ascii'))
 
-        container = merge_dicts(
+        # Notice: we always leave with exit code 0, so pods are never restarted
+        container = self._merge_containers(
             container,
             dict(name=name, image=docker_image,
                  command=['/bin/bash'],
-                 args=['-c', '{} ; {}'.format(create_trains_conf, create_init_script)])
+                 args=['-c', '{} ; {} ; exit 0'.format(create_clearml_conf, create_init_script)])
         )
 
         if template['spec']['containers']:
-            template['spec']['containers'][0] = merge_dicts(template['spec']['containers'][0], container)
+            template['spec']['containers'][0] = self._merge_containers(template['spec']['containers'][0], container)
         else:
             template['spec']['containers'].append(container)
 
-        fp, yaml_file = tempfile.mkstemp(prefix='trains_k8stmpl_', suffix='.yml')
+        if self._docker_force_pull:
+            for c in template['spec']['containers']:
+                c.setdefault('imagePullPolicy', 'Always')
+
+        fp, yaml_file = tempfile.mkstemp(prefix='clearml_k8stmpl_', suffix='.yml')
         os.close(fp)
         with open(yaml_file, 'wt') as f:
             yaml.dump(template, f)
@@ -379,6 +516,7 @@ class K8sIntegration(Worker):
             task_id=task_id,
             docker_image=docker_image,
             queue_id=queue,
+            namespace=self.namespace
         )
         # make sure we provide a list
         if isinstance(kubectl_cmd, str):
@@ -396,15 +534,20 @@ class K8sIntegration(Worker):
 
         return output, error
 
-    def _kubectl_run(self, create_trains_conf, docker_image, labels, queue, task_data, task_id, queue_name):
+    def _kubectl_run(
+        self, create_clearml_conf, docker_image, docker_args, docker_bash, labels, queue, task_data, task_id
+    ):
         if callable(self.kubectl_cmd):
-            kubectl_cmd = self.kubectl_cmd(task_id, docker_image, queue, task_data, queue_name)
+            kubectl_cmd = self.kubectl_cmd(task_id, docker_image, docker_args, queue, task_data)
         else:
             kubectl_cmd = self.kubectl_cmd.format(
-                queue_name=queue_name,
                 task_id=task_id,
                 docker_image=docker_image,
-                queue_id=queue
+                docker_args=" ".join(self._get_docker_args(
+                    docker_args, flags={"-e", "--env"}, convert=lambda env: '--env={}'.format(env))
+                ),
+                queue_id=queue,
+                namespace=self.namespace,
             )
         # make sure we provide a list
         if isinstance(kubectl_cmd, str):
@@ -418,6 +561,9 @@ class K8sIntegration(Worker):
         if self.pod_requests:
             kubectl_cmd += ['--requests', ",".join(self.pod_requests)]
 
+        if self._docker_force_pull and not any(x.startswith("--image-pull-policy=") for x in kubectl_cmd):
+            kubectl_cmd += ["--image-pull-policy='always'"]
+
         container_bash_script = [self.container_bash_script] if isinstance(self.container_bash_script, str) \
             else self.container_bash_script
         container_bash_script = ' ; '.join(container_bash_script)
@@ -428,8 +574,11 @@ class K8sIntegration(Worker):
             "--",
             "/bin/sh",
             "-c",
-            "{} ; {}".format(create_trains_conf, container_bash_script.format(
-                extra_bash_init_cmd=self.extra_bash_init_script, task_id=task_id)),
+            "{} ; {}".format(create_clearml_conf, container_bash_script.format(
+                extra_bash_init_cmd=self.extra_bash_init_script or "",
+                extra_docker_bash_script=docker_bash or "",
+                task_id=task_id
+            )),
         ]
         process = subprocess.Popen(kubectl_cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
         output, error = process.communicate()
@@ -444,7 +593,7 @@ class K8sIntegration(Worker):
         :param queues: IDs of queues to pull tasks from
         :type queues: list of ``Text``
         :param worker_params: Worker command line arguments
-        :type worker_params: ``trains_agent.helper.process.WorkerParams``
+        :type worker_params: ``clearml_agent.helper.process.WorkerParams``
         """
         events_service = self.get_service(Events)
 
@@ -462,7 +611,7 @@ class K8sIntegration(Worker):
             # iterate over queues (priority style, queues[0] is highest)
             for queue in queues:
                 # delete old completed / failed pods
-                get_bash_output(self.KUBECTL_DELETE_CMD)
+                get_bash_output(self.KUBECTL_DELETE_CMD.format(namespace=self.namespace, selector=self.AGENT_LABEL))
 
                 # get next task in queue
                 try:
@@ -514,3 +663,27 @@ class K8sIntegration(Worker):
     @classmethod
     def get_ssh_server_bash(cls, ssh_port_number):
         return ' ; '.join(line.format(port=ssh_port_number) for line in cls.BASH_INSTALL_SSH_CMD)
+
+    @staticmethod
+    def _merge_containers(c1, c2):
+        def merge_env(k, d1, d2, not_set):
+            if k != "env":
+                return not_set
+            # Merge environment lists, second list overrides first
+            return list({
+                item['name']: item for envs in (d1, d2) for item in envs
+            }.values())
+
+        return merge_dicts(
+            c1, c2, custom_merge_func=merge_env
+        )
+
+    @staticmethod
+    def _safe_k8s_label_value(value):
+        """ Conform string to k8s standards for a label value """
+        value = value.lower().strip()
+        value = re.sub(r'^[^A-Za-z0-9]+', '', value)  # strip leading non-alphanumeric chars
+        value = re.sub(r'[^A-Za-z0-9]+$', '', value)  # strip trailing non-alphanumeric chars
+        value = re.sub(r'\W+', '-', value)  # allow only word chars (this removed "." which is supported, but nvm)
+        value = re.sub(r'-+', '-', value)  # don't leave messy "--" after replacing previous chars
+        return value[:63]

clearml_agent/helper/base.py

@@ -1,4 +1,4 @@
-""" TRAINS-AGENT Stdout Helper Functions  """
+""" CLEARML-AGENT Stdout Helper Functions  """
 from __future__ import print_function, unicode_literals
 
 import io
@@ -24,12 +24,11 @@ import pyhocon
 import yaml
 from attr import fields_dict
 from pathlib2 import Path
-from tqdm import tqdm
 
 import six
 from six.moves import reduce
-from trains_agent.errors import CommandFailedError
-from trains_agent.helper.dicts import filter_keys
+from clearml_agent.errors import CommandFailedError
+from clearml_agent.helper.dicts import filter_keys
 
 pretty_lines = False
 
@@ -380,11 +379,11 @@ AllDumper.add_multi_representer(object, lambda dumper, data: dumper.represent_st
 
 
 def error(message):
-    print('\ntrains_agent: ERROR: {}\n'.format(message))
+    print('\nclearml_agent: ERROR: {}\n'.format(message))
 
 
 def warning(message):
-    print('trains_agent: Warning: {}'.format(message))
+    print('clearml_agent: Warning: {}'.format(message))
 
 
 class TqdmStream(object):
@@ -399,12 +398,6 @@ class TqdmStream(object):
         self.buffer.write('\n')
 
 
-class TqdmLog(tqdm):
-
-    def __init__(self, iterable=None, file=None, **kwargs):
-        super(TqdmLog, self).__init__(iterable, file=TqdmStream(file or sys.stderr), **kwargs)
-
-
 def url_join(first, *rest):
     """
     Join url parts similarly to Path.join

clearml_agent/helper/check_update.py

@@ -21,14 +21,14 @@ def start_check_update_daemon():
 
 def _check_new_version_available():
     cur_version = __version__
-    update_server_releases = requests.get('https://updates.trains.allegro.ai/updates',
-                                          data=json.dumps({"versions": {"trains-agent": str(cur_version)}}),
+    update_server_releases = requests.get('https://updates.clear.ml/updates',
+                                          data=json.dumps({"versions": {"clearml-agent": str(cur_version)}}),
                                           timeout=3.0)
     if update_server_releases.ok:
         update_server_releases = update_server_releases.json()
     else:
         return None
-    trains_answer = update_server_releases.get("trains-agent", {})
+    trains_answer = update_server_releases.get("clearml-agent", {})
     latest_version = trains_answer.get("version")
     cur_version = cur_version
     latest_version = latest_version or ''
@@ -48,7 +48,7 @@ def _check_update_daemon():
             if latest_version:
                 if latest_version[1]:
                     sep = os.linesep
-                    print('TRAINS-AGENT new package available: UPGRADE to v{} is recommended!\nRelease Notes:\n{}'.format(
+                    print('CLEARML-AGENT new package available: UPGRADE to v{} is recommended!\nRelease Notes:\n{}'.format(
                         latest_version[0], sep.join(latest_version[2])))
                 else:
                     print('TRAINS-SERVER new version available: upgrade to v{} is recommended!'.format(

clearml_agent/helper/console.py

@@ -9,7 +9,7 @@ from attr import attrs, attrib
 
 import six
 from six import binary_type, text_type
-from trains_agent.helper.base import nonstrict_in_place_sort
+from clearml_agent.helper.base import nonstrict_in_place_sort
 
 
 def print_text(text, newline=True):

clearml_agent/helper/dicts.py

@@ -0,0 +1,23 @@
+from typing import Callable, Dict, Any, Optional
+
+_not_set = object()
+
+
+def filter_keys(filter_, dct):  # type: (Callable[[Any], bool], Dict) -> Dict
+    return {key: value for key, value in dct.items() if filter_(key)}
+
+
+def merge_dicts(dict1, dict2, custom_merge_func=None):
+    # type: (Any, Any, Optional[Callable[[str, Any, Any, Any], Any]]) -> Any
+    """ Recursively merges dict2 into dict1 """
+    if not isinstance(dict1, dict) or not isinstance(dict2, dict):
+        return dict2
+    for k in dict2:
+        if k in dict1:
+            res = None
+            if custom_merge_func:
+                res = custom_merge_func(k, dict1[k], dict2[k], _not_set)
+            dict1[k] = merge_dicts(dict1[k], dict2[k], custom_merge_func) if res is _not_set else res
+        else:
+            dict1[k] = dict2[k]
+    return dict1

clearml_agent/helper/gpu/gpustat.py

@@ -20,6 +20,7 @@ import platform
 import sys
 import time
 from datetime import datetime
+from typing import Optional
 
 import psutil
 from ..gpu import pynvml as N
@@ -390,3 +391,38 @@ def new_query(shutdown=False, per_process_stats=False, get_driver_info=False):
     '''
     return GPUStatCollection.new_query(shutdown=shutdown, per_process_stats=per_process_stats,
                                        get_driver_info=get_driver_info)
+
+
+def get_driver_cuda_version():
+    # type: () -> Optional[str]
+    """
+    :return: Return detected CUDA version from driver. On fail return value is None.
+        Example: `110` is cuda version 11.0
+    """
+    # noinspection PyBroadException
+    try:
+        N.nvmlInit()
+    except BaseException:
+        return None
+
+    # noinspection PyBroadException
+    try:
+        cuda_version = str(N.nvmlSystemGetCudaDriverVersion())
+    except BaseException:
+        # noinspection PyBroadException
+        try:
+            cuda_version = str(N.nvmlSystemGetCudaDriverVersion_v2())
+        except BaseException:
+            cuda_version = ''
+
+    # noinspection PyBroadException
+    try:
+        N.nvmlShutdown()
+    except BaseException:
+        return None
+
+    # for some reason we get CUDA version 11020 instead of 11200, so this is the fix
+    if cuda_version and len(cuda_version) >= 4 and cuda_version[2] == '0' and cuda_version[3] != '0':
+        return cuda_version[:2]+cuda_version[3]
+
+    return cuda_version[:3] if cuda_version else None

clearml_agent/helper/os/folder_cache.py

@@ -0,0 +1,225 @@
+import os
+import shutil
+from logging import warning
+from random import random
+from time import time
+from typing import List, Optional, Sequence
+
+import psutil
+from pathlib2 import Path
+
+from .locks import FileLock
+
+
+class FolderCache(object):
+    _lock_filename = '.clearml.lock'
+    _lock_timeout_seconds = 30
+    _temp_entry_prefix = '_temp.'
+
+    def __init__(self, cache_folder, max_cache_entries=5, min_free_space_gb=None):
+        self._cache_folder = Path(os.path.expandvars(cache_folder)).expanduser().absolute()
+        self._cache_folder.mkdir(parents=True, exist_ok=True)
+        self._max_cache_entries = max_cache_entries
+        self._last_copied_entry_folder = None
+        self._min_free_space_gb = min_free_space_gb if min_free_space_gb and min_free_space_gb > 0 else None
+        self._lock = FileLock((self._cache_folder / self._lock_filename).as_posix())
+
+    def get_cache_folder(self):
+        # type: () -> Path
+        """
+        :return: Return the base cache folder
+        """
+        return self._cache_folder
+
+    def copy_cached_entry(self, keys, destination):
+        # type: (List[str], Path) -> Optional[Path]
+        """
+        Copy a cached entry into a destination directory, if the cached entry does not exist return None
+        :param keys:
+        :param destination:
+        :return: Target path, None if cached entry does not exist
+        """
+        self._last_copied_entry_folder = None
+        if not keys:
+            return None
+
+        # lock so we make sure no one deletes it before we copy it
+        # noinspection PyBroadException
+        try:
+            self._lock.acquire(timeout=self._lock_timeout_seconds)
+        except BaseException as ex:
+            warning('Could not lock cache folder {}: {}'.format(self._cache_folder, ex))
+            return None
+
+        src = None
+        try:
+            src = self.get_entry(keys)
+            if src:
+                destination = Path(destination).absolute()
+                destination.mkdir(parents=True, exist_ok=True)
+                shutil.rmtree(destination.as_posix())
+                shutil.copytree(src.as_posix(), dst=destination.as_posix(), symlinks=True)
+        except BaseException as ex:
+            warning('Could not copy cache folder {} to {}: {}'.format(src, destination, ex))
+            self._lock.release()
+            return None
+
+        # release Lock
+        self._lock.release()
+
+        self._last_copied_entry_folder = src
+        return destination if src else None
+
+    def get_entry(self, keys):
+        # type: (List[str]) -> Optional[Path]
+        """
+        Return a folder (a sub-folder of inside the cache_folder) matching one of the keys
+        :param keys: List of keys, return the first match to one of the keys, notice keys cannot contain '.'
+        :return: Path to the sub-folder or None if none was found
+        """
+        if not keys:
+            return None
+        # conform keys
+        keys = [keys] if isinstance(keys, str) else keys
+        keys = sorted([k.replace('.', '_') for k in keys])
+        for cache_folder in self._cache_folder.glob('*'):
+            if cache_folder.is_dir() and any(True for k in cache_folder.name.split('.') if k in keys):
+                cache_folder.touch()
+                return cache_folder
+        return None
+
+    def add_entry(self, keys, source_folder, exclude_sub_folders=None):
+        # type: (List[str], Path, Optional[Sequence[str]]) -> bool
+        """
+        Add a local folder into the cache, copy all sub-folders inside `source_folder`
+        excluding folders matching `exclude_sub_folders` list
+        :param keys: Cache entry keys list (str)
+        :param source_folder: Folder to copy into the cache
+        :param exclude_sub_folders: List of sub-folders to exclude from the copy operation
+        :return: return True is new entry was added to cache
+        """
+        if not keys:
+            return False
+
+        keys = [keys] if isinstance(keys, str) else keys
+        keys = sorted([k.replace('.', '_') for k in keys])
+
+        # If entry already exists skip it
+        cached_entry = self.get_entry(keys)
+        if cached_entry:
+            # make sure the entry contains all keys
+            cached_keys = cached_entry.name.split('.')
+            if set(keys) - set(cached_keys):
+                # noinspection PyBroadException
+                try:
+                    self._lock.acquire(timeout=self._lock_timeout_seconds)
+                except BaseException as ex:
+                    warning('Could not lock cache folder {}: {}'.format(self._cache_folder, ex))
+                    # failed locking do nothing
+                    return True
+                keys = sorted(list(set(keys) | set(cached_keys)))
+                dst = cached_entry.parent / '.'.join(keys)
+                # rename
+                try:
+                    shutil.move(src=cached_entry.as_posix(), dst=dst.as_posix())
+                except BaseException as ex:
+                    warning('Could not rename cache entry {} to {}: ex'.format(
+                        cached_entry.as_posix(), dst.as_posix(), ex))
+                # release lock
+                self._lock.release()
+            return True
+
+        # make sure we remove old entries
+        self._remove_old_entries()
+
+        # if we do not have enough free space, do nothing.
+        if not self._check_min_free_space():
+            warning('Could not add cache entry, not enough free space on drive, '
+                    'free space threshold {} GB. Clearing all cache entries!'.format(self._min_free_space_gb))
+            self._remove_old_entries(max_cache_entries=0)
+            return False
+
+        # create the new entry for us
+        exclude_sub_folders = exclude_sub_folders or []
+        source_folder = Path(source_folder).absolute()
+        # create temp folder
+        temp_folder = \
+            self._temp_entry_prefix + \
+            '{}.{}'.format(str(time()).replace('.', '_'), str(random()).replace('.', '_'))
+        temp_folder = self._cache_folder / temp_folder
+        temp_folder.mkdir(parents=True, exist_ok=False)
+
+        for f in source_folder.glob('*'):
+            if f.name in exclude_sub_folders:
+                continue
+            if f.is_dir():
+                shutil.copytree(
+                    src=f.as_posix(), dst=(temp_folder / f.name).as_posix(),
+                    symlinks=True, ignore_dangling_symlinks=True)
+            else:
+                shutil.copy(
+                    src=f.as_posix(), dst=(temp_folder / f.name).as_posix(),
+                    follow_symlinks=False)
+
+        # rename the target folder
+        target_cache_folder = self._cache_folder / '.'.join(keys)
+        # if we failed moving it means someone else created the cached entry before us, we can just leave
+        # noinspection PyBroadException
+        try:
+            shutil.move(src=temp_folder.as_posix(), dst=target_cache_folder.as_posix())
+        except BaseException:
+            # noinspection PyBroadException
+            try:
+                shutil.rmtree(path=temp_folder.as_posix())
+            except BaseException:
+                return False
+
+        return True
+
+    def get_last_copied_entry(self):
+        # type: () -> Optional[Path]
+        """
+        :return: the last copied cached entry folder inside the cache
+        """
+        return self._last_copied_entry_folder
+
+    def _remove_old_entries(self, max_cache_entries=None):
+        # type: (Optional[int]) -> ()
+        """
+        Notice we only keep self._max_cache_entries-1, assuming we will be adding a new entry soon
+        :param int max_cache_entries: if not None use instead of self._max_cache_entries
+        """
+        folder_entries = [(cache_folder, cache_folder.stat().st_mtime)
+                          for cache_folder in self._cache_folder.glob('*')
+                          if cache_folder.is_dir() and not cache_folder.name.startswith(self._temp_entry_prefix)]
+        folder_entries = sorted(folder_entries, key=lambda x: x[1], reverse=True)
+
+        # lock so we make sure no one deletes it before we copy it
+        # noinspection PyBroadException
+        try:
+            self._lock.acquire(timeout=self._lock_timeout_seconds)
+        except BaseException as ex:
+            warning('Could not lock cache folder {}: {}'.format(self._cache_folder, ex))
+            return
+
+        number_of_entries_to_keep = self._max_cache_entries - 1 \
+            if max_cache_entries is None else max(0, int(max_cache_entries))
+        for folder, ts in folder_entries[number_of_entries_to_keep:]:
+            try:
+                shutil.rmtree(folder.as_posix(), ignore_errors=True)
+            except BaseException as ex:
+                warning('Could not delete cache entry {}: {}'.format(folder.as_posix(), ex))
+
+        self._lock.release()
+
+    def _check_min_free_space(self):
+        # type: () -> bool
+        """
+        :return: return False if we hit the free space limit.
+        If not free space limit provided, always return True
+        """
+        if not self._min_free_space_gb or not self._cache_folder:
+            return True
+        free_space = float(psutil.disk_usage(self._cache_folder.as_posix()).free)
+        free_space /= 2**30
+        return free_space > self._min_free_space_gb

clearml_agent/helper/os/locks.py

@@ -0,0 +1,211 @@
+import os
+import time
+import tempfile
+import contextlib
+
+from .portalocker import constants, exceptions, lock, unlock
+
+
+current_time = getattr(time, "monotonic", time.time)
+
+DEFAULT_TIMEOUT = 10 ** 8
+DEFAULT_CHECK_INTERVAL = 0.25
+LOCK_METHOD = constants.LOCK_EX | constants.LOCK_NB
+
+__all__ = [
+    'FileLock',
+    'open_atomic',
+]
+
+
+@contextlib.contextmanager
+def open_atomic(filename, binary=True):
+    """Open a file for atomic writing. Instead of locking this method allows
+    you to write the entire file and move it to the actual location. Note that
+    this makes the assumption that a rename is atomic on your platform which
+    is generally the case but not a guarantee.
+
+    http://docs.python.org/library/os.html#os.rename
+
+    >>> filename = 'test_file.txt'
+    >>> if os.path.exists(filename):
+    ...     os.remove(filename)
+
+    >>> with open_atomic(filename) as fh:
+    ...     written = fh.write(b'test')
+    >>> assert os.path.exists(filename)
+    >>> os.remove(filename)
+
+    """
+    assert not os.path.exists(filename), '%r exists' % filename
+    path, name = os.path.split(filename)
+
+    # Create the parent directory if it doesn't exist
+    if path and not os.path.isdir(path):  # pragma: no cover
+        os.makedirs(path)
+
+    temp_fh = tempfile.NamedTemporaryFile(
+        mode=binary and 'wb' or 'w',
+        dir=path,
+        delete=False,
+    )
+    yield temp_fh
+    temp_fh.flush()
+    os.fsync(temp_fh.fileno())
+    temp_fh.close()
+    try:
+        os.rename(temp_fh.name, filename)
+    finally:
+        try:
+            os.remove(temp_fh.name)
+        except Exception:  # noqa
+            pass
+
+
+class FileLock(object):
+
+    def __init__(
+            self, filename, mode='a', timeout=DEFAULT_TIMEOUT,
+            check_interval=DEFAULT_CHECK_INTERVAL, fail_when_locked=False,
+            flags=LOCK_METHOD, **file_open_kwargs):
+        """Lock manager with build-in timeout
+
+        filename -- filename
+        mode -- the open mode, 'a' or 'ab' should be used for writing
+        truncate -- use truncate to emulate 'w' mode, None is disabled, 0 is
+            truncate to 0 bytes
+        timeout -- timeout when trying to acquire a lock
+        check_interval -- check interval while waiting
+        fail_when_locked -- after the initial lock failed, return an error
+            or lock the file
+        **file_open_kwargs -- The kwargs for the `open(...)` call
+
+        fail_when_locked is useful when multiple threads/processes can race
+        when creating a file. If set to true than the system will wait till
+        the lock was acquired and then return an AlreadyLocked exception.
+
+        Note that the file is opened first and locked later. So using 'w' as
+        mode will result in truncate _BEFORE_ the lock is checked.
+        """
+
+        if 'w' in mode:
+            truncate = True
+            mode = mode.replace('w', 'a')
+        else:
+            truncate = False
+
+        self.fh = None
+        self.filename = filename
+        self.mode = mode
+        self.truncate = truncate
+        self.timeout = timeout
+        self.check_interval = check_interval
+        self.fail_when_locked = fail_when_locked
+        self.flags = flags
+        self.file_open_kwargs = file_open_kwargs
+
+    def acquire(
+            self, timeout=None, check_interval=None, fail_when_locked=None):
+        """Acquire the locked filehandle"""
+        if timeout is None:
+            timeout = self.timeout
+        if timeout is None:
+            timeout = 0
+
+        if check_interval is None:
+            check_interval = self.check_interval
+
+        if fail_when_locked is None:
+            fail_when_locked = self.fail_when_locked
+
+        # If we already have a filehandle, return it
+        fh = self.fh
+        if fh:
+            return fh
+
+        # Get a new filehandler
+        fh = self._get_fh()
+        try:
+            # Try to lock
+            fh = self._get_lock(fh)
+        except exceptions.LockException as exception:
+            # Try till the timeout has passed
+            timeoutend = current_time() + timeout
+            while timeoutend > current_time():
+                # Wait a bit
+                time.sleep(check_interval)
+
+                # Try again
+                try:
+
+                    # We already tried to the get the lock
+                    # If fail_when_locked is true, then stop trying
+                    if fail_when_locked:
+                        raise exceptions.AlreadyLocked(exception)
+
+                    else:  # pragma: no cover
+                        # We've got the lock
+                        fh = self._get_lock(fh)
+                        break
+
+                except exceptions.LockException:
+                    pass
+
+            else:
+                # We got a timeout... reraising
+                raise exceptions.LockException(exception)
+
+        # Prepare the filehandle (truncate if needed)
+        fh = self._prepare_fh(fh)
+
+        self.fh = fh
+        return fh
+
+    def release(self):
+        """Releases the currently locked file handle"""
+        if self.fh:
+            # noinspection PyBroadException
+            try:
+                unlock(self.fh)
+            except Exception:
+                pass
+            # noinspection PyBroadException
+            try:
+                self.fh.close()
+            except Exception:
+                pass
+            self.fh = None
+
+    def _get_fh(self):
+        """Get a new filehandle"""
+        return open(self.filename, self.mode, **self.file_open_kwargs)
+
+    def _get_lock(self, fh):
+        """
+        Try to lock the given filehandle
+
+        returns LockException if it fails"""
+        lock(fh, self.flags)
+        return fh
+
+    def _prepare_fh(self, fh):
+        """
+        Prepare the filehandle for usage
+
+        If truncate is a number, the file will be truncated to that amount of
+        bytes
+        """
+        if self.truncate:
+            fh.seek(0)
+            fh.truncate(0)
+
+        return fh
+
+    def __enter__(self):
+        return self.acquire()
+
+    def __exit__(self, type_, value, tb):
+        self.release()
+
+    def __delete__(self, instance):  # pragma: no cover
+        instance.release()

clearml_agent/helper/os/portalocker.py

@@ -0,0 +1,193 @@
+import os
+import sys
+
+
+class exceptions:
+    class BaseLockException(Exception):
+        # Error codes:
+        LOCK_FAILED = 1
+
+        def __init__(self, *args, **kwargs):
+            self.fh = kwargs.pop('fh', None)
+            Exception.__init__(self, *args, **kwargs)
+
+    class LockException(BaseLockException):
+        pass
+
+    class AlreadyLocked(BaseLockException):
+        pass
+
+    class FileToLarge(BaseLockException):
+        pass
+
+
+class constants:
+    # The actual tests will execute the code anyhow so the following code can
+    # safely be ignored from the coverage tests
+    if os.name == 'nt':  # pragma: no cover
+        import msvcrt
+
+        LOCK_EX = 0x1  #: exclusive lock
+        LOCK_SH = 0x2  #: shared lock
+        LOCK_NB = 0x4  #: non-blocking
+        LOCK_UN = msvcrt.LK_UNLCK  #: unlock
+
+        LOCKFILE_FAIL_IMMEDIATELY = 1
+        LOCKFILE_EXCLUSIVE_LOCK = 2
+
+    elif os.name == 'posix':  # pragma: no cover
+        import fcntl
+
+        LOCK_EX = fcntl.LOCK_EX  #: exclusive lock
+        LOCK_SH = fcntl.LOCK_SH  #: shared lock
+        LOCK_NB = fcntl.LOCK_NB  #: non-blocking
+        LOCK_UN = fcntl.LOCK_UN  #: unlock
+
+    else:  # pragma: no cover
+        raise RuntimeError('PortaLocker only defined for nt and posix platforms')
+
+
+if os.name == 'nt':  # pragma: no cover
+    import msvcrt
+
+    if sys.version_info.major == 2:
+        lock_length = -1
+    else:
+        lock_length = int(2**31 - 1)
+
+    def lock(file_, flags):
+        if flags & constants.LOCK_SH:
+            import win32file
+            import pywintypes
+            import winerror
+            __overlapped = pywintypes.OVERLAPPED()
+            if sys.version_info.major == 2:
+                if flags & constants.LOCK_NB:
+                    mode = constants.LOCKFILE_FAIL_IMMEDIATELY
+                else:
+                    mode = 0
+
+            else:
+                if flags & constants.LOCK_NB:
+                    mode = msvcrt.LK_NBRLCK
+                else:
+                    mode = msvcrt.LK_RLCK
+
+            # is there any reason not to reuse the following structure?
+            hfile = win32file._get_osfhandle(file_.fileno())
+            try:
+                win32file.LockFileEx(hfile, mode, 0, -0x10000, __overlapped)
+            except pywintypes.error as exc_value:
+                # error: (33, 'LockFileEx', 'The process cannot access the file
+                # because another process has locked a portion of the file.')
+                if exc_value.winerror == winerror.ERROR_LOCK_VIOLATION:
+                    raise exceptions.LockException(
+                        exceptions.LockException.LOCK_FAILED,
+                        exc_value.strerror,
+                        fh=file_)
+                else:
+                    # Q:  Are there exceptions/codes we should be dealing with
+                    # here?
+                    raise
+        else:
+            mode = constants.LOCKFILE_EXCLUSIVE_LOCK
+            if flags & constants.LOCK_NB:
+                mode |= constants.LOCKFILE_FAIL_IMMEDIATELY
+
+            if flags & constants.LOCK_NB:
+                mode = msvcrt.LK_NBLCK
+            else:
+                mode = msvcrt.LK_LOCK
+
+            # windows locks byte ranges, so make sure to lock from file start
+            try:
+                savepos = file_.tell()
+                if savepos:
+                    # [ ] test exclusive lock fails on seek here
+                    # [ ] test if shared lock passes this point
+                    file_.seek(0)
+                    # [x] check if 0 param locks entire file (not documented in
+                    #     Python)
+                    # [x] fails with "IOError: [Errno 13] Permission denied",
+                    #     but -1 seems to do the trick
+
+                try:
+                    msvcrt.locking(file_.fileno(), mode, lock_length)
+                except IOError as exc_value:
+                    # [ ] be more specific here
+                    raise exceptions.LockException(
+                        exceptions.LockException.LOCK_FAILED,
+                        exc_value.strerror,
+                        fh=file_)
+                finally:
+                    if savepos:
+                        file_.seek(savepos)
+            except IOError as exc_value:
+                raise exceptions.LockException(
+                    exceptions.LockException.LOCK_FAILED, exc_value.strerror,
+                    fh=file_)
+
+    def unlock(file_):
+        try:
+            savepos = file_.tell()
+            if savepos:
+                file_.seek(0)
+
+            try:
+                msvcrt.locking(file_.fileno(), constants.LOCK_UN, lock_length)
+            except IOError as exc_value:
+                if exc_value.strerror == 'Permission denied':
+                    import pywintypes
+                    import win32file
+                    import winerror
+                    __overlapped = pywintypes.OVERLAPPED()
+                    hfile = win32file._get_osfhandle(file_.fileno())
+                    try:
+                        win32file.UnlockFileEx(
+                            hfile, 0, -0x10000, __overlapped)
+                    except pywintypes.error as exc_value:
+                        if exc_value.winerror == winerror.ERROR_NOT_LOCKED:
+                            # error: (158, 'UnlockFileEx',
+                            #         'The segment is already unlocked.')
+                            # To match the 'posix' implementation, silently
+                            # ignore this error
+                            pass
+                        else:
+                            # Q:  Are there exceptions/codes we should be
+                            # dealing with here?
+                            raise
+                else:
+                    raise exceptions.LockException(
+                        exceptions.LockException.LOCK_FAILED,
+                        exc_value.strerror,
+                        fh=file_)
+            finally:
+                if savepos:
+                    file_.seek(savepos)
+        except IOError as exc_value:
+            raise exceptions.LockException(
+                exceptions.LockException.LOCK_FAILED, exc_value.strerror,
+                fh=file_)
+
+elif os.name == 'posix':  # pragma: no cover
+    import fcntl
+
+    def lock(file_, flags):
+        locking_exceptions = IOError,
+        try:  # pragma: no cover
+            locking_exceptions += BlockingIOError,
+        except NameError:  # pragma: no cover
+            pass
+
+        try:
+            fcntl.flock(file_.fileno(), flags)
+        except locking_exceptions as exc_value:
+            # The exception code varies on different systems so we'll catch
+            # every IO error
+            raise exceptions.LockException(exc_value, fh=file_)
+
+    def unlock(file_):
+        fcntl.flock(file_.fileno(), constants.LOCK_UN)
+
+else:  # pragma: no cover
+    raise RuntimeError('PortaLocker only defined for nt and posix platforms')

clearml_agent/helper/package/base.py

@@ -0,0 +1,263 @@
+from __future__ import unicode_literals
+
+import abc
+from collections import OrderedDict
+from contextlib import contextmanager
+from typing import Text, Iterable, Union, Optional, Dict, List
+from pathlib2 import Path
+from hashlib import md5
+
+import six
+from clearml_agent.helper.base import mkstemp, safe_remove_file, join_lines, select_for_platform
+from clearml_agent.helper.console import ensure_binary
+from clearml_agent.helper.os.folder_cache import FolderCache
+from clearml_agent.helper.process import Executable, Argv, PathLike
+
+
+@six.add_metaclass(abc.ABCMeta)
+class PackageManager(object):
+    """
+    ABC for classes providing python package management interface
+    """
+
+    _selected_manager = None
+    _cwd = None
+    _pip_version = None
+    _config_cache_folder = 'agent.venvs_cache.path'
+    _config_cache_max_entries = 'agent.venvs_cache.max_entries'
+    _config_cache_free_space_threshold = 'agent.venvs_cache.free_space_threshold_gb'
+
+    def __init__(self):
+        self._cache_manager = None
+
+    @abc.abstractproperty
+    def bin(self):
+        # type: () -> PathLike
+        pass
+
+    @abc.abstractmethod
+    def create(self):
+        pass
+
+    @abc.abstractmethod
+    def remove(self):
+        pass
+
+    @abc.abstractmethod
+    def install_from_file(self, path):
+        pass
+
+    @abc.abstractmethod
+    def freeze(self):
+        pass
+
+    @abc.abstractmethod
+    def load_requirements(self, requirements):
+        pass
+
+    @abc.abstractmethod
+    def install_packages(self, *packages):
+        # type: (Iterable[Text]) -> None
+        """
+        Install packages, upgrading depends on config
+        """
+        pass
+
+    @abc.abstractmethod
+    def _install(self, *packages):
+        # type: (Iterable[Text]) -> None
+        """
+        Run install command
+        """
+        pass
+
+    @abc.abstractmethod
+    def uninstall_packages(self, *packages):
+        # type: (Iterable[Text]) -> None
+        pass
+
+    def upgrade_pip(self):
+        result = self._install(
+            select_for_platform(windows='pip{}', linux='pip{}').format(self.get_pip_version()), "--upgrade")
+        packages = self.run_with_env(('list',), output=True).splitlines()
+        # p.split is ('pip', 'x.y.z')
+        pip = [p.split() for p in packages if len(p.split()) == 2 and p.split()[0] == 'pip']
+        if pip:
+            # noinspection PyBroadException
+            try:
+                from .requirements import MarkerRequirement
+                pip = pip[0][1].split('.')
+                MarkerRequirement.pip_new_version = bool(int(pip[0]) >= 20)
+            except Exception:
+                pass
+        return result
+
+    def get_python_command(self, extra=()):
+        # type: (...) -> Executable
+        return Argv(self.bin, *extra)
+
+    @contextmanager
+    def temp_file(self, prefix, contents, suffix=".txt"):
+        # type: (Union[Text, Iterable[Text]], Iterable[Text], Text) -> Text
+        """
+        Write contents to a temporary file, yielding its path. Finally, delete it.
+        :param prefix: file name prefix
+        :param contents: text lines to write
+        :param suffix: file name suffix
+        """
+        f, temp_path = mkstemp(suffix=suffix, prefix=prefix)
+        with f:
+            f.write(
+                contents
+                if isinstance(contents, six.text_type)
+                else join_lines(contents)
+            )
+        try:
+            yield temp_path
+        finally:
+            if not self.session.debug_mode:
+                safe_remove_file(temp_path)
+
+    def set_selected_package_manager(self):
+        # set this instance as the selected package manager
+        # this is helpful when we want out of context requirement installations
+        PackageManager._selected_manager = self
+
+    @property
+    def cwd(self):
+        return self._cwd
+
+    @cwd.setter
+    def cwd(self, value):
+        self._cwd = value
+
+    @classmethod
+    def out_of_scope_install_package(cls, package_name, *args):
+        if PackageManager._selected_manager is not None:
+            try:
+                result = PackageManager._selected_manager._install(package_name, *args)
+                if result not in (0, None, True):
+                    return False
+            except Exception:
+                return False
+        return True
+
+    @classmethod
+    def out_of_scope_freeze(cls):
+        if PackageManager._selected_manager is not None:
+            try:
+                return PackageManager._selected_manager.freeze()
+            except Exception:
+                pass
+        return []
+
+    @classmethod
+    def set_pip_version(cls, version):
+        if not version:
+            return
+        version = version.replace(' ', '')
+        if ('=' in version) or ('~' in version) or ('<' in version) or ('>' in version):
+            cls._pip_version = version
+        else:
+            cls._pip_version = "=="+version
+
+    @classmethod
+    def get_pip_version(cls):
+        return cls._pip_version or ''
+
+    def get_cached_venv(self, requirements, docker_cmd, python_version, cuda_version, destination_folder):
+        # type: (Dict, Optional[Union[dict, str]], Optional[str], Optional[str], Path) -> Optional[Path]
+        """
+        Copy a cached copy of the venv (based on the requirements) into destination_folder.
+        Return None if failed or cached entry does not exist
+        """
+        if not self._get_cache_manager():
+            return None
+
+        keys = self._generate_reqs_hash_keys(requirements, docker_cmd, python_version, cuda_version)
+        return self._get_cache_manager().copy_cached_entry(keys, destination_folder)
+
+    def add_cached_venv(
+            self,
+            requirements,  # type: Union[Dict, List[Dict]]
+            docker_cmd,   # type: Optional[Union[dict, str]]
+            python_version,  # type: Optional[str]
+            cuda_version,  # type: Optional[str]
+            source_folder,  # type:  Path
+            exclude_sub_folders=None  # type: Optional[List[str]]
+    ):
+        # type: (...) -> ()
+        """
+        Copy the local venv folder into the venv cache (keys are based on the requirements+python+docker).
+        """
+        if not self._get_cache_manager():
+            return
+        keys = self._generate_reqs_hash_keys(requirements, docker_cmd, python_version, cuda_version)
+        return self._get_cache_manager().add_entry(
+            keys=keys, source_folder=source_folder, exclude_sub_folders=exclude_sub_folders)
+
+    def get_cache_folder(self):
+        # type: () -> Optional[Path]
+        if not self._get_cache_manager():
+            return
+        return self._get_cache_manager().get_cache_folder()
+
+    def get_last_used_entry_cache(self):
+        # type: () -> Optional[Path]
+        """
+        :return: the last used cached folder entry
+        """
+        if not self._get_cache_manager():
+            return
+        return self._get_cache_manager().get_last_copied_entry()
+
+    @classmethod
+    def _generate_reqs_hash_keys(cls, requirements_list, docker_cmd, python_version, cuda_version):
+        # type: (Union[Dict, List[Dict]], Optional[Union[dict, str]], Optional[str], Optional[str]) -> List[str]
+        requirements_list = requirements_list or dict()
+        if not isinstance(requirements_list, (list, tuple)):
+            requirements_list = [requirements_list]
+        docker_cmd = dict(docker_cmd=docker_cmd) if isinstance(docker_cmd, str) else docker_cmd or dict()
+        docker_cmd = OrderedDict(sorted(docker_cmd.items(), key=lambda t: t[0]))
+        if 'docker_cmd' in docker_cmd:
+            # we only take the first part of the docker_cmd which is the docker image name
+            docker_cmd['docker_cmd'] = docker_cmd['docker_cmd'].strip('\r\n\t ').split(' ')[0]
+
+        keys = []
+        strip_chars = '\n\r\t '
+        for requirements in requirements_list:
+            pip, conda = ('pip', 'conda')
+            pip_reqs = requirements.get(pip, '')
+            conda_reqs = requirements.get(conda, '')
+            if isinstance(pip_reqs, str):
+                pip_reqs = pip_reqs.split('\n')
+            if isinstance(conda_reqs, str):
+                conda_reqs = conda_reqs.split('\n')
+            pip_reqs = sorted([p.strip(strip_chars) for p in pip_reqs
+                               if p.strip(strip_chars) and not p.strip(strip_chars).startswith('#')])
+            conda_reqs = sorted([p.strip(strip_chars) for p in conda_reqs
+                                 if p.strip(strip_chars) and not p.strip(strip_chars).startswith('#')])
+            if not pip_reqs and not conda_reqs:
+                continue
+            hash_text = '{class_type}\n{docker_cmd}\n{cuda_ver}\n{python_version}\n{pip_reqs}\n{conda_reqs}'.format(
+                class_type=str(cls),
+                docker_cmd=str(docker_cmd or ''),
+                cuda_ver=str(cuda_version or ''),
+                python_version=str(python_version or ''),
+                pip_reqs=str(pip_reqs or ''),
+                conda_reqs=str(conda_reqs or ''),
+            )
+            keys.append(md5(ensure_binary(hash_text)).hexdigest())
+        return sorted(list(set(keys)))
+
+    def _get_cache_manager(self):
+        if not self._cache_manager:
+            cache_folder = self.session.config.get(self._config_cache_folder, None)
+            if not cache_folder:
+                return None
+
+            max_entries = int(self.session.config.get(self._config_cache_max_entries, 10))
+            free_space_threshold = float(self.session.config.get(self._config_cache_free_space_threshold, 0))
+            self._cache_manager = FolderCache(
+                cache_folder, max_cache_entries=max_entries, min_free_space_gb=free_space_threshold)
+        return self._cache_manager

clearml_agent/helper/package/conda_api.py

@@ -15,14 +15,14 @@ import yaml
 from time import time
 from attr import attrs, attrib, Factory
 from pathlib2 import Path
-from trains_agent.external.requirements_parser import parse
-from trains_agent.external.requirements_parser.requirement import Requirement
+from clearml_agent.external.requirements_parser import parse
+from clearml_agent.external.requirements_parser.requirement import Requirement
 
-from trains_agent.errors import CommandFailedError
-from trains_agent.helper.base import rm_tree, NonStrictAttrs, select_for_platform, is_windows_platform, ExecutionInfo
-from trains_agent.helper.process import Argv, Executable, DEVNULL, CommandSequence, PathLike
-from trains_agent.helper.package.requirements import SimpleVersion
-from trains_agent.session import Session
+from clearml_agent.errors import CommandFailedError
+from clearml_agent.helper.base import rm_tree, NonStrictAttrs, select_for_platform, is_windows_platform, ExecutionInfo
+from clearml_agent.helper.process import Argv, Executable, DEVNULL, CommandSequence, PathLike
+from clearml_agent.helper.package.requirements import SimpleVersion
+from clearml_agent.session import Session
 from .base import PackageManager
 from .pip_api.venv import VirtualenvPip
 from .requirements import RequirementsManager, MarkerRequirement
@@ -69,6 +69,7 @@ class CondaAPI(PackageManager):
         :param python: base python version to use (e.g python3.6)
         :param path: path of env
         """
+        super(CondaAPI, self).__init__()
         self.session = session
         self.python = python
         self.source = None
@@ -132,7 +133,7 @@ class CondaAPI(PackageManager):
         if self.env_read_only:
             print('Conda environment in read-only mode, skipping pip upgrade.')
             return ''
-        return self._install(select_for_platform(windows='"pip{}"', linux='pip{}').format(self.pip.get_pip_version()))
+        return self._install(select_for_platform(windows='pip{}', linux='pip{}').format(self.pip.get_pip_version()))
 
     def create(self):
         """
@@ -140,19 +141,7 @@ class CondaAPI(PackageManager):
         """
         if self.conda_env_as_base_docker and self.conda_pre_build_env_path:
             if Path(self.conda_pre_build_env_path).is_dir():
-                print("Using pre-existing Conda environment from {}".format(self.conda_pre_build_env_path))
-                self.path = Path(self.conda_pre_build_env_path)
-                self.source = ("conda", "activate", self.path.as_posix())
-                self.pip = CondaPip(
-                    session=self.session,
-                    source=self.source,
-                    python=self.python,
-                    requirements_manager=self.requirements_manager,
-                    path=self.path,
-                )
-                conda_env = self._get_conda_sh()
-                self.source = self.pip.source = CommandSequence(('source', conda_env.as_posix()), self.source)
-                self.env_read_only = True
+                self._init_existing_environment(self.conda_pre_build_env_path)
                 return self
             elif Path(self.conda_pre_build_env_path).is_file():
                 print("Restoring Conda environment from {}".format(self.conda_pre_build_env_path))
@@ -210,6 +199,21 @@ class CondaAPI(PackageManager):
             pass
         return self
 
+    def _init_existing_environment(self, conda_pre_build_env_path):
+        print("Using pre-existing Conda environment from {}".format(conda_pre_build_env_path))
+        self.path = Path(conda_pre_build_env_path)
+        self.source = ("conda", "activate", self.path.as_posix())
+        self.pip = CondaPip(
+            session=self.session,
+            source=self.source,
+            python=self.python,
+            requirements_manager=self.requirements_manager,
+            path=self.path,
+        )
+        conda_env = self._get_conda_sh()
+        self.source = self.pip.source = CommandSequence(('source', conda_env.as_posix()), self.source)
+        self.env_read_only = True
+
     def remove(self):
         """
         Delete a conda environment.
@@ -284,17 +288,11 @@ class CondaAPI(PackageManager):
         """
         Try to install packages from conda. Install packages which are not available from conda with pip.
         """
-        try:
-            self._install_from_file(path)
-            return
-        except PackageNotFoundError as e:
-            pip_packages = [e.pkg]
-        except PackagesNotFoundError as e:
-            pip_packages = package_set(e.packages)
-        with self.temp_file("conda_reqs", _package_diff(path, pip_packages)) as reqs:
-            self.install_from_file(reqs)
-        with self.temp_file("pip_reqs", pip_packages) as reqs:
-            self.pip.install_from_file(reqs)
+        requirements = {}
+        # assume requirements.txt
+        with open(path, 'rt') as f:
+            requirements['pip'] = f.read()
+        self.load_requirements(requirements)
 
     def freeze(self, freeze_full_environment=False):
         requirements = self.pip.freeze()
@@ -507,6 +505,8 @@ class CondaAPI(PackageManager):
             reqs.append(m)
 
         # if we have a conda list, the rest should be installed with pip,
+        # this means  any experiment that was executed with pip environment,
+        # will be installed using pip
         if requirements.get('conda', None) is not None:
             for r in requirements['pip']:
                 try:
@@ -520,7 +520,7 @@ class CondaAPI(PackageManager):
                 # skip over local files (we cannot change the version to a local file)
                 if m.local_file:
                     continue
-                m_name = m.name.lower()
+                m_name = (m.name or '').lower()
                 if m_name in conda_supported_req_names:
                     # this package is in the conda list,
                     # make sure that if we changed version and we match it in conda
@@ -557,7 +557,7 @@ class CondaAPI(PackageManager):
         # conform conda packages (version/name)
         for r in reqs:
             # change _ to - in name but not the prefix _ (as this is conda prefix)
-            if not r.name.startswith('_') and not requirements.get('conda', None):
+            if r.name and not r.name.startswith('_') and not requirements.get('conda', None):
                 r.name = r.name.replace('_', '-')
             # remove .post from version numbers, it fails ~= version, and change == to ~=
             if r.specs and r.specs[0]:
@@ -573,6 +573,8 @@ class CondaAPI(PackageManager):
             conda_env['dependencies'] = [clean_ver(r) for r in reqs]
             with self.temp_file("conda_env", yaml.dump(conda_env), suffix=".yml") as name:
                 print('Conda: Trying to install requirements:\n{}'.format(conda_env['dependencies']))
+                if self.session.debug_mode:
+                    print('{}:\n{}'.format(name, yaml.dump(conda_env)))
                 result = self._run_command(
                     ("env", "update", "-p", self.path, "--file", name)
                 )
@@ -603,6 +605,8 @@ class CondaAPI(PackageManager):
                 pip_req_str = [r.tostr() for r in pip_requirements if r.name not in ('pip', 'virtualenv', )]
                 print('Conda: Installing requirements: step 2 - using pip:\n{}'.format(pip_req_str))
                 PackageManager._selected_manager = self.pip
+                if self.session.debug_mode:
+                    print('pip requirements.txt:\n{}'.format('\n'.join(pip_req_str)))
                 self.pip.load_requirements({'pip': '\n'.join(pip_req_str)})
             except Exception as e:
                 print(e)
@@ -646,12 +650,16 @@ class CondaAPI(PackageManager):
             ansi_escape = re.compile(r'(?:\x1B[@-_]|[\x80-\x9F])[0-?]*[ -/]*[@-~]')
             return ansi_escape.sub('', line)
 
+        # make sure we are not running it with our own PYTHONPATH
+        env = dict(**os.environ)
+        env.pop('PYTHONPATH', None)
+
         command = Argv(*command)  # type: Executable
         if not raw:
             command = (self.conda,) + command + ("--quiet", "--json")
         try:
             print('Executing Conda: {}'.format(command.serialize()))
-            result = command.get_output(stdin=DEVNULL, **kwargs)
+            result = command.get_output(stdin=DEVNULL, env=env, **kwargs)
             if self.session.debug_mode:
                 print(result)
         except Exception as e:
@@ -671,6 +679,8 @@ class CondaAPI(PackageManager):
         return result
 
     def get_python_command(self, extra=()):
+        if not self.source:
+            self._init_existing_environment(self.path)
         return CommandSequence(self.source, self.pip.get_python_command(extra=extra))
 
     def _get_conda_sh(self):

clearml_agent/helper/package/external_req.py

@@ -2,6 +2,8 @@ import re
 from collections import OrderedDict
 from typing import Text
 
+from pathlib2 import Path
+
 from .base import PackageManager
 from .requirements import SimpleSubstitution
 from ..base import safe_furl as furl
@@ -10,13 +12,27 @@ from ..base import safe_furl as furl
 class ExternalRequirements(SimpleSubstitution):
 
     name = "external_link"
+    cwd = None
 
     def __init__(self, *args, **kwargs):
         super(ExternalRequirements, self).__init__(*args, **kwargs)
         self.post_install_req = []
         self.post_install_req_lookup = OrderedDict()
+        self.post_install_local_req_lookup = OrderedDict()
 
     def match(self, req):
+        # match local folder building:
+        if self.is_local_folder_package(req):
+            # noinspection PyBroadException
+            try:
+                folder_path = req.req.line.strip().split('#')[0].strip()
+                if self.cwd and not Path(folder_path).is_absolute():
+                    folder_path = (Path(self.cwd) / Path(folder_path)).absolute().as_posix()
+                self.post_install_local_req_lookup['file://{}'.format(folder_path)] = req.req.line
+            except Exception:
+                pass
+            return True
+
         # match both editable or code or unparsed
         if not (not req.name or req.req and (req.req.editable or req.req.vcs)):
             return False
@@ -103,4 +119,45 @@ class ExternalRequirements(SimpleSubstitution):
                                        if r not in self.post_install_req_lookup]
             list_of_requirements[k] += [self.post_install_req_lookup.get(r, '')
                                         for r in self.post_install_req_lookup.keys() if r in original_requirements]
+
+            if self.post_install_local_req_lookup:
+                original_requirements = list_of_requirements[k]
+                list_of_requirements[k] = [
+                    r for r in original_requirements
+                    if len(r.split('@', 1)) != 2 or r.split('@', 1)[1].strip() not in self.post_install_local_req_lookup]
+
+                list_of_requirements[k] += [
+                    self.post_install_local_req_lookup.get(r.split('@', 1)[1].strip(), '')
+                    for r in original_requirements
+                    if len(r.split('@', 1)) == 2 and r.split('@', 1)[1].strip() in self.post_install_local_req_lookup]
+
         return list_of_requirements
+
+    @classmethod
+    def is_local_folder_package(cls, req):
+        # noinspection PyBroadException
+        try:
+            if not req.name and req.req and not req.req.editable and not req.req.vcs and \
+                    req.req.line and req.req.line.strip().split('#')[0] and \
+                    not req.req.line.strip().split('#')[0].lower().endswith('.whl'):
+                return True
+        except Exception:
+            pass
+        return False
+
+
+class OnlyExternalRequirements(ExternalRequirements):
+    def __init__(self, *args, **kwargs):
+        super(OnlyExternalRequirements, self).__init__(*args, **kwargs)
+
+    def match(self, req):
+        return not super(OnlyExternalRequirements, self).match(req)
+
+    def replace(self, req):
+        """
+        Replace a requirement
+        :raises: ValueError if version is pre-release
+        """
+        # Do not store the skipped requirements
+        # mark skip package
+        return Text('')

clearml_agent/helper/package/pip_api/system.py

@@ -1,11 +1,12 @@
+import os
 import sys
 from itertools import chain
 from typing import Text, Optional
 
-from trains_agent.definitions import PIP_EXTRA_INDICES, PROGRAM_NAME
-from trains_agent.helper.package.base import PackageManager
-from trains_agent.helper.process import Argv, DEVNULL
-from trains_agent.session import Session
+from clearml_agent.definitions import PIP_EXTRA_INDICES, PROGRAM_NAME
+from clearml_agent.helper.package.base import PackageManager
+from clearml_agent.helper.process import Argv, DEVNULL
+from clearml_agent.session import Session
 
 
 class SystemPip(PackageManager):
@@ -17,6 +18,7 @@ class SystemPip(PackageManager):
         """
         Program interface to the system pip.
         """
+        super(SystemPip, self).__init__()
         self._bin = interpreter or sys.executable
         self.session = session
 
@@ -81,7 +83,10 @@ class SystemPip(PackageManager):
         :param kwargs: kwargs for get_output/check_output command
         """
         command = self._make_command(command)
-        return (command.get_output if output else command.check_call)(stdin=DEVNULL, **kwargs)
+        # make sure we are not running it with our own PYTHONPATH
+        env = dict(**os.environ)
+        env.pop('PYTHONPATH', None)
+        return (command.get_output if output else command.check_call)(stdin=DEVNULL, env=env, **kwargs)
 
     def _make_command(self, command):
         return Argv(self.bin, '-m', 'pip', '--disable-pip-version-check', *command)

clearml_agent/helper/package/pip_api/venv.py

@@ -2,10 +2,10 @@ from typing import Any
 
 from pathlib2 import Path
 
-from trains_agent.helper.base import select_for_platform, rm_tree, ExecutionInfo
-from trains_agent.helper.package.base import PackageManager
-from trains_agent.helper.process import Argv, PathLike
-from trains_agent.session import Session
+from clearml_agent.helper.base import select_for_platform, rm_tree, ExecutionInfo
+from clearml_agent.helper.package.base import PackageManager
+from clearml_agent.helper.process import Argv, PathLike
+from clearml_agent.session import Session
 from ..pip_api.system import SystemPip
 from ..requirements import RequirementsManager
 

clearml_agent/helper/package/poetry_api.py

@@ -5,8 +5,8 @@ import attr
 import sys
 import os
 from pathlib2 import Path
-from trains_agent.helper.process import Argv, DEVNULL, check_if_command_exists
-from trains_agent.session import Session, POETRY
+from clearml_agent.helper.process import Argv, DEVNULL, check_if_command_exists
+from clearml_agent.session import Session, POETRY
 
 
 def prop_guard(prop, log_prop=None):