Version bump to v1.15.0

Store worker statistics under worker id and not internal redis key
Fix unit tests

.gitignore

@@ -1,3 +1,4 @@
+syntax: glob
 .idea
 apierrors/errors
 static/build.json
@@ -11,10 +12,10 @@ test-reports
 .pytest_cache
 venv
 *.noseids
-build
 *.egg-info
 .cache
 .mypy_cache
 dist
 code.tar.gz
 server/schema/services/_cache.json
+server/apierrors/errors/*

LICENSE

@@ -1,7 +1,7 @@
                      Server Side Public License
                      VERSION 1, OCTOBER 16, 2018
 
-                    Copyright © 2018 MongoDB, Inc.
+                    Copyright © 2019 allegro.ai, Inc.
 
   Everyone is permitted to copy and distribute verbatim copies of this
   license document, but changing it is not allowed.

README.md

@@ -1,317 +1,244 @@
-# TRAINS Server
+<div align="center">
 
-##  Auto-Magical Experiment Manager & Version Control for AI
+<img src="docs/clearml_server_logo.png" width="250px">
+
+**ClearML - Auto-Magical Suite of tools to streamline your ML workflow 
+</br>Experiment Manager, ML-Ops and Data-Management**
 
 [![GitHub license](https://img.shields.io/badge/license-SSPL-green.svg)](https://img.shields.io/badge/license-SSPL-green.svg)
 [![Python versions](https://img.shields.io/badge/python-3.6%20%7C%203.7-blue.svg)](https://img.shields.io/badge/python-3.6%20%7C%203.7-blue.svg)
 [![GitHub version](https://img.shields.io/github/release-pre/allegroai/trains-server.svg)](https://img.shields.io/github/release-pre/allegroai/trains-server.svg)
-[![PyPI status](https://img.shields.io/badge/status-beta-yellow.svg)](https://img.shields.io/badge/status-beta-yellow.svg)
+[![Artifact Hub](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/allegroai)](https://artifacthub.io/packages/search?repo=allegroai)
 
-## Introduction
+</div>
 
-The **trains-server** is the backend service infrastructure for [TRAINS](https://github.com/allegroai/trains).
+---
+<div align="center">
+
+**Note regarding Apache Log4j2 Remote Code Execution (RCE) Vulnerability - CVE-2021-44228 - ESA-2021-31**
+
+</div>
+
+According to [ElasticSearch's latest report](https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476), 
+supported versions of Elasticsearch (6.8.9+, 7.8+) used with recent versions of the JDK (JDK9+) **are not susceptible to either remote code execution or information leakage**
+due to Elasticsearch’s usage of the Java Security Manager.
+
+**As the latest version of ClearML Server uses Elasticsearch 7.10+ with JDK15, it is not affected by these vulnerabilities.**
+
+As a precaution, we've upgraded the ES version to 7.16.2 and added the mitigation recommended by ElasticSearch to our latest [docker-compose.yml](https://github.com/allegroai/clearml-server/blob/cfccbe05c158b75e520581f86e9668291da5c70a/docker/docker-compose.yml#L42) file.
+
+While previous Elasticsearch versions (5.6.11+, 6.4.0+ and 7.0.0+) used by older ClearML Server versions are only susceptible to the information leakage vulnerability
+(which in any case **does not permit access to data within the Elasticsearch cluster**), 
+we still recommend upgrading to the latest version of ClearML Server. Alternatively, you can apply the mitigation as implemented in our latest 
+[docker-compose.yml](https://github.com/allegroai/clearml-server/blob/cfccbe05c158b75e520581f86e9668291da5c70a/docker/docker-compose.yml#L42) file.    
+
+**Update 15 December**: A further vulnerability (CVE-2021-45046) was disclosed on December 14th.
+ElasticSearch's guidance for Elasticsearch remains unchanged by this new vulnerability, thus **not affecting ClearML Server**.
+
+**Update 22 December**: To keep with ElasticSearch's recommendations, we've upgraded the ES version to the newly released 7.16.2
+
+---
+
+## ClearML Server
+#### *Formerly known as Trains Server*
+
+The **ClearML Server** is the backend service infrastructure for [ClearML](https://github.com/allegroai/clearml).
 It allows multiple users to collaborate and manage their experiments.
-By default, TRAINS is set up to work with the TRAINS demo server, which is open to anyone and resets periodically. 
-In order to host your own server, you will need to install **trains-server** and point TRAINS to it.
+**ClearML** offers a [free hosted service](https://app.clear.ml/), which is maintained by **ClearML** and open to anyone.
+In order to host your own server, you will need to launch the **ClearML Server** and point **ClearML** to it.
 
-**trains-server** contains the following components:
+The **ClearML Server** contains the following components:
 
-* The TRAINS Web-App, a single-page UI for experiment management and browsing
+* The **ClearML** Web-App, a single-page UI for experiment management and browsing
 * RESTful API for:
     * Documenting and logging experiment information, statistics and results
     * Querying experiments history, logs and results
 * Locally-hosted file server for storing images and models making them easily accessible using the Web-App
 
-You can quickly setup your **trains-server** using a pre-built Docker image (see [Installation](#installation)).
+You can quickly [deploy](#launching-the-clearml-server)  your **ClearML Server** using Docker, AWS EC2 AMI, or Kubernetes. 
 
-When new releases are available, you can upgrade your pre-built Docker image (see [Upgrade](#upgrade)).
-
-## System diagram
+## System design
 
 
-![Alt Text](https://github.com/allegroai/trains/blob/master/docs/system_diagram.png?raw=true)
+![Alt Text](docs/ClearML_Server_Diagram.png)
 
+The **ClearML Server** has two supported configurations:
+- Single IP (domain) with the following open ports
+    - Web application on port 8080
+    - API service on port 8008
+    - File storage service on port 8081
 
-## Install / Upgrade - AWS
-
-Use our pre-installed Amazon Machine Image for easy deployment in AWS. 
-
-Details and instructions can be found  [here](docs/install_aws.md).
-
-## Installation - Docker
-
-This section contains the instructions to setup and launch a pre-built Docker image for the **trains-server**.
-This is the quickest way to get started with your own server. 
-Alternatively, you can build the entire trains-server architecture using the code available in our repositories.
-
-**Please Note**:
-* This Docker image was tested with Linux, only. For Windows users, we recommend running the server
-on a Linux virtual machine.
-
-* All command-line instructions below assume you're using `bash`.
+- Sub-Domain configuration with default http/s ports (80 or 443)
+    - Web application on sub-domain: app.\*.\*
+    - API service on sub-domain: api.\*.\*
+    - File storage service on sub-domain: files.\*.\*
+    
+## Launching The ClearML Server
 
 ### Prerequisites
 
-Make sure you are logged in as a user with sudo privileges.
+The ports 8080/8081/8008 must be available for the **ClearML Server** services.
+   
+For example, to see if port `8080` is in use:
 
-### Setup
+* Linux or macOS: 
+   
+        sudo lsof -Pn -i4 | grep :8080 | grep LISTEN
 
-#### Step 1: Install Docker CE
+* Windows:
 
-In order to run the pre-packaged **trains-server**, install Docker.
-
-* See [Supported platforms](https://docs.docker.com/install//#support) in the Docker documentation for instructions
-
-* For example, to install in [Ubuntu](https://docs.docker.com/install/linux/docker-ce/ubuntu/) / Mint (x86_64/amd64):
-
-    ```bash
-    sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common
-    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
-    . /etc/os-release
-    sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $UBUNTU_CODENAME stable"
-    sudo apt-get update
-    sudo apt-get install -y docker-ce
-    ```
-
-#### Step 2: Setup the Docker daemon
-
-To run the ElasticSearch Docker container, setup the Docker daemon by modifying the default
-values required by Elastic in your Docker configuration file (see [Notes for production use and defaults](https://www.elastic.co/guide/en/elasticsearch/reference/master/docker.html#_notes_for_production_use_and_defaults)). We provide instructions for the most common Docker configuration files.
-
-Edit or create the Docker configuration file:
-
-* If your system contains a `/etc/sysconfig/docker` Docker configuration file, edit it.
-
-    Add the options in quotes to the available arguments in the `OPTIONS` section:
-
-    ```bash
-    OPTIONS="--default-ulimit nofile=1024:65536 --default-ulimit memlock=-1:-1"
-    ```
-
-* Otherwise, edit `/etc/docker/daemon.json` (if it exists) or create it (if it does not exist).
-
-    Add or modify the `defaults-ulimits` section as shown below. Be sure the `defaults-ulimits` section contains the `nofile` and `memlock` sub-sections and values shown.
-
-    **Note**: Your configuration file may contain other sections. If so, confirm that the sections are separated by commas (valid JSON format). For more information about Docker configuration files, see [Daemon configuration file](https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file) in the Docker documentation.
-
-    The **trains-server** required defaults values are:
-
-    ```json
-    {
-        "default-ulimits": {
-            "nofile": {
-                "name": "nofile",
-                "hard": 65536,
-                "soft": 1024
-            },
-            "memlock":
-            {
-                "name": "memlock",
-                "soft": -1,
-                "hard": -1
-            }
-        }
-    }
-    ```
-
-#### Step 3: Restart the Docker daemon
-
-After modifying the configuration file, restart the Docker daemon:
-
-```bash
-sudo service docker stop
-sudo service docker start
-```
-
-#### Step 4: Set the Maximum Number of Memory Map Areas
-
-The maximum number of memory map areas a process can use is defined
-using the `vm.max_map_count` kernel setting.
-
-Elastic requires that `vm.max_map_count` is at least 262144 (see [Production mode](https://www.elastic.co/guide/en/elasticsearch/reference/master/docker.html#docker-cli-run-prod-mode)).
-
-* For CentOS 7, Ubuntu 16.04, Mint 18.3, Ubuntu 18.04 and Mint 19 users, we tested the following commands to set
-`vm.max_map_count`:
-
-    ```bash
-    sudo echo "vm.max_map_count=262144" > /tmp/99-trains.conf
-    sudo mv /tmp/99-trains.conf /etc/sysctl.d/99-trains.conf
-    sudo sysctl -w vm.max_map_count=262144
-    ```
-
-* For information about setting this parameter on other systems, see the [elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-prod-mode) documentation.
-
-#### Step 5: Choose a Data Directory
-
-Choose a directory on your system in which all data maintained by the **trains-server** is stored.
-Create this directory, and set its owner and group to `uid` 1000. The data stored in this directory will include the database, uploaded files and logs.
-
-For example, if your data directory is `/opt/trains`, then use the following command:
-
-```bash
-sudo mkdir -p /opt/trains/data/elastic && sudo chown -R 1000:1000 /opt/trains
-```
-
-### Optional Configuration
-
-The **trains-server** default configuration can be easily overridden using external configuration files. By default, the server will look for these files in `/opt/trains/config`.
-
-If the configuration is changed while the server is running, the server should be restarted for changes to take effect.
-
-
-#### Fixed users mode (basic users management)
-
-In this mode, the server authenticates users based only on a pre-configured users list.
-
-Enable this feature by placing `apiserver.conf` file under `/opt/trains/config`.
- 
-**Notice**: In order for the changes to take effect, the *trains-apiserver* (docker) will need to be restarted.
-
-Sample fixed user configuration file `/opt/trains/config/apiserver.conf`:
-
-    auth {
-        # Fixed users login credetials 
-        # No other user will be able to login
-        fixed_users {
-            enabled: true
-            users: [
-                {
-                    username: "jane"
-                    password: "12345678"
-                    name: "Jane Doe"
-                },
-                {
-                    username: "john"
-                    password: "12345678"
-                    name: "John Doe"
-                },
-            ]
-        }
-    }
- 
-
-#### Non-responsive experiments watchdog
-
-This watchdog monitors experiments that were not updated for a given period of time, and marks them as `stopped`. The watchdog is always active.
-
-To change the watchdog's timeouts, place a `services.conf` file under `/opt/trains/config`, containing for example:
-
-    tasks {
-        non_responsive_tasks_watchdog {
-            # In-progress tasks that haven't been updated for at least 'value' seconds will be stopped by the watchdog
-            threshold_sec: 7200
-        
-            # Watchdog will sleep for this number of seconds after each cycle
-            watch_interval_sec: 900
-        }
-    }
-
-### Launching Docker Containers
-
-**Note**:
-* If your data directory is not `/opt/trains`, please find and replace `/opt/trains` in the following commands with your data directory path
-
-* Make sure ports `8008`, `8080` and `8081` are not in use before starting the docker containers, as the containers will fail to initialize if these ports are already taken. If the following commands shows no output, the ports are available:
-    ```bash
-    sudo netstat -tplna | egrep "8008|8080|8081"
-    ```  
-
-To launch the Docker containers, use the following commands: 
-
-```bash
-sudo docker run -d --restart="always" --name="trains-elastic" -e "ES_JAVA_OPTS=-Xms2g -Xmx2g" -e "bootstrap.memory_lock=true" -e "cluster.name=trains" -e "discovery.zen.minimum_master_nodes=1" -e "node.name=trains" -e "script.inline=true" -e "script.update=true" -e "thread_pool.bulk.queue_size=2000" -e "thread_pool.search.queue_size=10000" -e "xpack.security.enabled=false" -e "xpack.monitoring.enabled=false" -e "cluster.routing.allocation.node_initial_primaries_recoveries=500" -e "node.ingest=true" -e "http.compression_level=7" -e "reindex.remote.whitelist=*.*" -e "script.painless.regex.enabled=true" --network="host" -v /opt/trains/data/elastic:/usr/share/elasticsearch/data docker.elastic.co/elasticsearch/elasticsearch:5.6.16
-```
-
-```bash
-sudo docker run -d --restart="always" --name="trains-mongo" -v /opt/trains/data/mongo/db:/data/db -v /opt/trains/data/mongo/configdb:/data/configdb --network="host" mongo:3.6.5
-```
-
-```bash
-sudo docker run -d --restart="always" --name="trains-fileserver" --network="host" -v /opt/trains/logs:/var/log/trains -v /opt/trains/data/fileserver:/mnt/fileserver allegroai/trains:latest fileserver
-```
-
-```bash
-sudo docker run -d --restart="always" --name="trains-apiserver" --network="host" -v /opt/trains/logs:/var/log/trains -v /opt/trains/config:/opt/trains/config allegroai/trains:latest apiserver
-```
-
-```bash
-sudo docker run -d --restart="always" --name="trains-webserver" -p 8080:80 allegroai/trains:latest webserver
-```
-
-After the **trains-server** Dockers are up, the following are available:
-
-* API server on port `8008`
-* Web server on port `8080`
-* File server on port `8081`
-
-### Configuring **TRAINS** client
-
-Once you have installed the **trains-server**, make sure to configure **TRAINS** [client](https://github.com/allegroai/trains) to use your locally installed server (and not the demo server).
-
-If you have already installed **TRAINS**, run the `trains-init` command for an interactive setup or edit your `trains.conf` file and make sure the `api.host` value is configured as follows:
-
-    api {
-        host: "http://localhost:8008"
-    }
-
-See [Installing and Configuring TRAINS](https://github.com/allegroai/trains#installing-and-configuring-trains) for more details.
-
-## What next?
-
-Now that the **trains-server** is installed, and TRAINS is configured to use it, 
-you can [use](https://github.com/allegroai/trains#using-trains) TRAINS in your experiments and view them in the web server, 
-for example http://localhost:8080
-
-## Upgrade
-
-We are constantly updating, improving and adding to the **trains-server**.
-New releases will include new pre-built Docker images.
-When we release a new version and include a new pre-built Docker image for it, upgrade as follows:
-
-1. Shut down and remove each of your Docker instances using the following commands:
-
-        sudo docker stop <docker-name>
-        sudo docker rm -v <docker-name>
-
-    The Docker names are (see [Launching Docker Containers](#launching-docker-containers)):
-
-    * `trains-elastic`
-    * `trains-mongo`
-    * `trains-fileserver`
-    * `trains-apiserver`
-    * `trains-webserver`
-
-2. Pull the new **trains-server** docker image using the following command:
-
-        sudo docker pull allegroai/trains:latest
+        netstat -an |find /i "8080"
+   
+### Launching   
     
-    If you wish to pull a different version, replace `latest` with the required version number, for example:
+Launch The **ClearML Server** in any of the following formats:
 
-        sudo docker pull allegroai/trains:0.10.0
-        
-3. We highly recommend backing up your data directory!. A simple way to do that is using `tar`:
+- Pre-built [AWS EC2 AMI](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_aws_ec2_ami)
+- Pre-built [GCP Custom Image](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_gcp)
+- Pre-built Docker Image
+    - [Linux](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_linux_mac)
+    - [macOS](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_linux_mac)
+    - [Windows 10](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_win)
+- Kubernetes    
+    - [Kubernetes Helm](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_kubernetes_helm)
+    - Manual [Kubernetes installation](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_kubernetes)
 
-    For example, if your data directory is `/opt/trains`, use the following command:
+## Connecting ClearML to your ClearML Server
 
-        sudo tar czvf ~/trains_backup.tgz /opt/trains/data
+In order to set up the **ClearML** client to work with your **ClearML Server**:
+- Run the `clearml-init` command for an interactive setup.
+- Or manually edit `~/clearml.conf` file, making sure the server settings (`api_server`, `web_server`, `file_server`) are configured correctly, for example:
 
-    This back ups all data to an archive in your home directory.
+        api {
+            # API server on port 8008
+            api_server: "http://localhost:8008"
 
-    To restore this example backup, use the following command:
+            # web_server on port 8080
+            web_server: "http://localhost:8080"
 
-        sudo rm -R /opt/trains/data
-        sudo tar -xzf ~/trains_backup.tgz -C /opt/trains/data
+            # file server on port 8081
+            files_server: "http://localhost:8081"
+        }
 
-4. Launch the newly released Docker image (see [Launching Docker Containers](#launching-docker-containers)).
+**Note**: If you have set up your **ClearML Server** in a sub-domain configuration, then there is no need to specify a port number,
+it will be inferred from the http/s scheme.
+
+After launching the **ClearML Server** and configuring the **ClearML** client to use the **ClearML Server**,
+you can [use](https://github.com/allegroai/clearml) **ClearML** in your experiments and view them in your **ClearML Server** web server,
+for example http://localhost:8080.  
+For more information about the ClearML client, see [**ClearML**](https://github.com/allegroai/clearml).
+
+## ClearML-Agent Services  <a name="services"></a> 
+
+As of version 0.15 of **ClearML Server**, dockerized deployment includes a **ClearML-Agent Services** container running as 
+part of the docker container collection.
+
+ClearML-Agent Services is an extension of ClearML-Agent that provides the ability to launch long-lasting jobs 
+that previously had to be executed on local / dedicated machines. It allows a single agent to 
+launch multiple dockers (Tasks) for different use cases. To name a few use cases, auto-scaler service (spinning instances 
+when the need arises and the budget allows), Controllers (Implementing pipelines and more sophisticated DevOps logic),
+Optimizer (such as Hyper-parameter Optimization or sweeping), and Application (such as interactive Bokeh apps for 
+increased data transparency)
+
+ClearML-Agent Services container will spin **any** task enqueued into the dedicated `services` queue. 
+Every task launched by ClearML-Agent Services  will be registered as a new node in the system, 
+providing tracking and transparency capabilities.  
+You can also run the ClearML-Agent Services manually, see details in [ClearML-agent services mode](https://github.com/allegroai/clearml-agent#clearml-agent-services-mode-)
+
+**Note**: It is the user's responsibility to make sure the proper tasks are pushed into the `services` queue. 
+Do not enqueue training / inference tasks into the `services` queue, as it will put unnecessary load on the server.
+
+## Advanced Functionality
+
+The **ClearML Server** provides a few additional useful features, which can be manually enabled:
+ 
+* [Web login authentication](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_config#web-login-authentication)
+* [Non-responsive experiments watchdog](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_config#non-responsive-task-watchdog)  
+
+## Restarting ClearML Server
+
+To restart the **ClearML Server**, you must first stop the containers, and then restart them.
+
+   ```bash
+   docker-compose down
+   docker-compose -f docker-compose.yml up
+   ```
+
+## Upgrading <a name="upgrade"></a>
+
+**ClearML Server** releases are also reflected in the [docker compose configuration file](https://github.com/allegroai/trains-server/blob/master/docker/docker-compose.yml).  
+We strongly encourage you to keep your **ClearML Server** up to date, by keeping up with the current release.
+
+**Note**: The following upgrade instructions use the Linux OS as an example.
+
+To upgrade your existing **ClearML Server** deployment:
+
+1. Shut down the docker containers
+   ```bash
+   docker-compose down
+   ```
+
+1. We highly recommend backing up your data directory before upgrading.
+
+   Assuming your data directory is `/opt/clearml`, to archive all data into `~/clearml_backup.tgz` execute:
+
+   ```bash
+   sudo tar czvf ~/clearml_backup.tgz /opt/clearml/data
+   ```    
+
+   <details>
+   <summary>Restore instructions:</summary>
+
+   To restore this example backup, execute:
+   ```bash
+   sudo rm -R /opt/clearml/data
+   sudo tar -xzf ~/clearml_backup.tgz -C /opt/clearml/data
+   ```
+   </details>
+
+1. Download the latest `docker-compose.yml` file.
+
+   ```bash
+   curl https://raw.githubusercontent.com/allegroai/trains-server/master/docker/docker-compose.yml -o docker-compose.yml 
+   ```
+
+1. Configure the ClearML-Agent Services (not supported on Windows installation). 
+   If `CLEARML_HOST_IP` is not provided, ClearML-Agent Services will use the external 
+   public address of the **ClearML Server**. If `CLEARML_AGENT_GIT_USER` / `CLEARML_AGENT_GIT_PASS` are not provided, 
+   the ClearML-Agent Services will not be able to access any private repositories for running service tasks.
+   
+   ```bash
+   export CLEARML_HOST_IP=server_host_ip_here
+   export CLEARML_AGENT_GIT_USER=git_username_here
+   export CLEARML_AGENT_GIT_PASS=git_password_here
+   ```
+
+1. Spin up the docker containers, it will automatically pull the latest **ClearML Server** build    
+   ```bash
+   docker-compose -f docker-compose.yml pull
+   docker-compose -f docker-compose.yml up
+   ```
+
+**\* If something went wrong along the way, check our FAQ: [Common Docker Upgrade Errors](https://clear.ml/docs/latest/docs/faq/).**
+
+
+## Community & Support
+
+If you have any questions, look to the ClearML [FAQ](https://clear.ml/docs/latest/docs/faq), or
+tag your questions on [stackoverflow](https://stackoverflow.com/questions/tagged/clearml) with '**clearml**' tag.
+
+For feature requests or bug reports, please use [GitHub issues](https://github.com/allegroai/clearml-server/issues).
+
+Additionally, you can always find us at *clearml@allegro.ai*
 
 ## License
 
 [Server Side Public License v1.0](https://github.com/mongodb/mongo/blob/master/LICENSE-Community.txt)
 
-**trains-server** relies on both [MongoDB](https://github.com/mongodb/mongo) and [ElasticSearch](https://github.com/elastic/elasticsearch).
-With the recent changes in both MongoDB's and ElasticSearch's OSS license, we feel it is our responsibility as a 
+The **ClearML Server** relies on both [MongoDB](https://github.com/mongodb/mongo) and [ElasticSearch](https://github.com/elastic/elasticsearch).
+With the recent changes in both MongoDB's and ElasticSearch's OSS license, we feel it is our responsibility as a
 member of the community to support the projects we love and cherish.
-We believe the cause for the license change in both cases is more than just, 
+We believe the cause for the license change in both cases is more than just,
 and chose [SSPL](https://www.mongodb.com/licensing/server-side-public-license) because it is the more general and flexible of the two licenses.
 
 This is our way to say - we support you guys!

apiserver/LICENSE

@@ -1,7 +1,7 @@
                      Server Side Public License
                      VERSION 1, OCTOBER 16, 2018
 
-                    Copyright © 2018 MongoDB, Inc.
+                    Copyright © 2019 allegro.ai, Inc.
 
   Everyone is permitted to copy and distribute verbatim copies of this
   license document, but changing it is not allowed.

apiserver/apierrors/__init__.py

@@ -0,0 +1,6 @@
+from .apierror import APIError
+from .base import BaseError
+
+from apiserver.apierrors_generator import ErrorsGenerator
+
+ErrorsGenerator.generate_python_files()

apiserver/apierrors/apierror.py

@@ -1,9 +1,10 @@
 class APIError(Exception):
-    def __init__(self, msg, code=500, subcode=0, **_):
+    def __init__(self, msg, code=500, subcode=0, error_data=None, **_):
         super(APIError, self).__init__()
         self._msg = msg
         self._code = code
         self._subcode = subcode
+        self._error_data = error_data or {}
 
     @property
     def msg(self):
@@ -17,5 +18,9 @@ class APIError(Exception):
     def subcode(self):
         return self._subcode
 
+    @property
+    def error_data(self):
+        return self._error_data
+
     def __str__(self):
         return self.msg

apiserver/apierrors/base.py

@@ -1,9 +1,13 @@
-import six
-from boltons.typeutils import classproperty
 from typing import Tuple
 
+import six
+from boltons.iterutils import is_collection, remap
+from boltons.typeutils import classproperty
+
 from .apierror import APIError
 
+jsonable_types = (dict, list, tuple, str, int, float, bool, type(None))
+
 
 class BaseError(APIError):
     _default_code = 500
@@ -19,15 +23,26 @@ class BaseError(APIError):
                 f"{k}={self._format_kwarg(v)}" for k, v in kwargs.items()
             )
             message += f": {kwargs_msg}"
-        params = kwargs.copy()
-        params.update(
-            code=self._default_code, subcode=self._default_subcode, msg=message
+
+        super(BaseError, self).__init__(
+            code=self._default_code,
+            subcode=self._default_subcode,
+            msg=message,
+            error_data=self._to_safe_json_types(kwargs),
         )
-        super(BaseError, self).__init__(**params)
+
+    @staticmethod
+    def _to_safe_json_types(data):
+        def visit(_, k, v):
+            if not isinstance(v, jsonable_types):
+                v = str(v)
+            return k, v
+
+        return remap(data, visit=visit)
 
     @staticmethod
     def _format_kwarg(value):
-        if isinstance(value, (tuple, list)):
+        if is_collection(value):
             return f'({", ".join(str(v) for v in value)})'
         elif isinstance(value, six.string_types):
             return value

apiserver/apierrors/errors.conf

@@ -0,0 +1,156 @@
+301 {
+    _: "moved_permanently"
+    1: ["not_supported", "this endpoint is no longer supported for the requested API version"]
+}
+
+400 {
+    _: "bad_request"
+    1: ["not_supported", "endpoint is not supported"]
+    2: ["request_path_has_invalid_version", "request path has invalid version"]
+    5: ["invalid_headers", "invalid headers"]
+    6: ["impersonation_error", "impersonation error"]
+
+    10: ["invalid_id", "invalid object id"]
+    11: ["missing_required_fields", "missing required fields"]
+    12: ["validation_error", "validation error"]
+    13: ["fields_not_allowed_for_role", "fields not allowed for role"]
+    14: ["invalid fields", "fields not defined for object"]
+    15: ["fields_conflict", "conflicting fields"]
+    16: ["fields_value_error", "invalid value for fields"]
+    17: ["batch_contains_no_items", "batch request contains no items"]
+    18: ["batch_validation_error", "batch request validation error"]
+    19: ["invalid_lucene_syntax", "malformed lucene query"]
+    20: ["fields_type_error", "invalid type for fields"]
+    21: ["invalid_regex_error", "malformed regular expression"]
+    22: ["invalid_email_address", "malformed email address"]
+    23: ["invalid_domain_name", "malformed domain name"]
+    24: ["not_public_object", "object is not public"]
+
+    # Auth / Login
+    75: ["invalid_access_key", "access key not found"]
+
+    # Tasks
+    100: ["task_error", "general task error"]
+    101: ["invalid_task_id", "invalid task id"]
+    102: ["task_validation_error", "task validation error"]
+    110: ["invalid_task_status", "invalid task status"]
+    111: ["task_not_started", "task not started (invalid task status)"]
+    112: ["task_in_progress", "task in progress (invalid task status)"]
+    113: ["task_published", "task published (invalid task status)"]
+    114: ["task_status_unknown", "task unknown (invalid task status)"]
+    120: ["invalid_task_execution_progress", "invalid task execution progress"]
+    121: ["failed_changing_task_status", "failed changing task status. probably someone changed it before you"]
+    122: ["missing_task_fields", "task is missing expected fields"]
+    123: ["task_cannot_be_deleted", "task cannot be deleted"]
+    125: ["task_has_jobs_running", "task has jobs that haven't completed yet"]
+    126: ["invalid_task_type", "invalid task type for this operations"]
+    127: ["invalid_task_input", "invalid task output"]
+    128: ["invalid_task_output", "invalid task output"]
+    129: ["task_publish_in_progress", "Task publish in progress"]
+    130: ["task_not_found", "task not found"]
+    131: ["events_not_added", "events not added"]
+
+    # Reports
+    150: ["operation_supported_on_reports_only", "passed task is not report"]
+
+    # Pipelines
+    160: ["cannot_remove_all_runs", "at least one pipeline run should be left"]
+
+    # Models
+    200: ["model_error", "general task error"]
+    201: ["invalid_model_id", "invalid model id"]
+    202: ["model_not_ready", "model is not ready"]
+    203: ["model_is_ready", "model is ready"]
+    204: ["invalid_model_uri", "invalid model URI"]
+    205: ["model_in_use", "model is used by tasks"]
+    206: ["model_creating_task_exists", "task that created this model exists"]
+
+    # Users
+    300: ["invalid_user", "invalid user"]
+    301: ["invalid_user_id", "invalid user id"]
+    302: ["user_id_exists", "user id already exists"]
+    305: ["invalid_preferences_update", "Malformed key and/or value"]
+
+    # Projects
+    401: ["invalid_project_id", "invalid project id"]
+    402: ["project_has_tasks", "project has associated tasks"]
+    403: ["project_not_found", "project not found"]
+    405: ["project_has_models", "project has associated models"]
+    406: ["project_has_datasets", "project has associated non-empty datasets"]
+    407: ["invalid_project_name", "invalid project name"]
+    408: ["cannot_update_project_location", "Cannot update project location. Use projects.move instead"]
+    409: ["project_path_exceeds_max", "Project path exceed the maximum allowed depth"]
+    410: ["project_source_and_destination_are_the_same", "Project has the same source and destination paths"]
+    411: ["project_cannot_be_moved_under_itself", "Project can not be moved under itself in the projects hierarchy"]
+    412: ["project_cannot_be_merged_into_its_child", "Project can not be merged into its own child"]
+    413: ["project_has_pipelines", "project has associated pipelines with active controllers"]
+
+    # Queues
+    701: ["invalid_queue_id", "invalid queue id"]
+    702: ["queue_not_empty", "queue is not empty"]
+    703: ["invalid_queue_or_task_not_queued", "invalid queue id or task not in queue"]
+    704: ["removed_during_reposition", "task was removed by another party during reposition"]
+    705: ["failed_adding_during_reposition", "failed adding task back to queue during reposition"]
+    706: ["task_already_queued", "failed adding task to queue since task is already queued"]
+    707: ["no_default_queue", "no queue is tagged as the default queue for this company"]
+    708: ["multiple_default_queues", "more than one queue is tagged as the default queue for this company"]
+
+    # Database
+    800: ["data_validation_error", "data validation error"]
+    801: ["expected_unique_data", "value combination already exists (unique field already contains this value)"]
+
+    # Workers
+    1001: ["invalid_worker_id", "invalid worker id"]
+    1002: ["worker_registration_failed", "worker registration failed"]
+    1003: ["worker_registered", "worker is already registered"]
+    1004: ["worker_not_registered", "worker is not registered"]
+    1005: ["worker_stats_not_found", "worker stats not found"]
+
+    1104: ["invalid_scroll_id", "Invalid scroll id"]
+}
+
+401 {
+    _:  "unauthorized"
+    1:  ["not_authorized", "unauthorized (not authorized for endpoint)"]
+    2:  ["entity_not_allowed", "unauthorized (entity not allowed)"]
+    10: ["bad_auth_type", "unauthorized (bad authentication header type)"]
+    20: ["no_credentials", "unauthorized (missing credentials)"]
+    21: ["bad_credentials", "unauthorized (malformed credentials)"]
+    22: ["invalid_credentials", "unauthorized (invalid credentials)"]
+    30: ["invalid_token", "invalid token"]
+    31: ["blocked_token", "token is blocked"]
+    40: ["invalid_fixed_user", "fixed user ID was not found"]
+}
+
+403: {
+    _:  "forbidden"
+    10: ["routing_error", "forbidden (routing error)"]
+    12: ["blocked_internal_endpoint", "forbidden (blocked internal endpoint)"]
+    20: ["role_not_allowed", "forbidden (not allowed for role)"]
+    21: ["no_write_permission", "forbidden (modification not allowed)"]
+}
+
+410: {
+    _:  "gone"
+    1: ["not_supported", "thus endpoint is not supported any more"]
+}
+
+500 {
+    _:   "server_error"
+    0:   ["general_error", "general server error"]
+    1:   ["internal_error", "internal server error"]
+    2:   ["config_error", "configuration error"]
+    3:   ["build_info_error", "build info unavailable or corrupted"]
+    4:   ["low_disk_space", "Critical server error! Server reports low or insufficient disk space. Please resolve immediately by allocating additional disk space or freeing up storage space."]
+    10:  ["transaction_error", "a transaction call has returned with an error"]
+    # Database-related issues
+    100: ["data_error", "general data error"]
+    101: ["inconsistent_data", "inconsistent data encountered in document"]
+    102: ["database_unavailable", "database is temporarily unavailable"]
+    110: ["update_failed", "update failed"]
+
+    # Index-related issues
+    201: ["missing_index", "missing internal index"]
+
+    9999: ["not_implemented", "action is not yet implemented"]
+}

apiserver/apierrors_generator/__init__.py

@@ -0,0 +1 @@
+from .errors_generator import ErrorsGenerator

apiserver/apierrors_generator/__main__.py

@@ -0,0 +1,4 @@
+from .errors_generator import ErrorsGenerator
+
+if __name__ == '__main__':
+    ErrorsGenerator.generate_python_files()

apiserver/apierrors_generator/errors_generator.py

@@ -0,0 +1,31 @@
+from functools import reduce
+from pathlib import Path
+from typing import Union
+
+from pyhocon import ConfigFactory, ConfigTree
+
+from .generator import Generator
+
+
+class ErrorsGenerator:
+    _apierrors_path = Path(__file__).parents[1] / "apierrors"
+    _files = [_apierrors_path / "errors.conf"]
+
+    @classmethod
+    def _get_codes(cls):
+        return {
+            (k, v.pop("_")): v
+            for k, v in reduce(
+                ConfigTree.merge_configs, map(ConfigFactory.parse_file, cls._files),
+            ).items()
+        }
+
+    @classmethod
+    def add_errors_file(cls, path: Union[Path, str]):
+        cls._files.append(path)
+
+    @classmethod
+    def generate_python_files(cls):
+        Generator(cls._apierrors_path / "errors", format_pep8=False).make_errors(
+            cls._get_codes()
+        )

apiserver/apierrors_generator/generator.py

@@ -8,9 +8,12 @@ from pathlib import Path
 
 env = jinja2.Environment(
     loader=jinja2.FileSystemLoader(str(Path(__file__).parent)),
-    autoescape=jinja2.select_autoescape(disabled_extensions=('py',), default_for_string=False),
+    autoescape=jinja2.select_autoescape(
+        disabled_extensions=("py",), default_for_string=False
+    ),
     trim_blocks=True,
-    lstrip_blocks=True)
+    lstrip_blocks=True,
+)
 
 
 def env_filter(name=None):
@@ -19,14 +22,14 @@ def env_filter(name=None):
 
 @env_filter()
 def cls_name(name):
-    delims = list(map(re.escape, (' ', '_')))
-    parts = re.split('|'.join(delims), name)
-    return ''.join(x.capitalize() for x in parts)
+    delims = list(map(re.escape, (" ", "_")))
+    parts = re.split("|".join(delims), name)
+    return "".join(x.capitalize() for x in parts)
 
 
 class Generator(object):
-    _base_class_name = 'BaseError'
-    _base_class_module = 'apierrors.base'
+    _base_class_name = "BaseError"
+    _base_class_module = "apiserver.apierrors.base"
 
     def __init__(self, path, format_pep8=True, use_md5=True):
         self._use_md5 = use_md5
@@ -35,29 +38,37 @@ class Generator(object):
         self._path.mkdir(parents=True, exist_ok=True)
 
     def _make_init_file(self, path):
-        (self._path / path / '__init__.py').write_bytes('')
+        (self._path / path / "__init__.py").write_bytes(b"")
 
     def _do_render(self, file, template, context):
-        with file.open('w') as f:
+        with file.open("w") as f:
             result = template.render(
                 base_class_name=self._base_class_name,
                 base_class_module=self._base_class_module,
-                **context)
+                **context
+            )
             if self._format_pep8:
-                result = autopep8.fix_code(result, options={'aggressive': 1, 'verbose': 0, 'max_line_length': 120})
+                import autopep8
+
+                result = autopep8.fix_code(
+                    result,
+                    options={"aggressive": 1, "verbose": 0, "max_line_length": 120},
+                )
             f.write(result)
 
     def _make_section(self, name, code, subcodes):
         self._do_render(
-            file=(self._path / name).with_suffix('.py'),
-            template=env.get_template('templates/section.jinja2'),
-            context=dict(code=code, subcodes=list(subcodes.items()),))
+            file=(self._path / name).with_suffix(".py"),
+            template=env.get_template("templates/section.jinja2"),
+            context=dict(code=code, subcodes=list(subcodes.items()),),
+        )
 
     def _make_init(self, sections):
         self._do_render(
-            file=(self._path / '__init__.py'),
-            template=env.get_template('templates/init.jinja2'),
-            context=dict(sections=sections,))
+            file=(self._path / "__init__.py"),
+            template=env.get_template("templates/init.jinja2"),
+            context=dict(sections=sections,),
+        )
 
     def _key_to_str(self, data):
         if isinstance(data, dict):
@@ -66,11 +77,11 @@ class Generator(object):
 
     def _calc_digest(self, data):
         data = json.dumps(self._key_to_str(data), sort_keys=True)
-        return hashlib.md5(data.encode('utf8')).hexdigest()
+        return hashlib.md5(data.encode("utf8")).hexdigest()
 
     def make_errors(self, errors):
         digest = None
-        digest_file = self._path / 'digest.md5'
+        digest_file = self._path / "digest.md5"
         if self._use_md5:
             digest = self._calc_digest(errors)
             if digest_file.is_file():
@@ -79,7 +90,7 @@ class Generator(object):
 
         self._make_init(errors)
         for (code, section_name), subcodes in errors.items():
-            self._make_section(section_name, code, subcodes)
+            self._make_section(section_name, int(code), subcodes)
 
         if self._use_md5:
             digest_file.write_text(digest)

apiserver/apierrors_generator/templates/section.jinja2

@@ -5,5 +5,5 @@ from {{ base_class_module }} import {{ base_class_name }}
 {% for subcode, (name, msg) in subcodes %}
 
 
-{{ error_class(name|cls_name, msg, code, subcode) -}}
+{{ error_class(name|cls_name, msg, code, subcode|int) -}}
 {% endfor %}

apiserver/apimodels/__init__.py

@@ -0,0 +1,304 @@
+from enum import Enum
+from typing import Union, Type, Iterable
+
+import jsonmodels.errors
+import six
+from jsonmodels import fields
+from jsonmodels.fields import _LazyType, NotSet
+from jsonmodels.models import Base as ModelBase
+from jsonmodels.validators import Enum as EnumValidator
+from mongoengine.base import BaseDocument
+from validators import email as email_validator, domain as domain_validator
+
+from apiserver.apierrors import errors
+from apiserver.utilities.json import loads, dumps
+
+
+class EmailField(fields.StringField):
+    def validate(self, value):
+        super().validate(value)
+        if value is None:
+            return
+        if email_validator(value) is not True:
+            raise errors.bad_request.InvalidEmailAddress()
+
+
+class DomainField(fields.StringField):
+    def validate(self, value):
+        super().validate(value)
+        if value is None:
+            return
+        if domain_validator(value) is not True:
+            raise errors.bad_request.InvalidDomainName()
+
+
+def make_default(field_cls, default_value):
+    class _FieldWithDefault(field_cls):
+        def get_default_value(self):
+            return default_value
+
+    return _FieldWithDefault
+
+
+class ListField(fields.ListField):
+    def __init__(self, items_types=None, *args, default=NotSet, **kwargs):
+        if default is not NotSet and callable(default):
+            default = default()
+
+        super(ListField, self).__init__(items_types, *args, default=default, **kwargs)
+
+    def _cast_value(self, value):
+        try:
+            return super(ListField, self)._cast_value(value)
+        except TypeError:
+            if len(self.items_types) == 1 and issubclass(self.items_types[0], Enum):
+                return self.items_types[0](value)
+            return value
+
+    def validate_single_value(self, item):
+        super(ListField, self).validate_single_value(item)
+        if isinstance(item, ModelBase):
+            item.validate()
+
+
+class ScalarField(fields.BaseField):
+
+    """String field."""
+
+    types = (str, int, float, bool)
+
+
+class DictField(fields.BaseField):
+    types = (dict,)
+
+    def __init__(self, value_types=None, *args, **kwargs):
+        self.value_types = self._assign_types(value_types)
+        super(DictField, self).__init__(*args, **kwargs)
+
+    def get_default_value(self):
+        default = super(DictField, self).get_default_value()
+        if default is None and not self.required:
+            return {}
+        return default
+
+    @staticmethod
+    def _assign_types(value_types):
+        if value_types:
+            try:
+                value_types = tuple(value_types)
+            except TypeError:
+                value_types = (value_types,)
+        else:
+            value_types = tuple()
+
+        return tuple(
+            _LazyType(type_) if isinstance(type_, six.string_types) else type_
+            for type_ in value_types
+        )
+
+    def parse_value(self, values):
+        """Cast value to proper collection."""
+        result = self.get_default_value()
+
+        if values is None:
+            return result
+
+        if not self.value_types or not isinstance(values, dict):
+            return values
+
+        return {key: self._cast_value(value) for key, value in values.items()}
+
+    def _cast_value(self, value):
+        if isinstance(value, self.value_types):
+            return value
+        else:
+            if len(self.value_types) != 1:
+                tpl = 'Cannot decide which type to choose from "{types}".'
+                raise jsonmodels.errors.ValidationError(
+                    tpl.format(
+                        types=', '.join([t.__name__ for t in self.value_types])
+                    )
+                )
+            return self.value_types[0](**value)
+
+    def validate(self, value):
+        super(DictField, self).validate(value)
+
+        if not self.value_types:
+            return
+
+        if not value:
+            return
+
+        for item in value.values():
+            self.validate_single_value(item)
+
+    def validate_single_value(self, item):
+        if not self.value_types:
+            return
+
+        if not isinstance(item, self.value_types):
+            raise jsonmodels.errors.ValidationError(
+                "All items must be instances "
+                'of "{types}", and not "{type}".'.format(
+                    types=", ".join([t.__name__ for t in self.value_types]),
+                    type=type(item).__name__,
+                )
+            )
+
+    def _elem_to_struct(self, value):
+        try:
+            return value.to_struct()
+        except AttributeError:
+            return value
+
+    def to_struct(self, values):
+        return {k: self._elem_to_struct(v) for k, v in values.items()}
+
+
+class IntField(fields.IntField):
+    def parse_value(self, value):
+        try:
+            return super(IntField, self).parse_value(value)
+        except (ValueError, TypeError):
+            return value
+
+
+class NullableEnumValidator(EnumValidator):
+    """Validator for enums that allows a None value."""
+
+    def validate(self, value):
+        if value is not None:
+            super(NullableEnumValidator, self).validate(value)
+
+
+class EnumField(fields.StringField):
+    def __init__(
+        self,
+        values_or_type: Union[Iterable, Type[Enum]],
+        *args,
+        required=False,
+        default=None,
+        **kwargs
+    ):
+        choices = list(map(self.parse_value, values_or_type))
+        validator_cls = EnumValidator if required else NullableEnumValidator
+        kwargs.setdefault("validators", []).append(validator_cls(*choices))
+        super().__init__(
+            default=self.parse_value(default), required=required, *args, **kwargs
+        )
+
+    def parse_value(self, value):
+        if isinstance(value, Enum):
+            return str(value.value)
+        return super().parse_value(value)
+
+
+class ActualEnumField(fields.StringField):
+    def __init__(
+        self,
+        enum_class: Type[Enum],
+        *args,
+        validators=None,
+        required=False,
+        default=None,
+        **kwargs
+    ):
+        self.__enum = enum_class
+        self.types = (enum_class,)
+        # noinspection PyTypeChecker
+        choices = list(enum_class)
+        validator_cls = EnumValidator if required else NullableEnumValidator
+        validators = [*(validators or []), validator_cls(*choices)]
+        super().__init__(
+            default=self.parse_value(default) if default else NotSet,
+            *args,
+            required=required,
+            validators=validators,
+            **kwargs
+        )
+
+    def parse_value(self, value):
+        if value is NotSet and not self.required:
+            return self.get_default_value()
+        try:
+            # noinspection PyArgumentList
+            return self.__enum(value)
+        except ValueError:
+            return value
+
+    def to_struct(self, value):
+        return super().to_struct(value.value)
+
+
+class JsonSerializableMixin:
+    def to_json(self: ModelBase):
+        return dumps(self.to_struct())
+
+    @classmethod
+    def from_json(cls: Type[ModelBase], s):
+        return cls(**loads(s))
+
+
+def callable_default(cls: Type[fields.BaseField]) -> Type[fields.BaseField]:
+    class _Wrapped(cls):
+        _callable_default = None
+
+        def get_default_value(self):
+            if self._callable_default:
+                return self._callable_default()
+            return super(_Wrapped, self).get_default_value()
+
+        def __init__(self, *args, default=None, **kwargs):
+            if default and callable(default):
+                self._callable_default = default
+                default = default()
+            super(_Wrapped, self).__init__(*args, default=default, **kwargs)
+
+    return _Wrapped
+
+
+class MongoengineFieldsDict(DictField):
+    """
+    DictField representing mongoengine field names/value mapping.
+    Used to convert mongoengine-style field/subfield notation to user-presentable syntax, including handling update
+        operators.
+    """
+
+    mongoengine_update_operators = (
+        "inc",
+        "dec",
+        "push",
+        "push_all",
+        "pop",
+        "pull",
+        "pull_all",
+        "add_to_set",
+    )
+
+    @staticmethod
+    def _normalize_mongo_value(value):
+        if isinstance(value, BaseDocument):
+            return value.to_mongo()
+        return value
+
+    @classmethod
+    def _normalize_mongo_field_path(cls, path, value):
+        parts = path.split("__")
+        if len(parts) > 1:
+            if parts[0] == "set":
+                parts = parts[1:]
+            elif parts[0] == "unset":
+                parts = parts[1:]
+                value = None
+            elif parts[0] in cls.mongoengine_update_operators:
+                return None, None
+        return ".".join(parts), cls._normalize_mongo_value(value)
+
+    def parse_value(self, value):
+        value = super(MongoengineFieldsDict, self).parse_value(value)
+        return {
+            k: v
+            for k, v in (self._normalize_mongo_field_path(*p) for p in value.items())
+            if k is not None
+        }

apiserver/apimodels/auth.py

@@ -1,11 +1,11 @@
-from jsonmodels.fields import IntField, StringField, BoolField, EmbeddedField
+from jsonmodels.fields import IntField, StringField, BoolField, EmbeddedField, DateTimeField
 from jsonmodels.models import Base
 from jsonmodels.validators import Max, Enum
 
-from apimodels import ListField, EnumField
-from config import config
-from database.model.auth import Role
-from database.utils import get_options
+from apiserver.apimodels import ListField, EnumField
+from apiserver.config_repo import config
+from apiserver.database.model.auth import Role
+from apiserver.database.utils import get_options
 
 
 class GetTokenRequest(Base):
@@ -75,10 +75,17 @@ class CreateUserResponse(Base):
 class Credentials(Base):
     access_key = StringField(required=True)
     secret_key = StringField(required=True)
+    label = StringField()
 
 
 class CredentialsResponse(Credentials):
     secret_key = StringField()
+    last_used = DateTimeField(default=None)
+    last_used_from = StringField()
+
+
+class CreateCredentialsRequest(Base):
+    label = StringField()
 
 
 class CreateCredentialsResponse(Base):
@@ -89,6 +96,11 @@ class GetCredentialsResponse(Base):
     credentials = ListField(CredentialsResponse)
 
 
+class EditCredentialsRequest(Base):
+    access_key = StringField(required=True)
+    label = StringField()
+
+
 class RevokeCredentialsRequest(Base):
     access_key = StringField(required=True)
 

apiserver/apimodels/base.py

@@ -0,0 +1,28 @@
+from jsonmodels import models, fields
+from jsonmodels.validators import Length
+
+from apiserver.apimodels import MongoengineFieldsDict, ListField
+
+
+class UpdateResponse(models.Base):
+    updated = fields.IntField(required=True)
+    fields = MongoengineFieldsDict()
+
+
+class PagedRequest(models.Base):
+    page = fields.IntField()
+    page_size = fields.IntField()
+
+
+class IdResponse(models.Base):
+    id = fields.StringField(required=True)
+
+
+class MakePublicRequest(models.Base):
+    ids = ListField(items_types=str, validators=[Length(minimum_value=1)])
+
+
+class MoveRequest(models.Base):
+    ids = ListField([str], validators=Length(minimum_value=1))
+    project = fields.StringField()
+    project_name = fields.StringField()

apiserver/apimodels/batch.py

@@ -0,0 +1,25 @@
+from typing import Sequence
+
+from jsonmodels.fields import StringField
+from jsonmodels.models import Base
+from jsonmodels.validators import Length
+
+from apiserver.apimodels import ListField
+from apiserver.apimodels.base import UpdateResponse
+
+
+class BatchRequest(Base):
+    ids: Sequence[str] = ListField([str], validators=Length(minimum_value=1))
+
+
+class BatchResponse(Base):
+    succeeded: Sequence[dict] = ListField([dict])
+    failed: Sequence[dict] = ListField([dict])
+
+
+class UpdateBatchItem(UpdateResponse):
+    id: str = StringField()
+
+
+class UpdateBatchResponse(BatchResponse):
+    succeeded: Sequence[UpdateBatchItem] = ListField(UpdateBatchItem)

apiserver/apimodels/custom_validators/__init__.py

@@ -0,0 +1,34 @@
+import validators
+from jsonmodels.errors import ValidationError
+
+
+class ForEach(object):
+    def __init__(self, validator):
+        self.validator = validator
+
+    def validate(self, values):
+        for value in values:
+            self.validator.validate(value)
+
+    def modify_schema(self, field_schema):
+        return self.validator.modify_schema(field_schema)
+
+
+class Hostname(object):
+
+    def validate(self, value):
+        if validators.domain(value) is not True:
+            raise ValidationError(f"Value '{value}' is not a valid hostname")
+
+    def modify_schema(self, field_schema):
+        field_schema["format"] = "hostname"
+
+
+class Email(object):
+
+    def validate(self, value):
+        if validators.email(value) is not True:
+            raise ValidationError(f"Value '{value}' is not a valid email address")
+
+    def modify_schema(self, field_schema):
+        field_schema["format"] = "email"

apiserver/apimodels/events.py

@@ -0,0 +1,231 @@
+from enum import auto
+from typing import Sequence, Optional
+
+from jsonmodels import validators
+from jsonmodels.fields import StringField, BoolField, EmbeddedField
+from jsonmodels.models import Base
+from jsonmodels.validators import Length, Min, Max
+
+from apiserver.apimodels import ListField, IntField, ActualEnumField
+from apiserver.bll.event.event_common import EventType
+from apiserver.bll.event.scalar_key import ScalarKeyEnum
+from apiserver.config_repo import config
+from apiserver.utilities.stringenum import StringEnum
+
+
+class TaskRequest(Base):
+    task: str = StringField(required=True)
+
+
+class ModelRequest(Base):
+    model: str = StringField(required=True)
+
+
+class HistogramRequestBase(Base):
+    samples: int = IntField(default=2000, validators=[Min(1), Max(6000)])
+    key: ScalarKeyEnum = ActualEnumField(ScalarKeyEnum, default=ScalarKeyEnum.iter)
+
+
+class MetricVariants(Base):
+    metric: str = StringField(required=True)
+    variants: Sequence[str] = ListField(items_types=str)
+
+
+class ScalarMetricsIterHistogramRequest(HistogramRequestBase):
+    task: str = StringField(required=True)
+    metrics: Sequence[MetricVariants] = ListField(items_types=MetricVariants)
+    model_events: bool = BoolField(default=False)
+
+
+class GetMetricsAndVariantsRequest(Base):
+    task: str = StringField(required=True)
+    model_events: bool = BoolField(default=False)
+
+
+class MultiTaskScalarMetricsIterHistogramRequest(HistogramRequestBase):
+    tasks: Sequence[str] = ListField(
+        items_types=str,
+        validators=[
+            Length(
+                minimum_value=1,
+                maximum_value=config.get(
+                    "services.tasks.multi_task_histogram_limit", 100
+                ),
+            )
+        ],
+    )
+    metrics: Sequence[MetricVariants] = ListField(items_types=MetricVariants)
+    model_events: bool = BoolField(default=False)
+
+
+class TaskMetric(Base):
+    task: str = StringField(required=True)
+    metric: str = StringField(default=None)
+    variants: Sequence[str] = ListField(items_types=str)
+
+
+class LegacyMetricEventsRequest(TaskRequest):
+    iters: int = IntField(default=1, validators=validators.Min(1))
+    scroll_id: str = StringField()
+    model_events: bool = BoolField(default=False)
+
+
+class MetricEventsRequest(Base):
+    metrics: Sequence[TaskMetric] = ListField(
+        items_types=TaskMetric, validators=[Length(minimum_value=1)]
+    )
+    iters: int = IntField(default=1, validators=validators.Min(1))
+    navigate_earlier: bool = BoolField(default=True)
+    refresh: bool = BoolField(default=False)
+    scroll_id: str = StringField()
+    model_events: bool = BoolField(default=False)
+
+
+class VectorMetricsIterHistogramRequest(Base):
+    task: str = StringField(required=True)
+    metric: str = StringField(required=True)
+    variant: str = StringField(required=True)
+    model_events: bool = BoolField(default=False)
+
+
+class GetVariantSampleRequest(Base):
+    task: str = StringField(required=True)
+    metric: str = StringField(required=True)
+    variant: str = StringField(required=True)
+    iteration: Optional[int] = IntField()
+    refresh: bool = BoolField(default=False)
+    scroll_id: Optional[str] = StringField()
+    navigate_current_metric: bool = BoolField(default=True)
+    model_events: bool = BoolField(default=False)
+
+
+class GetMetricSamplesRequest(Base):
+    task: str = StringField(required=True)
+    metric: str = StringField(required=True)
+    iteration: Optional[int] = IntField()
+    refresh: bool = BoolField(default=False)
+    scroll_id: Optional[str] = StringField()
+    navigate_current_metric: bool = BoolField(default=True)
+    model_events: bool = BoolField(default=False)
+
+
+class NextHistorySampleRequest(Base):
+    task: str = StringField(required=True)
+    scroll_id: Optional[str] = StringField()
+    navigate_earlier: bool = BoolField(default=True)
+    next_iteration: bool = BoolField(default=False)
+    model_events: bool = BoolField(default=False)
+
+
+class LogOrderEnum(StringEnum):
+    asc = auto()
+    desc = auto()
+
+
+class TaskEventsRequestBase(Base):
+    task: str = StringField(required=True)
+    batch_size: int = IntField(default=500)
+
+
+class TaskEventsRequest(TaskEventsRequestBase):
+    metrics: Sequence[MetricVariants] = ListField(items_types=MetricVariants)
+    event_type: EventType = ActualEnumField(EventType, default=EventType.all)
+    order: Optional[str] = ActualEnumField(LogOrderEnum, default=LogOrderEnum.asc)
+    scroll_id: str = StringField()
+    count_total: bool = BoolField(default=True)
+    model_events: bool = BoolField(default=False)
+
+
+class LegacyLogEventsRequest(TaskEventsRequestBase):
+    order: Optional[str] = ActualEnumField(LogOrderEnum, default=LogOrderEnum.desc)
+    scroll_id: str = StringField()
+
+
+class LogEventsRequest(TaskEventsRequestBase):
+    batch_size: int = IntField(default=5000)
+    navigate_earlier: bool = BoolField(default=True)
+    from_timestamp: Optional[int] = IntField()
+    order: Optional[str] = ActualEnumField(LogOrderEnum)
+
+
+class ScalarMetricsIterRawRequest(TaskEventsRequestBase):
+    batch_size: int = IntField()
+    key: ScalarKeyEnum = ActualEnumField(ScalarKeyEnum, default=ScalarKeyEnum.iter)
+    metric: MetricVariants = EmbeddedField(MetricVariants, required=True)
+    count_total: bool = BoolField(default=False)
+    scroll_id: str = StringField()
+    model_events: bool = BoolField(default=False)
+
+
+class IterationEvents(Base):
+    iter: int = IntField()
+    events: Sequence[dict] = ListField(items_types=dict)
+
+
+class MetricEvents(Base):
+    task: str = StringField()
+    iterations: Sequence[IterationEvents] = ListField(items_types=IterationEvents)
+
+
+class MetricEventsResponse(Base):
+    metrics: Sequence[MetricEvents] = ListField(items_types=MetricEvents)
+    scroll_id: str = StringField()
+
+
+class MultiTasksRequestBase(Base):
+    tasks: Sequence[str] = ListField(
+        items_types=str, validators=[Length(minimum_value=1)]
+    )
+    model_events: bool = BoolField(default=False)
+
+
+class SingleValueMetricsRequest(MultiTasksRequestBase):
+    metrics: Sequence[MetricVariants] = ListField(items_types=MetricVariants)
+
+
+class TaskMetricsRequest(MultiTasksRequestBase):
+    event_type: EventType = ActualEnumField(EventType, required=True)
+
+
+class MultiTaskMetricsRequest(MultiTasksRequestBase):
+    event_type: EventType = ActualEnumField(EventType, default=EventType.all)
+
+
+class LegacyMultiTaskEventsRequest(MultiTasksRequestBase):
+    iters: int = IntField(default=1, validators=validators.Min(1))
+    scroll_id: str = StringField()
+
+
+class MultiTaskPlotsRequest(MultiTasksRequestBase):
+    iters: int = IntField(default=1)
+    scroll_id: str = StringField()
+    no_scroll: bool = BoolField(default=False)
+    last_iters_per_task_metric: bool = BoolField(default=True)
+    metrics: Sequence[MetricVariants] = ListField(items_types=MetricVariants)
+
+
+class TaskPlotsRequest(Base):
+    task: str = StringField(required=True)
+    iters: int = IntField(default=1)
+    scroll_id: str = StringField()
+    no_scroll: bool = BoolField(default=False)
+    metrics: Sequence[MetricVariants] = ListField(items_types=MetricVariants)
+    model_events: bool = BoolField(default=False)
+
+
+class GetScalarMetricDataRequest(Base):
+    task: str = StringField(required=True)
+    metric: str = StringField(required=True)
+    scroll_id: str = StringField()
+    no_scroll: bool = BoolField(default=False)
+    model_events: bool = BoolField(default=False)
+
+
+class ClearScrollRequest(Base):
+    scroll_id: str = StringField()
+
+
+class ClearTaskLogRequest(Base):
+    task: str = StringField(required=True)
+    threshold_sec = IntField()
+    allow_locked = BoolField(default=False)

apiserver/apimodels/login.py

@@ -0,0 +1,35 @@
+from jsonmodels.fields import StringField, BoolField, EmbeddedField, ListField
+from jsonmodels.models import Base
+
+from apiserver.apimodels import DictField, callable_default
+
+
+class GetSupportedModesRequest(Base):
+    pass
+    # state = StringField(help_text="ASCII base64 encoded application state")
+    # callback_url_prefix = StringField()
+
+
+class BasicGuestMode(Base):
+    enabled = BoolField(default=False)
+    name = StringField()
+    username = StringField()
+    password = StringField()
+
+
+class BasicMode(Base):
+    enabled = BoolField(default=False)
+    guest = callable_default(EmbeddedField)(BasicGuestMode, default=BasicGuestMode)
+
+
+class ServerErrors(Base):
+    missed_es_upgrade = BoolField(default=False)
+    es_connection_error = BoolField(default=False)
+
+
+class GetSupportedModesResponse(Base):
+    basic = EmbeddedField(BasicMode)
+    server_errors = EmbeddedField(ServerErrors)
+    sso = DictField([str, type(None)])
+    sso_providers = ListField([dict])
+    authenticated = BoolField(default=False)

apiserver/apimodels/metadata.py

@@ -0,0 +1,24 @@
+from typing import Sequence
+
+from jsonmodels import validators
+from jsonmodels.fields import StringField, BoolField
+from jsonmodels.models import Base
+
+from apiserver.apimodels import ListField
+
+
+class MetadataItem(Base):
+    key = StringField(required=True)
+    type = StringField(required=True)
+    value = StringField(required=True)
+
+
+class DeleteMetadata(Base):
+    keys: Sequence[str] = ListField(str, validators=validators.Length(minimum_value=1))
+
+
+class AddOrUpdateMetadata(Base):
+    metadata: Sequence[MetadataItem] = ListField(
+        [MetadataItem], validators=validators.Length(minimum_value=1)
+    )
+    replace_metadata = BoolField(default=False)

apiserver/apimodels/models.py

@@ -0,0 +1,99 @@
+from jsonmodels import models, fields
+from six import string_types
+
+from apiserver.apimodels import ListField, DictField
+from apiserver.apimodels.base import UpdateResponse
+from apiserver.apimodels.batch import BatchRequest
+from apiserver.apimodels.metadata import (
+    MetadataItem,
+    DeleteMetadata,
+    AddOrUpdateMetadata,
+)
+
+
+class GetFrameworksRequest(models.Base):
+    projects = fields.ListField(items_types=[str])
+
+
+class CreateModelRequest(models.Base):
+    name = fields.StringField(required=True)
+    uri = fields.StringField(required=True)
+    labels = DictField(value_types=string_types + (int,))
+    tags = ListField(items_types=string_types)
+    system_tags = ListField(items_types=string_types)
+    comment = fields.StringField()
+    public = fields.BoolField(default=False)
+    project = fields.StringField()
+    parent = fields.StringField()
+    framework = fields.StringField()
+    design = DictField()
+    ready = fields.BoolField(default=True)
+    ui_cache = DictField()
+    task = fields.StringField()
+    metadata = DictField(value_types=[MetadataItem])
+
+
+class CreateModelResponse(models.Base):
+    id = fields.StringField(required=True)
+    created = fields.BoolField(required=True)
+
+
+class ModelRequest(models.Base):
+    model = fields.StringField(required=True)
+
+
+class TaskRequest(models.Base):
+    task = fields.StringField(required=True)
+
+
+class UpdateForTaskRequest(TaskRequest):
+    uri = fields.StringField()
+    iteration = fields.IntField()
+    override_model_id = fields.StringField()
+
+
+class UpdateModelRequest(ModelRequest):
+    task = fields.StringField()
+    iteration = fields.IntField()
+
+
+class DeleteModelRequest(ModelRequest):
+    force = fields.BoolField(default=False)
+    delete_external_artifacts = fields.BoolField(default=True)
+
+
+class ModelsDeleteManyRequest(BatchRequest):
+    force = fields.BoolField(default=False)
+    delete_external_artifacts = fields.BoolField(default=True)
+
+
+class PublishModelRequest(ModelRequest):
+    force_publish_task = fields.BoolField(default=False)
+    publish_task = fields.BoolField(default=True)
+
+
+class ModelTaskPublishResponse(models.Base):
+    id = fields.StringField(required=True)
+    data = fields.EmbeddedField(UpdateResponse)
+
+
+class PublishModelResponse(UpdateResponse):
+    published_task = fields.EmbeddedField(ModelTaskPublishResponse)
+
+
+class ModelsPublishManyRequest(BatchRequest):
+    force_publish_task = fields.BoolField(default=False)
+    publish_task = fields.BoolField(default=True)
+
+
+class DeleteMetadataRequest(DeleteMetadata):
+    model = fields.StringField(required=True)
+
+
+class AddOrUpdateMetadataRequest(AddOrUpdateMetadata):
+    model = fields.StringField(required=True)
+
+
+class ModelsGetRequest(models.Base):
+    include_stats = fields.BoolField(default=False)
+    allow_public = fields.BoolField(default=True)

apiserver/apimodels/organization.py

@@ -0,0 +1,62 @@
+from enum import auto
+from typing import Sequence
+
+from jsonmodels import fields, models
+from jsonmodels.validators import Length
+
+from apiserver.apimodels import DictField, ActualEnumField, ScalarField
+from apiserver.utilities.stringenum import StringEnum
+
+
+class Filter(models.Base):
+    tags = fields.ListField([str])
+    system_tags = fields.ListField([str])
+
+
+class TagsRequest(models.Base):
+    include_system = fields.BoolField(default=False)
+    filter = fields.EmbeddedField(Filter)
+
+
+class EntitiesCountRequest(models.Base):
+    projects = DictField()
+    tasks = DictField()
+    models = DictField()
+    pipelines = DictField()
+    datasets = DictField()
+    reports = DictField()
+    active_users = fields.ListField(str)
+    search_hidden = fields.BoolField(default=False)
+    allow_public = fields.BoolField(default=True)
+
+
+class EntityType(StringEnum):
+    task = auto()
+    model = auto()
+
+
+class ValueMapping(models.Base):
+    key = ScalarField(nullable=True)
+    value = ScalarField(nullable=True)
+
+
+class FieldMapping(models.Base):
+    field = fields.StringField(required=True)
+    name = fields.StringField()
+    values: Sequence[ValueMapping] = fields.ListField(items_types=[ValueMapping])
+
+
+class PrepareDownloadForGetAllRequest(models.Base):
+    entity_type = ActualEnumField(EntityType)
+    allow_public = fields.BoolField(default=True)
+    search_hidden = fields.BoolField(default=False)
+    only_fields = fields.ListField(
+        items_types=[str], validators=[Length(1)], required=True
+    )
+    field_mappings: Sequence[FieldMapping] = fields.ListField(
+        items_types=[FieldMapping], validators=[Length(1)], required=True
+    )
+
+
+class DownloadForGetAllRequest(models.Base):
+    prepare_id = fields.StringField(required=True)

apiserver/apimodels/pipelines.py

@@ -0,0 +1,21 @@
+from jsonmodels import models, fields
+from jsonmodels.validators import Length
+
+from apiserver.apimodels import ListField
+
+
+class Arg(models.Base):
+    name = fields.StringField(required=True)
+    value = fields.StringField(required=True)
+
+
+class DeleteRunsRequest(models.Base):
+    project = fields.StringField(required=True)
+    ids = ListField([str], required=True, validators=[Length(1)])
+
+
+class StartPipelineRequest(models.Base):
+    task = fields.StringField(required=True)
+    queue = fields.StringField(required=True)
+    args = ListField(Arg)
+    verify_watched_queue = fields.BoolField(default=False)

apiserver/apimodels/projects.py

@@ -0,0 +1,103 @@
+from enum import Enum, auto
+
+from jsonmodels import models, fields
+
+from apiserver.apimodels import ListField, ActualEnumField, DictField
+from apiserver.apimodels.organization import TagsRequest
+from apiserver.database.model import EntityVisibility
+from apiserver.utilities.stringenum import StringEnum
+
+
+class ProjectRequest(models.Base):
+    project = fields.StringField(required=True)
+
+
+class MergeRequest(ProjectRequest):
+    destination_project = fields.StringField()
+
+
+class MoveRequest(ProjectRequest):
+    new_location = fields.StringField()
+
+
+class DeleteRequest(ProjectRequest):
+    force = fields.BoolField(default=False)
+    delete_contents = fields.BoolField(default=False)
+    delete_external_artifacts = fields.BoolField(default=True)
+
+
+class ProjectOrNoneRequest(models.Base):
+    project = fields.StringField()
+    include_subprojects = fields.BoolField(default=True)
+
+
+class GetUniqueMetricsRequest(ProjectOrNoneRequest):
+    model_metrics = fields.BoolField(default=False)
+    ids = fields.ListField(str)
+
+
+class GetParamsRequest(ProjectOrNoneRequest):
+    page = fields.IntField(default=0)
+    page_size = fields.IntField(default=500)
+
+
+class ProjectTagsRequest(TagsRequest):
+    projects = ListField(str)
+
+
+class MultiProjectRequest(models.Base):
+    projects = fields.ListField(items_types=[str, type(None)])
+    include_subprojects = fields.BoolField(default=True)
+
+
+class ProjectTaskParentsRequest(MultiProjectRequest):
+    tasks_state = ActualEnumField(EntityVisibility)
+    task_name = fields.StringField()
+
+
+class EntityTypeEnum(StringEnum):
+    task = auto()
+    model = auto()
+
+
+class ProjectUserNamesRequest(MultiProjectRequest):
+    entity = ActualEnumField(EntityTypeEnum, default=EntityTypeEnum.task)
+
+
+class MultiProjectPagedRequest(MultiProjectRequest):
+    allow_public = fields.BoolField(default=True)
+    page = fields.IntField(default=0)
+    page_size = fields.IntField(default=500)
+
+
+class ProjectHyperparamValuesRequest(MultiProjectPagedRequest):
+    section = fields.StringField(required=True)
+    name = fields.StringField(required=True)
+    pattern = fields.StringField()
+
+
+class ProjectModelMetadataValuesRequest(MultiProjectPagedRequest):
+    key = fields.StringField(required=True)
+
+
+class ProjectChildrenType(Enum):
+    pipeline = "pipeline"
+    report = "report"
+    dataset = "dataset"
+
+
+class ProjectsGetRequest(models.Base):
+    include_dataset_stats = fields.BoolField(default=False)
+    include_stats = fields.BoolField(default=False)
+    include_stats_filter = DictField()
+    stats_with_children = fields.BoolField(default=True)
+    stats_for_state = ActualEnumField(EntityVisibility, default=EntityVisibility.active)
+    non_public = fields.BoolField(default=False)  # legacy, use allow_public instead
+    active_users = fields.ListField(str)
+    check_own_contents = fields.BoolField(default=False)
+    shallow_search = fields.BoolField(default=False)
+    search_hidden = fields.BoolField(default=False)
+    allow_public = fields.BoolField(default=True)
+    children_type = ActualEnumField(ProjectChildrenType)
+    children_tags = fields.ListField(str)
+    children_tags_filter = DictField()

apiserver/apimodels/queues.py

@@ -0,0 +1,91 @@
+from jsonmodels import validators
+from jsonmodels.fields import StringField, IntField, BoolField, FloatField
+from jsonmodels.models import Base
+
+from apiserver.apimodels import ListField, DictField
+from apiserver.apimodels.metadata import (
+    MetadataItem,
+    DeleteMetadata,
+    AddOrUpdateMetadata,
+)
+
+
+class GetDefaultResp(Base):
+    id = StringField(required=True)
+    name = StringField(required=True)
+
+
+class CreateRequest(Base):
+    name = StringField(required=True)
+    tags = ListField(items_types=[str])
+    system_tags = ListField(items_types=[str])
+    metadata = DictField(value_types=[MetadataItem])
+
+
+class QueueRequest(Base):
+    queue = StringField(required=True)
+
+
+class GetByIdRequest(QueueRequest):
+    max_task_entries = IntField()
+
+
+class GetAllRequest(Base):
+    max_task_entries = IntField()
+    search_hidden = BoolField(default=False)
+
+
+class GetNextTaskRequest(QueueRequest):
+    queue = StringField(required=True)
+    get_task_info = BoolField(default=False)
+    task = StringField()
+
+
+class DeleteRequest(QueueRequest):
+    force = BoolField(default=False)
+
+
+class UpdateRequest(QueueRequest):
+    name = StringField()
+    tags = ListField(items_types=[str])
+    system_tags = ListField(items_types=[str])
+    metadata = DictField(value_types=[MetadataItem])
+
+
+class TaskRequest(QueueRequest):
+    task = StringField(required=True)
+
+
+class MoveTaskRequest(TaskRequest):
+    count = IntField(default=1)
+
+
+class MoveTaskResponse(Base):
+    position = IntField()
+
+
+class GetMetricsRequest(Base):
+    queue_ids = ListField([str])
+    from_date = FloatField(required=True, validators=validators.Min(0))
+    to_date = FloatField(required=True, validators=validators.Min(0))
+    interval = IntField(required=True, validators=validators.Min(1))
+    refresh = BoolField(default=False)
+
+
+class QueueMetrics(Base):
+    queue = StringField()
+    dates = ListField(int)
+    avg_waiting_times = ListField([float, int])
+    queue_lengths = ListField(int)
+
+
+class GetMetricsResponse(Base):
+    queues = ListField(QueueMetrics)
+
+
+class DeleteMetadataRequest(DeleteMetadata):
+    queue = StringField(required=True)
+
+
+class AddOrUpdateMetadataRequest(AddOrUpdateMetadata):
+    queue = StringField(required=True)

apiserver/apimodels/reports.py

@@ -0,0 +1,84 @@
+from typing import Sequence
+
+from jsonmodels import validators
+from jsonmodels.fields import StringField, ListField, BoolField, EmbeddedField, IntField
+from jsonmodels.models import Base
+from jsonmodels.validators import Length
+
+from apiserver.apimodels.events import MetricVariants, HistogramRequestBase
+
+
+class UpdateReportRequest(Base):
+    task = StringField(required=True)
+    name = StringField(nullable=True, validators=Length(minimum_value=3))
+    tags = ListField(items_types=[str])
+    comment = StringField()
+    report = StringField()
+    report_assets = ListField(items_types=[str])
+
+
+class CreateReportRequest(Base):
+    name = StringField(required=True, validators=Length(minimum_value=3))
+    tags = ListField(items_types=[str])
+    comment = StringField()
+    report = StringField()
+    project = StringField()
+    report_assets = ListField(items_types=[str])
+
+
+class PublishReportRequest(Base):
+    task = StringField(required=True)
+    message = StringField(default="")
+
+
+class ArchiveReportRequest(Base):
+    task = StringField(required=True)
+    message = StringField(default="")
+
+
+class ShareReportRequest(Base):
+    task = StringField(required=True)
+    share = BoolField(default=True)
+
+
+class DeleteReportRequest(Base):
+    task = StringField(required=True)
+    force = BoolField(default=False)
+
+
+class MoveReportRequest(Base):
+    task = StringField(required=True)
+    project = StringField()
+    project_name = StringField()
+
+
+class EventsRequest(Base):
+    iters = IntField(default=1, validators=validators.Min(1))
+    metrics: Sequence[MetricVariants] = ListField(items_types=MetricVariants)
+
+
+class PlotEventsRequest(EventsRequest):
+    last_iters_per_task_metric: bool = BoolField(default=True)
+
+
+class ScalarMetricsIterHistogram(HistogramRequestBase):
+    metrics: Sequence[MetricVariants] = ListField(items_types=MetricVariants)
+
+
+class SingleValueMetrics(Base):
+    pass
+
+
+class GetTasksDataRequest(Base):
+    debug_images: EventsRequest = EmbeddedField(EventsRequest)
+    plots: PlotEventsRequest = EmbeddedField(PlotEventsRequest)
+    scalar_metrics_iter_histogram: ScalarMetricsIterHistogram = EmbeddedField(
+        ScalarMetricsIterHistogram
+    )
+    single_value_metrics: SingleValueMetrics = EmbeddedField(SingleValueMetrics)
+    allow_public = BoolField(default=True)
+    model_events: bool = BoolField(default=False)
+
+
+class GetAllRequest(Base):
+    allow_public = BoolField(default=True)

apiserver/apimodels/server.py

@@ -0,0 +1,19 @@
+from jsonmodels.fields import BoolField, DateTimeField, StringField
+from jsonmodels.models import Base
+
+
+class ReportStatsOptionRequest(Base):
+    enabled = BoolField(default=None, nullable=True)
+
+
+class GetConfigRequest(Base):
+    path = StringField()
+
+
+class ReportStatsOptionResponse(Base):
+    supported = BoolField(default=True)
+    enabled = BoolField()
+    enabled_time = DateTimeField(nullable=True)
+    enabled_version = StringField(nullable=True)
+    enabled_user = StringField(nullable=True)
+    current_version = StringField()

apiserver/apimodels/tasks.py

@@ -0,0 +1,340 @@
+from typing import Sequence
+
+from jsonmodels import models
+from jsonmodels.fields import StringField, BoolField, IntField, EmbeddedField
+from jsonmodels.validators import Enum, Length
+
+from apiserver.apimodels import DictField, ListField
+from apiserver.apimodels.base import UpdateResponse
+from apiserver.apimodels.batch import BatchRequest, UpdateBatchItem, BatchResponse
+from apiserver.database.model.task.task import (
+    TaskType,
+    ArtifactModes,
+    DEFAULT_ARTIFACT_MODE,
+    TaskModelTypes,
+)
+from apiserver.database.utils import get_options
+
+
+class ArtifactTypeData(models.Base):
+    preview = StringField()
+    content_type = StringField()
+    data_hash = StringField()
+
+
+class Artifact(models.Base):
+    key = StringField(required=True)
+    type = StringField(required=True)
+    mode = StringField(
+        validators=Enum(*get_options(ArtifactModes)), default=DEFAULT_ARTIFACT_MODE
+    )
+    uri = StringField()
+    hash = StringField()
+    content_size = IntField()
+    timestamp = IntField()
+    type_data = EmbeddedField(ArtifactTypeData)
+    display_data = ListField([list])
+
+
+class StartedResponse(UpdateResponse):
+    started = IntField()
+
+
+class EnqueueResponse(UpdateResponse):
+    queued = IntField()
+    queue_watched = BoolField()
+
+
+class EnqueueBatchItem(UpdateBatchItem):
+    queued: bool = BoolField()
+
+
+class EnqueueManyResponse(BatchResponse):
+    succeeded: Sequence[EnqueueBatchItem] = ListField(EnqueueBatchItem)
+    queue_watched = BoolField()
+
+
+class DequeueResponse(UpdateResponse):
+    dequeued = IntField()
+
+
+class DequeueBatchItem(UpdateBatchItem):
+    dequeued: bool = BoolField()
+
+
+class DequeueManyResponse(BatchResponse):
+    succeeded: Sequence[DequeueBatchItem] = ListField(DequeueBatchItem)
+
+
+class ResetResponse(UpdateResponse):
+    dequeued = DictField()
+    events = DictField()
+    deleted_models = IntField()
+    urls = DictField()
+
+
+class ResetBatchItem(UpdateBatchItem):
+    dequeued: bool = BoolField()
+    deleted_models = IntField()
+    urls = DictField()
+
+
+class ResetManyResponse(BatchResponse):
+    succeeded: Sequence[ResetBatchItem] = ListField(ResetBatchItem)
+
+
+class TaskRequest(models.Base):
+    task = StringField(required=True)
+
+
+class TaskUpdateRequest(TaskRequest):
+    force = BoolField(default=False)
+
+
+class UpdateRequest(TaskUpdateRequest):
+    status_reason = StringField(default="")
+    status_message = StringField(default="")
+
+
+class DequeueRequest(UpdateRequest):
+    remove_from_all_queues = BoolField(default=False)
+    new_status = StringField()
+
+
+class EnqueueRequest(UpdateRequest):
+    queue = StringField()
+    queue_name = StringField()
+    verify_watched_queue = BoolField(default=False)
+
+
+class DeleteRequest(UpdateRequest):
+    move_to_trash = BoolField(default=True)
+    return_file_urls = BoolField(default=False)
+    delete_output_models = BoolField(default=True)
+    delete_external_artifacts = BoolField(default=True)
+
+
+class SetRequirementsRequest(TaskRequest):
+    requirements = DictField(required=True)
+
+
+class CompletedRequest(UpdateRequest):
+    publish = BoolField(default=False)
+
+
+class CompletedResponse(UpdateResponse):
+    published = IntField(default=0)
+
+
+class PublishRequest(UpdateRequest):
+    publish_model = BoolField(default=True)
+
+
+class TaskData(models.Base):
+    """
+    This is a partial description of task can be updated incrementally
+    """
+
+
+class CreateRequest(TaskData):
+    name = StringField(required=True)
+    type = StringField(required=True, validators=Enum(*get_options(TaskType)))
+
+
+class PingRequest(TaskRequest):
+    pass
+
+
+class GetTypesRequest(models.Base):
+    projects = ListField(items_types=[str])
+
+
+class TaskInputModel(models.Base):
+    name = StringField()
+    model = StringField()
+
+
+class CloneRequest(TaskRequest):
+    new_task_name = StringField()
+    new_task_comment = StringField()
+    new_task_tags = ListField([str])
+    new_task_system_tags = ListField([str])
+    new_task_parent = StringField()
+    new_task_project = StringField()
+    new_task_hyperparams = DictField()
+    new_task_configuration = DictField()
+    new_task_container = DictField()
+    new_task_input_models = ListField([TaskInputModel])
+    execution_overrides = DictField()
+    validate_references = BoolField(default=False)
+    new_project_name = StringField()
+
+
+class AddOrUpdateArtifactsRequest(TaskUpdateRequest):
+    artifacts = ListField([Artifact], validators=Length(minimum_value=1))
+
+
+class ArtifactId(models.Base):
+    key = StringField(required=True)
+    mode = StringField(
+        validators=Enum(*get_options(ArtifactModes)), default=DEFAULT_ARTIFACT_MODE
+    )
+
+
+class DeleteArtifactsRequest(TaskUpdateRequest):
+    artifacts = ListField([ArtifactId], validators=Length(minimum_value=1))
+
+
+class ResetRequest(UpdateRequest):
+    clear_all = BoolField(default=False)
+    return_file_urls = BoolField(default=False)
+    delete_output_models = BoolField(default=True)
+    delete_external_artifacts = BoolField(default=True)
+
+
+class MultiTaskRequest(models.Base):
+    tasks = ListField([str], validators=Length(minimum_value=1))
+
+
+class GetHyperParamsRequest(MultiTaskRequest):
+    pass
+
+
+class HyperParamItem(models.Base):
+    section = StringField(required=True, validators=Length(minimum_value=1))
+    name = StringField(required=True, validators=Length(minimum_value=1))
+    value = StringField(required=True)
+    type = StringField()
+    description = StringField()
+
+
+class ReplaceHyperparams(object):
+    none = "none"
+    section = "section"
+    all = "all"
+
+
+class EditHyperParamsRequest(TaskUpdateRequest):
+    hyperparams: Sequence[HyperParamItem] = ListField(
+        [HyperParamItem], validators=Length(minimum_value=1)
+    )
+    replace_hyperparams = StringField(
+        validators=Enum(*get_options(ReplaceHyperparams)),
+        default=ReplaceHyperparams.none,
+    )
+
+
+class HyperParamKey(models.Base):
+    section = StringField(required=True, validators=Length(minimum_value=1))
+    name = StringField(nullable=True)
+
+
+class DeleteHyperParamsRequest(TaskUpdateRequest):
+    hyperparams: Sequence[HyperParamKey] = ListField(
+        [HyperParamKey], validators=Length(minimum_value=1)
+    )
+
+
+class GetConfigurationsRequest(MultiTaskRequest):
+    names = ListField([str])
+
+
+class GetConfigurationNamesRequest(MultiTaskRequest):
+    skip_empty = BoolField(default=True)
+
+
+class Configuration(models.Base):
+    name = StringField(required=True, validators=Length(minimum_value=1))
+    value = StringField(required=True)
+    type = StringField()
+    description = StringField()
+
+
+class EditConfigurationRequest(TaskUpdateRequest):
+    configuration: Sequence[Configuration] = ListField(
+        [Configuration], validators=Length(minimum_value=1)
+    )
+    replace_configuration = BoolField(default=False)
+
+
+class DeleteConfigurationRequest(TaskUpdateRequest):
+    configuration: Sequence[str] = ListField([str], validators=Length(minimum_value=1))
+
+
+class ArchiveRequest(MultiTaskRequest):
+    status_reason = StringField(default="")
+    status_message = StringField(default="")
+
+
+class ArchiveResponse(models.Base):
+    archived = IntField()
+
+
+class TaskBatchRequest(BatchRequest):
+    status_reason = StringField(default="")
+    status_message = StringField(default="")
+
+
+class StopManyRequest(TaskBatchRequest):
+    force = BoolField(default=False)
+
+
+class DequeueManyRequest(TaskBatchRequest):
+    remove_from_all_queues = BoolField(default=False)
+    new_status = StringField()
+
+
+class EnqueueManyRequest(TaskBatchRequest):
+    queue = StringField()
+    queue_name = StringField()
+    validate_tasks = BoolField(default=False)
+    verify_watched_queue = BoolField(default=False)
+
+
+class DeleteManyRequest(TaskBatchRequest):
+    move_to_trash = BoolField(default=True)
+    return_file_urls = BoolField(default=False)
+    delete_output_models = BoolField(default=True)
+    force = BoolField(default=False)
+    delete_external_artifacts = BoolField(default=True)
+
+
+class ResetManyRequest(TaskBatchRequest):
+    clear_all = BoolField(default=False)
+    return_file_urls = BoolField(default=False)
+    delete_output_models = BoolField(default=True)
+    force = BoolField(default=False)
+    delete_external_artifacts = BoolField(default=True)
+
+
+class PublishManyRequest(TaskBatchRequest):
+    publish_model = BoolField(default=True)
+    force = BoolField(default=False)
+
+
+class AddUpdateModelRequest(TaskRequest):
+    name = StringField(required=True)
+    model = StringField(required=True)
+    type = StringField(required=True, validators=Enum(*get_options(TaskModelTypes)))
+    iteration = IntField()
+
+
+class ModelItemKey(models.Base):
+    name = StringField(required=True)
+    type = StringField(required=True, validators=Enum(*get_options(TaskModelTypes)))
+
+
+class DeleteModelsRequest(TaskRequest):
+    models: Sequence[ModelItemKey] = ListField(
+        [ModelItemKey], validators=Length(minimum_value=1)
+    )
+
+
+class GetAllReq(models.Base):
+    allow_public = BoolField(default=True)
+    search_hidden = BoolField(default=False)
+
+
+class UpdateTagsRequest(BatchRequest):
+    add_tags = ListField([str])
+    remove_tags = ListField([str])

apiserver/apimodels/users.py

@@ -1,7 +1,11 @@
 from jsonmodels.fields import StringField
 from jsonmodels.models import Base
 
-from apimodels import DictField
+from apiserver.apimodels import DictField
+
+
+class UserRequest(Base):
+    user = StringField(required=True)
 
 
 class CreateRequest(Base):

apiserver/apimodels/workers.py

@@ -0,0 +1,186 @@
+from enum import Enum
+
+import six
+from jsonmodels import validators
+from jsonmodels.fields import (
+    StringField,
+    EmbeddedField,
+    DateTimeField,
+    IntField,
+    FloatField,
+    BoolField,
+)
+from jsonmodels.models import Base
+
+from apiserver.apimodels import ListField, EnumField, JsonSerializableMixin
+from apiserver.config_repo import config
+
+
+class WorkerRequest(Base):
+    worker = StringField(required=True)
+    tags = ListField(str)
+    system_tags = ListField(str)
+
+
+class RegisterRequest(WorkerRequest):
+    timeout = IntField(
+        default=int(config.get("services.workers.default_worker_timeout_sec", 10 * 60))
+    )
+    """ registration timeout in seconds (default is 10min) """
+    queues = ListField(six.string_types)  # list of queues this worker listens to
+
+
+class MachineStats(Base):
+    cpu_usage = ListField(six.integer_types + (float,))
+    cpu_temperature = ListField(six.integer_types + (float,))
+    gpu_usage = ListField(six.integer_types + (float,))
+    gpu_temperature = ListField(six.integer_types + (float,))
+    gpu_memory_free = ListField(six.integer_types + (float,))
+    gpu_memory_used = ListField(six.integer_types + (float,))
+    memory_used = FloatField()
+    memory_free = FloatField()
+    network_tx = FloatField()
+    network_rx = FloatField()
+    disk_free_home = FloatField()
+    disk_free_temp = FloatField()
+    disk_read = FloatField()
+    disk_write = FloatField()
+
+
+class StatusReportRequest(WorkerRequest):
+    task = StringField()  # task the worker is running on
+    queue = StringField()  # queue from which task was taken
+    queues = ListField(
+        str
+    )  # list of queues this worker listens to. if None, this will not update the worker's queues list.
+    timestamp = IntField(required=True)
+    machine_stats = EmbeddedField(MachineStats)
+
+
+class IdNameEntry(Base):
+    id = StringField(required=True)
+    name = StringField()
+
+
+class WorkerEntry(Base, JsonSerializableMixin):
+    key = StringField()  # not required due to migration issues
+    id = StringField(required=True)
+    user = EmbeddedField(IdNameEntry)
+    company = EmbeddedField(IdNameEntry)
+    ip = StringField()
+    task = EmbeddedField(IdNameEntry)
+    project = EmbeddedField(IdNameEntry)
+    queue = StringField()  # queue from which current task was taken
+    queues = ListField(str)  # list of queues this worker listens to
+    register_time = DateTimeField(required=True)
+    register_timeout = IntField(required=True)
+    last_activity_time = DateTimeField(required=True)
+    last_report_time = DateTimeField()
+    tags = ListField(str)
+    system_tags = ListField(str)
+
+
+class CurrentTaskEntry(IdNameEntry):
+    running_time = IntField()
+    last_iteration = IntField()
+
+
+class QueueEntry(IdNameEntry):
+    next_task = EmbeddedField(IdNameEntry)
+    num_tasks = IntField()
+
+
+class WorkerResponseEntry(WorkerEntry):
+    task = EmbeddedField(CurrentTaskEntry)
+    queue = EmbeddedField(QueueEntry)
+    queues = ListField(QueueEntry)
+
+
+class GetAllRequest(Base):
+    last_seen = IntField(default=3600)
+    tags = ListField(str)
+    system_tags = ListField(str)
+
+
+class GetAllResponse(Base):
+    workers = ListField(WorkerResponseEntry)
+
+
+class GetCountRequest(GetAllRequest):
+    last_seen = IntField(default=0)
+
+
+class StatsBase(Base):
+    worker_ids = ListField(str)
+
+
+class StatsReportBase(StatsBase):
+    from_date = FloatField(required=True, validators=validators.Min(0))
+    to_date = FloatField(required=True, validators=validators.Min(0))
+    interval = IntField(required=True, validators=validators.Min(1))
+
+
+class AggregationType(Enum):
+    avg = "avg"
+    min = "min"
+    max = "max"
+
+
+class StatItem(Base):
+    key = StringField(required=True)
+    aggregation = EnumField(AggregationType, default=AggregationType.avg)
+
+
+class GetStatsRequest(StatsReportBase):
+    items = ListField(
+        StatItem, required=True, validators=validators.Length(minimum_value=1)
+    )
+    split_by_variant = BoolField(default=False)
+
+
+class AggregationStats(Base):
+    aggregation = EnumField(AggregationType)
+    values = ListField(float)
+
+
+class MetricStats(Base):
+    metric = StringField()
+    variant = StringField()
+    dates = ListField(int)
+    stats = ListField(AggregationStats)
+
+
+class WorkerStatistics(Base):
+    worker = StringField()
+    metrics = ListField(MetricStats)
+
+
+class GetStatsResponse(Base):
+    workers = ListField(WorkerStatistics)
+
+
+class GetMetricKeysRequest(StatsBase):
+    pass
+
+
+class MetricCategory(Base):
+    name = StringField()
+    metric_keys = ListField(str)
+
+
+class GetMetricKeysResponse(Base):
+    categories = ListField(MetricCategory)
+
+
+class GetActivityReportRequest(StatsReportBase):
+    pass
+
+
+class ActivityReportSeries(Base):
+    dates = ListField(int)
+    counts = ListField(int)
+
+
+class GetActivityReportResponse(Base):
+    total = EmbeddedField(ActivityReportSeries)
+    active = EmbeddedField(ActivityReportSeries)

apiserver/bll/auth/__init__.py

@@ -1,17 +1,21 @@
 from datetime import datetime
 
-import database
-from apierrors import errors
-from apimodels.auth import GetTokenResponse, CreateUserRequest, Credentials as CredModel
-from apimodels.users import CreateRequest as Users_CreateRequest
-from bll.user import UserBLL
-from config import config
-from config.info import get_version, get_build_number
-from database.errors import translate_errors_context
-from database.model.auth import User, Role, Credentials
-from database.model.company import Company
-from service_repo import APICall, ServiceRepo
-from service_repo.auth import Identity, Token, get_client_id, get_secret_key
+from apiserver import database
+from apiserver.apierrors import errors
+from apiserver.apimodels.auth import (
+    GetTokenResponse,
+    CreateUserRequest,
+    Credentials as CredModel,
+)
+from apiserver.apimodels.users import CreateRequest as Users_CreateRequest
+from apiserver.bll.user import UserBLL
+from apiserver.config_repo import config
+from apiserver.config.info import get_version, get_build_number
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.auth import User, Role, Credentials
+from apiserver.database.model.company import Company
+from apiserver.service_repo import APICall, ServiceRepo
+from apiserver.service_repo.auth import Identity, Token, get_client_id, get_secret_key
 
 log = config.logger("AuthBLL")
 
@@ -57,9 +61,10 @@ class AuthBLL:
                 api_version=str(ServiceRepo.max_endpoint_version()),
                 server_version=str(get_version()),
                 server_build=str(get_build_number()),
+                feature_set="basic",
             )
 
-            return GetTokenResponse(token=token.decode("ascii"))
+            return GetTokenResponse(token=token)
 
     @staticmethod
     def create_user(request: CreateUserRequest, call: APICall = None) -> str:
@@ -144,7 +149,7 @@ class AuthBLL:
 
     @classmethod
     def create_credentials(
-        cls, user_id: str, company_id: str, role: str = None
+        cls, user_id: str, company_id: str, role: str = None, label: str = None,
     ) -> CredModel:
 
         with translate_errors_context():
@@ -153,9 +158,11 @@ class AuthBLL:
             if not user:
                 raise errors.bad_request.InvalidUserId(**query)
 
-            cred = CredModel(access_key=get_client_id(), secret_key=get_secret_key())
+            cred = CredModel(
+                access_key=get_client_id(), secret_key=get_secret_key(), label=label
+            )
             user.credentials.append(
-                Credentials(key=cred.access_key, secret=cred.secret_key)
+                Credentials(key=cred.access_key, secret=cred.secret_key, label=label)
             )
             user.save()
 

apiserver/bll/event/event_common.py

@@ -0,0 +1,166 @@
+import base64
+import zlib
+from enum import Enum
+from typing import Union, Sequence, Mapping, Tuple
+
+from boltons.typeutils import classproperty
+from elasticsearch import Elasticsearch
+
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.task.task import Task
+from apiserver.utilities.dicts import nested_get
+
+
+class EventType(Enum):
+    metrics_scalar = "training_stats_scalar"
+    metrics_vector = "training_stats_vector"
+    metrics_image = "training_debug_image"
+    metrics_plot = "plot"
+    task_log = "log"
+    all = "*"
+
+
+SINGLE_SCALAR_ITERATION = -(2 ** 31)
+MetricVariants = Mapping[str, Sequence[str]]
+TaskCompanies = Mapping[str, Sequence[Task]]
+
+
+class EventSettings:
+    _max_es_allowed_aggregation_buckets = 10000
+
+    @classproperty
+    def max_workers(self):
+        return config.get("services.events.events_retrieval.max_metrics_concurrency", 4)
+
+    @classproperty
+    def state_expiration_sec(self):
+        return config.get(
+            f"services.events.events_retrieval.state_expiration_sec", 3600
+        )
+
+    @classproperty
+    def max_es_buckets(self):
+        percentage = (
+            min(
+                100,
+                config.get(
+                    "services.events.events_retrieval.dynamic_metrics_count_threshold",
+                    80,
+                ),
+            )
+            / 100
+        )
+        return int(self._max_es_allowed_aggregation_buckets * percentage)
+
+
+def get_index_name(company_id: Union[str, Sequence[str]], event_type: str):
+    event_type = event_type.lower().replace(" ", "_")
+    if isinstance(company_id, str):
+        company_id = [company_id]
+
+    return ",".join(f"events-{event_type}-{(c_id or '').lower()}" for c_id in company_id)
+
+
+def check_empty_data(es: Elasticsearch, company_id: str, event_type: EventType) -> bool:
+    es_index = get_index_name(company_id, event_type.value)
+    if not es.indices.exists(index=es_index):
+        return True
+    return False
+
+
+def search_company_events(
+    es: Elasticsearch,
+    company_id: Union[str, Sequence[str]],
+    event_type: EventType,
+    body: dict,
+    **kwargs,
+) -> dict:
+    es_index = get_index_name(company_id, event_type.value)
+    return es.search(index=es_index, body=body, **kwargs)
+
+
+def delete_company_events(
+    es: Elasticsearch, company_id: str, event_type: EventType, body: dict, **kwargs
+) -> dict:
+    es_index = get_index_name(company_id, event_type.value)
+    return es.delete_by_query(index=es_index, body=body, conflicts="proceed", **kwargs)
+
+
+def count_company_events(
+    es: Elasticsearch, company_id: str, event_type: EventType, body: dict, **kwargs
+) -> dict:
+    es_index = get_index_name(company_id, event_type.value)
+    return es.count(index=es_index, body=body, **kwargs)
+
+
+def get_max_metric_and_variant_counts(
+    es: Elasticsearch,
+    company_id: Union[str, Sequence[str]],
+    event_type: EventType,
+    query: dict,
+    **kwargs,
+) -> Tuple[int, int]:
+    dynamic = config.get(
+        "services.events.events_retrieval.dynamic_metrics_count", False
+    )
+    max_metrics_count = config.get(
+        "services.events.events_retrieval.max_metrics_count", 100
+    )
+    max_variants_count = config.get(
+        "services.events.events_retrieval.max_variants_count", 100
+    )
+    if not dynamic:
+        return max_metrics_count, max_variants_count
+
+    es_req: dict = {
+        "size": 0,
+        "query": query,
+        "aggs": {"metrics_count": {"cardinality": {"field": "metric"}}},
+    }
+    with translate_errors_context():
+        es_res = search_company_events(
+            es, company_id=company_id, event_type=event_type, body=es_req, **kwargs,
+        )
+
+    metrics_count = nested_get(
+        es_res, ("aggregations", "metrics_count", "value"), max_metrics_count
+    )
+    if not metrics_count:
+        return max_metrics_count, max_variants_count
+
+    return metrics_count, int(EventSettings.max_es_buckets / metrics_count)
+
+
+def get_metric_variants_condition(metric_variants: MetricVariants,) -> Sequence:
+    conditions = [
+        {
+            "bool": {
+                "must": [
+                    {"term": {"metric": metric}},
+                    {"terms": {"variant": variants}},
+                ]
+            }
+        }
+        if variants
+        else {"term": {"metric": metric}}
+        for metric, variants in metric_variants.items()
+    ]
+
+    return {"bool": {"should": conditions}}
+
+
+class PlotFields:
+    valid_plot = "valid_plot"
+    plot_len = "plot_len"
+    plot_str = "plot_str"
+    plot_data = "plot_data"
+    source_urls = "source_urls"
+
+
+def uncompress_plot(event: dict):
+    plot_data = event.pop(PlotFields.plot_data, None)
+    if plot_data and event.get(PlotFields.plot_str) is None:
+        event[PlotFields.plot_str] = zlib.decompress(
+            base64.b64decode(plot_data)
+        ).decode()

apiserver/bll/event/event_metrics.py

@@ -0,0 +1,596 @@
+import itertools
+import math
+from collections import defaultdict
+from concurrent.futures.thread import ThreadPoolExecutor
+from functools import partial
+from operator import itemgetter
+from typing import Sequence, Tuple, Mapping
+
+from boltons.iterutils import bucketize
+from elasticsearch import Elasticsearch
+
+from apiserver.bll.event.event_common import (
+    EventType,
+    EventSettings,
+    search_company_events,
+    check_empty_data,
+    MetricVariants,
+    get_metric_variants_condition,
+    get_max_metric_and_variant_counts,
+    SINGLE_SCALAR_ITERATION,
+    TaskCompanies,
+)
+from apiserver.bll.event.scalar_key import ScalarKey, ScalarKeyEnum
+from apiserver.bll.query import Builder as QueryBuilder
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.utilities.dicts import nested_get
+
+log = config.logger(__file__)
+
+
+class EventMetrics:
+    MAX_AGGS_ELEMENTS_COUNT = 50
+    MAX_SAMPLE_BUCKETS = 6000
+
+    def __init__(self, es: Elasticsearch):
+        self.es = es
+
+    def get_scalar_metrics_average_per_iter(
+        self,
+        company_id: str,
+        task_id: str,
+        samples: int,
+        key: ScalarKeyEnum,
+        metric_variants: MetricVariants = None,
+    ) -> dict:
+        """
+        Get scalar metric histogram per metric and variant
+        The amount of points in each histogram should not exceed
+        the requested samples
+        """
+        event_type = EventType.metrics_scalar
+        if check_empty_data(self.es, company_id=company_id, event_type=event_type):
+            return {}
+
+        return self._get_scalar_average_per_iter_core(
+            task_id=task_id,
+            company_id=company_id,
+            event_type=event_type,
+            samples=samples,
+            key=ScalarKey.resolve(key),
+            metric_variants=metric_variants,
+        )
+
+    def _get_scalar_average_per_iter_core(
+        self,
+        task_id: str,
+        company_id: str,
+        event_type: EventType,
+        samples: int,
+        key: ScalarKey,
+        run_parallel: bool = True,
+        metric_variants: MetricVariants = None,
+    ) -> dict:
+        intervals = self._get_task_metric_intervals(
+            company_id=company_id,
+            event_type=event_type,
+            task_id=task_id,
+            samples=samples,
+            field=key.field,
+            metric_variants=metric_variants,
+        )
+        if not intervals:
+            return {}
+        interval_groups = self._group_task_metric_intervals(intervals)
+
+        get_scalar_average = partial(
+            self._get_scalar_average,
+            task_id=task_id,
+            company_id=company_id,
+            event_type=event_type,
+            key=key,
+        )
+        if run_parallel:
+            with ThreadPoolExecutor(max_workers=EventSettings.max_workers) as pool:
+                metrics = itertools.chain.from_iterable(
+                    pool.map(get_scalar_average, interval_groups)
+                )
+        else:
+            metrics = itertools.chain.from_iterable(
+                get_scalar_average(group) for group in interval_groups
+            )
+
+        ret = defaultdict(dict)
+        for metric_key, metric_values in metrics:
+            ret[metric_key].update(metric_values)
+
+        return ret
+
+    def compare_scalar_metrics_average_per_iter(
+        self,
+        companies: TaskCompanies,
+        samples,
+        key: ScalarKeyEnum,
+        metric_variants: MetricVariants = None,
+    ):
+        """
+        Compare scalar metrics for different tasks per metric and variant
+        The amount of points in each histogram should not exceed the requested samples
+        """
+        event_type = EventType.metrics_scalar
+        companies = {
+            company_id: tasks
+            for company_id, tasks in companies.items()
+            if not check_empty_data(
+                self.es, company_id=company_id, event_type=event_type
+            )
+        }
+        if not companies:
+            return {}
+
+        get_scalar_average_per_iter = partial(
+            self._get_scalar_average_per_iter_core,
+            event_type=event_type,
+            samples=samples,
+            key=ScalarKey.resolve(key),
+            metric_variants=metric_variants,
+            run_parallel=False,
+        )
+        task_ids, company_ids = zip(
+            *(
+                (t.id, t.company)
+                for t in itertools.chain.from_iterable(companies.values())
+            )
+        )
+        with ThreadPoolExecutor(max_workers=EventSettings.max_workers) as pool:
+            task_metrics = zip(
+                task_ids, pool.map(get_scalar_average_per_iter, task_ids, company_ids)
+            )
+
+        task_names = {
+            t.id: t.name for t in itertools.chain.from_iterable(companies.values())
+        }
+        res = defaultdict(lambda: defaultdict(dict))
+        for task_id, task_data in task_metrics:
+            task_name = task_names[task_id]
+            for metric_key, metric_data in task_data.items():
+                for variant_key, variant_data in metric_data.items():
+                    variant_data["name"] = task_name
+                    res[metric_key][variant_key][task_id] = variant_data
+
+        return res
+
+    def get_task_single_value_metrics(
+        self,
+        companies: TaskCompanies,
+        metric_variants: MetricVariants = None,
+    ) -> Mapping[str, dict]:
+        """
+        For the requested tasks return all the events delivered for the single iteration (-2**31)
+        """
+        companies = {
+            company_id: [t.id for t in tasks]
+            for company_id, tasks in companies.items()
+            if not check_empty_data(
+                self.es, company_id=company_id, event_type=EventType.metrics_scalar
+            )
+        }
+        if not companies:
+            return {}
+
+        with ThreadPoolExecutor(max_workers=EventSettings.max_workers) as pool:
+            task_events = list(
+                itertools.chain.from_iterable(
+                    pool.map(
+                        partial(
+                            self._get_task_single_value_metrics,
+                            metric_variants=metric_variants,
+                        ),
+                        companies.items(),
+                    )
+                ),
+            )
+
+        def _get_value(event: dict):
+            return {
+                field: event.get(field)
+                for field in ("metric", "variant", "value", "timestamp")
+            }
+
+        return {
+            task: [_get_value(e) for e in events]
+            for task, events in bucketize(task_events, itemgetter("task")).items()
+        }
+
+    def _get_task_single_value_metrics(
+        self, tasks: Tuple[str, Sequence[str]], metric_variants: MetricVariants = None
+    ) -> Sequence[dict]:
+        company_id, task_ids = tasks
+        must = [
+            {"terms": {"task": task_ids}},
+            {"term": {"iter": SINGLE_SCALAR_ITERATION}},
+        ]
+        if metric_variants:
+            must.append(get_metric_variants_condition(metric_variants))
+
+        es_req = {
+            "size": 10000,
+            "query": {"bool": {"must": must}},
+        }
+        with translate_errors_context():
+            es_res = search_company_events(
+                body=es_req,
+                es=self.es,
+                company_id=company_id,
+                event_type=EventType.metrics_scalar,
+            )
+            if not es_res["hits"]["total"]["value"]:
+                return []
+
+        return [hit["_source"] for hit in es_res["hits"]["hits"]]
+
+    MetricInterval = Tuple[str, str, int, int]
+    MetricIntervalGroup = Tuple[int, Sequence[Tuple[str, str]]]
+
+    @classmethod
+    def _group_task_metric_intervals(
+        cls, intervals: Sequence[MetricInterval]
+    ) -> Sequence[MetricIntervalGroup]:
+        """
+        Group task metric intervals so that the following conditions are meat:
+            - All the metrics in the same group have the same interval (with 10% rounding)
+            - The amount of metrics in the group does not exceed MAX_AGGS_ELEMENTS_COUNT
+            - The total count of samples in the group does not exceed MAX_SAMPLE_BUCKETS
+        """
+        metric_interval_groups = []
+        interval_group = []
+        group_interval_upper_bound = 0
+        group_max_interval = 0
+        group_samples = 0
+        for metric, variant, interval, size in sorted(intervals, key=itemgetter(2)):
+            if (
+                interval > group_interval_upper_bound
+                or (group_samples + size) > cls.MAX_SAMPLE_BUCKETS
+                or len(interval_group) >= cls.MAX_AGGS_ELEMENTS_COUNT
+            ):
+                if interval_group:
+                    metric_interval_groups.append((group_max_interval, interval_group))
+                    interval_group = []
+                group_max_interval = interval
+                group_interval_upper_bound = interval + int(interval * 0.1)
+                group_samples = 0
+            interval_group.append((metric, variant))
+            group_samples += size
+            group_max_interval = max(group_max_interval, interval)
+        if interval_group:
+            metric_interval_groups.append((group_max_interval, interval_group))
+
+        return metric_interval_groups
+
+    def _get_task_metric_intervals(
+        self,
+        company_id: str,
+        event_type: EventType,
+        task_id: str,
+        samples: int,
+        field: str = "iter",
+        metric_variants: MetricVariants = None,
+    ) -> Sequence[MetricInterval]:
+        """
+        Calculate interval per task metric variant so that the resulting
+        amount of points does not exceed sample.
+        Return the list og metric variant intervals as the following tuple:
+        (metric, variant, interval, samples)
+        """
+        must = self._task_conditions(task_id)
+        if metric_variants:
+            must.append(get_metric_variants_condition(metric_variants))
+        query = {"bool": {"must": must}}
+        search_args = dict(es=self.es, company_id=company_id, event_type=event_type)
+        max_metrics, max_variants = get_max_metric_and_variant_counts(
+            query=query,
+            **search_args,
+        )
+        max_variants = int(max_variants // 2)
+        es_req = {
+            "size": 0,
+            "query": query,
+            "aggs": {
+                "metrics": {
+                    "terms": {
+                        "field": "metric",
+                        "size": max_metrics,
+                        "order": {"_key": "asc"},
+                    },
+                    "aggs": {
+                        "variants": {
+                            "terms": {
+                                "field": "variant",
+                                "size": max_variants,
+                                "order": {"_key": "asc"},
+                            },
+                            "aggs": {
+                                "count": {"value_count": {"field": field}},
+                                "min_index": {"min": {"field": field}},
+                                "max_index": {"max": {"field": field}},
+                            },
+                        }
+                    },
+                }
+            },
+        }
+
+        es_res = search_company_events(body=es_req, **search_args)
+
+        aggs_result = es_res.get("aggregations")
+        if not aggs_result:
+            return []
+
+        return [
+            self._build_metric_interval(metric["key"], variant["key"], variant, samples)
+            for metric in aggs_result["metrics"]["buckets"]
+            for variant in metric["variants"]["buckets"]
+        ]
+
+    @staticmethod
+    def _build_metric_interval(
+        metric: str, variant: str, data: dict, samples: int
+    ) -> Tuple[str, str, int, int]:
+        """
+        Calculate index interval per metric_variant variant so that the
+        total amount of intervals does not exceeds the samples
+        Return the interval and resulting amount of intervals
+        """
+        count = nested_get(data, ("count", "value"), default=0)
+        if count < samples:
+            return metric, variant, 1, count
+
+        min_index = nested_get(data, ("min_index", "value"), default=0)
+        max_index = nested_get(data, ("max_index", "value"), default=min_index)
+        index_range = max_index - min_index + 1
+        interval = max(1, math.ceil(float(index_range) / samples))
+        max_samples = math.ceil(float(index_range) / interval)
+        return (
+            metric,
+            variant,
+            interval,
+            max_samples,
+        )
+
+    MetricData = Tuple[str, dict]
+
+    def _get_scalar_average(
+        self,
+        metrics_interval: MetricIntervalGroup,
+        task_id: str,
+        company_id: str,
+        event_type: EventType,
+        key: ScalarKey,
+    ) -> Sequence[MetricData]:
+        """
+        Retrieve scalar histograms per several metric variants that share the same interval
+        """
+        interval, metrics = metrics_interval
+        aggregation = self._add_aggregation_average(key.get_aggregation(interval))
+        query = self._get_task_metrics_query(task_id=task_id, metrics=metrics)
+        search_args = dict(es=self.es, company_id=company_id, event_type=event_type)
+        max_metrics, max_variants = get_max_metric_and_variant_counts(
+            query=query,
+            **search_args,
+        )
+        max_variants = int(max_variants // 2)
+        es_req = {
+            "size": 0,
+            "query": query,
+            "aggs": {
+                "metrics": {
+                    "terms": {
+                        "field": "metric",
+                        "size": max_metrics,
+                        "order": {"_key": "asc"},
+                    },
+                    "aggs": {
+                        "variants": {
+                            "terms": {
+                                "field": "variant",
+                                "size": max_variants,
+                                "order": {"_key": "asc"},
+                            },
+                            "aggs": aggregation,
+                        }
+                    },
+                }
+            },
+        }
+
+        with translate_errors_context():
+            es_res = search_company_events(body=es_req, **search_args)
+
+        aggs_result = es_res.get("aggregations")
+        if not aggs_result:
+            return {}
+
+        metrics = [
+            (
+                metric["key"],
+                {
+                    variant["key"]: {
+                        "name": variant["key"],
+                        **key.get_iterations_data(variant),
+                    }
+                    for variant in metric["variants"]["buckets"]
+                },
+            )
+            for metric in aggs_result["metrics"]["buckets"]
+        ]
+        return metrics
+
+    @staticmethod
+    def _add_aggregation_average(aggregation):
+        average_agg = {"avg_val": {"avg": {"field": "value"}}}
+        return {
+            key: {**value, "aggs": {**value.get("aggs", {}), **average_agg}}
+            for key, value in aggregation.items()
+        }
+
+    @staticmethod
+    def _task_conditions(task_id: str) -> list:
+        return [
+            {"term": {"task": task_id}},
+            {"range": {"iter": {"gt": SINGLE_SCALAR_ITERATION}}},
+        ]
+
+    @classmethod
+    def _get_task_metrics_query(
+        cls,
+        task_id: str,
+        metrics: Sequence[Tuple[str, str]],
+    ):
+        must = cls._task_conditions(task_id)
+        if metrics:
+            should = [
+                {
+                    "bool": {
+                        "must": [
+                            {"term": {"metric": metric}},
+                            {"term": {"variant": variant}},
+                        ]
+                    }
+                }
+                for metric, variant in metrics
+            ]
+            must.append({"bool": {"should": should}})
+
+        return {"bool": {"must": must}}
+
+    def get_multi_task_metrics(self, companies: TaskCompanies, event_type: EventType) -> Mapping[str, list]:
+        """
+        For the requested tasks return reported metrics and variants
+        """
+        tasks_ids = {
+            company: [t.id for t in tasks]
+            for company, tasks in companies.items()
+        }
+        with ThreadPoolExecutor(EventSettings.max_workers) as pool:
+            companies_res: Sequence = list(
+                pool.map(
+                    partial(
+                        self._get_multi_task_metrics,
+                        event_type=event_type,
+                    ),
+                    tasks_ids.items(),
+                )
+            )
+
+        if len(companies_res) == 1:
+            return companies_res[0]
+
+        res = defaultdict(set)
+        for c_res in companies_res:
+            for m, vars_ in c_res.items():
+                res[m].update(vars_)
+
+        return {
+            k: list(v)
+            for k, v in res.items()
+        }
+
+    def _get_multi_task_metrics(
+        self, company_tasks: Tuple[str, Sequence[str]], event_type: EventType
+    ) -> Mapping[str, list]:
+        company_id, task_ids = company_tasks
+        if check_empty_data(self.es, company_id, event_type):
+            return {}
+
+        search_args = dict(
+            es=self.es,
+            company_id=company_id,
+            event_type=event_type,
+        )
+        query = QueryBuilder.terms("task", task_ids)
+        max_metrics, max_variants = get_max_metric_and_variant_counts(
+            query=query,
+            **search_args,
+        )
+        es_req = {
+            "size": 0,
+            "query": query,
+            "aggs": {
+                "metrics": {
+                    "terms": {
+                        "field": "metric",
+                        "size": max_metrics,
+                        "order": {"_key": "asc"},
+                    },
+                    "aggs": {
+                        "variants": {
+                            "terms": {
+                                "field": "variant",
+                                "size": max_variants,
+                                "order": {"_key": "asc"},
+                            },
+                        }
+                    }
+                }
+            },
+        }
+
+        es_res = search_company_events(
+            body=es_req,
+            **search_args,
+        )
+        aggs_result = es_res.get("aggregations")
+        if not aggs_result:
+            return {}
+
+        return {
+            mb["key"]: [vb["key"] for vb in mb["variants"]["buckets"]]
+            for mb in aggs_result["metrics"]["buckets"]
+        }
+
+    def get_task_metrics(
+        self, company_id, task_ids: Sequence, event_type: EventType
+    ) -> Sequence:
+        """
+        For the requested tasks return reported metrics per task
+        """
+        if check_empty_data(self.es, company_id, event_type):
+            return {}
+
+        with ThreadPoolExecutor(EventSettings.max_workers) as pool:
+            res = pool.map(
+                partial(
+                    self._get_task_metrics,
+                    company_id=company_id,
+                    event_type=event_type,
+                ),
+                task_ids,
+            )
+        return list(zip(task_ids, res))
+
+    def _get_task_metrics(
+        self, task_id: str, company_id: str, event_type: EventType
+    ) -> Sequence:
+        es_req = {
+            "size": 0,
+            "query": {"bool": {"must": self._task_conditions(task_id)}},
+            "aggs": {
+                "metrics": {
+                    "terms": {
+                        "field": "metric",
+                        "size": EventSettings.max_es_buckets,
+                        "order": {"_key": "asc"},
+                    }
+                }
+            },
+        }
+
+        es_res = search_company_events(
+            self.es, company_id=company_id, event_type=event_type, body=es_req
+        )
+
+        return [
+            metric["key"]
+            for metric in nested_get(es_res, ("aggregations", "metrics", "buckets"), default=[])
+        ]

apiserver/bll/event/events_iterator.py

@@ -0,0 +1,197 @@
+from typing import Optional, Tuple, Sequence, Any
+
+import attr
+import jsonmodels.models
+import jwt
+from elasticsearch import Elasticsearch
+from jwt.algorithms import get_default_algorithms
+
+from apiserver.bll.event.event_common import (
+    check_empty_data,
+    search_company_events,
+    EventType,
+    MetricVariants,
+    get_metric_variants_condition,
+    count_company_events,
+)
+from apiserver.bll.event.scalar_key import ScalarKeyEnum, ScalarKey
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+
+
+@attr.s(auto_attribs=True)
+class TaskEventsResult:
+    total_events: int = 0
+    next_scroll_id: str = None
+    events: list = attr.Factory(list)
+
+
+class EventsIterator:
+    def __init__(self, es: Elasticsearch):
+        self.es = es
+
+    def get_task_events(
+        self,
+        event_type: EventType,
+        company_id: str,
+        task_id: str,
+        batch_size: int,
+        navigate_earlier: bool = True,
+        from_key_value: Optional[Any] = None,
+        metric_variants: MetricVariants = None,
+        key: ScalarKeyEnum = ScalarKeyEnum.timestamp,
+        **kwargs,
+    ) -> TaskEventsResult:
+        if check_empty_data(self.es, company_id, event_type):
+            return TaskEventsResult()
+
+        from_key_value = kwargs.pop("from_timestamp", from_key_value)
+
+        res = TaskEventsResult()
+        res.events, res.total_events = self._get_events(
+            event_type=event_type,
+            company_id=company_id,
+            task_id=task_id,
+            batch_size=batch_size,
+            navigate_earlier=navigate_earlier,
+            from_key_value=from_key_value,
+            metric_variants=metric_variants,
+            key=ScalarKey.resolve(key),
+        )
+        return res
+
+    def count_task_events(
+        self,
+        event_type: EventType,
+        company_id: str,
+        task_ids: Sequence[str],
+        metric_variants: MetricVariants = None,
+    ) -> int:
+        if check_empty_data(self.es, company_id, event_type):
+            return 0
+
+        query, _ = self._get_initial_query_and_must(task_ids, metric_variants)
+        es_req = {
+            "query": query,
+        }
+
+        with translate_errors_context():
+            es_result = count_company_events(
+                self.es, company_id=company_id, event_type=event_type, body=es_req,
+            )
+
+            return es_result["count"]
+
+    def _get_events(
+        self,
+        event_type: EventType,
+        company_id: str,
+        task_id: str,
+        batch_size: int,
+        navigate_earlier: bool,
+        key: ScalarKey,
+        from_key_value: Optional[Any],
+        metric_variants: MetricVariants = None,
+    ) -> Tuple[Sequence[dict], int]:
+        """
+        Return up to 'batch size' events starting from the previous key-field value (timestamp or iter) either in the
+        direction of earlier events (navigate_earlier=True) or in the direction of later events.
+        If from_key_field is not set then start either from latest or earliest.
+        For the last key-field value all the events are brought (even if the resulting size exceeds batch_size)
+        so that events with this value will not be lost between the calls.
+        """
+        query, must = self._get_initial_query_and_must([task_id], metric_variants)
+
+        # retrieve the next batch of events
+        es_req = {
+            "size": batch_size,
+            "query": query,
+            "sort": {key.field: "desc" if navigate_earlier else "asc"},
+        }
+
+        if from_key_value:
+            es_req["search_after"] = [from_key_value]
+
+        with translate_errors_context():
+            es_result = search_company_events(
+                self.es, company_id=company_id, event_type=event_type, body=es_req,
+            )
+            hits = es_result["hits"]["hits"]
+            hits_total = es_result["hits"]["total"]["value"]
+            if not hits:
+                return [], hits_total
+
+            events = [hit["_source"] for hit in hits]
+
+            # retrieve the events that match the last event timestamp
+            # but did not make it into the previous call due to batch_size limitation
+            es_req = {
+                "size": 10000,
+                "query": {
+                    "bool": {
+                        "must": must + [{"term": {key.field: events[-1][key.field]}}]
+                    }
+                },
+            }
+            es_result = search_company_events(
+                self.es, company_id=company_id, event_type=event_type, body=es_req,
+            )
+            last_second_hits = es_result["hits"]["hits"]
+            if not last_second_hits or len(last_second_hits) < 2:
+                # if only one element is returned for the last timestamp
+                # then it is already present in the events
+                return events, hits_total
+
+            already_present_ids = set(hit["_id"] for hit in hits)
+            last_second_events = [
+                hit["_source"]
+                for hit in last_second_hits
+                if hit["_id"] not in already_present_ids
+            ]
+
+            # return the list merged from original query results +
+            # leftovers from the last timestamp
+            return (
+                [*events, *last_second_events],
+                hits_total,
+            )
+
+    @staticmethod
+    def _get_initial_query_and_must(
+        task_ids: Sequence[str], metric_variants: MetricVariants = None
+    ) -> Tuple[dict, list]:
+        if not metric_variants:
+            query = {"terms": {"task": task_ids}}
+            must = [query]
+        else:
+            must = [
+                {"terms": {"task": task_ids}},
+                get_metric_variants_condition(metric_variants),
+            ]
+            query = {"bool": {"must": must}}
+        return query, must
+
+
+class Scroll(jsonmodels.models.Base):
+    def get_scroll_id(self) -> str:
+        return jwt.encode(
+            self.to_struct(),
+            key=config.get(
+                "services.events.events_retrieval.scroll_id_key", "1234567890"
+            ),
+        )
+
+    @classmethod
+    def from_scroll_id(cls, scroll_id: str):
+        try:
+            return cls(
+                **jwt.decode(
+                    scroll_id,
+                    key=config.get(
+                        "services.events.events_retrieval.scroll_id_key", "1234567890"
+                    ),
+                    algorithms=get_default_algorithms(),
+                )
+            )
+        except jwt.PyJWTError:
+            raise ValueError("Invalid Scroll ID")

apiserver/bll/event/history_debug_image_iterator.py

@@ -0,0 +1,455 @@
+import operator
+from operator import attrgetter
+from typing import Sequence, Tuple, Optional, Mapping
+
+import attr
+from boltons.iterutils import first, bucketize
+from elasticsearch import Elasticsearch
+from jsonmodels.fields import StringField, IntField, BoolField, ListField
+from jsonmodels.models import Base
+from redis.client import StrictRedis
+
+from apiserver.utilities.dicts import nested_get
+from .event_common import (
+    EventType,
+    EventSettings,
+    check_empty_data,
+    search_company_events,
+    get_max_metric_and_variant_counts,
+)
+from apiserver.apimodels import JsonSerializableMixin
+from apiserver.bll.redis_cache_manager import RedisCacheManager
+from apiserver.apierrors import errors
+
+
+class VariantState(Base):
+    name: str = StringField(required=True)
+    metric: str = StringField(default=None)
+    min_iteration: int = IntField()
+    max_iteration: int = IntField()
+
+
+class DebugImageSampleState(Base, JsonSerializableMixin):
+    id: str = StringField(required=True)
+    iteration: int = IntField()
+    variant: str = StringField()
+    task: str = StringField()
+    metric: str = StringField()
+    variant_states: Sequence[VariantState] = ListField([VariantState])
+    warning: str = StringField()
+    navigate_current_metric = BoolField(default=True)
+
+
+@attr.s(auto_attribs=True)
+class VariantSampleResult(object):
+    scroll_id: str = None
+    event: dict = None
+    min_iteration: int = None
+    max_iteration: int = None
+
+
+class HistoryDebugImageIterator:
+    event_type = EventType.metrics_image
+
+    def __init__(self, redis: StrictRedis, es: Elasticsearch):
+        self.es = es
+        self.cache_manager = RedisCacheManager(
+            state_class=DebugImageSampleState,
+            redis=redis,
+            expiration_interval=EventSettings.state_expiration_sec,
+        )
+
+    def get_next_sample(
+        self,
+        company_id: str,
+        task: str,
+        state_id: str,
+        navigate_earlier: bool,
+        next_iteration: bool,
+    ) -> VariantSampleResult:
+        """
+        Get the sample for next/prev variant on the current iteration
+        If does not exist then try getting sample for the first/last variant from next/prev iteration
+        """
+        res = VariantSampleResult(scroll_id=state_id)
+        state = self.cache_manager.get_state(state_id)
+        if not state or state.task != task:
+            raise errors.bad_request.InvalidScrollId(scroll_id=state_id)
+
+        if check_empty_data(self.es, company_id=company_id, event_type=self.event_type):
+            return res
+
+        if next_iteration:
+            event = self._get_next_for_another_iteration(
+                company_id=company_id, navigate_earlier=navigate_earlier, state=state
+            )
+        else:
+            # noinspection PyArgumentList
+            event = first(
+                f(company_id=company_id, navigate_earlier=navigate_earlier, state=state)
+                for f in (
+                    self._get_next_for_current_iteration,
+                    self._get_next_for_another_iteration,
+                )
+            )
+        if not event:
+            return res
+
+        self._fill_res_and_update_state(event=event, res=res, state=state)
+        self.cache_manager.set_state(state=state)
+        return res
+
+    @staticmethod
+    def _fill_res_and_update_state(
+        event: dict, res: VariantSampleResult, state: DebugImageSampleState
+    ):
+        state.variant = event["variant"]
+        state.metric = event["metric"]
+        state.iteration = event["iter"]
+        res.event = event
+        var_state = first(
+            vs
+            for vs in state.variant_states
+            if vs.name == state.variant and vs.metric == state.metric
+        )
+        if var_state:
+            res.min_iteration = var_state.min_iteration
+            res.max_iteration = var_state.max_iteration
+
+    @staticmethod
+    def _get_metric_conditions(variants: Sequence[VariantState]) -> dict:
+        metrics = bucketize(variants, key=attrgetter("metric"))
+
+        def _get_variants_conditions(metric_variants: Sequence[VariantState]) -> dict:
+            variants_conditions = [
+                {
+                    "bool": {
+                        "must": [
+                            {"term": {"variant": v.name}},
+                            {"range": {"iter": {"gte": v.min_iteration}}},
+                        ]
+                    }
+                }
+                for v in metric_variants
+            ]
+            return {"bool": {"should": variants_conditions}}
+
+        metrics_conditions = [
+            {
+                "bool": {
+                    "must": [
+                        {"term": {"metric": metric}},
+                        _get_variants_conditions(metric_variants),
+                    ]
+                }
+            }
+            for metric, metric_variants in metrics.items()
+        ]
+        return {"bool": {"should": metrics_conditions}}
+
+    def _get_next_for_current_iteration(
+        self, company_id: str, navigate_earlier: bool, state: DebugImageSampleState
+    ) -> Optional[dict]:
+        """
+        Get the sample for next (if navigate_earlier is False) or previous variant sorted by name for the same iteration
+        Only variants for which the iteration falls into their valid range are considered
+        Return None if no such variant or sample is found
+        """
+        if state.navigate_current_metric:
+            variants = [
+                var_state
+                for var_state in state.variant_states
+                if var_state.metric == state.metric
+            ]
+        else:
+            variants = state.variant_states
+
+        cmp = operator.lt if navigate_earlier else operator.gt
+        variants = [
+            var_state
+            for var_state in variants
+            if cmp((var_state.metric, var_state.name), (state.metric, state.variant))
+            and var_state.min_iteration <= state.iteration
+        ]
+        if not variants:
+            return
+
+        must_conditions = [
+            {"term": {"task": state.task}},
+            {"term": {"iter": state.iteration}},
+            self._get_metric_conditions(variants),
+            {"exists": {"field": "url"}},
+        ]
+        order = "desc" if navigate_earlier else "asc"
+        es_req = {
+            "size": 1,
+            "sort": [{"metric": order}, {"variant": order}],
+            "query": {"bool": {"must": must_conditions}},
+        }
+
+        es_res = search_company_events(
+            self.es,
+            company_id=company_id,
+            event_type=self.event_type,
+            body=es_req,
+        )
+
+        hits = nested_get(es_res, ("hits", "hits"))
+        if not hits:
+            return
+
+        return hits[0]["_source"]
+
+    def _get_next_for_another_iteration(
+        self, company_id: str, navigate_earlier: bool, state: DebugImageSampleState
+    ) -> Optional[dict]:
+        """
+        Get the sample for the first variant for the next iteration (if navigate_earlier is set to False)
+        or from the last variant for the previous iteration (otherwise)
+        The variants for which the sample falls in invalid range are discarded
+        If no suitable sample is found then None is returned
+        """
+        if state.navigate_current_metric:
+            variants = [
+                var_state
+                for var_state in state.variant_states
+                if var_state.metric == state.metric
+            ]
+        else:
+            variants = state.variant_states
+
+        if navigate_earlier:
+            range_operator = "lt"
+            order = "desc"
+            variants = [
+                var_state
+                for var_state in variants
+                if var_state.min_iteration < state.iteration
+            ]
+        else:
+            range_operator = "gt"
+            order = "asc"
+            variants = variants
+
+        if not variants:
+            return
+
+        must_conditions = [
+            {"term": {"task": state.task}},
+            self._get_metric_conditions(variants),
+            {"range": {"iter": {range_operator: state.iteration}}},
+            {"exists": {"field": "url"}},
+        ]
+        es_req = {
+            "size": 1,
+            "sort": [{"iter": order}, {"metric": order}, {"variant": order}],
+            "query": {"bool": {"must": must_conditions}},
+        }
+        es_res = search_company_events(
+            self.es,
+            company_id=company_id,
+            event_type=self.event_type,
+            body=es_req,
+        )
+
+        hits = nested_get(es_res, ("hits", "hits"))
+        if not hits:
+            return
+
+        return hits[0]["_source"]
+
+    def get_sample_for_variant(
+        self,
+        company_id: str,
+        task: str,
+        metric: str,
+        variant: str,
+        iteration: Optional[int] = None,
+        refresh: bool = False,
+        state_id: str = None,
+        navigate_current_metric: bool = True,
+    ) -> VariantSampleResult:
+        """
+        Get the sample for the requested iteration or the latest before it
+        If the iteration is not passed then get the latest event
+        """
+        res = VariantSampleResult()
+        if check_empty_data(self.es, company_id=company_id, event_type=self.event_type):
+            return res
+
+        def init_state(state_: DebugImageSampleState):
+            state_.task = task
+            state_.metric = metric
+            state_.navigate_current_metric = navigate_current_metric
+            self._reset_variant_states(company_id=company_id, state=state_)
+
+        def validate_state(state_: DebugImageSampleState):
+            if (
+                state_.task != task
+                or state_.navigate_current_metric != navigate_current_metric
+                or (state_.navigate_current_metric and state_.metric != metric)
+            ):
+                raise errors.bad_request.InvalidScrollId(
+                    "Task and metric stored in the state do not match the passed ones",
+                    scroll_id=state_.id,
+                )
+            # fix old variant states:
+            for vs in state_.variant_states:
+                if vs.metric is None:
+                    vs.metric = metric
+            if refresh:
+                self._reset_variant_states(company_id=company_id, state=state_)
+
+        state: DebugImageSampleState
+        with self.cache_manager.get_or_create_state(
+            state_id=state_id, init_state=init_state, validate_state=validate_state,
+        ) as state:
+            res.scroll_id = state.id
+
+            var_state = first(
+                vs
+                for vs in state.variant_states
+                if vs.name == variant and vs.metric == metric
+            )
+            if not var_state:
+                return res
+
+            res.min_iteration = var_state.min_iteration
+            res.max_iteration = var_state.max_iteration
+
+            must_conditions = [
+                {"term": {"task": task}},
+                {"term": {"metric": metric}},
+                {"term": {"variant": variant}},
+                {"exists": {"field": "url"}},
+            ]
+            if iteration is not None:
+                must_conditions.append(
+                    {
+                        "range": {
+                            "iter": {"lte": iteration, "gte": var_state.min_iteration}
+                        }
+                    }
+                )
+            else:
+                must_conditions.append(
+                    {"range": {"iter": {"gte": var_state.min_iteration}}}
+                )
+
+            es_req = {
+                "size": 1,
+                "sort": {"iter": "desc"},
+                "query": {"bool": {"must": must_conditions}},
+            }
+
+            es_res = search_company_events(
+                self.es,
+                company_id=company_id,
+                event_type=self.event_type,
+                body=es_req,
+            )
+
+            hits = nested_get(es_res, ("hits", "hits"))
+            if not hits:
+                return res
+
+            self._fill_res_and_update_state(
+                event=hits[0]["_source"], res=res, state=state
+            )
+            return res
+
+    def _reset_variant_states(self, company_id: str, state: DebugImageSampleState):
+        metrics = self._get_metric_variant_iterations(
+            company_id=company_id,
+            task=state.task,
+            metric=state.metric if state.navigate_current_metric else None,
+        )
+        state.variant_states = [
+            VariantState(
+                metric=metric,
+                name=var_name,
+                min_iteration=min_iter,
+                max_iteration=max_iter,
+            )
+            for metric, variants in metrics.items()
+            for var_name, min_iter, max_iter in variants
+        ]
+
+    def _get_metric_variant_iterations(
+        self, company_id: str, task: str, metric: str,
+    ) -> Mapping[str, Sequence[Tuple[str, int, int]]]:
+        """
+        Return valid min and max iterations that the task reported events of the required type
+        """
+        must = [
+            {"term": {"task": task}},
+            {"exists": {"field": "url"}},
+        ]
+        if metric is not None:
+            must.append({"term": {"metric": metric}})
+        query = {"bool": {"must": must}}
+
+        search_args = dict(
+            es=self.es, company_id=company_id, event_type=self.event_type,
+        )
+        max_metrics, max_variants = get_max_metric_and_variant_counts(
+            query=query, **search_args
+        )
+        max_variants = int(max_variants // 2)
+        es_req: dict = {
+            "size": 0,
+            "query": query,
+            "aggs": {
+                "metrics": {
+                    "terms": {
+                        "field": "metric",
+                        "size": max_metrics,
+                        "order": {"_key": "asc"},
+                    },
+                    "aggs": {
+                        "variants": {
+                            "terms": {
+                                "field": "variant",
+                                "size": max_variants,
+                                "order": {"_key": "asc"},
+                            },
+                            "aggs": {
+                                "last_iter": {"max": {"field": "iter"}},
+                                "urls": {
+                                    # group by urls and choose the minimal iteration
+                                    # from all the maximal iterations per url
+                                    "terms": {
+                                        "field": "url",
+                                        "order": {"max_iter": "asc"},
+                                        "size": 1,
+                                    },
+                                    "aggs": {
+                                        # find max iteration for each url
+                                        "max_iter": {"max": {"field": "iter"}}
+                                    },
+                                },
+                            },
+                        }
+                    },
+                }
+            },
+        }
+
+        es_res = search_company_events(body=es_req, **search_args)
+
+        def get_variant_data(variant_bucket: dict) -> Tuple[str, int, int]:
+            variant = variant_bucket["key"]
+            urls = nested_get(variant_bucket, ("urls", "buckets"))
+            min_iter = int(urls[0]["max_iter"]["value"])
+            max_iter = int(variant_bucket["last_iter"]["value"])
+            return variant, min_iter, max_iter
+
+        return {
+            metric_bucket["key"]: [
+                get_variant_data(variant_bucket)
+                for variant_bucket in nested_get(metric_bucket, ("variants", "buckets"))
+            ]
+            for metric_bucket in nested_get(
+                es_res, ("aggregations", "metrics", "buckets")
+            )
+        }

apiserver/bll/event/history_plots_iterator.py

@@ -0,0 +1,316 @@
+from typing import Sequence, Tuple, Optional, Mapping
+
+import attr
+from boltons.iterutils import first
+from elasticsearch import Elasticsearch
+from jsonmodels.fields import StringField, IntField, ListField, BoolField
+from jsonmodels.models import Base
+from redis.client import StrictRedis
+
+from .event_common import (
+    EventType,
+    uncompress_plot,
+    EventSettings,
+    check_empty_data,
+    search_company_events,
+)
+from apiserver.apimodels import JsonSerializableMixin
+from apiserver.utilities.dicts import nested_get
+from apiserver.bll.redis_cache_manager import RedisCacheManager
+from apiserver.apierrors import errors
+
+
+class MetricState(Base):
+    name: str = StringField(default=None)
+    min_iteration: int = IntField()
+    max_iteration: int = IntField()
+
+
+class PlotsSampleState(Base, JsonSerializableMixin):
+    id: str = StringField(required=True)
+    iteration: int = IntField()
+    task: str = StringField()
+    metric: str = StringField()
+    metric_states: Sequence[MetricState] = ListField([MetricState])
+    warning: str = StringField()
+    navigate_current_metric = BoolField(default=True)
+
+
+@attr.s(auto_attribs=True)
+class MetricSamplesResult(object):
+    scroll_id: str = None
+    events: list = []
+    min_iteration: int = None
+    max_iteration: int = None
+
+
+class HistoryPlotsIterator:
+    event_type = EventType.metrics_plot
+
+    def __init__(self, redis: StrictRedis, es: Elasticsearch):
+        self.es = es
+        self.cache_manager = RedisCacheManager(
+            state_class=PlotsSampleState,
+            redis=redis,
+            expiration_interval=EventSettings.state_expiration_sec,
+        )
+
+    def get_next_sample(
+        self,
+        company_id: str,
+        task: str,
+        state_id: str,
+        navigate_earlier: bool,
+        next_iteration: bool,
+    ) -> MetricSamplesResult:
+        """
+        Get the samples for next/prev metric on the current iteration
+        If does not exist then try getting sample for the first/last metric from next/prev iteration
+        """
+        res = MetricSamplesResult(scroll_id=state_id)
+        state = self.cache_manager.get_state(state_id)
+        if not state or state.task != task:
+            raise errors.bad_request.InvalidScrollId(scroll_id=state_id)
+
+        if check_empty_data(self.es, company_id=company_id, event_type=self.event_type):
+            return res
+
+        if navigate_earlier:
+            range_operator = "lt"
+            order = "desc"
+        else:
+            range_operator = "gt"
+            order = "asc"
+
+        must_conditions = [
+            {"term": {"task": state.task}},
+        ]
+        if state.navigate_current_metric:
+            must_conditions.append({"term": {"metric": state.metric}})
+
+        next_iteration_condition = {
+            "range": {"iter": {range_operator: state.iteration}}
+        }
+        if next_iteration or state.navigate_current_metric:
+            must_conditions.append(next_iteration_condition)
+        else:
+            next_metric_condition = {
+                "bool": {
+                    "must": [
+                        {"term": {"iter": state.iteration}},
+                        {"range": {"metric": {range_operator: state.metric}}},
+                    ]
+                }
+            }
+            must_conditions.append(
+                {"bool": {"should": [next_metric_condition, next_iteration_condition]}}
+            )
+
+        events = self._get_metric_events_for_condition(
+            company_id=company_id,
+            task=state.task,
+            order=order,
+            must_conditions=must_conditions,
+        )
+
+        if not events:
+            return res
+
+        self._fill_res_and_update_state(events=events, res=res, state=state)
+        self.cache_manager.set_state(state=state)
+        return res
+
+    def get_samples_for_metric(
+        self,
+        company_id: str,
+        task: str,
+        metric: str,
+        iteration: Optional[int] = None,
+        refresh: bool = False,
+        state_id: str = None,
+        navigate_current_metric: bool = True,
+    ) -> MetricSamplesResult:
+        """
+        Get the sample for the requested iteration or the latest before it
+        If the iteration is not passed then get the latest event
+        """
+        res = MetricSamplesResult()
+        if check_empty_data(self.es, company_id=company_id, event_type=self.event_type):
+            return res
+
+        def init_state(state_: PlotsSampleState):
+            state_.task = task
+            state_.metric = metric
+            state_.navigate_current_metric = navigate_current_metric
+            self._reset_metric_states(company_id=company_id, state=state_)
+
+        def validate_state(state_: PlotsSampleState):
+            if (
+                state_.task != task
+                or state_.navigate_current_metric != navigate_current_metric
+                or (state_.navigate_current_metric and state_.metric != metric)
+            ):
+                raise errors.bad_request.InvalidScrollId(
+                    "Task and metric stored in the state do not match the passed ones",
+                    scroll_id=state_.id,
+                )
+            if refresh:
+                self._reset_metric_states(company_id=company_id, state=state_)
+
+        state: PlotsSampleState
+        with self.cache_manager.get_or_create_state(
+            state_id=state_id, init_state=init_state, validate_state=validate_state,
+        ) as state:
+            res.scroll_id = state.id
+
+            metric_state = first(ms for ms in state.metric_states if ms.name == metric)
+            if not metric_state:
+                return res
+
+            res.min_iteration = metric_state.min_iteration
+            res.max_iteration = metric_state.max_iteration
+
+            must_conditions = [
+                {"term": {"task": task}},
+                {"term": {"metric": metric}},
+            ]
+            if iteration is not None:
+                must_conditions.append({"range": {"iter": {"lte": iteration}}})
+
+            events = self._get_metric_events_for_condition(
+                company_id=company_id,
+                task=state.task,
+                order="desc",
+                must_conditions=must_conditions,
+            )
+            if not events:
+                return res
+
+            self._fill_res_and_update_state(events=events, res=res, state=state)
+            return res
+
+    def _reset_metric_states(self, company_id: str, state: PlotsSampleState):
+        metrics = self._get_metric_iterations(
+            company_id=company_id,
+            task=state.task,
+            metric=state.metric if state.navigate_current_metric else None,
+        )
+        state.metric_states = [
+            MetricState(name=metric, min_iteration=min_iter, max_iteration=max_iter)
+            for metric, (min_iter, max_iter) in metrics.items()
+        ]
+
+    def _get_metric_iterations(
+        self, company_id: str, task: str, metric: str,
+    ) -> Mapping[str, Tuple[int, int]]:
+        """
+        Return valid min and max iterations that the task reported events of the required type
+        """
+        must = [
+            {"term": {"task": task}},
+        ]
+        if metric is not None:
+            must.append({"term": {"metric": metric}})
+        query = {"bool": {"must": must}}
+
+        es_req: dict = {
+            "size": 0,
+            "query": query,
+            "aggs": {
+                "metrics": {
+                    "terms": {
+                        "field": "metric",
+                        "size": 5000,
+                        "order": {"_key": "asc"},
+                    },
+                    "aggs": {
+                        "last_iter": {"max": {"field": "iter"}},
+                        "first_iter": {"min": {"field": "iter"}},
+                    },
+                }
+            },
+        }
+
+        es_res = search_company_events(
+            body=es_req,
+            es=self.es,
+            company_id=company_id,
+            event_type=self.event_type,
+        )
+
+        return {
+            metric_bucket["key"]: (
+                int(metric_bucket["first_iter"]["value"]),
+                int(metric_bucket["last_iter"]["value"]),
+            )
+            for metric_bucket in nested_get(
+                es_res, ("aggregations", "metrics", "buckets")
+            )
+        }
+
+    @staticmethod
+    def _fill_res_and_update_state(
+        events: Sequence[dict], res: MetricSamplesResult, state: PlotsSampleState
+    ):
+        for event in events:
+            uncompress_plot(event)
+        state.metric = events[0]["metric"]
+        state.iteration = events[0]["iter"]
+        res.events = events
+        metric_state = first(
+            ms for ms in state.metric_states if ms.name == state.metric
+        )
+        if metric_state:
+            res.min_iteration = metric_state.min_iteration
+            res.max_iteration = metric_state.max_iteration
+
+    def _get_metric_events_for_condition(
+        self, company_id: str, task: str, order: str, must_conditions: Sequence
+    ) -> Sequence:
+        es_req = {
+            "size": 0,
+            "query": {"bool": {"must": must_conditions}},
+            "aggs": {
+                "iters": {
+                    "terms": {"field": "iter", "size": 1, "order": {"_key": order}},
+                    "aggs": {
+                        "metrics": {
+                            "terms": {
+                                "field": "metric",
+                                "size": 1,
+                                "order": {"_key": order},
+                            },
+                            "aggs": {
+                                "events": {
+                                    "top_hits": {
+                                        "sort": {"variant": {"order": "asc"}},
+                                        "size": 100,
+                                    }
+                                }
+                            },
+                        },
+                    },
+                }
+            },
+        }
+        es_res = search_company_events(
+            self.es,
+            company_id=company_id,
+            event_type=self.event_type,
+            body=es_req,
+        )
+
+        aggs_result = es_res.get("aggregations")
+        if not aggs_result:
+            return []
+
+        for level in ("iters", "metrics"):
+            level_data = aggs_result[level]["buckets"]
+            if not level_data:
+                return []
+            aggs_result = level_data[0]
+
+        return [
+            hit["_source"]
+            for hit in nested_get(aggs_result, ("events", "hits", "hits"))
+        ]

apiserver/bll/event/metric_debug_images_iterator.py

@@ -0,0 +1,53 @@
+from typing import Sequence, Tuple, Callable
+
+from elasticsearch import Elasticsearch
+from redis.client import StrictRedis
+
+from apiserver.utilities.dicts import nested_get
+from .event_common import EventType
+from .metric_events_iterator import MetricEventsIterator, VariantState
+
+
+class MetricDebugImagesIterator(MetricEventsIterator):
+    def __init__(self, redis: StrictRedis, es: Elasticsearch):
+        super().__init__(redis, es, EventType.metrics_image)
+
+    def _get_extra_conditions(self) -> Sequence[dict]:
+        return [{"exists": {"field": "url"}}]
+
+    def _get_variant_state_aggs(self) -> Tuple[dict, Callable[[dict, VariantState], None]]:
+        aggs = {
+            "urls": {
+                "terms": {
+                    "field": "url",
+                    "order": {"max_iter": "desc"},
+                    "size": 1,  # we need only one url from the most recent iteration
+                },
+                "aggs": {
+                    "max_iter": {"max": {"field": "iter"}},
+                    "iters": {
+                        "top_hits": {
+                            "sort": {"iter": {"order": "desc"}},
+                            "size": 2,  # need two last iterations so that we can take
+                            # the second one as invalid
+                            "_source": "iter",
+                        }
+                    },
+                },
+            }
+        }
+
+        def fill_variant_state_data(variant_bucket: dict,  state: VariantState):
+            """If the image urls get recycled then fill the last_invalid_iteration field"""
+            top_iter_url = nested_get(variant_bucket, ("urls", "buckets"))[0]
+            iters = nested_get(top_iter_url, ("iters", "hits", "hits"))
+            if len(iters) > 1:
+                state.last_invalid_iteration = nested_get(iters[1], ("_source", "iter"))
+
+        return aggs, fill_variant_state_data
+
+    def _process_event(self, event: dict) -> dict:
+        return event
+
+    def _get_same_variant_events_order(self) -> dict:
+        return {"url": {"order": "desc"}}

apiserver/bll/event/metric_events_iterator.py

@@ -0,0 +1,451 @@
+import abc
+from concurrent.futures.thread import ThreadPoolExecutor
+from datetime import datetime
+from functools import partial
+from operator import itemgetter
+from typing import Sequence, Tuple, Optional, Mapping, Callable
+
+import attr
+from boltons.iterutils import first
+from elasticsearch import Elasticsearch
+from jsonmodels.fields import StringField, ListField, IntField
+from jsonmodels.models import Base
+from redis import StrictRedis
+
+from apiserver.apimodels import JsonSerializableMixin
+from apiserver.bll.event.event_common import (
+    EventSettings,
+    check_empty_data,
+    search_company_events,
+    EventType,
+    get_metric_variants_condition,
+    get_max_metric_and_variant_counts,
+)
+from apiserver.bll.redis_cache_manager import RedisCacheManager
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.task.metrics import MetricEventStats
+from apiserver.database.model.task.task import Task
+from apiserver.utilities.dicts import nested_get
+
+
+class VariantState(Base):
+    variant: str = StringField(required=True)
+    last_invalid_iteration: int = IntField()
+
+
+class MetricState(Base):
+    metric: str = StringField(required=True)
+    variants: Sequence[VariantState] = ListField([VariantState], required=True)
+    timestamp: int = IntField(default=0)
+
+
+class TaskScrollState(Base):
+    task: str = StringField(required=True)
+    metrics: Sequence[MetricState] = ListField([MetricState], required=True)
+    last_min_iter: Optional[int] = IntField()
+    last_max_iter: Optional[int] = IntField()
+
+    def reset(self):
+        """Reset the scrolling state for the metric"""
+        self.last_min_iter = self.last_max_iter = None
+
+
+class MetricEventsScrollState(Base, JsonSerializableMixin):
+    id: str = StringField(required=True)
+    tasks: Sequence[TaskScrollState] = ListField([TaskScrollState])
+    warning: str = StringField()
+
+
+@attr.s(auto_attribs=True)
+class MetricEventsResult(object):
+    metric_events: Sequence[tuple] = []
+    next_scroll_id: str = None
+
+
+class MetricEventsIterator:
+    def __init__(self, redis: StrictRedis, es: Elasticsearch, event_type: EventType):
+        self.es = es
+        self.event_type = event_type
+        self.cache_manager = RedisCacheManager(
+            state_class=MetricEventsScrollState,
+            redis=redis,
+            expiration_interval=EventSettings.state_expiration_sec,
+        )
+
+    def get_task_events(
+        self,
+        companies: Mapping[str, str],
+        task_metrics: Mapping[str, dict],
+        iter_count: int,
+        navigate_earlier: bool = True,
+        refresh: bool = False,
+        state_id: str = None,
+    ) -> MetricEventsResult:
+        companies = {
+            task_id: company_id
+            for task_id, company_id in companies.items()
+            if not check_empty_data(
+                self.es, company_id=company_id, event_type=self.event_type
+            )
+        }
+        if not companies:
+            return MetricEventsResult()
+
+        def init_state(state_: MetricEventsScrollState):
+            state_.tasks = self._init_task_states(companies, task_metrics)
+
+        def validate_state(state_: MetricEventsScrollState):
+            """
+            Validate that the metrics stored in the state are the same
+            as requested in the current call.
+            Refresh the state if requested
+            """
+            if refresh:
+                self._reinit_outdated_task_states(companies, state_, task_metrics)
+
+        with self.cache_manager.get_or_create_state(
+            state_id=state_id, init_state=init_state, validate_state=validate_state
+        ) as state:
+            res = MetricEventsResult(next_scroll_id=state.id)
+            specific_variants_requested = any(
+                variants
+                for t, metrics in task_metrics.items()
+                if metrics
+                for m, variants in metrics.items()
+            )
+            with ThreadPoolExecutor(EventSettings.max_workers) as pool:
+                res.metric_events = list(
+                    pool.map(
+                        partial(
+                            self._get_task_metric_events,
+                            companies=companies,
+                            iter_count=iter_count,
+                            navigate_earlier=navigate_earlier,
+                            specific_variants_requested=specific_variants_requested,
+                        ),
+                        state.tasks,
+                    )
+                )
+
+            return res
+
+    def _reinit_outdated_task_states(
+        self,
+        companies: Mapping[str, str],
+        state: MetricEventsScrollState,
+        task_metrics: Mapping[str, dict],
+    ):
+        """
+        Determine the metrics for which new event_type events were added
+        since their states were initialized and re-init these states
+        """
+        tasks = Task.objects(id__in=list(task_metrics)).only("id", "metric_stats")
+
+        def get_last_update_times_for_task_metrics(
+            task: Task,
+        ) -> Mapping[str, datetime]:
+            """For metrics that reported event_type events get mapping of the metric name to the last update times"""
+            metric_stats: Mapping[str, MetricEventStats] = task.metric_stats
+            if not metric_stats:
+                return {}
+
+            requested_metrics = task_metrics[task.id]
+            return {
+                stats.metric: stats.event_stats_by_type[
+                    self.event_type.value
+                ].last_update
+                for stats in metric_stats.values()
+                if self.event_type.value in stats.event_stats_by_type
+                and (not requested_metrics or stats.metric in requested_metrics)
+            }
+
+        update_times = {
+            task.id: get_last_update_times_for_task_metrics(task) for task in tasks
+        }
+        task_metric_states = {
+            task_state.task: {
+                metric_state.metric: metric_state for metric_state in task_state.metrics
+            }
+            for task_state in state.tasks
+        }
+        task_metrics_to_recalc = {}
+        for task, metrics_times in update_times.items():
+            old_metric_states = task_metric_states[task]
+            metrics_to_recalc = {
+                m: task_metrics[task].get(m)
+                for m, t in metrics_times.items()
+                if m not in old_metric_states or old_metric_states[m].timestamp < t
+            }
+            if metrics_to_recalc:
+                task_metrics_to_recalc[task] = metrics_to_recalc
+
+        updated_task_states = self._init_task_states(companies, task_metrics_to_recalc)
+
+        def merge_with_updated_task_states(
+            old_state: TaskScrollState, updates: Sequence[TaskScrollState]
+        ) -> TaskScrollState:
+            task = old_state.task
+            updated_state = first(uts for uts in updates if uts.task == task)
+            if not updated_state:
+                old_state.reset()
+                return old_state
+
+            updated_metrics = [m.metric for m in updated_state.metrics]
+            return TaskScrollState(
+                task=task,
+                metrics=[
+                    *updated_state.metrics,
+                    *(
+                        old_metric
+                        for old_metric in old_state.metrics
+                        if old_metric.metric not in updated_metrics
+                    ),
+                ],
+            )
+
+        state.tasks = [
+            merge_with_updated_task_states(task_state, updated_task_states)
+            for task_state in state.tasks
+        ]
+
+    def _init_task_states(
+        self, companies: Mapping[str, str], task_metrics: Mapping[str, dict]
+    ) -> Sequence[TaskScrollState]:
+        """
+        Returned initialized metric scroll stated for the requested task metrics
+        """
+        with ThreadPoolExecutor(EventSettings.max_workers) as pool:
+            task_metric_states = pool.map(
+                partial(self._init_metric_states_for_task, companies=companies),
+                task_metrics.items(),
+            )
+
+        return [
+            TaskScrollState(task=task, metrics=metric_states,)
+            for task, metric_states in zip(task_metrics, task_metric_states)
+        ]
+
+    @abc.abstractmethod
+    def _get_extra_conditions(self) -> Sequence[dict]:
+        pass
+
+    @abc.abstractmethod
+    def _get_variant_state_aggs(
+        self,
+    ) -> Tuple[dict, Callable[[dict, VariantState], None]]:
+        pass
+
+    def _init_metric_states_for_task(
+        self, task_metrics: Tuple[str, dict], companies: Mapping[str, str]
+    ) -> Sequence[MetricState]:
+        """
+        Return metric scroll states for the task filled with the variant states
+        for the variants that reported any event_type events
+        """
+        task, metrics = task_metrics
+        company_id = companies[task]
+        must = [{"term": {"task": task}}, *self._get_extra_conditions()]
+        if metrics:
+            must.append(get_metric_variants_condition(metrics))
+        query = {"bool": {"must": must}}
+
+        search_args = dict(
+            es=self.es, company_id=company_id, event_type=self.event_type
+        )
+        max_metrics, max_variants = get_max_metric_and_variant_counts(
+            query=query, **search_args
+        )
+        max_variants = int(max_variants // 2)
+        variant_state_aggs, fill_variant_state_data = self._get_variant_state_aggs()
+        es_req: dict = {
+            "size": 0,
+            "query": query,
+            "aggs": {
+                "metrics": {
+                    "terms": {
+                        "field": "metric",
+                        "size": max_metrics,
+                        "order": {"_key": "asc"},
+                    },
+                    "aggs": {
+                        "last_event_timestamp": {"max": {"field": "timestamp"}},
+                        "variants": {
+                            "terms": {
+                                "field": "variant",
+                                "size": max_variants,
+                                "order": {"_key": "asc"},
+                            },
+                            **(
+                                {"aggs": variant_state_aggs}
+                                if variant_state_aggs
+                                else {}
+                            ),
+                        },
+                    },
+                }
+            },
+        }
+
+        with translate_errors_context():
+            es_res = search_company_events(body=es_req, **search_args)
+        if "aggregations" not in es_res:
+            return []
+
+        def init_variant_state(variant: dict):
+            """
+            Return new variant state for the passed variant bucket
+            """
+            state = VariantState(variant=variant["key"])
+            if fill_variant_state_data:
+                fill_variant_state_data(variant, state)
+
+            return state
+
+        return [
+            MetricState(
+                metric=metric["key"],
+                timestamp=nested_get(metric, ("last_event_timestamp", "value")),
+                variants=[
+                    init_variant_state(variant)
+                    for variant in nested_get(metric, ("variants", "buckets"))
+                ],
+            )
+            for metric in nested_get(es_res, ("aggregations", "metrics", "buckets"))
+        ]
+
+    @abc.abstractmethod
+    def _process_event(self, event: dict) -> dict:
+        pass
+
+    @abc.abstractmethod
+    def _get_same_variant_events_order(self) -> dict:
+        pass
+
+    def _get_task_metric_events(
+        self,
+        task_state: TaskScrollState,
+        companies: Mapping[str, str],
+        iter_count: int,
+        navigate_earlier: bool,
+        specific_variants_requested: bool,
+    ) -> Tuple:
+        """
+        Return task metric events grouped by iterations
+        Update task scroll state
+        """
+        if not task_state.metrics:
+            return task_state.task, []
+
+        if task_state.last_max_iter is None:
+            # the first fetch is always from the latest iteration to the earlier ones
+            navigate_earlier = True
+
+        must_conditions = [
+            {"term": {"task": task_state.task}},
+            {"terms": {"metric": [m.metric for m in task_state.metrics]}},
+            *self._get_extra_conditions(),
+        ]
+
+        range_condition = None
+        if navigate_earlier and task_state.last_min_iter is not None:
+            range_condition = {"lt": task_state.last_min_iter}
+        elif not navigate_earlier and task_state.last_max_iter is not None:
+            range_condition = {"gt": task_state.last_max_iter}
+        if range_condition:
+            must_conditions.append({"range": {"iter": range_condition}})
+
+        metrics_count = len(task_state.metrics)
+        max_variants = int(EventSettings.max_es_buckets / (metrics_count * iter_count))
+        es_req = {
+            "size": 0,
+            "query": {"bool": {"must": must_conditions}},
+            "aggs": {
+                "iters": {
+                    "terms": {
+                        "field": "iter",
+                        "size": iter_count,
+                        "order": {"_key": "desc" if navigate_earlier else "asc"},
+                    },
+                    "aggs": {
+                        "metrics": {
+                            "terms": {
+                                "field": "metric",
+                                "size": metrics_count,
+                                "order": {"_key": "asc"},
+                            },
+                            "aggs": {
+                                "variants": {
+                                    "terms": {
+                                        "field": "variant",
+                                        "size": max_variants,
+                                        "order": {"_key": "asc"},
+                                    },
+                                    "aggs": {
+                                        "events": {
+                                            "top_hits": {
+                                                "sort": self._get_same_variant_events_order()
+                                            }
+                                        }
+                                    },
+                                }
+                            },
+                        }
+                    },
+                }
+            },
+        }
+        with translate_errors_context():
+            es_res = search_company_events(
+                self.es,
+                company_id=companies[task_state.task],
+                event_type=self.event_type,
+                body=es_req,
+            )
+        if "aggregations" not in es_res:
+            return task_state.task, []
+
+        invalid_iterations = {
+            (m.metric, v.variant): v.last_invalid_iteration
+            for m in task_state.metrics
+            for v in m.variants
+        }
+        allow_uninitialized = (
+            False
+            if specific_variants_requested
+            else config.get(
+                "services.events.events_retrieval.debug_images.allow_uninitialized_variants",
+                False,
+            )
+        )
+
+        def is_valid_event(event: dict) -> bool:
+            key = event.get("metric"), event.get("variant")
+            if key not in invalid_iterations:
+                return allow_uninitialized
+
+            max_invalid = invalid_iterations[key]
+            return max_invalid is None or event.get("iter") > max_invalid
+
+        def get_iteration_events(it_: dict) -> Sequence:
+            return [
+                self._process_event(ev["_source"])
+                for m in nested_get(it_, ("metrics", "buckets"))
+                for v in nested_get(m, ("variants", "buckets"))
+                for ev in nested_get(v, ("events", "hits", "hits"))
+                if is_valid_event(ev["_source"])
+            ]
+
+        iterations = []
+        for it in nested_get(es_res, ("aggregations", "iters", "buckets")):
+            events = get_iteration_events(it)
+            if events:
+                iterations.append({"iter": it["key"], "events": events})
+
+        if not navigate_earlier:
+            iterations.sort(key=itemgetter("iter"), reverse=True)
+        if iterations:
+            task_state.last_max_iter = iterations[0]["iter"]
+            task_state.last_min_iter = iterations[-1]["iter"]
+
+        return task_state.task, iterations

apiserver/bll/event/metric_plots_iterator.py

@@ -0,0 +1,25 @@
+from typing import Sequence
+
+from elasticsearch import Elasticsearch
+from redis.client import StrictRedis
+
+from .event_common import EventType, uncompress_plot
+from .metric_events_iterator import MetricEventsIterator
+
+
+class MetricPlotsIterator(MetricEventsIterator):
+    def __init__(self, redis: StrictRedis, es: Elasticsearch):
+        super().__init__(redis, es, EventType.metrics_plot)
+
+    def _get_extra_conditions(self) -> Sequence[dict]:
+        return []
+
+    def _get_variant_state_aggs(self):
+        return None, None
+
+    def _process_event(self, event: dict) -> dict:
+        uncompress_plot(event)
+        return event
+
+    def _get_same_variant_events_order(self) -> dict:
+        return {"timestamp": {"order": "desc"}}

apiserver/bll/event/scalar_key.py

@@ -0,0 +1,173 @@
+"""
+Module for polymorphism over different types of X axes in scalar aggregations
+"""
+from abc import ABC, abstractmethod
+from enum import auto
+
+from typing import Any
+
+from apiserver.utilities import extract_properties_to_lists
+from apiserver.utilities.stringenum import StringEnum
+from apiserver.config_repo import config
+
+log = config.logger(__file__)
+
+
+class ScalarKeyEnum(StringEnum):
+    """
+    String enum representing X axes key
+    """
+
+    iter = auto()
+    timestamp = auto()
+    iso_time = auto()
+
+
+class ScalarKey(ABC):
+    """
+    Abstract scalar key
+    """
+
+    _enum_to_key = {}
+    bucket_key_key = "key"
+
+    @property
+    @abstractmethod
+    def enum_value(self) -> ScalarKeyEnum:
+        """
+        Enum value accepted in API requests
+        """
+        pass
+
+    @property
+    @abstractmethod
+    def name(self) -> str:
+        """
+        Key name. Used as arbitrary internal key in elasticsearch queries
+        """
+        pass
+
+    @property
+    @abstractmethod
+    def field(self) -> str:
+        """
+        Event key to aggregate by
+        """
+        pass
+
+    @abstractmethod
+    def get_aggregation(self, interval: int) -> dict:
+        """
+        Get aggregation for this type of key
+        :param interval: elasticsearch aggregation interval
+        """
+        pass
+
+    def __init_subclass__(cls, **kwargs):
+        """
+        Save a mapping from enum values to key class
+        """
+        if cls.enum_value not in ScalarKeyEnum:
+            raise ValueError(f"{cls.enum_value!r} not in {ScalarKeyEnum.__name__}")
+        if cls.enum_value in cls._enum_to_key:
+            log.warning(
+                f"'{cls.enum_value.value}' is already registered to {ScalarKey.__name__}"
+            )
+        cls._enum_to_key[cls.enum_value] = cls
+
+    @classmethod
+    def resolve(cls, key: ScalarKeyEnum):
+        """
+        Create a key instance from enum instance
+        """
+        return cls._enum_to_key[key]()
+
+    def get_iterations_data(self, iter_buckets: dict) -> dict:
+        """
+        Convert a list of bucket entries to `x`s array and `y`s array
+        """
+        return extract_properties_to_lists(
+            ("x", "y"),
+            iter_buckets[self.name]["buckets"],
+            self._get_iterations_data_single,
+        )
+
+    def _get_iterations_data_single(self, iter_data):
+        """
+        Extract x value and y value from a single bucket item
+        """
+        return int(iter_data[self.bucket_key_key]), iter_data["avg_val"]["value"]
+
+    def cast_value(self, value: Any) -> Any:
+        """Cast value to appropriate type"""
+        return value
+
+
+class TimestampKey(ScalarKey):
+    """
+    Aggregate by timestamp in milliseconds since epoch
+    """
+
+    name = "timestamp"
+    field = "timestamp"
+    enum_value = ScalarKeyEnum.timestamp
+
+    def get_aggregation(self, interval: int) -> dict:
+        return {
+            self.name: {
+                "date_histogram": {
+                    "field": "timestamp",
+                    "fixed_interval": f"{interval}ms",
+                    "min_doc_count": 1,
+                }
+            }
+        }
+
+    def cast_value(self, value: Any) -> int:
+        return int(value)
+
+
+class IterKey(ScalarKey):
+    """
+    Aggregate by iteration number
+    """
+
+    name = "iters"
+    field = "iter"
+    enum_value = ScalarKeyEnum.iter
+
+    def get_aggregation(self, interval: int) -> dict:
+        return {
+            self.name: {
+                "histogram": {"field": "iter", "interval": interval, "min_doc_count": 1}
+            }
+        }
+
+    def cast_value(self, value: Any) -> int:
+        return int(value)
+
+
+class ISOTimeKey(ScalarKey):
+    """
+    Aggregate by time formatted as ISO strings
+    """
+
+    name = "iso_time"
+    field = "timestamp"
+    enum_value = ScalarKeyEnum.iso_time
+    bucket_key_key = "key_as_string"
+
+    def get_aggregation(self, interval: int) -> dict:
+        return {
+            self.name: {
+                "date_histogram": {
+                    "field": "timestamp",
+                    "fixed_interval": f"{interval}ms",
+                    "min_doc_count": 1,
+                    "format": "strict_date_time",
+                }
+            }
+        }
+
+    def _get_iterations_data_single(self, iter_data):
+        return iter_data[self.bucket_key_key], iter_data["avg_val"]["value"]

apiserver/bll/model/__init__.py

@@ -0,0 +1,268 @@
+from datetime import datetime
+from typing import Callable, Tuple, Sequence, Dict, Optional
+
+from mongoengine import Q
+
+from apiserver.apierrors import errors
+from apiserver.apimodels.models import ModelTaskPublishResponse
+from apiserver.bll.task.utils import deleted_prefix, get_last_metric_updates
+from apiserver.config_repo import config
+from apiserver.database.model import EntityVisibility
+from apiserver.database.model.model import Model
+from apiserver.database.model.task.task import Task, TaskStatus
+from apiserver.service_repo.auth import Identity
+from .metadata import Metadata
+
+
+class ModelBLL:
+    event_bll = None
+
+    @classmethod
+    def get_company_model_by_id(
+        cls, company_id: str, model_id: str, only_fields=None
+    ) -> Model:
+        query = dict(company=company_id, id=model_id)
+        qs = Model.objects(**query)
+        if only_fields:
+            qs = qs.only(*only_fields)
+        model = qs.first()
+        if not model:
+            raise errors.bad_request.InvalidModelId(**query)
+        return model
+
+    @staticmethod
+    def assert_exists(
+        company_id, model_ids, only=None, allow_public=False, return_models=True,
+    ) -> Optional[Sequence[Model]]:
+        model_ids = [model_ids] if isinstance(model_ids, str) else model_ids
+        ids = set(model_ids)
+        query = Q(id__in=ids)
+
+        q = Model.get_many(
+            company=company_id,
+            query=query,
+            allow_public=allow_public,
+            return_dicts=False,
+        )
+        if only:
+            q = q.only(*only)
+
+        if q.count() != len(ids):
+            raise errors.bad_request.InvalidModelId(ids=model_ids)
+
+        if return_models:
+            return list(q)
+
+    @classmethod
+    def publish_model(
+        cls,
+        model_id: str,
+        company_id: str,
+        identity: Identity,
+        force_publish_task: bool = False,
+        publish_task_func: Callable[[str, str, Identity, bool], dict] = None,
+    ) -> Tuple[int, ModelTaskPublishResponse]:
+        model = cls.get_company_model_by_id(company_id=company_id, model_id=model_id)
+        if model.ready:
+            raise errors.bad_request.ModelIsReady(company=company_id, model=model_id)
+
+        user_id = identity.user
+        published_task = None
+        if model.task and publish_task_func:
+            task = (
+                Task.objects(id=model.task, company=company_id)
+                .only("id", "status")
+                .first()
+            )
+            if task and task.status != TaskStatus.published:
+                task_publish_res = publish_task_func(
+                    model.task, company_id, identity, force_publish_task
+                )
+                published_task = ModelTaskPublishResponse(
+                    id=model.task, data=task_publish_res
+                )
+
+        now = datetime.utcnow()
+        updated = model.update(
+            upsert=False,
+            ready=True,
+            last_update=now,
+            last_change=now,
+            last_changed_by=user_id,
+        )
+        return updated, published_task
+
+    @classmethod
+    def delete_model(
+        cls, model_id: str, company_id: str, user_id: str, force: bool, delete_external_artifacts: bool = True,
+    ) -> Tuple[int, Model]:
+        model = cls.get_company_model_by_id(
+            company_id=company_id,
+            model_id=model_id,
+            only_fields=("id", "task", "project", "uri"),
+        )
+        deleted_model_id = f"{deleted_prefix}{model_id}"
+
+        using_tasks = Task.objects(models__input__model=model_id).only("id")
+        if using_tasks:
+            if not force:
+                raise errors.bad_request.ModelInUse(
+                    "as execution model, use force=True to delete",
+                    num_tasks=len(using_tasks),
+                )
+            # update deleted model id in using tasks
+            Task._get_collection().update_many(
+                filter={"_id": {"$in": [t.id for t in using_tasks]}},
+                update={"$set": {"models.input.$[elem].model": deleted_model_id}},
+                array_filters=[{"elem.model": model_id}],
+                upsert=False,
+            )
+
+        if model.task:
+            task = Task.objects(id=model.task).first()
+            if task:
+                now = datetime.utcnow()
+                if task.status == TaskStatus.published:
+                    if not force:
+                        raise errors.bad_request.ModelCreatingTaskExists(
+                            "and published, use force=True to delete", task=model.task
+                        )
+                    Task._get_collection().update_one(
+                        filter={"_id": model.task, "models.output.model": model_id},
+                        update={
+                            "$set": {
+                                "models.output.$[elem].model": deleted_model_id,
+                                "output.error": f"model deleted on {now.isoformat()}",
+                                "last_change": now,
+                                "last_changed_by": user_id,
+                            },
+                        },
+                        array_filters=[{"elem.model": model_id}],
+                        upsert=False,
+                    )
+                else:
+                    task.update(
+                        pull__models__output__model=model_id,
+                        set__last_change=now,
+                        set__last_changed_by=user_id,
+                    )
+
+        delete_external_artifacts = delete_external_artifacts and config.get(
+            "services.async_urls_delete.enabled", True
+        )
+        if delete_external_artifacts:
+            from apiserver.bll.task.task_cleanup import (
+                collect_debug_image_urls,
+                collect_plot_image_urls,
+                _schedule_for_delete,
+            )
+            urls = set()
+            urls.update(collect_debug_image_urls(company_id, model_id))
+            urls.update(collect_plot_image_urls(company_id, model_id))
+            if model.uri:
+                urls.add(model.uri)
+            if urls:
+                _schedule_for_delete(
+                    task_id=model_id,
+                    company=company_id,
+                    user=user_id,
+                    urls=urls,
+                    can_delete_folders=False,
+                )
+
+        if not cls.event_bll:
+            from apiserver.bll.event import EventBLL
+            cls.event_bll = EventBLL()
+
+        cls.event_bll.delete_task_events(company_id, model_id, allow_locked=True, model=True)
+        del_count = Model.objects(id=model_id, company=company_id).delete()
+        return del_count, model
+
+    @classmethod
+    def archive_model(cls, model_id: str, company_id: str, user_id: str):
+        cls.get_company_model_by_id(
+            company_id=company_id, model_id=model_id, only_fields=("id",)
+        )
+        now = datetime.utcnow()
+        archived = Model.objects(company=company_id, id=model_id).update(
+            add_to_set__system_tags=EntityVisibility.archived.value,
+            last_change=now,
+            last_changed_by=user_id,
+        )
+
+        return archived
+
+    @classmethod
+    def unarchive_model(cls, model_id: str, company_id: str, user_id: str):
+        cls.get_company_model_by_id(
+            company_id=company_id, model_id=model_id, only_fields=("id",)
+        )
+        now = datetime.utcnow()
+        unarchived = Model.objects(company=company_id, id=model_id).update(
+            pull__system_tags=EntityVisibility.archived.value,
+            last_change=now,
+            last_changed_by=user_id,
+        )
+
+        return unarchived
+
+    @classmethod
+    def get_model_stats(
+        cls, company: str, model_ids: Sequence[str],
+    ) -> Dict[str, dict]:
+        if not model_ids:
+            return {}
+
+        result = Model.aggregate(
+            [
+                {
+                    "$match": {
+                        "company": {"$in": [None, "", company]},
+                        "_id": {"$in": model_ids},
+                    }
+                },
+                {
+                    "$addFields": {
+                        "labels_count": {"$size": {"$objectToArray": "$labels"}}
+                    }
+                },
+                {"$project": {"labels_count": 1}},
+            ]
+        )
+        return {r.pop("_id"): r for r in result}
+
+    @staticmethod
+    def update_statistics(
+        company_id: str,
+        user_id: str,
+        model_id: str,
+        last_update: datetime = None,
+        last_iteration_max: int = None,
+        last_scalar_events: Dict[str, Dict[str, dict]] = None,
+    ):
+        last_update = last_update or datetime.utcnow()
+        updates = {
+            "last_update": datetime.utcnow(),
+            "last_change": last_update,
+            "last_changed_by": user_id,
+        }
+        if last_iteration_max is not None:
+            updates.update(max__last_iteration=last_iteration_max)
+
+        raw_updates = {}
+        if last_scalar_events is not None:
+            raw_updates = {}
+            if last_scalar_events is not None:
+                get_last_metric_updates(
+                    task_id=model_id,
+                    last_scalar_events=last_scalar_events,
+                    raw_updates=raw_updates,
+                    extra_updates=updates,
+                    model_events=True,
+                )
+
+        ret = Model.objects(id=model_id).update_one(**updates)
+        if ret and raw_updates:
+            Model.objects(id=model_id).update_one(__raw__=[{"$set": raw_updates}])
+
+        return ret

apiserver/bll/model/metadata.py

@@ -0,0 +1,107 @@
+from typing import Sequence, Union, Mapping
+
+from mongoengine import Document
+
+from apiserver.apierrors import errors
+from apiserver.apimodels.metadata import MetadataItem
+from apiserver.database.model.base import GetMixin
+from apiserver.utilities.parameter_key_escaper import (
+    ParameterKeyEscaper,
+    mongoengine_safe,
+)
+from apiserver.config_repo import config
+
+log = config.logger(__file__)
+
+
+class Metadata:
+    @staticmethod
+    def metadata_from_api(
+        api_data: Union[Mapping[str, MetadataItem], Sequence[MetadataItem]]
+    ) -> dict:
+        if not api_data:
+            return {}
+
+        if isinstance(api_data, dict):
+            return {
+                ParameterKeyEscaper.escape(k): v.to_struct()
+                for k, v in api_data.items()
+            }
+
+        return {
+            ParameterKeyEscaper.escape(item.key): item.to_struct() for item in api_data
+        }
+
+    @classmethod
+    def edit_metadata(
+        cls,
+        obj: Document,
+        items: Sequence[MetadataItem],
+        replace_metadata: bool,
+        **more_updates,
+    ) -> int:
+        update_cmds = dict()
+        metadata = cls.metadata_from_api(items)
+        if replace_metadata:
+            update_cmds["set__metadata"] = metadata
+        else:
+            for key, value in metadata.items():
+                update_cmds[f"set__metadata__{mongoengine_safe(key)}"] = value
+
+        return obj.update(**update_cmds, **more_updates)
+
+    @classmethod
+    def delete_metadata(cls, obj: Document, keys: Sequence[str], **more_updates) -> int:
+        return obj.update(
+            **{
+                f"unset__metadata__{ParameterKeyEscaper.escape(key)}": 1
+                for key in set(keys)
+            },
+            **more_updates,
+        )
+
+    @staticmethod
+    def _process_path(path: str):
+        """
+        Frontend does a partial escaping on the path so the all '.' in key names are escaped
+        Need to unescape and apply a full mongo escaping
+        """
+        parts = path.split(".")
+        if len(parts) < 2 or len(parts) > 3:
+            raise errors.bad_request.ValidationError("invalid field", path=path)
+        return ".".join(
+            ParameterKeyEscaper.escape(ParameterKeyEscaper.unescape(p)) for p in parts
+        )
+
+    @classmethod
+    def escape_paths(cls, paths: Sequence[str]) -> Sequence[str]:
+        for prefix in (
+            "metadata.",
+            "-metadata.",
+        ):
+            paths = [
+                cls._process_path(path) if path.startswith(prefix) else path
+                for path in paths
+            ]
+        return paths
+
+    @classmethod
+    def escape_query_parameters(cls, call_data: dict) -> dict:
+        if not call_data:
+            return call_data
+
+        keys = list(call_data)
+        call_data = {
+            safe_key: call_data[key]
+            for key, safe_key in zip(keys, Metadata.escape_paths(keys))
+        }
+
+        projection = GetMixin.get_projection(call_data)
+        if projection:
+            GetMixin.set_projection(call_data, Metadata.escape_paths(projection))
+
+        ordering = GetMixin.get_ordering(call_data)
+        if ordering:
+            GetMixin.set_ordering(call_data, Metadata.escape_paths(ordering))
+
+        return call_data

apiserver/bll/organization/__init__.py

@@ -0,0 +1,111 @@
+from collections import defaultdict
+from enum import Enum
+from typing import Sequence, Dict, Type
+
+from apiserver.apierrors import errors
+from apiserver.bll.util import update_project_time
+from apiserver.config_repo import config
+from apiserver.database.model.model import AttributedDocument
+from apiserver.database.model.model import Model
+from apiserver.database.model.task.task import Task
+from apiserver.redis_manager import redman
+from .tags_cache import _TagsCache
+
+log = config.logger(__file__)
+
+
+class Tags(Enum):
+    Task = "task"
+    Model = "model"
+
+
+class OrgBLL:
+    def __init__(self, redis=None):
+        self.redis = redis or redman.connection("apiserver")
+        self._task_tags = _TagsCache(Task, self.redis)
+        self._model_tags = _TagsCache(Model, self.redis)
+
+    def edit_entity_tags(
+        self,
+        company_id,
+        entity_cls: Type[AttributedDocument],
+        entity_ids: Sequence[str],
+        add_tags: Sequence[str],
+        remove_tags: Sequence[str],
+    ) -> int:
+        if entity_cls not in (Task, Model):
+            raise errors.bad_request.ValidationError(
+                "Tags editing can be called on tasks or models only"
+            )
+        if not entity_ids:
+            raise errors.bad_request.ValidationError(
+                "No entity ids provided for editing tags"
+            )
+        if not (add_tags or remove_tags):
+            raise errors.bad_request.ValidationError(
+                "Either add tags or remove tags should be provided"
+            )
+
+        updated = 0
+        if add_tags:
+            updated += entity_cls.objects(company=company_id, id__in=entity_ids).update(
+                add_to_set__tags=add_tags
+            )
+        if remove_tags:
+            updated += entity_cls.objects(company=company_id, id__in=entity_ids).update(
+                pull_all__tags=remove_tags
+            )
+        if not updated:
+            return 0
+
+        projects = entity_cls.objects(company=company_id, id__in=entity_ids).distinct(
+            "project"
+        )
+        update_project_time(project_ids=projects)
+        self.update_tags(
+            company_id,
+            entity=Tags.Task if entity_cls is Task else Tags.Model,
+            projects=projects,
+            tags=add_tags or remove_tags
+        )
+        return updated
+
+    def get_tags(
+        self,
+        company_id: str,
+        entity: Tags,
+        include_system: bool = False,
+        filter_: Dict[str, Sequence[str]] = None,
+        projects: Sequence[str] = None,
+    ) -> dict:
+        tags_cache = self._get_tags_cache_for_entity(entity)
+        if not projects:
+            return tags_cache.get_tags(
+                company_id, include_system=include_system, filter_=filter_
+            )
+
+        ret = defaultdict(set)
+        for project in projects:
+            project_tags = tags_cache.get_tags(
+                company_id,
+                include_system=include_system,
+                filter_=filter_,
+                project=project,
+            )
+            for field, tags in project_tags.items():
+                ret[field] |= tags
+
+        return ret
+
+    def update_tags(
+        self, company_id: str, entity: Tags, projects: Sequence[str], tags=None, system_tags=None,
+    ):
+        tags_cache = self._get_tags_cache_for_entity(entity)
+        tags_cache.update_tags(company_id, projects, tags, system_tags)
+
+    def reset_tags(self, company_id: str, entity: Tags, projects: Sequence[str]):
+        tags_cache = self._get_tags_cache_for_entity(entity)
+        tags_cache.reset_tags(company_id, projects=projects)
+
+    def _get_tags_cache_for_entity(self, entity: Tags) -> _TagsCache:
+        return self._task_tags if entity == Tags.Task else self._model_tags

apiserver/bll/organization/tags_cache.py

@@ -0,0 +1,148 @@
+from itertools import chain
+from typing import Sequence, Union, Type, Dict
+
+from mongoengine import Q
+from redis import Redis
+
+from apiserver.config_repo import config
+from apiserver.bll.project import project_ids_with_children
+from apiserver.database.model import EntityVisibility
+from apiserver.database.model.base import GetMixin
+from apiserver.database.model.model import Model
+from apiserver.database.model.task.task import Task
+
+log = config.logger(__file__)
+_settings_prefix = "services.organization"
+
+
+class _TagsCache:
+    _tags_field = "tags"
+    _system_tags_field = "system_tags"
+    _dummy_tag = "__dummy__"
+    # prepend our list in redis with this tag since empty lists are auto deleted
+
+    def __init__(self, db_cls: Union[Type[Model], Type[Task]], redis: Redis):
+        self.db_cls = db_cls
+        self.redis = redis
+
+    @property
+    def _tags_cache_expiration_seconds(self):
+        return config.get(f"{_settings_prefix}.tags_cache.expiration_seconds", 3600)
+
+    def _get_tags_from_db(
+        self,
+        company_id: str,
+        field: str,
+        project: str = None,
+        filter_: Dict[str, Sequence[str]] = None,
+    ) -> set:
+        query = Q(company=company_id)
+        if filter_:
+            for name, vals in filter_.items():
+                if vals:
+                    query &= GetMixin.get_list_field_query(name, vals)
+        if project:
+            query &= Q(project__in=project_ids_with_children([project]))
+        else:
+            query &= Q(system_tags__nin=[EntityVisibility.hidden.value])
+
+        return self.db_cls.objects(query).distinct(field)
+
+    def _get_tags_cache_key(
+        self,
+        company_id: str,
+        field: str,
+        project: str = None,
+        filter_: Dict[str, Sequence[str]] = None,
+    ):
+        """
+        Project None means 'from all company projects'
+        The key is built in the way that scanning company keys for 'all company projects'
+        will not return the keys related to the particular company projects and vice versa.
+        So that we can have a fine grain control on what redis keys to invalidate
+        """
+        filter_str = None
+        if filter_:
+            filter_str = "_".join(
+                ["filter", *chain.from_iterable([f, *v] for f, v in filter_.items())]
+            )
+        key_parts = [field, company_id, project, self.db_cls.__name__, filter_str]
+        return "_".join(filter(None, key_parts))
+
+    def get_tags(
+        self,
+        company_id: str,
+        include_system: bool = False,
+        filter_: Dict[str, Sequence[str]] = None,
+        project: str = None,
+    ) -> dict:
+        """
+        Get tags and optionally system tags for the company
+        Return the dictionary of tags per tags field name
+        The function retrieves both cached values from Redis in one call
+        and re calculates any of them if missing in Redis
+        """
+        fields = [self._tags_field]
+        if include_system:
+            fields.append(self._system_tags_field)
+
+        ret = {}
+        for field in fields:
+            redis_key = self._get_tags_cache_key(
+                company_id, field=field, project=project, filter_=filter_
+            )
+            cached_tags = self.redis.lrange(redis_key, 0, -1)
+            if cached_tags:
+                tags = [c.decode() for c in cached_tags[1:]]
+            else:
+                tags = list(
+                    self._get_tags_from_db(
+                        company_id, field=field, project=project, filter_=filter_
+                    )
+                )
+                self.redis.rpush(redis_key, self._dummy_tag, *tags)
+                self.redis.expire(redis_key, self._tags_cache_expiration_seconds)
+
+            ret[field] = set(tags)
+
+        return ret
+
+    def update_tags(self, company_id: str, projects: Sequence[str], tags=None, system_tags=None):
+        """
+        Updates tags. If reset is set then both tags and system_tags
+        are recalculated. Otherwise only those that are not 'None'
+        """
+        fields = [
+            field
+            for field, update in (
+                (self._tags_field, tags),
+                (self._system_tags_field, system_tags),
+            )
+            if update is not None
+        ]
+        if not fields:
+            return
+
+        self._delete_redis_keys(company_id, projects=projects, fields=fields)
+
+    def reset_tags(self, company_id: str, projects: Sequence[str]):
+        self._delete_redis_keys(
+            company_id,
+            projects=projects,
+            fields=(self._tags_field, self._system_tags_field),
+        )
+
+    def _delete_redis_keys(
+        self, company_id: str, projects: [Sequence[str]], fields: Sequence[str]
+    ):
+        redis_keys = list(
+            chain.from_iterable(
+                self.redis.keys(
+                    self._get_tags_cache_key(company_id, field=f, project=p) + "*"
+                )
+                for f in fields
+                for p in set(projects) | {None}
+            )
+        )
+        if redis_keys:
+            self.redis.delete(*redis_keys)

apiserver/bll/project/__init__.py

@@ -0,0 +1,3 @@
+from .project_bll import ProjectBLL
+from .project_queries import ProjectQueries
+from .sub_projects import _ids_with_children as project_ids_with_children

apiserver/bll/project/project_cleanup.py

@@ -0,0 +1,327 @@
+from collections import defaultdict
+from datetime import datetime
+from typing import Tuple, Set, Sequence
+
+import attr
+from mongoengine import Q
+
+from apiserver.apierrors import errors
+from apiserver.bll.event import EventBLL
+from apiserver.bll.task.task_cleanup import (
+    collect_debug_image_urls,
+    collect_plot_image_urls,
+    TaskUrls,
+    _schedule_for_delete,
+)
+from apiserver.config_repo import config
+from apiserver.database.model import EntityVisibility
+from apiserver.database.model.model import Model
+from apiserver.database.model.project import Project
+from apiserver.database.model.task.task import Task, ArtifactModes, TaskType, TaskStatus
+from .project_bll import (
+    ProjectBLL,
+    pipeline_tag,
+    pipelines_project_name,
+    dataset_tag,
+    datasets_project_name,
+    reports_tag,
+)
+from .sub_projects import _ids_with_children
+
+log = config.logger(__file__)
+event_bll = EventBLL()
+
+
+@attr.s(auto_attribs=True)
+class DeleteProjectResult:
+    deleted: int = 0
+    disassociated_tasks: int = 0
+    deleted_models: int = 0
+    deleted_tasks: int = 0
+    urls: TaskUrls = None
+
+
+def _get_child_project_ids(
+    project_id: str,
+) -> Tuple[Sequence[str], Sequence[str], Sequence[str]]:
+    project_ids = _ids_with_children([project_id])
+    pipeline_ids = list(
+        Project.objects(
+            id__in=project_ids,
+            system_tags__in=[pipeline_tag],
+            basename__ne=pipelines_project_name,
+        ).scalar("id")
+    )
+    dataset_ids = list(
+        Project.objects(
+            id__in=project_ids,
+            system_tags__in=[dataset_tag],
+            basename__ne=datasets_project_name,
+        ).scalar("id")
+    )
+    return project_ids, pipeline_ids, dataset_ids
+
+
+def validate_project_delete(company: str, project_id: str):
+    project = Project.get_for_writing(
+        company=company, id=project_id, _only=("id", "path", "system_tags")
+    )
+    if not project:
+        raise errors.bad_request.InvalidProjectId(id=project_id)
+
+    project_ids, pipeline_ids, dataset_ids = _get_child_project_ids(project_id)
+    ret = {}
+    if pipeline_ids:
+        pipelines_with_active_controllers = Task.objects(
+            project__in=pipeline_ids,
+            type=TaskType.controller,
+            system_tags__nin=[EntityVisibility.archived.value],
+        ).distinct("project")
+        ret["pipelines"] = len(pipelines_with_active_controllers)
+    else:
+        ret["pipelines"] = 0
+    if dataset_ids:
+        datasets_with_data = Task.objects(
+            project__in=dataset_ids,
+            system_tags__nin=[EntityVisibility.archived.value],
+        ).distinct("project")
+        ret["datasets"] = len(datasets_with_data)
+    else:
+        ret["datasets"] = 0
+
+    project_ids = list(set(project_ids) - set(pipeline_ids) - set(dataset_ids))
+    if project_ids:
+        in_project_query = Q(project__in=project_ids)
+        for cls in (Task, Model):
+            query = (
+                in_project_query & Q(system_tags__nin=[reports_tag])
+                if cls is Task
+                else in_project_query
+            )
+            ret[f"{cls.__name__.lower()}s"] = cls.objects(query).count()
+            ret[f"non_archived_{cls.__name__.lower()}s"] = cls.objects(
+                query & Q(system_tags__nin=[EntityVisibility.archived.value])
+            ).count()
+        ret["reports"] = Task.objects(
+            in_project_query & Q(system_tags__in=[reports_tag])
+        ).count()
+        ret["non_archived_reports"] = Task.objects(
+            in_project_query
+            & Q(
+                system_tags__in=[reports_tag],
+                system_tags__nin=[EntityVisibility.archived.value],
+            )
+        ).count()
+    else:
+        for cls in (Task, Model):
+            ret[f"{cls.__name__.lower()}s"] = 0
+            ret[f"non_archived_{cls.__name__.lower()}s"] = 0
+        ret["reports"] = 0
+        ret["non_archived_reports"] = 0
+
+    return ret
+
+
+def delete_project(
+    company: str,
+    user: str,
+    project_id: str,
+    force: bool,
+    delete_contents: bool,
+    delete_external_artifacts: bool,
+) -> Tuple[DeleteProjectResult, Set[str]]:
+    project = Project.get_for_writing(
+        company=company, id=project_id, _only=("id", "path", "system_tags")
+    )
+    if not project:
+        raise errors.bad_request.InvalidProjectId(id=project_id)
+
+    delete_external_artifacts = delete_external_artifacts and config.get(
+        "services.async_urls_delete.enabled", True
+    )
+    project_ids, pipeline_ids, dataset_ids = _get_child_project_ids(project_id)
+    if not force:
+        if pipeline_ids:
+            active_controllers = Task.objects(
+                project__in=pipeline_ids,
+                type=TaskType.controller,
+                system_tags__nin=[EntityVisibility.archived.value],
+            ).only("id")
+            if active_controllers:
+                raise errors.bad_request.ProjectHasPipelines(
+                    "please archive all the controllers or use force=true",
+                    id=project_id,
+                )
+        if dataset_ids:
+            datasets_with_data = Task.objects(
+                project__in=dataset_ids,
+                system_tags__nin=[EntityVisibility.archived.value],
+            ).only("id")
+            if datasets_with_data:
+                raise errors.bad_request.ProjectHasDatasets(
+                    "please delete all the dataset versions or use force=true",
+                    id=project_id,
+                )
+
+        regular_projects = list(set(project_ids) - set(pipeline_ids) - set(dataset_ids))
+        if regular_projects:
+            for cls, error in (
+                (Task, errors.bad_request.ProjectHasTasks),
+                (Model, errors.bad_request.ProjectHasModels),
+            ):
+                non_archived = cls.objects(
+                    project__in=regular_projects,
+                    system_tags__nin=[EntityVisibility.archived.value],
+                ).only("id")
+                if non_archived:
+                    raise error("use force=true", id=project_id)
+
+    if not delete_contents:
+        disassociated = defaultdict(int)
+        for cls in ProjectBLL.child_classes:
+            disassociated[cls] = cls.objects(project__in=project_ids).update(
+                project=None
+            )
+        res = DeleteProjectResult(disassociated_tasks=disassociated[Task])
+    else:
+        deleted_models, model_event_urls, model_urls = _delete_models(
+            company=company, user=user, projects=project_ids
+        )
+        deleted_tasks, task_event_urls, artifact_urls = _delete_tasks(
+            company=company, user=user, projects=project_ids
+        )
+        event_urls = task_event_urls | model_event_urls
+        if delete_external_artifacts:
+            scheduled = _schedule_for_delete(
+                task_id=project_id,
+                company=company,
+                user=user,
+                urls=event_urls | model_urls | artifact_urls,
+                can_delete_folders=True,
+            )
+            for urls in (event_urls, model_urls, artifact_urls):
+                urls.difference_update(scheduled)
+        res = DeleteProjectResult(
+            deleted_tasks=deleted_tasks,
+            deleted_models=deleted_models,
+            urls=TaskUrls(
+                model_urls=list(model_urls),
+                event_urls=list(event_urls),
+                artifact_urls=list(artifact_urls),
+            ),
+        )
+
+    affected = {*project_ids, *(project.path or [])}
+    res.deleted = Project.objects(id__in=project_ids).delete()
+
+    return res, affected
+
+
+def _delete_tasks(
+    company: str, user: str, projects: Sequence[str]
+) -> Tuple[int, Set, Set]:
+    """
+    Delete only the task themselves and their non published version.
+    Child models under the same project are deleted separately.
+    Children tasks should be deleted in the same api call.
+    If any child entities are left in another projects then updated their parent task to None
+    """
+    tasks = Task.objects(project__in=projects).only("id", "execution__artifacts")
+    if not tasks:
+        return 0, set(), set()
+
+    task_ids = list({t.id for t in tasks})
+    now = datetime.utcnow()
+    Task.objects(parent__in=task_ids, project__nin=projects).update(
+        parent=None,
+        last_change=now,
+        last_changed_by=user,
+    )
+    Model.objects(task__in=task_ids, project__nin=projects).update(
+        task=None,
+        last_change=now,
+        last_changed_by=user,
+    )
+
+    event_urls = collect_debug_image_urls(company, task_ids) | collect_plot_image_urls(
+        company, task_ids
+    )
+    artifact_urls = set()
+    for task in tasks:
+        if task.execution and task.execution.artifacts:
+            artifact_urls.update(
+                {
+                    a.uri
+                    for a in task.execution.artifacts.values()
+                    if a.mode == ArtifactModes.output and a.uri
+                }
+            )
+
+    event_bll.delete_multi_task_events(company, task_ids)
+    deleted = tasks.delete()
+    return deleted, event_urls, artifact_urls
+
+
+def _delete_models(
+    company: str, user: str, projects: Sequence[str]
+) -> Tuple[int, Set[str], Set[str]]:
+    """
+    Delete project models and update the tasks from other projects
+    that reference them to reference None.
+    """
+    models = Model.objects(project__in=projects).only("task", "id", "uri")
+    if not models:
+        return 0, set(), set()
+
+    model_ids = list({m.id for m in models})
+    deleted = "__DELETED__"
+    Task._get_collection().update_many(
+        filter={
+            "project": {"$nin": projects},
+            "models.input.model": {"$in": model_ids},
+        },
+        update={"$set": {"models.input.$[elem].model": deleted}},
+        array_filters=[{"elem.model": {"$in": model_ids}}],
+        upsert=False,
+    )
+
+    model_tasks = list({m.task for m in models if m.task})
+    if model_tasks:
+        now = datetime.utcnow()
+        # update published tasks
+        Task._get_collection().update_many(
+            filter={
+                "_id": {"$in": model_tasks},
+                "project": {"$nin": projects},
+                "models.output.model": {"$in": model_ids},
+                "status": TaskStatus.published,
+            },
+            update={
+                "$set": {
+                    "models.output.$[elem].model": deleted,
+                    "last_change": now,
+                    "last_changed_by": user,
+                }
+            },
+            array_filters=[{"elem.model": {"$in": model_ids}}],
+            upsert=False,
+        )
+        # update unpublished tasks
+        Task.objects(
+            id__in=model_tasks,
+            project__nin=projects,
+            status__ne=TaskStatus.published,
+        ).update(
+            pull__models__output__model__in=model_ids,
+            set__last_change=now,
+            set__last_changed_by=user,
+        )
+
+    event_urls = collect_debug_image_urls(company, model_ids) | collect_plot_image_urls(
+        company, model_ids
+    )
+    model_urls = {m.uri for m in models if m.uri}
+
+    event_bll.delete_multi_task_events(company, model_ids, model=True)
+    deleted = models.delete()
+    return deleted, event_urls, model_urls

apiserver/bll/project/project_queries.py

@@ -0,0 +1,407 @@
+import json
+from collections import OrderedDict
+from datetime import datetime
+from typing import (
+    Sequence,
+    Optional,
+    Tuple,
+)
+
+from redis import StrictRedis
+
+from apiserver.config_repo import config
+from apiserver.database.model.model import Model
+from apiserver.database.model.task.task import Task
+from apiserver.redis_manager import redman
+from apiserver.utilities.dicts import nested_get
+from apiserver.utilities.parameter_key_escaper import ParameterKeyEscaper
+from .sub_projects import _ids_with_children
+
+log = config.logger(__file__)
+
+
+class ProjectQueries:
+    def __init__(self, redis=None):
+        self.redis: StrictRedis = redis or redman.connection("apiserver")
+
+    @staticmethod
+    def _get_project_constraint(
+        project_ids: Sequence[str], include_subprojects: bool
+    ) -> dict:
+        """
+        If passed projects is None means top level projects
+        If passed projects is empty means no project filtering
+        """
+        if include_subprojects:
+            if not project_ids:
+                return {}
+            project_ids = _ids_with_children(project_ids)
+
+        if project_ids is None:
+            project_ids = [None]
+        if not project_ids:
+            return {}
+
+        return {"project": {"$in": project_ids}}
+
+    @staticmethod
+    def _get_company_constraint(company_id: str, allow_public: bool = True) -> dict:
+        if allow_public:
+            return {"company": {"$in": [None, "", company_id]}}
+
+        return {"company": company_id}
+
+    @classmethod
+    def get_aggregated_project_parameters(
+        cls,
+        company_id,
+        project_ids: Sequence[str],
+        include_subprojects: bool,
+        page: int = 0,
+        page_size: int = 500,
+    ) -> Tuple[int, int, Sequence[dict]]:
+        page = max(0, page)
+        page_size = max(1, page_size)
+        pipeline = [
+            {
+                "$match": {
+                    **cls._get_company_constraint(company_id),
+                    **cls._get_project_constraint(project_ids, include_subprojects),
+                    "hyperparams": {"$exists": True, "$gt": {}},
+                }
+            },
+            {"$project": {"sections": {"$objectToArray": "$hyperparams"}}},
+            {"$unwind": "$sections"},
+            {
+                "$project": {
+                    "section": "$sections.k",
+                    "names": {"$objectToArray": "$sections.v"},
+                }
+            },
+            {"$unwind": "$names"},
+            {"$group": {"_id": {"section": "$section", "name": "$names.k"}}},
+            {"$sort": OrderedDict({"_id.section": 1, "_id.name": 1})},
+            {"$skip": page * page_size},
+            {"$limit": page_size},
+            {
+                "$group": {
+                    "_id": 1,
+                    "total": {"$sum": 1},
+                    "results": {"$push": "$$ROOT"},
+                }
+            },
+        ]
+
+        result = next(Task.aggregate(pipeline), None)
+
+        total = 0
+        remaining = 0
+        results = []
+
+        if result:
+            total = int(result.get("total", -1))
+            results = [
+                {
+                    "section": ParameterKeyEscaper.unescape(
+                        nested_get(r, ("_id", "section"))
+                    ),
+                    "name": ParameterKeyEscaper.unescape(
+                        nested_get(r, ("_id", "name"))
+                    ),
+                }
+                for r in result.get("results", [])
+            ]
+            remaining = max(0, total - (len(results) + page * page_size))
+
+        return total, remaining, results
+
+    ParamValues = Tuple[int, Sequence[str]]
+
+    def _get_cached_param_values(
+        self, key: str, last_update: datetime, allowed_delta_sec=0
+    ) -> Optional[ParamValues]:
+        try:
+            cached = self.redis.get(key)
+            if not cached:
+                return
+
+            data = json.loads(cached)
+            cached_last_update = datetime.fromtimestamp(data["last_update"])
+            if (last_update - cached_last_update).total_seconds() <= allowed_delta_sec:
+                return data["total"], data["values"]
+        except Exception as ex:
+            log.error(f"Error retrieving params cached values: {str(ex)}")
+
+    def get_task_hyperparam_distinct_values(
+        self,
+        company_id: str,
+        project_ids: Sequence[str],
+        section: str,
+        name: str,
+        include_subprojects: bool,
+        allow_public: bool = True,
+        pattern: str = None,
+        page: int = 0,
+        page_size: int = 500,
+    ) -> ParamValues:
+        page = max(0, page)
+        page_size = max(1, page_size)
+        company_constraint = self._get_company_constraint(company_id, allow_public)
+        project_constraint = self._get_project_constraint(
+            project_ids, include_subprojects
+        )
+        key_path = f"hyperparams.{ParameterKeyEscaper.escape(section)}.{ParameterKeyEscaper.escape(name)}"
+        last_updated_task = (
+            Task.objects(
+                **company_constraint,
+                **project_constraint,
+                **{f"{key_path.replace('.', '__')}__exists": True},
+            )
+            .only("last_update")
+            .order_by("-last_update")
+            .limit(1)
+            .first()
+        )
+        if not last_updated_task:
+            return 0, []
+
+        redis_key = "_".join(
+            str(part)
+            for part in (
+                "hyperparam_values",
+                company_id,
+                "_".join(project_ids),
+                section,
+                name,
+                allow_public,
+                pattern,
+                page,
+                page_size,
+            )
+        )
+        last_update = last_updated_task.last_update or datetime.utcnow()
+        cached_res = self._get_cached_param_values(
+            key=redis_key,
+            last_update=last_update,
+            allowed_delta_sec=config.get(
+                "services.tasks.hyperparam_values.cache_allowed_outdate_sec", 60
+            ),
+        )
+        if cached_res:
+            return cached_res
+
+        match_condition = {
+            **company_constraint,
+            **project_constraint,
+            key_path: {"$exists": True},
+        }
+        if pattern:
+            match_condition["$expr"] = {
+                "$regexMatch": {
+                    "input": f"${key_path}.value",
+                    "regex": pattern,
+                    "options": "i",
+                }
+            }
+
+        pipeline = [
+            {"$match": match_condition},
+            {"$project": {"value": f"${key_path}.value"}},
+            {"$group": {"_id": "$value"}},
+            {"$sort": {"_id": 1}},
+            {"$skip": page * page_size},
+            {"$limit": page_size},
+            {
+                "$group": {
+                    "_id": 1,
+                    "total": {"$sum": 1},
+                    "results": {"$push": "$$ROOT._id"},
+                }
+            },
+        ]
+
+        result = next(Task.aggregate(pipeline, collation=Task._numeric_locale), None)
+        if not result:
+            return 0, []
+
+        total = int(result.get("total", 0))
+        values = result.get("results", [])
+
+        ttl = config.get("services.tasks.hyperparam_values.cache_ttl_sec", 86400)
+        cached = dict(last_update=last_update.timestamp(), total=total, values=values)
+        self.redis.setex(redis_key, ttl, json.dumps(cached))
+
+        return total, values
+
+    @classmethod
+    def get_unique_metric_variants(
+        cls,
+        company_id,
+        project_ids: Sequence[str],
+        include_subprojects: bool,
+        ids: Sequence[str],
+        model_metrics: bool = False,
+    ):
+        pipeline = [
+            {
+                "$match": {
+                    **cls._get_company_constraint(company_id),
+                    **cls._get_project_constraint(project_ids, include_subprojects),
+                    **({"_id": {"$in": ids}} if ids else {}),
+                }
+            },
+            {"$project": {"metrics": {"$objectToArray": "$last_metrics"}}},
+            {"$unwind": "$metrics"},
+            {
+                "$project": {
+                    "metric": "$metrics.k",
+                    "variants": {"$objectToArray": "$metrics.v"},
+                }
+            },
+            {"$unwind": "$variants"},
+            {
+                "$group": {
+                    "_id": {
+                        "metric": "$variants.v.metric",
+                        "variant": "$variants.v.variant",
+                    },
+                    "metrics": {
+                        "$addToSet": {
+                            "metric": "$variants.v.metric",
+                            "metric_hash": "$metric",
+                            "variant": "$variants.v.variant",
+                            "variant_hash": "$variants.k",
+                        }
+                    },
+                }
+            },
+            {"$sort": OrderedDict({"_id.metric": 1, "_id.variant": 1})},
+        ]
+
+        entity_cls = Model if model_metrics else Task
+        result = entity_cls.aggregate(pipeline)
+        return [r["metrics"][0] for r in result]
+
+    @classmethod
+    def get_model_metadata_keys(
+        cls,
+        company_id,
+        project_ids: Sequence[str],
+        include_subprojects: bool,
+        page: int = 0,
+        page_size: int = 500,
+    ) -> Tuple[int, int, Sequence[dict]]:
+        page = max(0, page)
+        page_size = max(1, page_size)
+        pipeline = [
+            {
+                "$match": {
+                    **cls._get_company_constraint(company_id),
+                    **cls._get_project_constraint(project_ids, include_subprojects),
+                    "metadata": {"$exists": True, "$gt": {}},
+                }
+            },
+            {"$project": {"metadata": {"$objectToArray": "$metadata"}}},
+            {"$unwind": "$metadata"},
+            {"$group": {"_id": "$metadata.k"}},
+            {"$sort": {"_id": 1}},
+            {"$skip": page * page_size},
+            {"$limit": page_size},
+            {
+                "$group": {
+                    "_id": 1,
+                    "total": {"$sum": 1},
+                    "results": {"$push": "$$ROOT"},
+                }
+            },
+        ]
+
+        result = next(Model.aggregate(pipeline), None)
+
+        total = 0
+        remaining = 0
+        results = []
+
+        if result:
+            total = int(result.get("total", -1))
+            results = [
+                ParameterKeyEscaper.unescape(r.get("_id"))
+                for r in result.get("results", [])
+            ]
+            remaining = max(0, total - (len(results) + page * page_size))
+
+        return total, remaining, results
+
+    def get_model_metadata_distinct_values(
+        self,
+        company_id: str,
+        project_ids: Sequence[str],
+        key: str,
+        include_subprojects: bool,
+        allow_public: bool = True,
+        page: int = 0,
+        page_size: int = 500,
+    ) -> ParamValues:
+        page = max(0, page)
+        page_size = max(1, page_size)
+        company_constraint = self._get_company_constraint(company_id, allow_public)
+        project_constraint = self._get_project_constraint(
+            project_ids, include_subprojects
+        )
+        key_path = f"metadata.{ParameterKeyEscaper.escape(key)}"
+        last_updated_model = (
+            Model.objects(
+                **company_constraint,
+                **project_constraint,
+                **{f"{key_path.replace('.', '__')}__exists": True},
+            )
+            .only("last_update")
+            .order_by("-last_update")
+            .limit(1)
+            .first()
+        )
+        if not last_updated_model:
+            return 0, []
+
+        redis_key = f"modelmetadata_values_{company_id}_{'_'.join(project_ids)}_{key}_{allow_public}_{page}_{page_size}"
+        last_update = last_updated_model.last_update or datetime.utcnow()
+        cached_res = self._get_cached_param_values(
+            key=redis_key, last_update=last_update
+        )
+        if cached_res:
+            return cached_res
+
+        pipeline = [
+            {
+                "$match": {
+                    **company_constraint,
+                    **project_constraint,
+                    key_path: {"$exists": True},
+                }
+            },
+            {"$project": {"value": f"${key_path}.value"}},
+            {"$group": {"_id": "$value"}},
+            {"$sort": {"_id": 1}},
+            {"$skip": page * page_size},
+            {"$limit": page_size},
+            {
+                "$group": {
+                    "_id": 1,
+                    "total": {"$sum": 1},
+                    "results": {"$push": "$$ROOT._id"},
+                }
+            },
+        ]
+
+        result = next(Model.aggregate(pipeline, collation=Model._numeric_locale), None)
+        if not result:
+            return 0, []
+
+        total = int(result.get("total", 0))
+        values = result.get("results", [])
+
+        ttl = config.get("services.models.metadata_values.cache_ttl_sec", 86400)
+        cached = dict(last_update=last_update.timestamp(), total=total, values=values)
+        self.redis.setex(redis_key, ttl, json.dumps(cached))
+
+        return total, values

apiserver/bll/project/sub_projects.py

@@ -0,0 +1,198 @@
+import itertools
+from datetime import datetime
+from typing import Tuple, Optional, Sequence, Mapping
+
+from apiserver import database
+from apiserver.apierrors import errors
+from apiserver.database.model import EntityVisibility
+from apiserver.database.model.project import Project
+
+name_separator = "/"
+
+
+def _get_project_depth(project_name: str) -> int:
+    return len(list(filter(None, project_name.split(name_separator))))
+
+
+def _validate_project_name(project_name: str, raise_if_empty=True) -> Tuple[str, str]:
+    """
+    Remove redundant '/' characters. Ensure that the project name is not empty
+    Return the cleaned up project name and location
+    """
+    name_parts = [p.strip() for p in project_name.split(name_separator) if p]
+    if not name_parts:
+        if raise_if_empty:
+            raise errors.bad_request.InvalidProjectName(name=project_name)
+        return "", ""
+
+    return name_separator.join(name_parts), name_separator.join(name_parts[:-1])
+
+
+def _ensure_project(
+    company: str, user: str, name: str, creation_params: dict = None
+) -> Optional[Project]:
+    """
+    Makes sure that the project with the given name exists
+    If needed auto-create the project and all the missing projects in the path to it
+    Return the project
+    """
+    name, location = _validate_project_name(name, raise_if_empty=False)
+    if not name:
+        return None
+
+    project = _get_writable_project_from_name(company, name)
+    if project:
+        return project
+
+    now = datetime.utcnow()
+    project = Project(
+        id=database.utils.id(),
+        user=user,
+        company=company,
+        created=now,
+        last_update=now,
+        name=name,
+        basename=name.split("/")[-1],
+        **(creation_params or dict(description="")),
+    )
+    parent = _ensure_project(company, user, location, creation_params=creation_params)
+    _save_under_parent(project=project, parent=parent)
+    if parent:
+        parent.update(last_update=now)
+
+    return project
+
+
+def _save_under_parent(project: Project, parent: Optional[Project]):
+    """
+    Save the project under the given parent project or top level (parent=None)
+    Check that the project location matches the parent name
+    """
+    location, _, _ = project.name.rpartition(name_separator)
+    if not parent:
+        if location:
+            raise ValueError(
+                f"Project location {location} does not match empty parent name"
+            )
+        project.parent = None
+        project.path = []
+        project.save()
+        return
+
+    if location != parent.name:
+        raise ValueError(
+            f"Project location {location} does not match parent name {parent.name}"
+        )
+    project.parent = parent.id
+    project.path = [*(parent.path or []), parent.id]
+    project.save()
+
+
+def _get_writable_project_from_name(
+    company,
+    name,
+    _only: Optional[Sequence[str]] = ("id", "name", "path", "company", "parent"),
+) -> Optional[Project]:
+    """
+    Return a project from name. If the project not found then return None
+    """
+    qs = Project.objects(company=company, name=name)
+    if _only:
+        qs = qs.only(*_only)
+    return qs.first()
+
+
+ProjectsChildren = Mapping[str, Sequence[Project]]
+
+
+def _get_sub_projects(
+    project_ids: Sequence[str],
+    _only: Sequence[str] = ("id", "path"),
+    search_hidden=True,
+    allowed_ids: Sequence[str] = None,
+) -> ProjectsChildren:
+    """
+    Return the list of child projects of all the levels for the parent project ids
+    """
+    query = dict(path__in=project_ids)
+    if not search_hidden:
+        query["system_tags__nin"] = [EntityVisibility.hidden.value]
+    if allowed_ids:
+        query["id__in"] = allowed_ids
+
+    qs = Project.objects(**query)
+    if _only:
+        _only = set(_only) | {"path"}
+        qs = qs.only(*_only)
+    subprojects = list(qs)
+
+    return {
+        pid: [s for s in subprojects if pid in (s.path or [])] for pid in project_ids
+    }
+
+
+def _ids_with_parents(project_ids: Sequence[str]) -> Sequence[str]:
+    """
+    Return project ids with all the parent projects
+    """
+    projects = Project.objects(id__in=project_ids).only("id", "path")
+    parent_ids = set(itertools.chain.from_iterable(p.path for p in projects if p.path))
+    return list({*(p.id for p in projects), *parent_ids})
+
+
+def _ids_with_children(project_ids: Sequence[str]) -> Sequence[str]:
+    """
+    Return project ids with the ids of all the subprojects
+    """
+    children_ids = Project.objects(path__in=project_ids).scalar("id")
+    return list({*project_ids, *children_ids})
+
+
+def _update_subproject_names(
+    project: Project,
+    children: Sequence[Project],
+    old_name: str,
+    update_path: bool = False,
+    old_path: Sequence[str] = None,
+) -> int:
+    """
+    Update sub project names when the base project name changes
+    Optionally update the paths
+    """
+    updated = 0
+    now = datetime.utcnow()
+    for child in children:
+        child_suffix = name_separator.join(
+            child.name.split(name_separator)[len(old_name.split(name_separator)):]
+        )
+        updates = {
+            "name": name_separator.join((project.name, child_suffix)),
+            "last_update": now,
+        }
+        if update_path:
+            updates["path"] = project.path + child.path[len(old_path):]
+        updated += child.update(upsert=False, **updates)
+
+    return updated
+
+
+def _reposition_project_with_children(
+    project: Project, children: Sequence[Project], parent: Project
+) -> int:
+    new_location = parent.name if parent else None
+    old_name = project.name
+    old_path = project.path
+    project.name = name_separator.join(
+        filter(None, (new_location, project.name.split(name_separator)[-1]))
+    )
+    project.last_update = datetime.utcnow()
+    _save_under_parent(project, parent=parent)
+
+    moved = 1 + _update_subproject_names(
+        project=project,
+        children=children,
+        old_name=old_name,
+        update_path=True,
+        old_path=old_path,
+    )
+    return moved

apiserver/bll/query/__init__.py

@@ -0,0 +1 @@
+from .builder import Builder

apiserver/bll/query/builder.py

@@ -0,0 +1,36 @@
+from typing import Optional, Sequence, Iterable, Union
+
+from apiserver.config_repo import config
+
+log = config.logger(__file__)
+
+RANGE_IGNORE_VALUE = -1
+
+
+class Builder:
+    @staticmethod
+    def dates_range(from_date: Union[int, float], to_date: Union[int, float]) -> dict:
+        return {
+            "range": {
+                "timestamp": {
+                    "gte": int(from_date),
+                    "lte": int(to_date),
+                    "format": "epoch_second",
+                }
+            }
+        }
+
+    @staticmethod
+    def terms(field: str, values: Iterable[str]) -> dict:
+        return {"terms": {field: list(values)}}
+
+    @staticmethod
+    def normalize_range(
+        range_: Sequence[Union[int, float]],
+        ignore_value: Union[int, float] = RANGE_IGNORE_VALUE,
+    ) -> Optional[Sequence[Union[int, float]]]:
+        if not range_ or set(range_) == {ignore_value}:
+            return None
+        if len(range_) < 2:
+            return [range_[0]] * 2
+        return range_

apiserver/bll/queue/__init__.py

@@ -0,0 +1 @@
+from .queue_bll import QueueBLL

apiserver/bll/queue/queue_bll.py

@@ -0,0 +1,475 @@
+from collections import defaultdict
+from datetime import datetime
+from typing import Sequence, Optional, Tuple, Union
+
+from elasticsearch import Elasticsearch
+from mongoengine import Q
+
+from apiserver import database
+from apiserver.database.model.task.task import Task, TaskStatus
+from apiserver.es_factory import es_factory
+from apiserver.apierrors import errors
+from apiserver.bll.queue.queue_metrics import QueueMetrics
+from apiserver.bll.workers import WorkerBLL
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.queue import Queue, Entry
+
+log = config.logger(__file__)
+MOVE_FIRST = "first"
+MOVE_LAST = "last"
+
+
+class QueueBLL(object):
+    def __init__(self, worker_bll: WorkerBLL = None, es: Elasticsearch = None):
+        self.worker_bll = worker_bll or WorkerBLL()
+        self.es = es or es_factory.connect("workers")
+        self._metrics = QueueMetrics(self.es)
+
+    @property
+    def metrics(self) -> QueueMetrics:
+        return self._metrics
+
+    @staticmethod
+    def create(
+        company_id: str,
+        name: str,
+        tags: Optional[Sequence[str]] = None,
+        system_tags: Optional[Sequence[str]] = None,
+        metadata: Optional[dict] = None,
+    ) -> Queue:
+        """Creates a queue"""
+        with translate_errors_context():
+            now = datetime.utcnow()
+            queue = Queue(
+                id=database.utils.id(),
+                company=company_id,
+                created=now,
+                name=name,
+                tags=tags or [],
+                system_tags=system_tags or [],
+                metadata=metadata,
+                last_update=now,
+            )
+            queue.save()
+            return queue
+
+    def get_by_name(
+        self, company_id: str, queue_name: str, only: Optional[Sequence[str]] = None,
+    ) -> Queue:
+        qs = Queue.objects(name=queue_name, company=company_id)
+        if only:
+            qs = qs.only(*only)
+
+        return qs.first()
+
+    @staticmethod
+    def _get_task_entries_projection(max_task_entries: int) -> dict:
+        return dict(slice__entries=max_task_entries)
+
+    def get_by_id(
+        self,
+        company_id: str,
+        queue_id: str,
+        only: Optional[Sequence[str]] = None,
+        max_task_entries: int = None,
+    ) -> Queue:
+        """
+        Get queue by id
+        :raise errors.bad_request.InvalidQueueId: if the queue is not found
+        """
+        with translate_errors_context():
+            query = dict(id=queue_id, company=company_id)
+            qs = Queue.objects(**query)
+            if only:
+                qs = qs.only(*only)
+            if max_task_entries:
+                qs = qs.fields(**self._get_task_entries_projection(max_task_entries))
+            queue = qs.first()
+            if not queue:
+                raise errors.bad_request.InvalidQueueId(**query)
+
+            return queue
+
+    @classmethod
+    def get_queue_with_task(cls, company_id: str, queue_id: str, task_id: str) -> Queue:
+        with translate_errors_context():
+            query = dict(id=queue_id, company=company_id)
+            queue = Queue.objects(entries__task=task_id, **query).first()
+            if not queue:
+                raise errors.bad_request.InvalidQueueOrTaskNotQueued(
+                    task=task_id, **query
+                )
+
+            return queue
+
+    def get_default(self, company_id: str) -> Queue:
+        """
+        Get the default queue
+        :raise errors.bad_request.NoDefaultQueue: if the default queue not found
+        :raise errors.bad_request.MultipleDefaultQueues: if more than one default queue is found
+        """
+        with translate_errors_context():
+            res = Queue.objects(company=company_id, system_tags="default").only(
+                "id", "name"
+            )
+            if not res:
+                raise errors.bad_request.NoDefaultQueue()
+            if len(res) > 1:
+                raise errors.bad_request.MultipleDefaultQueues(
+                    queues=tuple(r.id for r in res)
+                )
+
+            return res.first()
+
+    def update(
+        self, company_id: str, queue_id: str, **update_fields
+    ) -> Tuple[int, dict]:
+        """
+        Partial update of the queue from update_fields
+        :raise errors.bad_request.InvalidQueueId: if the queue is not found
+        :return: number of updated objects and updated fields dictionary
+        """
+        with translate_errors_context():
+            # validate the queue exists
+            self.get_by_id(company_id=company_id, queue_id=queue_id, only=("id",))
+            return Queue.safe_update(company_id, queue_id, update_fields)
+
+    def delete(self, company_id: str, user_id: str, queue_id: str, force: bool) -> None:
+        """
+        Delete the queue
+        :raise errors.bad_request.InvalidQueueId: if the queue is not found
+        :raise errors.bad_request.QueueNotEmpty: if the queue is not empty and 'force' not set
+        """
+        with translate_errors_context():
+            queue = self.get_by_id(company_id=company_id, queue_id=queue_id)
+            if queue.entries:
+                if not force:
+                    raise errors.bad_request.QueueNotEmpty(
+                        "use force=true to delete", id=queue_id
+                    )
+                from apiserver.bll.task import ChangeStatusRequest
+
+                for item in queue.entries:
+                    try:
+                        task = Task.get(
+                            company=company_id,
+                            id=item.task,
+                            _only=[
+                                "id",
+                                "company",
+                                "status",
+                                "enqueue_status",
+                                "project",
+                            ],
+                        )
+                        if not task:
+                            continue
+
+                        ChangeStatusRequest(
+                            task=task,
+                            new_status=task.enqueue_status or TaskStatus.created,
+                            status_reason="Queue was deleted",
+                            status_message="",
+                            user_id=user_id,
+                            force=True,
+                        ).execute(enqueue_status=None)
+                    except Exception as ex:
+                        log.exception(
+                            f"Failed dequeuing task {item.task} from queue: {queue_id}"
+                        )
+
+            queue.delete()
+
+    def get_all(
+        self,
+        company_id: str,
+        query_dict: dict,
+        query: Q = None,
+        max_task_entries: int = None,
+        ret_params: dict = None,
+    ) -> Sequence[dict]:
+        """Get all the queues according to the query"""
+        with translate_errors_context():
+            return Queue.get_many(
+                company=company_id,
+                parameters=query_dict,
+                query_dict=query_dict,
+                query=query,
+                projection_fields=self._get_task_entries_projection(max_task_entries)
+                if max_task_entries
+                else None,
+                ret_params=ret_params,
+            )
+
+    def check_for_workers(self, company_id: str, queue_id: str) -> bool:
+        for worker in self.worker_bll.get_all(company_id):
+            if queue_id in worker.queues:
+                return True
+
+        return False
+
+    def get_queue_infos(
+        self,
+        company_id: str,
+        query_dict: dict,
+        query: Q = None,
+        max_task_entries: int = None,
+        ret_params: dict = None,
+    ) -> Sequence[dict]:
+        """
+        Get infos on all the company queues, including queue tasks and workers
+        """
+        projection = Queue.get_extra_projection("entries.task.name")
+        with translate_errors_context():
+            res = Queue.get_many_with_join(
+                company=company_id,
+                query_dict=query_dict,
+                query=query,
+                override_projection=projection,
+                projection_fields=self._get_task_entries_projection(max_task_entries)
+                if max_task_entries
+                else None,
+                ret_params=ret_params,
+            )
+
+            queue_workers = defaultdict(list)
+            for worker in self.worker_bll.get_all(company_id):
+                for queue in worker.queues:
+                    queue_workers[queue].append(worker)
+
+            for item in res:
+                item["workers"] = [
+                    {
+                        "name": w.id,
+                        "ip": w.ip,
+                        "key": w.key,
+                        "task": w.task.to_struct() if w.task else None,
+                    }
+                    for w in queue_workers.get(item["id"], [])
+                ]
+
+        return res
+
+    def add_task(self, company_id: str, queue_id: str, task_id: str) -> dict:
+        """
+        Add the task to the queue and return the queue update results
+        :raise errors.bad_request.TaskAlreadyQueued: if the task is already in the queue
+        :raise errors.bad_request.InvalidQueueOrTaskNotQueued: if the queue update operation failed
+        """
+        with translate_errors_context():
+            queue = self.get_by_id(company_id=company_id, queue_id=queue_id)
+            if any(e.task == task_id for e in queue.entries):
+                raise errors.bad_request.TaskAlreadyQueued(task=task_id)
+
+            entry = Entry(added=datetime.utcnow(), task=task_id)
+            query = dict(id=queue_id, company=company_id)
+            res = Queue.objects(entries__task__ne=task_id, **query).update_one(
+                push__entries=entry, last_update=datetime.utcnow(), upsert=False
+            )
+
+            queue.reload()
+            self.metrics.log_queue_metrics_to_es(company_id=company_id, queues=[queue])
+
+            if not res:
+                raise errors.bad_request.InvalidQueueOrTaskNotQueued(
+                    task=task_id, **query
+                )
+
+            return res
+
+    def get_next_task(
+        self, company_id: str, queue_id: str, task_id: str = None
+    ) -> Optional[Entry]:
+        """
+        Atomically pop and return the first task from the queue (or None)
+        :raise errors.bad_request.InvalidQueueId: if the queue does not exist
+        """
+        with translate_errors_context():
+            query = dict(id=queue_id, company=company_id)
+            queue = Queue.objects(
+                **query, **({"entries__0__task": task_id} if task_id else {})
+            ).modify(pop__entries=-1, upsert=False)
+            if not queue:
+                if not task_id or not Queue.objects(**query).first():
+                    raise errors.bad_request.InvalidQueueId(**query)
+                return
+
+            self.metrics.log_queue_metrics_to_es(company_id, queues=[queue])
+
+            if not queue.entries:
+                return
+
+            try:
+                Queue.objects(**query).update(last_update=datetime.utcnow())
+            except Exception:
+                log.exception("Error while updating Queue.last_update")
+
+            return queue.entries[0]
+
+    def remove_task(self, company_id: str, queue_id: str, task_id: str) -> int:
+        """
+        Removes the task from the queue and returns the number of removed items
+        :raise errors.bad_request.InvalidQueueOrTaskNotQueued: if the task is not found in the queue
+        """
+        with translate_errors_context():
+            queue = self.get_queue_with_task(
+                company_id=company_id, queue_id=queue_id, task_id=task_id
+            )
+
+            entries_to_remove = [e for e in queue.entries if e.task == task_id]
+            query = dict(id=queue_id, company=company_id)
+            res = Queue.objects(entries__task=task_id, **query).update_one(
+                pull_all__entries=entries_to_remove, last_update=datetime.utcnow()
+            )
+
+            queue.reload()
+            self.metrics.log_queue_metrics_to_es(company_id=company_id, queues=[queue])
+
+            return len(entries_to_remove) if res else 0
+
+    def reposition_task(
+        self, company_id: str, queue_id: str, task_id: str, move_count: Union[int, str],
+    ) -> int:
+        """
+        Moves the task in the queue to the position calculated by pos_func
+        Returns the updated task position in the queue
+        """
+
+        def get_queue_and_task_position():
+            q = self.get_queue_with_task(
+                company_id=company_id, queue_id=queue_id, task_id=task_id
+            )
+            return q, next(i for i, e in enumerate(q.entries) if e.task == task_id)
+
+        with translate_errors_context():
+            queue, position = get_queue_and_task_position()
+            if move_count == MOVE_FIRST:
+                new_position = 0
+            elif move_count == MOVE_LAST:
+                new_position = len(queue.entries) - 1
+            else:
+                new_position = position + move_count
+            if new_position == position:
+                return new_position
+
+            without_entry = {
+                "$filter": {
+                    "input": "$entries",
+                    "as": "entry",
+                    "cond": {"$ne": ["$$entry.task", task_id]},
+                }
+            }
+            task_entry = {
+                "$filter": {
+                    "input": "$entries",
+                    "as": "entry",
+                    "cond": {"$eq": ["$$entry.task", task_id]},
+                }
+            }
+            if move_count == MOVE_FIRST:
+                operations = [
+                    {
+                        "$set": {
+                            "entries": {"$concatArrays": [task_entry, without_entry]}
+                        }
+                    }
+                ]
+            elif move_count == MOVE_LAST:
+                operations = [
+                    {
+                        "$set": {
+                            "entries": {"$concatArrays": [without_entry, task_entry]}
+                        }
+                    }
+                ]
+            else:
+                operations = [
+                    {
+                        "$set": {
+                            "new_pos": {
+                                "$add": [
+                                    {"$indexOfArray": ["$entries.task", task_id]},
+                                    move_count,
+                                ]
+                            },
+                            "without_entry": without_entry,
+                            "task_entry": task_entry,
+                        }
+                    },
+                    {
+                        "$set": {
+                            "entries": {
+                                "$switch": {
+                                    "branches": [
+                                        {
+                                            "case": {"$lte": ["$new_pos", 0]},
+                                            "then": {
+                                                "$concatArrays": [
+                                                    "$task_entry",
+                                                    "$without_entry",
+                                                ]
+                                            },
+                                        },
+                                        {
+                                            "case": {
+                                                "$gte": [
+                                                    "$new_pos",
+                                                    {"$size": "$without_entry"},
+                                                ]
+                                            },
+                                            "then": {
+                                                "$concatArrays": [
+                                                    "$without_entry",
+                                                    "$task_entry",
+                                                ]
+                                            },
+                                        },
+                                    ],
+                                    "default": {
+                                        "$concatArrays": [
+                                            {"$slice": ["$without_entry", "$new_pos"]},
+                                            "$task_entry",
+                                            {
+                                                "$slice": [
+                                                    "$without_entry",
+                                                    "$new_pos",
+                                                    {"$size": "$without_entry"},
+                                                ]
+                                            },
+                                        ]
+                                    },
+                                }
+                            }
+                        }
+                    },
+                    {"$unset": ["new_pos", "without_entry", "task_entry"]},
+                ]
+
+            updated = Queue.objects(
+                id=queue_id, company=company_id, entries__task=task_id
+            ).update_one(__raw__=operations)
+
+            if not updated:
+                raise errors.bad_request.FailedAddingDuringReposition(task=task_id)
+
+            return get_queue_and_task_position()[1]
+
+    def count_entries(self, company: str, queue_id: str) -> Optional[int]:
+        res = next(
+            Queue.aggregate(
+                [
+                    {
+                        "$match": {
+                            "company": {"$in": [None, "", company]},
+                            "_id": queue_id,
+                        }
+                    },
+                    {"$project": {"count": {"$size": "$entries"}}},
+                ]
+            ),
+            None,
+        )
+        if res is None:
+            raise errors.bad_request.InvalidQueueId(queue_id=queue_id)
+        return int(res.get("count"))

apiserver/bll/queue/queue_metrics.py

@@ -0,0 +1,314 @@
+import json
+from collections import defaultdict
+from datetime import datetime
+from time import sleep
+from typing import Sequence
+
+from boltons.typeutils import classproperty
+from elasticsearch import Elasticsearch
+
+from apiserver.es_factory import es_factory
+from apiserver.apierrors.errors import bad_request
+from apiserver.bll.query import Builder as QueryBuilder
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.queue import Queue, Entry
+from apiserver.redis_manager import redman
+from apiserver.utilities.threads_manager import ThreadsManager
+
+log = config.logger(__file__)
+_conf = config.get("services.queues")
+_queue_metrics_key_pattern = "queue_metrics_{queue}"
+redis = redman.connection("apiserver")
+
+
+class EsKeys:
+    WAITING_TIME_FIELD = "average_waiting_time"
+    QUEUE_LENGTH_FIELD = "queue_length"
+    TIMESTAMP_FIELD = "timestamp"
+    QUEUE_FIELD = "queue"
+
+
+class QueueMetrics:
+    def __init__(self, es: Elasticsearch):
+        self.es = es
+
+    @staticmethod
+    def _queue_metrics_prefix_for_company(company_id: str) -> str:
+        """Returns the es index prefix for the company"""
+        return f"queue_metrics_{company_id.lower()}_"
+
+    @staticmethod
+    def _get_es_index_suffix():
+        """Get the index name suffix for storing current month data"""
+        return datetime.utcnow().strftime("%Y-%m")
+
+    @staticmethod
+    def _calc_avg_waiting_time(entries: Sequence[Entry]) -> float:
+        """
+        Calculate avg waiting time for the given tasks.
+        Return 0 if the list is empty
+        """
+        if not entries:
+            return 0
+
+        now = datetime.utcnow()
+        total_waiting_in_secs = sum((now - e.added).total_seconds() for e in entries)
+        return total_waiting_in_secs / len(entries)
+
+    def log_queue_metrics_to_es(self, company_id: str, queues: Sequence[Queue]) -> int:
+        """
+        Calculate and write queue statistics (avg waiting time and queue length) to Elastic
+        :return: True if the write to es was successful, false otherwise
+        """
+        es_index = (
+            self._queue_metrics_prefix_for_company(company_id)
+            + self._get_es_index_suffix()
+        )
+
+        timestamp = es_factory.get_timestamp_millis()
+
+        def make_doc(queue: Queue) -> dict:
+            entries = [e for e in queue.entries if e.added]
+            return {
+                EsKeys.TIMESTAMP_FIELD: timestamp,
+                EsKeys.QUEUE_FIELD: queue.id,
+                EsKeys.WAITING_TIME_FIELD: self._calc_avg_waiting_time(entries),
+                EsKeys.QUEUE_LENGTH_FIELD: len(entries),
+            }
+
+        logged = 0
+        for q in queues:
+            queue_doc = make_doc(q)
+            self.es.index(index=es_index, document=queue_doc)
+            redis_key = _queue_metrics_key_pattern.format(queue=q.id)
+            redis.set(redis_key, json.dumps(queue_doc))
+            logged += 1
+
+        return logged
+
+    def _log_current_metrics(self, company_id: str, queue_ids=Sequence[str]):
+        query = dict(company=company_id)
+        if queue_ids:
+            query["id__in"] = list(queue_ids)
+        queues = Queue.objects(**query)
+        self.log_queue_metrics_to_es(company_id, queues=list(queues))
+
+    def _search_company_metrics(self, company_id: str, es_req: dict) -> dict:
+        return self.es.search(
+            index=f"{self._queue_metrics_prefix_for_company(company_id)}*", body=es_req,
+        )
+
+    @classmethod
+    def _get_dates_agg(cls, interval) -> dict:
+        """
+        Aggregation for building date histogram with internal grouping per queue.
+        We are grouping by queue inside date histogram and not vice versa so that
+        it will be easy to average between queue metrics inside each date bucket.
+        Ignore empty buckets.
+        """
+        return {
+            "dates": {
+                "date_histogram": {
+                    "field": EsKeys.TIMESTAMP_FIELD,
+                    "fixed_interval": f"{interval}s",
+                    "min_doc_count": 1,
+                },
+                "aggs": {
+                    "queues": {
+                        "terms": {"field": EsKeys.QUEUE_FIELD},
+                        "aggs": cls._get_top_waiting_agg(),
+                    }
+                },
+            }
+        }
+
+    @classmethod
+    def _get_top_waiting_agg(cls) -> dict:
+        """
+        Aggregation for getting max waiting time and the corresponding queue length
+        inside each date->queue bucket
+        """
+        return {
+            "top_avg_waiting": {
+                "top_hits": {
+                    "sort": [
+                        {EsKeys.WAITING_TIME_FIELD: {"order": "desc"}},
+                        {EsKeys.QUEUE_LENGTH_FIELD: {"order": "desc"}},
+                    ],
+                    "_source": {
+                        "includes": [
+                            EsKeys.WAITING_TIME_FIELD,
+                            EsKeys.QUEUE_LENGTH_FIELD,
+                        ]
+                    },
+                    "size": 1,
+                }
+            }
+        }
+
+    def get_queue_metrics(
+        self,
+        company_id: str,
+        from_date: float,
+        to_date: float,
+        interval: int,
+        queue_ids: Sequence[str],
+        refresh: bool = False,
+    ) -> dict:
+        """
+        Get the company queue metrics in the specified time range.
+        Returned as date histograms of average values per queue and metric type.
+        The from_date is extended by 'metrics_before_from_date' seconds from
+        queues.conf due to possibly small amount of points. The default extension is 3600s
+        In case no queue ids are specified the avg across all the
+        company queues is calculated for each metric
+        """
+        if refresh:
+            self._log_current_metrics(company_id, queue_ids=queue_ids)
+
+        if from_date >= to_date:
+            raise bad_request.FieldsValueError("from_date must be less than to_date")
+
+        seconds_before = config.get("services.queues.metrics_before_from_date", 3600)
+        must_terms = [QueryBuilder.dates_range(from_date - seconds_before, to_date)]
+        if queue_ids:
+            must_terms.append(QueryBuilder.terms("queue", queue_ids))
+
+        es_req = {
+            "size": 0,
+            "query": {"bool": {"must": must_terms}},
+            "aggs": self._get_dates_agg(interval),
+        }
+
+        with translate_errors_context():
+            res = self._search_company_metrics(company_id, es_req)
+
+        if "aggregations" not in res:
+            return {}
+
+        date_metrics = [
+            dict(
+                timestamp=d["key"],
+                queue_metrics=self._extract_queue_metrics(d["queues"]["buckets"]),
+            )
+            for d in res["aggregations"]["dates"]["buckets"]
+            if d["doc_count"] > 0
+        ]
+        if queue_ids:
+            return self._datetime_histogram_per_queue(date_metrics)
+
+        return self._average_datetime_histogram(date_metrics)
+
+    @classmethod
+    def _datetime_histogram_per_queue(cls, date_metrics: Sequence[dict]) -> dict:
+        """
+        Build datetime histogram per queue from datetime histogram where every
+        bucket contains all the queues metrics
+        """
+        queues_data = defaultdict(list)
+        for date_data in date_metrics:
+            timestamp = date_data["timestamp"]
+            for queue, metrics in date_data["queue_metrics"].items():
+                queues_data[queue].append({"date": timestamp, **metrics})
+
+        return queues_data
+
+    @classmethod
+    def _average_datetime_histogram(cls, date_metrics: Sequence[dict]) -> dict:
+        """
+        Calculate weighted averages and total count for each bucket of date_metrics histogram.
+        If for any queue the data is missing then take it from the previous bucket
+        The result is returned as a dictionary with one key 'total'
+        """
+        queues_total = []
+        last_values = {}
+        for date_data in date_metrics:
+            date_metrics = date_data["queue_metrics"]
+            queue_metrics = {
+                **date_metrics,
+                **{k: v for k, v in last_values.items() if k not in date_metrics},
+            }
+
+            total_length = sum(m["queue_length"] for m in queue_metrics.values())
+            if total_length:
+                total_average = sum(
+                    m["avg_waiting_time"] * m["queue_length"] / total_length
+                    for m in queue_metrics.values()
+                )
+            else:
+                total_average = 0
+
+            queues_total.append(
+                dict(
+                    date=date_data["timestamp"],
+                    avg_waiting_time=total_average,
+                    queue_length=total_length,
+                )
+            )
+
+            for k, v in date_metrics.items():
+                last_values[k] = v
+
+        return dict(total=queues_total)
+
+    @classmethod
+    def _extract_queue_metrics(cls, queue_buckets: Sequence[dict]) -> dict:
+        """
+        Extract ES data for single date and queue bucket
+        """
+        queue_metrics = dict()
+        for queue_data in queue_buckets:
+            if not queue_data["doc_count"]:
+                continue
+            res = queue_data["top_avg_waiting"]["hits"]["hits"][0]["_source"]
+            queue_metrics[queue_data["key"]] = {
+                "queue_length": res[EsKeys.QUEUE_LENGTH_FIELD],
+                "avg_waiting_time": res[EsKeys.WAITING_TIME_FIELD],
+            }
+        return queue_metrics
+
+
+class MetricsRefresher:
+    threads = ThreadsManager()
+
+    @classproperty
+    def watch_interval_sec(self):
+        return _conf.get("metrics_refresh_interval_sec", 300)
+
+    @classmethod
+    @threads.register("queue_metrics_refresh_watchdog", daemon=True)
+    def start(cls, queue_metrics: QueueMetrics = None):
+        if not cls.watch_interval_sec:
+            return
+
+        if not queue_metrics:
+            from .queue_bll import QueueBLL
+
+            queue_metrics = QueueBLL().metrics
+
+        sleep(10)
+        while True:
+            try:
+                for queue in Queue.objects():
+                    timestamp = es_factory.get_timestamp_millis()
+                    doc_time = 0
+                    try:
+                        redis_key = _queue_metrics_key_pattern.format(queue=queue.id)
+                        data = redis.get(redis_key)
+                        if data:
+                            queue_doc = json.loads(data)
+                            doc_time = int(queue_doc.get(EsKeys.TIMESTAMP_FIELD))
+                    except Exception as ex:
+                        log.exception(
+                            f"Error reading queue metrics data for queue {queue.id}: {str(ex)}"
+                        )
+
+                    if (
+                        not doc_time
+                        or (timestamp - doc_time) > cls.watch_interval_sec * 1000
+                    ):
+                        queue_metrics.log_queue_metrics_to_es(queue.company, [queue])
+            except Exception as ex:
+                log.exception(f"Failed collecting queue metrics: {str(ex)}")
+            sleep(60)

apiserver/bll/redis_cache_manager.py

@@ -0,0 +1,87 @@
+from contextlib import contextmanager
+from typing import Optional, TypeVar, Generic, Type, Callable
+
+from redis import StrictRedis
+
+from apiserver import database
+
+T = TypeVar("T")
+
+
+def _do_nothing(_: T):
+    return
+
+
+class RedisCacheManager(Generic[T]):
+    """
+    Class for store/retrieve of state objects from redis
+
+    self.state_class - class of the state
+    self.redis - instance of redis
+    self.expiration_interval - expiration interval in seconds
+    """
+
+    def __init__(
+        self, state_class: Type[T], redis: StrictRedis, expiration_interval: int
+    ):
+        self.state_class = state_class
+        self.redis = redis
+        self.expiration_interval = expiration_interval
+
+    def set_state(self, state: T) -> None:
+        redis_key = self._get_redis_key(state.id)
+        self.redis.set(redis_key, state.to_json())
+        self.redis.expire(redis_key, self.expiration_interval)
+
+    def get_state(self, state_id) -> Optional[T]:
+        redis_key = self._get_redis_key(state_id)
+        response = self.redis.get(redis_key)
+        if response:
+            return self.state_class.from_json(response)
+
+    def delete_state(self, state_id) -> None:
+        self.redis.delete(self._get_redis_key(state_id))
+
+    def _get_redis_key(self, state_id):
+        return f"{self.state_class}/{state_id}"
+
+    def get_or_create_state_core(
+        self,
+        state_id=None,
+        init_state: Callable[[T], None] = _do_nothing,
+        validate_state: Callable[[T], None] = _do_nothing,
+    ) -> T:
+        state = self.get_state(state_id) if state_id else None
+        if state:
+            validate_state(state)
+        else:
+            state = self.state_class(id=database.utils.id())
+            init_state(state)
+
+        return state
+
+    @contextmanager
+    def get_or_create_state(
+        self,
+        state_id=None,
+        init_state: Callable[[T], None] = _do_nothing,
+        validate_state: Callable[[T], None] = _do_nothing,
+    ):
+        """
+        Try to retrieve state with the given id from the Redis cache if yes then validates it
+        If no then create a new one with randomly generated id
+        Yield the state and write it back to redis once the user code block exits
+        :param state_id: id of the state to retrieve
+        :param init_state: user callback to init the newly created state
+        If not passed then no init except for the id generation is done
+        :param validate_state: user callback to validate the state if retrieved from cache
+        Should throw an exception if the state is not valid. If not passed then no validation is done
+        """
+        state = self.get_or_create_state_core(
+            state_id=state_id, init_state=init_state, validate_state=validate_state
+        )
+
+        try:
+            yield state
+        finally:
+            self.set_state(state)

apiserver/bll/statistics/resource_monitor.py

@@ -0,0 +1,97 @@
+from datetime import datetime
+import operator
+from threading import Lock
+from time import sleep
+
+import attr
+import psutil
+
+from apiserver.utilities.threads_manager import ThreadsManager
+
+
+stat_threads = ThreadsManager("Statistics")
+
+
+@attr.s(auto_attribs=True)
+class Sample:
+    cpu_usage: float = 0.0
+    mem_used_gb: float = 0
+    mem_free_gb: float = 0
+
+    @classmethod
+    def _apply(cls, op, *samples):
+        return cls(
+            **{
+                field: op(*(getattr(sample, field) for sample in samples))
+                for field in attr.fields_dict(cls)
+            }
+        )
+
+    def min(self, sample):
+        return self._apply(min, self, sample)
+
+    def max(self, sample):
+        return self._apply(max, self, sample)
+
+    def avg(self, sample, count):
+        res = self._apply(lambda x: x * count, self)
+        res = self._apply(operator.add, res, sample)
+        res = self._apply(lambda x: x / (count + 1), res)
+        return res
+
+    @classmethod
+    def get_current_sample(cls) -> "Sample":
+        return cls(
+            cpu_usage=psutil.cpu_percent(),
+            mem_used_gb=psutil.virtual_memory().used / (1024 ** 3),
+            mem_free_gb=psutil.virtual_memory().free / (1024 ** 3),
+        )
+
+
+class ResourceMonitor:
+    class Accumulator:
+        def __init__(self):
+            sample = Sample.get_current_sample()
+            self.avg = sample
+            self.min = sample
+            self.max = sample
+            self.time = datetime.utcnow()
+            self.count = 1
+
+        def add_sample(self, sample: Sample):
+            self.min = self.min.min(sample)
+            self.max = self.max.max(sample)
+            self.avg = self.avg.avg(sample, self.count)
+            self.count += 1
+
+    sample_interval_sec = 5
+    _lock = Lock()
+    accumulator = Accumulator()
+
+    @classmethod
+    @stat_threads.register("resource_monitor", daemon=True)
+    def start(cls):
+        while True:
+            sleep(cls.sample_interval_sec)
+            sample = Sample.get_current_sample()
+            with cls._lock:
+                cls.accumulator.add_sample(sample)
+
+    @classmethod
+    def get_stats(cls) -> dict:
+        """ Returns current resource statistics and clears internal resource statistics """
+        with cls._lock:
+            min_ = attr.asdict(cls.accumulator.min)
+            max_ = attr.asdict(cls.accumulator.max)
+            avg = attr.asdict(cls.accumulator.avg)
+            interval = datetime.utcnow() - cls.accumulator.time
+            cls.accumulator = cls.Accumulator()
+
+        return {
+            "interval_sec": interval.total_seconds(),
+            "num_cores": psutil.cpu_count(),
+            **{
+                k: {"min": v, "max": max_[k], "avg": avg[k]}
+                for k, v in min_.items()
+            }
+        }

apiserver/bll/statistics/stats_reporter.py

@@ -0,0 +1,312 @@
+import logging
+import queue
+import random
+import time
+from datetime import timedelta, datetime
+from time import sleep
+from typing import Sequence, Optional
+
+import dpath
+import requests
+from requests.adapters import HTTPAdapter, Retry
+
+from apiserver.bll.query import Builder as QueryBuilder
+from apiserver.bll.util import get_server_uuid
+from apiserver.bll.workers import WorkerStats, WorkerBLL
+from apiserver.config_repo import config
+from apiserver.config.info import get_deployment_type
+from apiserver.database.model import Company, User
+from apiserver.database.model.queue import Queue
+from apiserver.database.model.task.task import Task
+from apiserver.utilities.dicts import nested_get
+from apiserver.utilities.json import dumps
+from apiserver.version import __version__ as current_version
+from .resource_monitor import ResourceMonitor, stat_threads
+
+log = config.logger(__file__)
+
+worker_bll = WorkerBLL()
+
+
+class StatisticsReporter:
+    send_queue = queue.Queue()
+    supported = config.get("apiserver.statistics.supported", True)
+
+    @classmethod
+    def start(cls):
+        if not cls.supported:
+            return
+        ResourceMonitor.start()
+        cls.start_sender()
+        cls.start_reporter()
+
+    @classmethod
+    @stat_threads.register("reporter", daemon=True)
+    def start_reporter(cls):
+        """
+        Periodically send statistics reports for companies who have opted in.
+        Note: in clearml we usually have only a single company
+        """
+        if not cls.supported:
+            return
+
+        report_interval = timedelta(
+            hours=config.get("apiserver.statistics.report_interval_hours", 24)
+        )
+        sleep(report_interval.total_seconds())
+        while True:
+            try:
+                for company in Company.objects(
+                    defaults__stats_option__enabled=True
+                ).only("id"):
+                    stats = cls.get_statistics(company.id)
+                    cls.send_queue.put(stats)
+
+            except Exception as ex:
+                log.exception(f"Failed collecting stats: {str(ex)}")
+
+            sleep(report_interval.total_seconds())
+
+    @classmethod
+    @stat_threads.register("sender", daemon=True)
+    def start_sender(cls):
+        if not cls.supported:
+            return
+
+        url = config.get("apiserver.statistics.url")
+
+        retries = config.get("apiserver.statistics.max_retries", 5)
+        max_backoff = config.get("apiserver.statistics.max_backoff_sec", 5)
+        session = requests.Session()
+        adapter = HTTPAdapter(max_retries=Retry(retries))
+        session.mount("http://", adapter)
+        session.mount("https://", adapter)
+        session.headers["Content-type"] = "application/json"
+
+        WarningFilter.attach()
+
+        while True:
+            try:
+                report = cls.send_queue.get()
+
+                # Set a random backoff factor each time we send a report
+                adapter.max_retries.backoff_factor = random.random() * max_backoff
+
+                session.post(url, data=dumps(report))
+
+            except Exception as ex:
+                pass
+
+    @classmethod
+    def get_statistics(cls, company_id: str) -> dict:
+        """
+        Returns a statistics report per company
+        """
+        return {
+            "time": datetime.utcnow(),
+            "company_id": company_id,
+            "server": {
+                "version": current_version,
+                "deployment": get_deployment_type(),
+                "uuid": get_server_uuid(),
+                "queues": {"count": Queue.objects(company=company_id).count()},
+                "users": {"count": User.objects(company=company_id).count()},
+                "resources": ResourceMonitor.get_stats(),
+                "experiments": next(
+                    iter(cls._get_experiments_stats(company_id).values()), {}
+                ),
+            },
+            "agents": cls._get_agents_statistics(company_id),
+        }
+
+    @classmethod
+    def _get_agents_statistics(cls, company_id: str) -> Sequence[dict]:
+        result = cls._get_resource_stats_per_agent(company_id, key="resources")
+        dpath.merge(
+            result, cls._get_experiments_stats_per_agent(company_id, key="experiments")
+        )
+        return [{"uuid": agent_id, **data} for agent_id, data in result.items()]
+
+    @classmethod
+    def _get_resource_stats_per_agent(cls, company_id: str, key: str) -> dict:
+        agent_resource_threshold_sec = timedelta(
+            hours=config.get("apiserver.statistics.report_interval_hours", 24)
+        ).total_seconds()
+        to_timestamp = int(time.time())
+        from_timestamp = to_timestamp - int(agent_resource_threshold_sec)
+        es_req = {
+            "size": 0,
+            "query": QueryBuilder.dates_range(from_timestamp, to_timestamp),
+            "aggs": {
+                "workers": {
+                    "terms": {"field": "worker"},
+                    "aggs": {
+                        "categories": {
+                            "terms": {"field": "category"},
+                            "aggs": {"count": {"cardinality": {"field": "variant"}}},
+                        },
+                        "metrics": {
+                            "terms": {"field": "metric"},
+                            "aggs": {
+                                "min": {"min": {"field": "value"}},
+                                "max": {"max": {"field": "value"}},
+                                "avg": {"avg": {"field": "value"}},
+                            },
+                        },
+                    },
+                }
+            },
+        }
+        res = cls._run_worker_stats_query(company_id, es_req)
+
+        def _get_cardinality_fields(categories: Sequence[dict]) -> dict:
+            names = {"cpu": "num_cores"}
+            return {
+                names[c["key"]]: nested_get(c, ("count", "value"))
+                for c in categories
+                if c["key"] in names
+            }
+
+        def _get_metric_fields(metrics: Sequence[dict]) -> dict:
+            names = {
+                "cpu_usage": "cpu_usage",
+                "memory_used": "mem_used_gb",
+                "memory_free": "mem_free_gb",
+            }
+            return {
+                names[m["key"]]: {
+                    "min": nested_get(m, ("min", "value")),
+                    "max": nested_get(m, ("max", "value")),
+                    "avg": nested_get(m, ("avg", "value")),
+                }
+                for m in metrics
+                if m["key"] in names
+            }
+
+        buckets = nested_get(res, ("aggregations", "workers", "buckets"), default=[])
+        return {
+            b["key"]: {
+                key: {
+                    "interval_sec": agent_resource_threshold_sec,
+                    **_get_cardinality_fields(nested_get(b, ("categories", "buckets"), [])),
+                    **_get_metric_fields(nested_get(b, ("metrics", "buckets"), [])),
+                }
+            }
+            for b in buckets
+        }
+
+    @classmethod
+    def _get_experiments_stats_per_agent(cls, company_id: str, key: str) -> dict:
+        agent_relevant_threshold = timedelta(
+            days=config.get("apiserver.statistics.agent_relevant_threshold_days", 30)
+        )
+        to_timestamp = int(time.time())
+        from_timestamp = to_timestamp - int(agent_relevant_threshold.total_seconds())
+        workers = cls._get_active_workers(company_id, from_timestamp, to_timestamp)
+        if not workers:
+            return {}
+
+        stats = cls._get_experiments_stats(company_id, list(workers.keys()))
+        return {
+            worker_id: {key: {**workers[worker_id], **stat}}
+            for worker_id, stat in stats.items()
+        }
+
+    @classmethod
+    def _get_active_workers(
+        cls, company_id, from_timestamp: int, to_timestamp: int
+    ) -> dict:
+        es_req = {
+            "size": 0,
+            "query": QueryBuilder.dates_range(from_timestamp, to_timestamp),
+            "aggs": {
+                "workers": {
+                    "terms": {"field": "worker"},
+                    "aggs": {"last_activity_time": {"max": {"field": "timestamp"}}},
+                }
+            },
+        }
+        res = cls._run_worker_stats_query(company_id, es_req)
+        buckets = nested_get(res, ("aggregations", "workers", "buckets"), default=[])
+        return {
+            b["key"]: {"last_activity_time": b["last_activity_time"]["value"]}
+            for b in buckets
+        }
+
+    @classmethod
+    def _run_worker_stats_query(cls, company_id, es_req) -> dict:
+        return worker_bll.es_client.search(
+            index=f"{WorkerStats.worker_stats_prefix_for_company(company_id)}*",
+            body=es_req,
+        )
+
+    @classmethod
+    def _get_experiments_stats(
+        cls, company_id, workers: Optional[Sequence] = None
+    ) -> dict:
+        pipeline = [
+            {
+                "$match": {
+                    "company": company_id,
+                    "started": {"$exists": True, "$ne": None},
+                    "last_update": {"$exists": True, "$ne": None},
+                    "status": {"$nin": ["created", "queued"]},
+                    **({"last_worker": {"$in": workers}} if workers else {}),
+                }
+            },
+            {
+                "$project": {
+                    "last_worker": 1,
+                    "last_update": 1,
+                    "started": 1,
+                    "last_iteration": 1,
+                }
+            },
+            {
+                "$group": {
+                    "_id": "$last_worker" if workers else None,
+                    "count": {"$sum": 1},
+                    "avg_run_time_sec": {
+                        "$avg": {
+                            "$divide": [
+                                {"$subtract": ["$last_update", "$started"]},
+                                1000,
+                            ]
+                        }
+                    },
+                    "avg_iterations": {"$avg": "$last_iteration"},
+                }
+            },
+            {
+                "$project": {
+                    "count": 1,
+                    "avg_run_time_sec": {"$trunc": "$avg_run_time_sec"},
+                    "avg_iterations": {"$trunc": "$avg_iterations"},
+                }
+            },
+        ]
+        return {
+            group["_id"]: {k: v for k, v in group.items() if k != "_id"}
+            for group in Task.aggregate(pipeline)
+        }
+
+
+class WarningFilter(logging.Filter):
+    @classmethod
+    def attach(cls):
+        from urllib3.connectionpool import (
+            ConnectionPool,
+        )  # required to make sure the logger is created
+
+        assert ConnectionPool  # make sure import is not optimized out
+
+        logging.getLogger("urllib3.connectionpool").addFilter(cls())
+
+    def filter(self, record):
+        if (
+            record.levelno == logging.WARNING
+            and len(record.args) > 2
+            and record.args[2] == "/stats"
+        ):
+            return False
+        return True

apiserver/bll/storage/__init__.py

@@ -0,0 +1,48 @@
+from copy import copy
+
+from boltons.cacheutils import cachedproperty
+from clearml.backend_config.bucket_config import (
+    S3BucketConfigurations,
+    AzureContainerConfigurations,
+    GSBucketConfigurations,
+)
+
+from apiserver.config_repo import config
+
+
+log = config.logger(__file__)
+
+
+class StorageBLL:
+    default_aws_configs: S3BucketConfigurations = None
+    conf = config.get("services.storage_credentials")
+
+    @cachedproperty
+    def _default_aws_configs(self) -> S3BucketConfigurations:
+        return S3BucketConfigurations.from_config(self.conf.get("aws.s3"))
+
+    @cachedproperty
+    def _default_azure_configs(self) -> AzureContainerConfigurations:
+        return AzureContainerConfigurations.from_config(self.conf.get("azure.storage"))
+
+    @cachedproperty
+    def _default_gs_configs(self) -> GSBucketConfigurations:
+        return GSBucketConfigurations.from_config(self.conf.get("google.storage"))
+
+    def get_azure_settings_for_company(
+        self,
+        company_id: str,
+    ) -> AzureContainerConfigurations:
+        return copy(self._default_azure_configs)
+
+    def get_gs_settings_for_company(
+        self,
+        company_id: str,
+    ) -> GSBucketConfigurations:
+        return copy(self._default_gs_configs)
+
+    def get_aws_settings_for_company(
+        self,
+        company_id: str,
+    ) -> S3BucketConfigurations:
+        return copy(self._default_aws_configs)

apiserver/bll/task/__init__.py

@@ -1,7 +1,5 @@
 from .task_bll import TaskBLL
 from .utils import (
     ChangeStatusRequest,
-    update_project_time,
     validate_status_change,
-    split_by,
 )

apiserver/bll/task/artifacts.py

@@ -0,0 +1,93 @@
+from operator import itemgetter
+from typing import Sequence
+
+from apiserver.apimodels.tasks import Artifact as ApiArtifact, ArtifactId
+from apiserver.bll.task.utils import get_task_for_update, update_task
+from apiserver.database.model.task.task import DEFAULT_ARTIFACT_MODE, Artifact
+from apiserver.database.utils import hash_field_name
+from apiserver.service_repo.auth import Identity
+from apiserver.utilities.dicts import nested_get, nested_set
+from apiserver.utilities.parameter_key_escaper import mongoengine_safe
+
+
+def get_artifact_id(artifact: dict):
+    """
+    Calculate id from 'key' and 'mode' fields
+    Return hash on on the id so that it will not contain mongo illegal characters
+    """
+    key_hash: str = hash_field_name(artifact["key"])
+    mode: str = artifact.get("mode", DEFAULT_ARTIFACT_MODE)
+    return f"{key_hash}_{mode}"
+
+
+def artifacts_prepare_for_save(fields: dict):
+    artifacts_field = ("execution", "artifacts")
+    artifacts = nested_get(fields, artifacts_field)
+    if artifacts is None:
+        return
+
+    nested_set(
+        fields, artifacts_field, value={get_artifact_id(a): a for a in artifacts}
+    )
+
+
+def artifacts_unprepare_from_saved(fields):
+    artifacts_field = ("execution", "artifacts")
+    artifacts = nested_get(fields, artifacts_field)
+    if artifacts is None:
+        return
+
+    nested_set(
+        fields,
+        artifacts_field,
+        value=sorted(artifacts.values(), key=itemgetter("key")),
+    )
+
+
+class Artifacts:
+    @classmethod
+    def add_or_update_artifacts(
+        cls,
+        company_id: str,
+        identity: Identity,
+        task_id: str,
+        artifacts: Sequence[ApiArtifact],
+        force: bool,
+    ) -> int:
+        task = get_task_for_update(
+            company_id=company_id, task_id=task_id, force=force, identity=identity
+        )
+
+        artifacts = {
+            get_artifact_id(a): Artifact(**a)
+            for a in (api_artifact.to_struct() for api_artifact in artifacts)
+        }
+
+        update_cmds = {
+            f"set__execution__artifacts__{mongoengine_safe(name)}": value
+            for name, value in artifacts.items()
+        }
+        return update_task(task, user_id=identity.user, update_cmds=update_cmds)
+
+    @classmethod
+    def delete_artifacts(
+        cls,
+        company_id: str,
+        identity: Identity,
+        task_id: str,
+        artifact_ids: Sequence[ArtifactId],
+        force: bool,
+    ) -> int:
+        task = get_task_for_update(
+            company_id=company_id, task_id=task_id, force=force, identity=identity
+        )
+
+        artifact_ids = [
+            get_artifact_id(a)
+            for a in (artifact_id.to_struct() for artifact_id in artifact_ids)
+        ]
+        delete_cmds = {
+            f"unset__execution__artifacts__{id_}": 1 for id_ in set(artifact_ids)
+        }
+
+        return update_task(task, user_id=identity.user, update_cmds=delete_cmds)

apiserver/bll/task/hyperparams.py

@@ -0,0 +1,262 @@
+from itertools import chain
+from operator import attrgetter
+from typing import Sequence, Dict
+
+from boltons import iterutils
+
+from apiserver.apierrors import errors
+from apiserver.apimodels.tasks import (
+    HyperParamKey,
+    HyperParamItem,
+    ReplaceHyperparams,
+    Configuration,
+)
+from apiserver.bll.task import TaskBLL
+from apiserver.bll.task.utils import get_task_for_update, update_task
+from apiserver.config_repo import config
+from apiserver.database.model.task.task import ParamsItem, Task, ConfigurationItem
+from apiserver.service_repo.auth import Identity
+from apiserver.utilities.parameter_key_escaper import (
+    ParameterKeyEscaper,
+    mongoengine_safe,
+)
+
+log = config.logger(__file__)
+task_bll = TaskBLL()
+
+
+class HyperParams:
+    _properties_section = "properties"
+
+    @classmethod
+    def get_params(cls, company_id: str, task_ids: Sequence[str]) -> Dict[str, dict]:
+        only = ("id", "hyperparams")
+        tasks = task_bll.assert_exists(
+            company_id=company_id,
+            task_ids=task_ids,
+            only=only,
+            allow_public=True,
+        )
+
+        return {
+            task.id: {"hyperparams": cls._get_params_list(items=task.hyperparams)}
+            for task in tasks
+        }
+
+    @classmethod
+    def _get_params_list(
+        cls, items: Dict[str, Dict[str, ParamsItem]]
+    ) -> Sequence[dict]:
+        ret = list(chain.from_iterable(v.values() for v in items.values()))
+        return [
+            p.to_proper_dict() for p in sorted(ret, key=attrgetter("section", "name"))
+        ]
+
+    @classmethod
+    def _normalize_params(cls, params: Sequence) -> bool:
+        """
+        Lower case properties section and return True if it is the only section
+        """
+        for p in params:
+            if p.section.lower() == cls._properties_section:
+                p.section = cls._properties_section
+
+        return all(p.section == cls._properties_section for p in params)
+
+    @classmethod
+    def delete_params(
+        cls,
+        company_id: str,
+        identity: Identity,
+        task_id: str,
+        hyperparams: Sequence[HyperParamKey],
+        force: bool,
+    ) -> int:
+        properties_only = cls._normalize_params(hyperparams)
+        task = get_task_for_update(
+            company_id=company_id,
+            task_id=task_id,
+            allow_all_statuses=properties_only,
+            force=force,
+            identity=identity,
+        )
+
+        with_param, without_param = iterutils.partition(
+            hyperparams, key=lambda p: bool(p.name)
+        )
+        sections_to_delete = {p.section for p in without_param}
+        delete_cmds = {
+            f"unset__hyperparams__{ParameterKeyEscaper.escape(section)}": 1
+            for section in sections_to_delete
+        }
+
+        for item in with_param:
+            section = ParameterKeyEscaper.escape(item.section)
+            if item.section in sections_to_delete:
+                raise errors.bad_request.FieldsConflict(
+                    "Cannot delete section field if the whole section was scheduled for deletion"
+                )
+            name = ParameterKeyEscaper.escape(item.name)
+            delete_cmds[f"unset__hyperparams__{section}__{name}"] = 1
+
+        return update_task(
+            task,
+            user_id=identity.user,
+            update_cmds=delete_cmds,
+            set_last_update=not properties_only,
+        )
+
+    @classmethod
+    def edit_params(
+        cls,
+        company_id: str,
+        identity: Identity,
+        task_id: str,
+        hyperparams: Sequence[HyperParamItem],
+        replace_hyperparams: str,
+        force: bool,
+    ) -> int:
+        properties_only = cls._normalize_params(hyperparams)
+        task = get_task_for_update(
+            company_id=company_id,
+            task_id=task_id,
+            allow_all_statuses=properties_only,
+            force=force,
+            identity=identity,
+        )
+
+        update_cmds = dict()
+        hyperparams = cls._db_dicts_from_list(hyperparams)
+        if replace_hyperparams == ReplaceHyperparams.all:
+            update_cmds["set__hyperparams"] = hyperparams
+        elif replace_hyperparams == ReplaceHyperparams.section:
+            for section, value in hyperparams.items():
+                update_cmds[f"set__hyperparams__{mongoengine_safe(section)}"] = value
+        else:
+            for section, section_params in hyperparams.items():
+                for name, value in section_params.items():
+                    update_cmds[
+                        f"set__hyperparams__{section}__{mongoengine_safe(name)}"
+                    ] = value
+
+        return update_task(
+            task,
+            user_id=identity.user,
+            update_cmds=update_cmds,
+            set_last_update=not properties_only,
+        )
+
+    @classmethod
+    def _db_dicts_from_list(cls, items: Sequence[HyperParamItem]) -> Dict[str, dict]:
+        sections = iterutils.bucketize(items, key=attrgetter("section"))
+        return {
+            ParameterKeyEscaper.escape(section): {
+                ParameterKeyEscaper.escape(param.name): ParamsItem(**param.to_struct())
+                for param in params
+            }
+            for section, params in sections.items()
+        }
+
+    @classmethod
+    def get_configurations(
+        cls, company_id: str, task_ids: Sequence[str], names: Sequence[str]
+    ) -> Dict[str, dict]:
+        only = ["id"]
+        if names:
+            only.extend(
+                f"configuration.{ParameterKeyEscaper.escape(name)}" for name in names
+            )
+        else:
+            only.append("configuration")
+        tasks = task_bll.assert_exists(
+            company_id=company_id,
+            task_ids=task_ids,
+            only=only,
+            allow_public=True,
+        )
+
+        return {
+            task.id: {
+                "configuration": [
+                    c.to_proper_dict()
+                    for c in sorted(task.configuration.values(), key=attrgetter("name"))
+                ]
+            }
+            for task in tasks
+        }
+
+    @classmethod
+    def get_configuration_names(
+        cls, company_id: str, task_ids: Sequence[str], skip_empty: bool
+    ) -> Dict[str, list]:
+        skip_empty_condition = {"$match": {"items.v.value": {"$nin": [None, ""]}}}
+        pipeline = [
+            {
+                "$match": {
+                    "company": {"$in": [None, "", company_id]},
+                    "_id": {"$in": task_ids},
+                }
+            },
+            {"$project": {"items": {"$objectToArray": "$configuration"}}},
+            {"$unwind": "$items"},
+            *([skip_empty_condition] if skip_empty else []),
+            {"$group": {"_id": "$_id", "names": {"$addToSet": "$items.k"}}},
+        ]
+
+        tasks = Task.aggregate(pipeline)
+
+        return {
+            task["_id"]: {
+                "names": sorted(
+                    ParameterKeyEscaper.unescape(name) for name in task["names"]
+                )
+            }
+            for task in tasks
+        }
+
+    @classmethod
+    def edit_configuration(
+        cls,
+        company_id: str,
+        identity: Identity,
+        task_id: str,
+        configuration: Sequence[Configuration],
+        replace_configuration: bool,
+        force: bool,
+    ) -> int:
+        task = get_task_for_update(
+            company_id=company_id, task_id=task_id, force=force, identity=identity
+        )
+
+        update_cmds = dict()
+        configuration = {
+            ParameterKeyEscaper.escape(c.name): ConfigurationItem(**c.to_struct())
+            for c in configuration
+        }
+        if replace_configuration:
+            update_cmds["set__configuration"] = configuration
+        else:
+            for name, value in configuration.items():
+                update_cmds[f"set__configuration__{mongoengine_safe(name)}"] = value
+
+        return update_task(task, user_id=identity.user, update_cmds=update_cmds)
+
+    @classmethod
+    def delete_configuration(
+        cls,
+        company_id: str,
+        identity: Identity,
+        task_id: str,
+        configuration: Sequence[str],
+        force: bool,
+    ) -> int:
+        task = get_task_for_update(
+            company_id=company_id, task_id=task_id, force=force, identity=identity
+        )
+
+        delete_cmds = {
+            f"unset__configuration__{ParameterKeyEscaper.escape(name)}": 1
+            for name in set(configuration)
+        }
+
+        return update_task(task, user_id=identity.user, update_cmds=delete_cmds)

apiserver/bll/task/non_responsive_tasks_watchdog.py

@@ -0,0 +1,99 @@
+from datetime import timedelta, datetime
+from time import sleep
+
+from apiserver.bll.util import update_project_time
+from apiserver.config_repo import config
+from apiserver.database.model.task.task import TaskStatus, Task
+from apiserver.utilities.threads_manager import ThreadsManager
+
+log = config.logger(__file__)
+
+
+class NonResponsiveTasksWatchdog:
+    threads = ThreadsManager()
+
+    class _Settings:
+        """
+        Retrieves watchdog settings from the config file
+        The properties are not cached so that the updates in
+        the config file are reflected
+        """
+
+        _prefix = "services.tasks.non_responsive_tasks_watchdog"
+
+        @property
+        def enabled(self):
+            return config.get(f"{self._prefix}.enabled", True)
+
+        @property
+        def watch_interval_sec(self):
+            return config.get(f"{self._prefix}.watch_interval_sec", 900)
+
+        @property
+        def threshold_sec(self):
+            return config.get(f"{self._prefix}.threshold_sec", 7200)
+
+    settings = _Settings()
+
+    @classmethod
+    @threads.register("non_responsive_tasks_watchdog", daemon=True)
+    def start(cls):
+        sleep(cls.settings.watch_interval_sec)
+        while True:
+            watch_interval = cls.settings.watch_interval_sec
+            if cls.settings.enabled:
+                try:
+                    stopped = cls.cleanup_tasks(
+                        threshold_sec=cls.settings.threshold_sec
+                    )
+                    log.info(f"{stopped} non-responsive tasks stopped")
+                except Exception as ex:
+                    log.exception(f"Failed stopping tasks: {str(ex)}")
+            sleep(watch_interval)
+
+    @classmethod
+    def cleanup_tasks(cls, threshold_sec):
+        relevant_status = (TaskStatus.in_progress,)
+        threshold = timedelta(seconds=threshold_sec)
+        ref_time = datetime.utcnow() - threshold
+        log.info(
+            f"Starting cleanup cycle for running tasks last updated before {ref_time}"
+        )
+
+        tasks = list(
+            Task.objects(status__in=relevant_status, last_update__lt=ref_time).only(
+                "id", "name", "status", "project", "last_update"
+            )
+        )
+        log.info(f"{len(tasks)} non-responsive tasks found")
+        if not tasks:
+            return 0
+
+        err_count = 0
+        project_ids = set()
+        now = datetime.utcnow()
+        for task in tasks:
+            log.info(
+                f"Stopping {task.id} ({task.name}), last updated at {task.last_update}"
+            )
+            # noinspection PyBroadException
+            try:
+                updated = Task.objects(id=task.id, status=task.status).update(
+                    status=TaskStatus.stopped,
+                    status_reason="Forced stop (non-responsive)",
+                    status_message="Forced stop (non-responsive)",
+                    status_changed=now,
+                    last_update=now,
+                    last_change=now,
+                    last_changed_by="__apiserver__",
+                )
+                if updated:
+                    project_ids.add(task.project)
+                else:
+                    err_count += 1
+            except Exception as ex:
+                log.error("Failed setting status: %s", str(ex))
+
+        update_project_time(list(project_ids))
+
+        return len(tasks) - err_count

apiserver/bll/task/param_utils.py

@@ -0,0 +1,216 @@
+import itertools
+from typing import Sequence, Tuple, Optional
+
+
+from apiserver.apierrors import errors
+from apiserver.database.model.task.task import Task
+from apiserver.utilities.dicts import nested_get, nested_delete, nested_set
+from apiserver.utilities.parameter_key_escaper import ParameterKeyEscaper
+
+
+hyperparams_default_section = "Args"
+hyperparams_legacy_type = "legacy"
+tf_define_section = "TF_DEFINE"
+
+
+def split_param_name(full_name: str, default_section: str) -> Tuple[Optional[str], str]:
+    """
+    Return parameter section and name. The section is either TF_DEFINE or the default one
+    """
+    if default_section is None:
+        return None, full_name
+
+    section, _, name = full_name.partition("/")
+    if section != tf_define_section:
+        return default_section, full_name
+
+    if not name:
+        raise errors.bad_request.ValidationError("Parameter name cannot be empty")
+    return section, name
+
+
+def _get_full_param_name(param: dict) -> str:
+    section = param.get("section")
+    if section != tf_define_section:
+        return param["name"]
+
+    return "/".join((section, param["name"]))
+
+
+def _remove_legacy_params(data: dict, with_sections: bool = False) -> int:
+    """
+    Remove the legacy params from the data dict and return the number of removed params
+    If the path not found then return 0
+    """
+    removed = 0
+    if not data:
+        return removed
+
+    if with_sections:
+        for section, section_data in list(data.items()):
+            removed += _remove_legacy_params(section_data)
+            if not section_data:
+                """If section is empty after removing legacy params then delete it"""
+                del data[section]
+    else:
+        for key, param in list(data.items()):
+            if param.get("type") == hyperparams_legacy_type:
+                removed += 1
+                del data[key]
+
+    return removed
+
+
+def _get_legacy_params(data: dict, with_sections: bool = False) -> Sequence[dict]:
+    """
+    Remove the legacy params from the data dict and return the number of removed params
+    If the path not found then return 0
+    """
+    if not data:
+        return []
+
+    if with_sections:
+        return list(
+            itertools.chain.from_iterable(
+                _get_legacy_params(section_data) for section_data in data.values()
+            )
+        )
+
+    return [
+        param for param in data.values() if param.get("type") == hyperparams_legacy_type
+    ]
+
+
+def params_prepare_for_save(fields: dict, previous_task: Task = None):
+    """
+    If legacy hyper params or configuration is passed then replace the corresponding section in the new structure
+    Escape all the section and param names for hyper params and configuration to make it mongo sage
+    """
+    for old_params_field, new_params_field, default_section in (
+        (("execution", "parameters"), "hyperparams", hyperparams_default_section),
+        (("execution", "model_desc"), "configuration", None),
+    ):
+        legacy_params = nested_get(fields, old_params_field)
+        if legacy_params is None:
+            continue
+
+        if (
+            not fields.get(new_params_field)
+            and previous_task
+            and previous_task[new_params_field]
+        ):
+            previous_data = previous_task.to_proper_dict().get(new_params_field)
+            removed = _remove_legacy_params(
+                previous_data, with_sections=default_section is not None
+            )
+            if not legacy_params and not removed:
+                # if we only need to delete legacy fields from the db
+                # but they are not there then there is no point to proceed
+                continue
+
+            fields_update = {new_params_field: previous_data}
+            params_unprepare_from_saved(fields_update)
+            fields.update(fields_update)
+
+        for full_name, value in legacy_params.items():
+            section, name = split_param_name(full_name, default_section)
+            new_path = list(filter(None, (new_params_field, section, name)))
+            new_param = dict(name=name, type=hyperparams_legacy_type, value=str(value))
+            if section is not None:
+                new_param["section"] = section
+            nested_set(fields, new_path, new_param)
+        nested_delete(fields, old_params_field)
+
+    def ensure_non_empty(k: str, desc: str) -> str:
+        if not k:
+            raise errors.bad_request.ValidationError(
+                f"Empty {desc} name is not allowed"
+            )
+        return k
+
+    params = fields.get("hyperparams")
+    if params:
+        escaped_params = {
+            ParameterKeyEscaper.escape(ensure_non_empty(key, "section")): {
+                ParameterKeyEscaper.escape(ensure_non_empty(k, "parameter")): v
+                for k, v in value.items()
+            }
+            for key, value in params.items()
+        }
+        fields["hyperparams"] = escaped_params
+
+    params = fields.get("configuration")
+    if params:
+        escaped_params = {
+            ParameterKeyEscaper.escape(ensure_non_empty(key, "configuration")): value
+            for key, value in params.items()
+        }
+        fields["configuration"] = escaped_params
+
+
+def params_unprepare_from_saved(fields, copy_to_legacy=False):
+    """
+    Unescape all section and param names for hyper params and configuration
+    If copy_to_legacy is set then copy hyperparams and configuration data to the legacy location for the old clients
+    """
+    for param_field in ("hyperparams", "configuration"):
+        params = fields.get(param_field)
+        if params:
+            unescaped_params = {
+                ParameterKeyEscaper.unescape(key): {
+                    ParameterKeyEscaper.unescape(k): v for k, v in value.items()
+                }
+                if isinstance(value, dict)
+                else value
+                for key, value in params.items()
+            }
+            fields[param_field] = unescaped_params
+
+    if copy_to_legacy:
+        for new_params_field, old_params_field, use_sections in (
+            ("hyperparams", ("execution", "parameters"), True),
+            ("configuration", ("execution", "model_desc"), False),
+        ):
+            legacy_params = _get_legacy_params(
+                fields.get(new_params_field), with_sections=use_sections
+            )
+            if legacy_params:
+                nested_set(
+                    fields,
+                    old_params_field,
+                    {_get_full_param_name(p): p["value"] for p in legacy_params},
+                )
+
+
+def _process_path(path: str):
+    """
+    Frontend does a partial escaping on the path so the all '.' in section and key names are escaped
+    Need to unescape and apply a full mongo escaping
+    """
+    parts = path.split(".")
+    if len(parts) < 2 or len(parts) > 4:
+        raise errors.bad_request.ValidationError("invalid task field", path=path)
+    return ".".join(
+        ParameterKeyEscaper.escape(ParameterKeyEscaper.unescape(p)) for p in parts
+    )
+
+
+def escape_paths(paths: Sequence[str]) -> Sequence[str]:
+    for old_prefix, new_prefix in (
+        ("execution.parameters", f"hyperparams.{hyperparams_default_section}"),
+        ("execution.model_desc", "configuration"),
+        ("execution.docker_cmd", "container"),
+    ):
+        path: str
+        paths = [path.replace(old_prefix, new_prefix) for path in paths]
+
+    for prefix in (
+        "hyperparams.",
+        "-hyperparams.",
+        "configuration.",
+        "-configuration.",
+    ):
+        paths = [
+            _process_path(path) if path.startswith(prefix) else path for path in paths
+        ]
+    return paths

apiserver/bll/task/task_bll.py

@@ -0,0 +1,509 @@
+from datetime import datetime
+from typing import Collection, Sequence, Tuple, Optional, Dict
+
+import six
+from mongoengine import Q
+from redis import StrictRedis
+from six import string_types
+
+import apiserver.database.utils as dbutils
+from apiserver.apierrors import errors, APIError
+from apiserver.apimodels.tasks import TaskInputModel
+from apiserver.bll.queue import QueueBLL
+from apiserver.bll.organization import OrgBLL, Tags
+from apiserver.bll.project import ProjectBLL
+from apiserver.bll.util import update_project_time
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.model import Model
+from apiserver.database.model.project import Project
+from apiserver.database.model.task.metrics import EventStats, MetricEventStats
+from apiserver.database.model.task.output import Output
+from apiserver.database.model.task.task import (
+    Task,
+    TaskStatus,
+    TaskSystemTags,
+    ArtifactModes,
+    ModelItem,
+    Models,
+    DEFAULT_ARTIFACT_MODE,
+    TaskModelNames,
+    TaskModelTypes,
+)
+from apiserver.database.model import EntityVisibility
+from apiserver.database.model.queue import Queue
+from apiserver.database.utils import (
+    get_company_or_none_constraint,
+    id as create_id,
+)
+from apiserver.es_factory import es_factory
+from apiserver.redis_manager import redman
+from apiserver.services.utils import validate_tags, escape_dict_field, escape_dict
+from .artifacts import artifacts_prepare_for_save
+from .param_utils import params_prepare_for_save
+from .utils import (
+    ChangeStatusRequest,
+    deleted_prefix,
+    get_last_metric_updates,
+)
+
+log = config.logger(__file__)
+org_bll = OrgBLL()
+queue_bll = QueueBLL()
+project_bll = ProjectBLL()
+
+
+class TaskBLL:
+    def __init__(self, events_es=None, redis=None):
+        self.events_es = events_es or es_factory.connect("events")
+        self.redis: StrictRedis = redis or redman.connection("apiserver")
+
+    @staticmethod
+    def get_by_id(
+        company_id,
+        task_id,
+        required_status=None,
+        only_fields=None,
+        allow_public=False,
+    ):
+        if only_fields:
+            if isinstance(only_fields, string_types):
+                only_fields = [only_fields]
+            else:
+                only_fields = list(only_fields)
+            only_fields = only_fields + ["status"]
+
+        tasks = Task.get_many(
+            company=company_id,
+            query=Q(id=task_id),
+            allow_public=allow_public,
+            override_projection=only_fields,
+            return_dicts=False,
+        )
+        task = None if not tasks else tasks[0]
+
+        if not task:
+            raise errors.bad_request.InvalidTaskId(id=task_id)
+
+        if required_status and not task.status == required_status:
+            raise errors.bad_request.InvalidTaskStatus(expected=required_status)
+
+        return task
+
+    @staticmethod
+    def assert_exists(
+        company_id, task_ids, only=None, allow_public=False, return_tasks=True
+    ) -> Optional[Sequence[Task]]:
+        task_ids = [task_ids] if isinstance(task_ids, six.string_types) else task_ids
+        with translate_errors_context():
+            ids = set(task_ids)
+            q = Task.get_many(
+                company=company_id,
+                query=Q(id__in=ids),
+                allow_public=allow_public,
+                return_dicts=False,
+            )
+            if only:
+                # Make sure to reset fields filters (some fields are excluded by default) since this
+                # is an internal call and specific fields were requested.
+                q = q.all_fields().only(*only)
+
+            if q.count() != len(ids):
+                raise errors.bad_request.InvalidTaskId(ids=task_ids)
+
+            if return_tasks:
+                return list(q)
+
+    @staticmethod
+    def create(company: str, user: str, fields: dict):
+        now = datetime.utcnow()
+        return Task(
+            id=create_id(),
+            user=user,
+            company=company,
+            created=now,
+            last_update=now,
+            last_change=now,
+            last_changed_by=user,
+            **fields,
+        )
+
+    @staticmethod
+    def validate_input_models(task, allow_only_public=False):
+        if not task.models.input:
+            return
+
+        company = None if allow_only_public else task.company
+        model_ids = set(m.model for m in task.models.input)
+        models = Model.objects(
+            Q(id__in=model_ids) & get_company_or_none_constraint(company)
+        ).only("id")
+        missing = model_ids - {m.id for m in models}
+        if missing:
+            raise errors.bad_request.InvalidModelId(models=missing)
+
+        return
+
+    @classmethod
+    def clone_task(
+        cls,
+        company_id: str,
+        user_id: str,
+        task_id: str,
+        name: Optional[str] = None,
+        comment: Optional[str] = None,
+        parent: Optional[str] = None,
+        project: Optional[str] = None,
+        tags: Optional[Sequence[str]] = None,
+        system_tags: Optional[Sequence[str]] = None,
+        hyperparams: Optional[dict] = None,
+        configuration: Optional[dict] = None,
+        container: Optional[dict] = None,
+        execution_overrides: Optional[dict] = None,
+        input_models: Optional[Sequence[TaskInputModel]] = None,
+        validate_references: bool = False,
+        new_project_name: str = None,
+    ) -> Tuple[Task, dict]:
+        validate_tags(tags, system_tags)
+        params_dict = {
+            field: value
+            for field, value in (
+                ("hyperparams", hyperparams),
+                ("configuration", configuration),
+            )
+            if value is not None
+        }
+
+        task = cls.get_by_id(company_id=company_id, task_id=task_id, allow_public=True)
+
+        now = datetime.utcnow()
+        if input_models:
+            input_models = [
+                ModelItem(model=m.model, name=m.name, updated=now) for m in input_models
+            ]
+
+        execution_dict = task.execution.to_proper_dict() if task.execution else {}
+        if execution_overrides:
+            execution_model = execution_overrides.pop("model", None)
+            if not input_models and execution_model:
+                input_models = [
+                    ModelItem(
+                        model=execution_model,
+                        name=TaskModelNames[TaskModelTypes.input],
+                        updated=now,
+                    )
+                ]
+
+            docker_cmd = execution_overrides.pop("docker_cmd", None)
+            if not container and docker_cmd:
+                image, _, arguments = docker_cmd.partition(" ")
+                container = {"image": image, "arguments": arguments}
+
+            artifacts_prepare_for_save({"execution": execution_overrides})
+
+            params_dict["execution"] = {}
+            for legacy_param in ("parameters", "configuration"):
+                legacy_value = execution_overrides.pop(legacy_param, None)
+                if legacy_value is not None:
+                    params_dict["execution"] = legacy_value
+
+            escape_dict_field(execution_overrides, "model_labels")
+
+            execution_dict.update(execution_overrides)
+
+        params_prepare_for_save(params_dict, previous_task=task)
+
+        artifacts = execution_dict.get("artifacts")
+        if artifacts:
+            execution_dict["artifacts"] = {
+                k: a
+                for k, a in artifacts.items()
+                if a.get("mode", DEFAULT_ARTIFACT_MODE) != ArtifactModes.output
+            }
+        execution_dict.pop("queue", None)
+
+        new_project_data = None
+        if not project and new_project_name:
+            # Use a project with the provided name, or create a new project
+            project = ProjectBLL.find_or_create(
+                project_name=new_project_name,
+                user=user_id,
+                company=company_id,
+                description="",
+            )
+            new_project_data = {"id": project, "name": new_project_name}
+
+        def clean_system_tags(input_tags: Sequence[str]) -> Sequence[str]:
+            if not input_tags:
+                return input_tags
+
+            return [
+                tag
+                for tag in input_tags
+                if tag
+                not in [TaskSystemTags.development, EntityVisibility.archived.value]
+            ]
+
+        def ensure_int_labels(execution: dict) -> dict:
+            if not execution:
+                return execution
+
+            model_labels = execution.get("model_labels")
+            if model_labels:
+                execution["model_labels"] = {k: int(v) for k, v in model_labels.items()}
+
+            return execution
+
+        parent_task = (
+            task.parent
+            if task.parent and not task.parent.startswith(deleted_prefix)
+            else task.id
+        )
+        new_task = Task(
+            id=create_id(),
+            user=user_id,
+            company=company_id,
+            created=now,
+            last_update=now,
+            last_change=now,
+            last_changed_by=user_id,
+            name=name or task.name,
+            comment=comment or task.comment,
+            parent=parent or parent_task,
+            project=project or task.project,
+            tags=tags or task.tags,
+            system_tags=system_tags or clean_system_tags(task.system_tags),
+            type=task.type,
+            script=task.script,
+            output=Output(destination=task.output.destination) if task.output else None,
+            models=Models(input=input_models or task.models.input),
+            container=escape_dict(container) or task.container,
+            execution=ensure_int_labels(execution_dict),
+            configuration=params_dict.get("configuration") or task.configuration,
+            hyperparams=params_dict.get("hyperparams") or task.hyperparams,
+        )
+        cls.validate(
+            new_task,
+            validate_models=validate_references or input_models,
+            validate_parent=validate_references or parent,
+            validate_project=validate_references or project,
+        )
+        new_task.save()
+
+        if task.project == new_task.project:
+            updated_tags = tags
+            updated_system_tags = system_tags
+        else:
+            updated_tags = new_task.tags
+            updated_system_tags = new_task.system_tags
+        org_bll.update_tags(
+            company_id,
+            Tags.Task,
+            projects=[new_task.project],
+            tags=updated_tags,
+            system_tags=updated_system_tags,
+        )
+        update_project_time(new_task.project)
+
+        return new_task, new_project_data
+
+    @classmethod
+    def validate(
+        cls,
+        task: Task,
+        validate_models=True,
+        validate_parent=True,
+        validate_project=True,
+    ):
+        """
+        Validate task properties according to the flag
+        Task project is always checked for being writable
+        in order to disable the modification of public projects
+        """
+        if (
+            validate_parent
+            and task.parent
+            and not task.parent.startswith(deleted_prefix)
+            and not Task.get(
+                company=task.company, id=task.parent, _only=("id",), include_public=True
+            )
+        ):
+            raise errors.bad_request.InvalidTaskId("invalid parent", parent=task.parent)
+
+        if task.project:
+            project = Project.get_for_writing(company=task.company, id=task.project)
+            if validate_project and not project:
+                raise errors.bad_request.InvalidProjectId(id=task.project)
+
+        if validate_models:
+            cls.validate_input_models(task)
+
+    @staticmethod
+    def set_last_update(
+        task_ids: Collection[str],
+        company_id: str,
+        user_id: str,
+        last_update: datetime,
+        **extra_updates,
+    ):
+        tasks = Task.objects(id__in=task_ids, company=company_id).only(
+            "status", "started"
+        )
+        count = 0
+        for task in tasks:
+            updates = extra_updates
+            if task.status == TaskStatus.in_progress and task.started:
+                updates = {
+                    "active_duration": (
+                        datetime.utcnow() - task.started
+                    ).total_seconds(),
+                    **extra_updates,
+                }
+            count += Task.objects(id=task.id, company=company_id).update(
+                upsert=False,
+                last_update=last_update,
+                last_change=last_update,
+                last_changed_by=user_id,
+                **updates,
+            )
+        return count
+
+    @staticmethod
+    def update_statistics(
+        task_id: str,
+        company_id: str,
+        user_id: str,
+        last_update: datetime = None,
+        last_iteration: int = None,
+        last_iteration_max: int = None,
+        last_scalar_events: Dict[str, Dict[str, dict]] = None,
+        last_events: Dict[str, Dict[str, dict]] = None,
+        **extra_updates,
+    ):
+        """
+        Update task statistics
+        :param task_id: Task's ID.
+        :param company_id: Task's company ID.
+        :param last_update: Last update time. If not provided, defaults to datetime.utcnow().
+        :param last_iteration: Last reported iteration. Use this to set a value regardless of current
+            task's last iteration value.
+        :param last_iteration_max: Last reported iteration. Use this to conditionally set a value only
+            if the current task's last iteration value is smaller than the provided value.
+        :param last_scalar_values: Last reported metrics summary for scalar events (value, metric, variant).
+        :param last_events: Last reported metrics summary (value, metric, event type).
+        :param extra_updates: Extra task updates to include in this update call.
+        :return:
+        """
+        last_update = last_update or datetime.utcnow()
+
+        if last_iteration is not None:
+            extra_updates.update(last_iteration=last_iteration)
+        elif last_iteration_max is not None:
+            extra_updates.update(max__last_iteration=last_iteration_max)
+
+        raw_updates = {}
+        if last_scalar_events is not None:
+            get_last_metric_updates(
+                task_id=task_id,
+                last_scalar_events=last_scalar_events,
+                raw_updates=raw_updates,
+                extra_updates=extra_updates,
+            )
+
+        if last_events is not None:
+
+            def events_per_type(metric_data_: Dict[str, dict]) -> Dict[str, EventStats]:
+                return {
+                    event_type: EventStats(last_update=event["timestamp"])
+                    for event_type, event in metric_data_.items()
+                }
+
+            metric_stats = {
+                dbutils.hash_field_name(metric_key): MetricEventStats(
+                    metric=metric_key, event_stats_by_type=events_per_type(metric_data)
+                )
+                for metric_key, metric_data in last_events.items()
+            }
+            extra_updates["metric_stats"] = metric_stats
+
+        ret = TaskBLL.set_last_update(
+            task_ids=[task_id],
+            company_id=company_id,
+            user_id=user_id,
+            last_update=last_update,
+            **extra_updates,
+        )
+        if ret and raw_updates:
+            Task.objects(id=task_id).update_one(__raw__=[{"$set": raw_updates}])
+
+        return ret
+
+    @staticmethod
+    def remove_task_from_all_queues(company_id: str, task_id: str) -> int:
+        return Queue.objects(company=company_id, entries__task=task_id).update(
+            pull__entries__task=task_id, last_update=datetime.utcnow()
+        )
+
+    @classmethod
+    def dequeue_and_change_status(
+        cls,
+        task: Task,
+        company_id: str,
+        user_id: str,
+        status_message: str,
+        status_reason: str,
+        remove_from_all_queues=False,
+        new_status=None,
+    ):
+        try:
+            cls.dequeue(task, company_id, silent_fail=True)
+        except APIError:
+            # dequeue may fail if the queue was deleted
+            pass
+
+        if remove_from_all_queues:
+            cls.remove_task_from_all_queues(company_id=company_id, task_id=task.id)
+
+        if task.status not in [TaskStatus.queued, TaskStatus.in_progress]:
+            return {"updated": 0}
+
+        return ChangeStatusRequest(
+            task=task,
+            new_status=new_status or task.enqueue_status or TaskStatus.created,
+            status_reason=status_reason,
+            status_message=status_message,
+            user_id=user_id,
+            force=True,
+        ).execute(enqueue_status=None)
+
+    @classmethod
+    def dequeue(cls, task: Task, company_id: str, silent_fail=False):
+        """
+        Dequeue the task from the queue
+        :param task: task to dequeue
+        :param company_id: task's company ID.
+        :param silent_fail: do not throw exceptions. APIError is still thrown
+        :raise errors.bad_request.InvalidTaskId: if the task's status is not queued
+        :raise errors.bad_request.MissingRequiredFields: if the task is not queued
+        :raise APIError or errors.server_error.TransactionError: if internal call to queues.remove_task fails
+        :return: the result of queues.remove_task call. None in case of silent failure
+        """
+        if task.status not in (TaskStatus.queued,):
+            if silent_fail:
+                return
+            raise errors.bad_request.InvalidTaskId(
+                status=task.status, expected=TaskStatus.queued
+            )
+
+        if not task.execution or not task.execution.queue:
+            if silent_fail:
+                return
+            raise errors.bad_request.MissingRequiredFields(
+                "task has no queue value", field="execution.queue"
+            )
+
+        return {
+            "removed": queue_bll.remove_task(
+                company_id=company_id, queue_id=task.execution.queue, task_id=task.id
+            )
+        }

apiserver/bll/task/task_cleanup.py

@@ -0,0 +1,366 @@
+from datetime import datetime
+from itertools import chain
+from operator import attrgetter
+from typing import Sequence, Set, Tuple, Union
+
+import attr
+from boltons.iterutils import partition, bucketize, first, chunked_iter
+from furl import furl
+from mongoengine import NotUniqueError
+from pymongo.errors import DuplicateKeyError
+
+from apiserver.apierrors import errors
+from apiserver.bll.event import EventBLL
+from apiserver.bll.event.event_bll import PlotFields
+from apiserver.bll.task.utils import deleted_prefix
+from apiserver.config_repo import config
+from apiserver.database.model.model import Model
+from apiserver.database.model.task.task import Task, TaskStatus, ArtifactModes
+from apiserver.database.model.url_to_delete import (
+    StorageType,
+    UrlToDelete,
+    FileType,
+    DeletionStatus,
+)
+from apiserver.database.utils import id as db_id
+
+log = config.logger(__file__)
+event_bll = EventBLL()
+
+
+@attr.s(auto_attribs=True)
+class TaskUrls:
+    model_urls: Sequence[str]
+    event_urls: Sequence[str]
+    artifact_urls: Sequence[str]
+
+    def __add__(self, other: "TaskUrls"):
+        if not other:
+            return self
+
+        return TaskUrls(
+            model_urls=list(set(self.model_urls) | set(other.model_urls)),
+            event_urls=list(set(self.event_urls) | set(other.event_urls)),
+            artifact_urls=list(set(self.artifact_urls) | set(other.artifact_urls)),
+        )
+
+
+@attr.s(auto_attribs=True)
+class CleanupResult:
+    """
+    Counts of objects modified in task cleanup operation
+    """
+
+    updated_children: int
+    updated_models: int
+    deleted_models: int
+    urls: TaskUrls = None
+
+    def __add__(self, other: "CleanupResult"):
+        if not other:
+            return self
+
+        return CleanupResult(
+            updated_children=self.updated_children + other.updated_children,
+            updated_models=self.updated_models + other.updated_models,
+            deleted_models=self.deleted_models + other.deleted_models,
+            urls=self.urls + other.urls if self.urls else other.urls,
+        )
+
+
+def collect_plot_image_urls(
+    company: str, task_or_model: Union[str, Sequence[str]]
+) -> Set[str]:
+    urls = set()
+    task_ids = task_or_model if isinstance(task_or_model, list) else [task_or_model]
+    for tasks in chunked_iter(task_ids, 100):
+        next_scroll_id = None
+        while True:
+            events, next_scroll_id = event_bll.get_plot_image_urls(
+                company_id=company, task_ids=tasks, scroll_id=next_scroll_id
+            )
+            if not events:
+                break
+            for event in events:
+                event_urls = event.get(PlotFields.source_urls)
+                if event_urls:
+                    urls.update(set(event_urls))
+
+    return urls
+
+
+def collect_debug_image_urls(
+    company: str, task_or_model: Union[str, Sequence[str]]
+) -> Set[str]:
+    """
+    Return the set of unique image urls
+    Uses DebugImagesIterator to make sure that we do not retrieve recycled urls
+    """
+    urls = set()
+    task_ids = task_or_model if isinstance(task_or_model, list) else [task_or_model]
+    for tasks in chunked_iter(task_ids, 100):
+        after_key = None
+        while True:
+            res, after_key = event_bll.get_debug_image_urls(
+                company_id=company,
+                task_ids=tasks,
+                after_key=after_key,
+            )
+            urls.update(res)
+            if not after_key:
+                break
+
+    return urls
+
+
+supported_storage_types = {
+    "s3://": StorageType.s3,
+    "azure://": StorageType.azure,
+    "gs://": StorageType.gs,
+}
+
+supported_storage_types.update(
+    {
+        p: StorageType.fileserver
+        for p in config.get(
+            "services.async_urls_delete.fileserver.url_prefixes",
+            ["https://", "http://"],
+        )
+    }
+)
+
+
+def _schedule_for_delete(
+    company: str,
+    user: str,
+    task_id: str,
+    urls: Set[str],
+    can_delete_folders: bool,
+) -> Set[str]:
+    urls_per_storage = bucketize(
+        urls,
+        key=lambda u: first(
+            type_
+            for prefix, type_ in supported_storage_types.items()
+            if u.startswith(prefix)
+        ),
+    )
+    urls_per_storage.pop(None, None)
+
+    processed_urls = set()
+    for storage_type, storage_urls in urls_per_storage.items():
+        delete_folders = (storage_type == StorageType.fileserver) and can_delete_folders
+        scheduled_to_delete = set()
+        for url in storage_urls:
+            folder = None
+            if delete_folders:
+                try:
+                    parsed = furl(url)
+                    if parsed.path and len(parsed.path.segments) > 1:
+                        folder = parsed.remove(
+                            args=True, fragment=True, path=parsed.path.segments[-1]
+                        ).url.rstrip("/")
+                except Exception as ex:
+                    pass
+
+            to_delete = folder or url
+            if to_delete in scheduled_to_delete:
+                processed_urls.add(url)
+                continue
+
+            try:
+                UrlToDelete(
+                    id=db_id(),
+                    company=company,
+                    user=user,
+                    url=to_delete,
+                    task=task_id,
+                    created=datetime.utcnow(),
+                    storage_type=storage_type,
+                    type=FileType.folder if folder else FileType.file,
+                ).save()
+            except (DuplicateKeyError, NotUniqueError):
+                existing = UrlToDelete.objects(company=company, url=to_delete).first()
+                if existing:
+                    existing.update(
+                        user=user,
+                        task=task_id,
+                        created=datetime.utcnow(),
+                        retry_count=0,
+                        unset__last_failure_time=1,
+                        unset__last_failure_reason=1,
+                        status=DeletionStatus.created,
+                    )
+            processed_urls.add(url)
+            scheduled_to_delete.add(to_delete)
+
+    return processed_urls
+
+
+def cleanup_task(
+    company: str,
+    user: str,
+    task: Task,
+    force: bool = False,
+    update_children=True,
+    return_file_urls=False,
+    delete_output_models=True,
+    delete_external_artifacts=True,
+) -> CleanupResult:
+    """
+    Validate task deletion and delete/modify all its output.
+    :param task: task object
+    :param force: whether to delete task with published outputs
+    :return: count of delete and modified items
+    """
+    published_models, draft_models, in_use_model_ids = verify_task_children_and_ouptuts(
+        task, force
+    )
+    delete_external_artifacts = delete_external_artifacts and config.get(
+        "services.async_urls_delete.enabled", True
+    )
+    event_urls, artifact_urls, model_urls = set(), set(), set()
+    if return_file_urls or delete_external_artifacts:
+        event_urls = collect_debug_image_urls(task.company, task.id)
+        event_urls.update(collect_plot_image_urls(task.company, task.id))
+        if task.execution and task.execution.artifacts:
+            artifact_urls = {
+                a.uri
+                for a in task.execution.artifacts.values()
+                if a.mode == ArtifactModes.output and a.uri
+            }
+        model_urls = {
+            m.uri for m in draft_models if m.uri and m.id not in in_use_model_ids
+        }
+
+    deleted_task_id = f"{deleted_prefix}{task.id}"
+    updated_children = 0
+    now = datetime.utcnow()
+    if update_children:
+        updated_children = Task.objects(parent=task.id).update(
+            parent=deleted_task_id,
+            last_change=now,
+            last_changed_by=user,
+        )
+
+    deleted_models = 0
+    updated_models = 0
+    for models, allow_delete in ((draft_models, True), (published_models, False)):
+        if not models:
+            continue
+        if delete_output_models and allow_delete:
+            model_ids = list({m.id for m in models if m.id not in in_use_model_ids})
+            if model_ids:
+                if return_file_urls or delete_external_artifacts:
+                    event_urls.update(collect_debug_image_urls(task.company, model_ids))
+                    event_urls.update(collect_plot_image_urls(task.company, model_ids))
+
+                event_bll.delete_multi_task_events(
+                    task.company,
+                    model_ids,
+                    model=True,
+                )
+                deleted_models += Model.objects(id__in=model_ids).delete()
+
+            if in_use_model_ids:
+                Model.objects(id__in=list(in_use_model_ids)).update(
+                    unset__task=1,
+                    set__last_change=now,
+                    set__last_changed_by=user,
+                )
+            continue
+
+        if update_children:
+            updated_models += Model.objects(id__in=[m.id for m in models]).update(
+                task=deleted_task_id,
+                last_change=now,
+                last_changed_by=user,
+            )
+        else:
+            Model.objects(id__in=[m.id for m in models]).update(
+                unset__task=1,
+                set__last_change=now,
+                set__last_changed_by=user,
+            )
+
+    event_bll.delete_task_events(task.company, task.id, allow_locked=force)
+
+    if delete_external_artifacts:
+        scheduled = _schedule_for_delete(
+            task_id=task.id,
+            company=company,
+            user=user,
+            urls=event_urls | model_urls | artifact_urls,
+            can_delete_folders=not in_use_model_ids and not published_models,
+        )
+        for urls in (event_urls, model_urls, artifact_urls):
+            urls.difference_update(scheduled)
+
+    return CleanupResult(
+        deleted_models=deleted_models,
+        updated_children=updated_children,
+        updated_models=updated_models,
+        urls=TaskUrls(
+            event_urls=list(event_urls),
+            artifact_urls=list(artifact_urls),
+            model_urls=list(model_urls),
+        )
+        if return_file_urls
+        else None,
+    )
+
+
+def verify_task_children_and_ouptuts(
+    task, force: bool
+) -> Tuple[Sequence[Model], Sequence[Model], Set[str]]:
+    if not force:
+        published_children_count = Task.objects(
+            parent=task.id, status=TaskStatus.published
+        ).count()
+        if published_children_count:
+            raise errors.bad_request.TaskCannotBeDeleted(
+                "has children, use force=True",
+                task=task.id,
+                children=published_children_count,
+            )
+
+    model_fields = ["id", "ready", "uri"]
+    published_models, draft_models = partition(
+        Model.objects(task=task.id).only(*model_fields),
+        key=attrgetter("ready"),
+    )
+    if not force and published_models:
+        raise errors.bad_request.TaskCannotBeDeleted(
+            "has output models, use force=True",
+            task=task.id,
+            models=len(published_models),
+        )
+
+    if task.models and task.models.output:
+        model_ids = [m.model for m in task.models.output]
+        for output_model in Model.objects(id__in=model_ids).only(*model_fields):
+            if output_model.ready:
+                if not force:
+                    raise errors.bad_request.TaskCannotBeDeleted(
+                        "has output model, use force=True",
+                        task=task.id,
+                        model=output_model.id,
+                    )
+                published_models.append(output_model)
+            else:
+                draft_models.append(output_model)
+
+    in_use_model_ids = {}
+    if draft_models:
+        model_ids = {m.id for m in draft_models}
+        dependent_tasks = Task.objects(models__input__model__in=list(model_ids)).only(
+            "id", "models"
+        )
+        in_use_model_ids = model_ids & {
+            m.model
+            for m in chain.from_iterable(
+                t.models.input for t in dependent_tasks if t.models
+            )
+        }
+
+    return published_models, draft_models, in_use_model_ids

apiserver/bll/task/task_operations.py

@@ -0,0 +1,530 @@
+from datetime import datetime
+from typing import Callable, Any, Tuple, Union, Sequence
+
+from apiserver.apierrors import errors, APIError
+from apiserver.bll.queue import QueueBLL
+from apiserver.bll.task import (
+    TaskBLL,
+    validate_status_change,
+    ChangeStatusRequest,
+)
+from apiserver.bll.task.task_cleanup import cleanup_task, CleanupResult
+from apiserver.bll.task.utils import get_task_with_write_access
+from apiserver.bll.util import update_project_time
+from apiserver.config_repo import config
+from apiserver.database.model import EntityVisibility
+from apiserver.database.model.model import Model
+from apiserver.database.model.task.output import Output
+from apiserver.database.model.task.task import (
+    TaskStatus,
+    Task,
+    TaskSystemTags,
+    TaskStatusMessage,
+    ArtifactModes,
+    Execution,
+    DEFAULT_LAST_ITERATION,
+)
+from apiserver.database.utils import get_options
+from apiserver.service_repo.auth import Identity
+from apiserver.utilities.dicts import nested_set
+
+log = config.logger(__file__)
+queue_bll = QueueBLL()
+
+
+def archive_task(
+    task: Union[str, Task],
+    company_id: str,
+    identity: Identity,
+    status_message: str,
+    status_reason: str,
+) -> int:
+    """
+    Deque and archive task
+    Return 1 if successful
+    """
+    if isinstance(task, str):
+        task = get_task_with_write_access(
+            task,
+            company_id=company_id,
+            identity=identity,
+            only=(
+                "id",
+                "company",
+                "execution",
+                "status",
+                "project",
+                "system_tags",
+                "enqueue_status",
+            ),
+        )
+
+    user_id = identity.user
+    try:
+        TaskBLL.dequeue_and_change_status(
+            task,
+            company_id=company_id,
+            user_id=user_id,
+            status_message=status_message,
+            status_reason=status_reason,
+            remove_from_all_queues=True,
+        )
+    except APIError:
+        # dequeue may fail if the task was not enqueued
+        pass
+
+    return task.update(
+        status_message=status_message,
+        status_reason=status_reason,
+        add_to_set__system_tags=EntityVisibility.archived.value,
+        last_change=datetime.utcnow(),
+        last_changed_by=user_id,
+    )
+
+
+def unarchive_task(
+    task_id: str,
+    company_id: str,
+    identity: Identity,
+    status_message: str,
+    status_reason: str,
+) -> int:
+    """
+    Unarchive task. Return 1 if successful
+    """
+    task = get_task_with_write_access(
+        task_id,
+        company_id=company_id,
+        identity=identity,
+        only=("id",),
+    )
+    return task.update(
+        status_message=status_message,
+        status_reason=status_reason,
+        pull__system_tags=EntityVisibility.archived.value,
+        last_change=datetime.utcnow(),
+        last_changed_by=identity.user,
+    )
+
+
+def dequeue_task(
+    task_id: str,
+    company_id: str,
+    identity: Identity,
+    status_message: str,
+    status_reason: str,
+    remove_from_all_queues: bool = False,
+    new_status=None,
+) -> Tuple[int, dict]:
+    if new_status and new_status not in get_options(TaskStatus):
+        raise errors.bad_request.ValidationError(f"Invalid task status: {new_status}")
+
+    # get the task without write access to make sure that it actually exists
+    task = Task.get(
+        id=task_id,
+        company=company_id,
+        _only=("id",),
+        include_public=True,
+    )
+    if not task:
+        TaskBLL.remove_task_from_all_queues(company_id, task_id=task_id)
+        return 1, {"updated": 0}
+
+    user_id = identity.user
+    task = get_task_with_write_access(
+        task_id,
+        company_id=company_id,
+        identity=identity,
+        only=(
+            "id",
+            "company",
+            "execution",
+            "status",
+            "project",
+            "enqueue_status",
+        ),
+    )
+
+    res = TaskBLL.dequeue_and_change_status(
+        task,
+        company_id=company_id,
+        user_id=user_id,
+        status_message=status_message,
+        status_reason=status_reason,
+        remove_from_all_queues=remove_from_all_queues,
+        new_status=new_status,
+    )
+    return 1, res
+
+
+def enqueue_task(
+    task_id: str,
+    company_id: str,
+    identity: Identity,
+    queue_id: str,
+    status_message: str,
+    status_reason: str,
+    queue_name: str = None,
+    validate: bool = False,
+    force: bool = False,
+) -> Tuple[int, dict]:
+    if queue_id and queue_name:
+        raise errors.bad_request.ValidationError(
+            "Either queue id or queue name should be provided"
+        )
+
+    if queue_name:
+        queue = queue_bll.get_by_name(
+            company_id=company_id, queue_name=queue_name, only=("id",)
+        )
+        if not queue:
+            queue = queue_bll.create(company_id=company_id, name=queue_name)
+        queue_id = queue.id
+
+    if not queue_id:
+        # try to get default queue
+        queue_id = queue_bll.get_default(company_id).id
+
+    task = get_task_with_write_access(
+        task_id=task_id, company_id=company_id, identity=identity
+    )
+
+    user_id = identity.user
+    if validate:
+        TaskBLL.validate(task)
+
+    res = ChangeStatusRequest(
+        task=task,
+        new_status=TaskStatus.queued,
+        status_reason=status_reason,
+        status_message=status_message,
+        allow_same_state_transition=False,
+        force=force,
+        user_id=user_id,
+    ).execute(enqueue_status=task.status)
+
+    try:
+        queue_bll.add_task(company_id=company_id, queue_id=queue_id, task_id=task.id)
+    except Exception:
+        # failed enqueueing, revert to previous state
+        ChangeStatusRequest(
+            task=task,
+            current_status_override=TaskStatus.queued,
+            new_status=task.status,
+            force=True,
+            status_reason="failed enqueueing",
+            user_id=user_id,
+        ).execute(enqueue_status=None)
+        raise
+
+    # set the current queue ID in the task
+    if task.execution:
+        Task.objects(id=task_id).update(execution__queue=queue_id, multi=False)
+    else:
+        Task.objects(id=task_id).update(execution=Execution(queue=queue_id), multi=False)
+
+    nested_set(res, ("fields", "execution.queue"), queue_id)
+    return 1, res
+
+
+def move_tasks_to_trash(tasks: Sequence[str]) -> int:
+    try:
+        collection_name = Task._get_collection_name()
+        trash_collection_name = f"{collection_name}__trash"
+        Task.aggregate(
+            [
+                {"$match": {"_id": {"$in": tasks}}},
+                {
+                    "$merge": {
+                        "into": trash_collection_name,
+                        "on": "_id",
+                        "whenMatched": "replace",
+                        "whenNotMatched": "insert",
+                    }
+                },
+            ],
+            allow_disk_use=True,
+        )
+    except Exception as ex:
+        log.error(f"Error copying tasks to trash {str(ex)}")
+
+    return Task.objects(id__in=tasks).delete()
+
+
+def delete_task(
+    task_id: str,
+    company_id: str,
+    identity: Identity,
+    move_to_trash: bool,
+    force: bool,
+    return_file_urls: bool,
+    delete_output_models: bool,
+    status_message: str,
+    status_reason: str,
+    delete_external_artifacts: bool,
+) -> Tuple[int, Task, CleanupResult]:
+    user_id = identity.user
+    task = get_task_with_write_access(
+        task_id, company_id=company_id, identity=identity
+    )
+
+    if (
+        task.status != TaskStatus.created
+        and EntityVisibility.archived.value not in task.system_tags
+        and not force
+    ):
+        raise errors.bad_request.TaskCannotBeDeleted(
+            "due to status, use force=True",
+            task=task.id,
+            expected=TaskStatus.created,
+            current=task.status,
+        )
+
+    try:
+        TaskBLL.dequeue_and_change_status(
+            task,
+            company_id=company_id,
+            user_id=user_id,
+            status_message=status_message,
+            status_reason=status_reason,
+            remove_from_all_queues=True,
+        )
+    except APIError:
+        # dequeue may fail if the task was not enqueued
+        pass
+
+    cleanup_res = cleanup_task(
+        company=company_id,
+        user=user_id,
+        task=task,
+        force=force,
+        return_file_urls=return_file_urls,
+        delete_output_models=delete_output_models,
+        delete_external_artifacts=delete_external_artifacts,
+    )
+
+    if move_to_trash:
+        # make sure that whatever changes were done to the task are saved
+        # the task itself will be deleted later in the move_tasks_to_trash operation
+        task.last_update = datetime.utcnow()
+        task.save()
+    else:
+        task.delete()
+
+    update_project_time(task.project)
+    return 1, task, cleanup_res
+
+
+def reset_task(
+    task_id: str,
+    company_id: str,
+    identity: Identity,
+    force: bool,
+    return_file_urls: bool,
+    delete_output_models: bool,
+    clear_all: bool,
+    delete_external_artifacts: bool,
+) -> Tuple[dict, CleanupResult, dict]:
+    user_id = identity.user
+    task = get_task_with_write_access(
+        task_id, company_id=company_id, identity=identity
+    )
+
+    if not force and task.status == TaskStatus.published:
+        raise errors.bad_request.InvalidTaskStatus(task_id=task.id, status=task.status)
+
+    dequeued = {}
+    updates = {}
+
+    try:
+        dequeued = TaskBLL.dequeue(task, company_id, silent_fail=True)
+    except APIError:
+        # dequeue may fail if the task was not enqueued
+        pass
+
+    TaskBLL.remove_task_from_all_queues(company_id=company_id, task_id=task.id)
+
+    cleaned_up = cleanup_task(
+        company=company_id,
+        user=user_id,
+        task=task,
+        force=force,
+        update_children=False,
+        return_file_urls=return_file_urls,
+        delete_output_models=delete_output_models,
+        delete_external_artifacts=delete_external_artifacts,
+    )
+
+    updates.update(
+        set__last_iteration=DEFAULT_LAST_ITERATION,
+        set__last_metrics={},
+        set__unique_metrics=[],
+        set__metric_stats={},
+        set__models__output=[],
+        set__runtime={},
+        unset__output__result=1,
+        unset__output__error=1,
+        unset__last_worker=1,
+        unset__last_worker_report=1,
+        unset__started=1,
+        unset__completed=1,
+        unset__published=1,
+        unset__active_duration=1,
+        unset__enqueue_status=1,
+    )
+
+    if clear_all:
+        updates.update(
+            set__execution=Execution(),
+            unset__script=1,
+        )
+    else:
+        updates.update(unset__execution__queue=1)
+        if task.execution and task.execution.artifacts:
+            updates.update(
+                set__execution__artifacts={
+                    key: artifact
+                    for key, artifact in task.execution.artifacts.items()
+                    if artifact.mode == ArtifactModes.input
+                }
+            )
+
+    res = ChangeStatusRequest(
+        task=task,
+        new_status=TaskStatus.created,
+        force=force,
+        status_reason="reset",
+        status_message="reset",
+        user_id=user_id,
+    ).execute(
+        **updates,
+    )
+
+    return dequeued, cleaned_up, res
+
+
+def publish_task(
+    task_id: str,
+    company_id: str,
+    identity: Identity,
+    force: bool,
+    publish_model_func: Callable[[str, str, Identity], Any] = None,
+    status_message: str = "",
+    status_reason: str = "",
+) -> dict:
+    user_id = identity.user
+    task = get_task_with_write_access(
+        task_id, company_id=company_id, identity=identity
+    )
+    if not force:
+        validate_status_change(task.status, TaskStatus.published)
+
+    previous_task_status = task.status
+    output = task.output or Output()
+    publish_failed = False
+
+    try:
+        # set state to publishing
+        task.status = TaskStatus.publishing
+        task.save()
+
+        # publish task models
+        if task.models and task.models.output and publish_model_func:
+            model_id = task.models.output[-1].model
+            model = (
+                Model.objects(id=model_id, company=company_id)
+                .only("id", "ready")
+                .first()
+            )
+            if model and not model.ready:
+                publish_model_func(model.id, company_id, identity)
+
+        # set task status to published, and update (or set) it's new output (view and models)
+        return ChangeStatusRequest(
+            task=task,
+            new_status=TaskStatus.published,
+            force=force,
+            status_reason=status_reason,
+            status_message=status_message,
+            user_id=user_id,
+        ).execute(published=datetime.utcnow(), output=output)
+
+    except Exception as ex:
+        publish_failed = True
+        raise ex
+    finally:
+        if publish_failed:
+            task.status = previous_task_status
+            task.save()
+
+
+def stop_task(
+    task_id: str,
+    company_id: str,
+    identity: Identity,
+    user_name: str,
+    status_reason: str,
+    force: bool,
+) -> dict:
+    """
+    Stop a running task. Requires task status 'in_progress' and
+    execution_progress 'running', or force=True. Development task or
+    task that has no associated worker is stopped immediately.
+    For a non-development task with worker only the status message
+    is set to 'stopping' to allow the worker to stop the task and report by itself
+    :return: updated task fields
+    """
+    user_id = identity.user
+    task = get_task_with_write_access(
+        task_id,
+        company_id=company_id,
+        identity=identity,
+        only=(
+            "status",
+            "project",
+            "tags",
+            "system_tags",
+            "last_worker",
+            "last_update",
+            "execution.queue",
+        ),
+    )
+
+    def is_run_by_worker(t: Task) -> bool:
+        """Checks if there is an active worker running the task"""
+        update_timeout = config.get("apiserver.workers.task_update_timeout", 600)
+        return (
+            t.last_worker
+            and t.last_update
+            and (datetime.utcnow() - t.last_update).total_seconds() < update_timeout
+        )
+
+    is_queued = task.status == TaskStatus.queued
+    set_stopped = (
+        is_queued
+        or TaskSystemTags.development in task.system_tags
+        or not is_run_by_worker(task)
+    )
+
+    if set_stopped:
+        if is_queued:
+            try:
+                TaskBLL.dequeue(task, company_id=company_id, silent_fail=True)
+            except APIError:
+                # dequeue may fail if the task was not enqueued
+                pass
+
+        new_status = TaskStatus.stopped
+        status_message = f"Stopped by {user_name}"
+    else:
+        new_status = task.status
+        status_message = TaskStatusMessage.stopping
+
+    return ChangeStatusRequest(
+        task=task,
+        new_status=new_status,
+        status_reason=status_reason,
+        status_message=status_message,
+        force=force,
+        user_id=user_id,
+    ).execute()

apiserver/bll/task/utils.py

@@ -0,0 +1,333 @@
+from datetime import datetime
+from typing import Sequence
+
+import attr
+import six
+from mongoengine import Q
+
+from apiserver.apierrors import errors
+from apiserver.bll.util import update_project_time
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.model import Model
+from apiserver.database.model.task.task import Task, TaskStatus, TaskSystemTags
+from apiserver.database.utils import get_options
+from apiserver.service_repo.auth import Identity
+from apiserver.utilities.attrs import typed_attrs
+
+valid_statuses = get_options(TaskStatus)
+deleted_prefix = "__DELETED__"
+
+
+@typed_attrs
+class ChangeStatusRequest(object):
+    task = attr.ib(type=Task)
+    new_status = attr.ib(
+        type=six.string_types, validator=attr.validators.in_(valid_statuses)
+    )
+    status_reason = attr.ib(type=six.string_types, default="")
+    status_message = attr.ib(type=six.string_types, default="")
+    force = attr.ib(type=bool, default=False)
+    allow_same_state_transition = attr.ib(type=bool, default=True)
+    current_status_override = attr.ib(default=None)
+    user_id = attr.ib(type=str, default=None)
+
+    def execute(self, **kwargs):
+        current_status = self.current_status_override or self.task.status
+        project_id = self.task.project
+
+        # Verify new status is allowed from current status (will throw exception if not valid)
+        self.validate_transition(current_status)
+
+        control = dict(upsert=False, multi=False, write_concern=None, full_result=False)
+
+        now = datetime.utcnow()
+        fields = dict(
+            status=self.new_status,
+            status_reason=self.status_reason,
+            status_message=self.status_message,
+            status_changed=now,
+            last_update=now,
+            last_change=now,
+            last_changed_by=self.user_id,
+        )
+
+        if self.new_status == TaskStatus.queued:
+            fields["pull__system_tags"] = TaskSystemTags.development
+
+        def safe_mongoengine_key(key):
+            return f"__{key}" if key in control else key
+
+        fields.update({safe_mongoengine_key(k): v for k, v in kwargs.items()})
+
+        with translate_errors_context():
+            # atomic change of task status by querying the task with the EXPECTED status before modifying it
+            params = fields.copy()
+            params.update(control)
+            updated = Task.objects(id=self.task.id, status=current_status).update(
+                **params
+            )
+
+        if not updated:
+            # failed to change status (someone else beat us to it?)
+            raise errors.bad_request.FailedChangingTaskStatus(
+                task_id=self.task.id,
+                current_status=current_status,
+                new_status=self.new_status,
+            )
+
+        update_project_time(project_id)
+
+        # make sure that _raw_ queries are not returned back to the client
+        fields.pop("__raw__", None)
+
+        return dict(updated=updated, fields=fields)
+
+    def validate_transition(self, current_status):
+        if self.force:
+            return
+        if self.new_status != current_status:
+            validate_status_change(current_status, self.new_status)
+        elif not self.allow_same_state_transition:
+            raise errors.bad_request.InvalidTaskStatus(
+                "Task already in requested status",
+                current_status=current_status,
+                new_status=self.new_status,
+            )
+
+
+def validate_status_change(current_status, new_status):
+    assert current_status in valid_statuses
+    assert new_status in valid_statuses
+
+    allowed_statuses = get_possible_status_changes(current_status)
+    if new_status not in allowed_statuses:
+        raise errors.bad_request.InvalidTaskStatus(
+            "Invalid status change",
+            current_status=current_status,
+            new_status=new_status,
+        )
+
+
+state_machine = {
+    TaskStatus.created: {TaskStatus.queued, TaskStatus.in_progress},
+    TaskStatus.queued: {TaskStatus.created, TaskStatus.in_progress, TaskStatus.stopped},
+    TaskStatus.in_progress: {
+        TaskStatus.stopped,
+        TaskStatus.failed,
+        TaskStatus.created,
+        TaskStatus.completed,
+    },
+    TaskStatus.stopped: {
+        TaskStatus.closed,
+        TaskStatus.created,
+        TaskStatus.failed,
+        TaskStatus.queued,
+        TaskStatus.in_progress,
+        TaskStatus.published,
+        TaskStatus.publishing,
+        TaskStatus.completed,
+    },
+    TaskStatus.closed: {
+        TaskStatus.created,
+        TaskStatus.failed,
+        TaskStatus.published,
+        TaskStatus.publishing,
+        TaskStatus.stopped,
+    },
+    TaskStatus.failed: {TaskStatus.created, TaskStatus.stopped, TaskStatus.published},
+    TaskStatus.publishing: {TaskStatus.published},
+    TaskStatus.published: set(),
+    TaskStatus.completed: {
+        TaskStatus.published,
+        TaskStatus.in_progress,
+        TaskStatus.created,
+    },
+}
+
+
+def get_possible_status_changes(current_status):
+    """
+    :param current_status:
+    :return possible states from current state
+    """
+    possible = state_machine.get(current_status)
+    if possible is None:
+        raise errors.server_error.InternalError(
+            f"Current status {current_status} not supported by state machine"
+        )
+
+    return possible
+
+
+def get_many_tasks_for_writing(
+    company_id: str,
+    identity: Identity,
+    query: Q = None,
+    only: Sequence = None,
+    throw_on_forbidden: bool = True,
+) -> Sequence[Task]:
+    if only:
+        missing = [f for f in ("company", ) if f not in only]
+        if missing:
+            only = [*only, *missing]
+
+    result = list(
+        Task.get_many(
+            company=company_id,
+            query=query,
+            override_projection=only,
+            allow_public=True,
+            return_dicts=False,
+        )
+    )
+
+    if not company_id:
+        return result
+
+    forbidden_tasks = {task.id for task in result if not task.company}
+    if forbidden_tasks:
+        if throw_on_forbidden:
+            raise errors.forbidden.NoWritePermission(
+                f"cannot modify public task(s), ids={tuple(forbidden_tasks)}"
+            )
+        result = [task for task in result if task.id not in forbidden_tasks]
+
+    return result
+
+
+def get_task_with_write_access(
+    task_id: str,
+    company_id: str,
+    identity: Identity,
+    only=None,
+) -> Task:
+    """
+    Gets a task that has a required write access
+    :except errors.bad_request.InvalidTaskId: if the task is not found
+    :except errors.forbidden.NoWritePermission: if write_access was required and the task cannot be modified
+    """
+    query = dict(id=task_id, company=company_id)
+
+    task = Task.get_for_writing(_only=only, **query)
+    if not task:
+        raise errors.bad_request.InvalidTaskId(**query)
+
+    return task
+
+
+def get_task_for_update(
+    company_id: str,
+    task_id: str,
+    identity: Identity,
+    allow_all_statuses: bool = False,
+    force: bool = False
+) -> Task:
+    """
+    Loads only task id and return the task only if it is updatable (status == 'created')
+    """
+    task = get_task_with_write_access(
+        task_id=task_id,
+        company_id=company_id,
+        only=("id", "status"),
+        identity=identity,
+    )
+
+    if allow_all_statuses:
+        return task
+
+    allowed_statuses = (
+        [TaskStatus.created, TaskStatus.in_progress] if force else [TaskStatus.created]
+    )
+    if task.status not in allowed_statuses:
+        raise errors.bad_request.InvalidTaskStatus(
+            expected=TaskStatus.created, status=task.status
+        )
+    return task
+
+
+def update_task(
+    task: Task, user_id: str, update_cmds: dict, set_last_update: bool = True
+):
+    now = datetime.utcnow()
+    last_updates = dict(last_change=now, last_changed_by=user_id)
+    if set_last_update:
+        last_updates.update(last_update=now)
+    return task.update(**update_cmds, **last_updates)
+
+
+def get_last_metric_updates(
+    task_id: str,
+    last_scalar_events: dict,
+    raw_updates: dict,
+    extra_updates: dict,
+    model_events: bool = False,
+):
+    max_values = config.get("services.tasks.max_last_metrics", 2000)
+    total_metrics = set()
+    if max_values:
+        query = dict(id=task_id)
+        to_add = sum(len(v) for m, v in last_scalar_events.items())
+        if to_add <= max_values:
+            query[f"unique_metrics__{max_values - to_add}__exists"] = True
+        db_cls = Model if model_events else Task
+        task = db_cls.objects(**query).only("unique_metrics").first()
+        if task and task.unique_metrics:
+            total_metrics = set(task.unique_metrics)
+
+    new_metrics = []
+
+    def add_last_metric_conditional_update(
+        metric_path: str, metric_value, iter_value: int, is_min: bool
+    ):
+        """
+        Build an aggregation for an atomic update of the min or max value and the corresponding iteration
+        """
+        if is_min:
+            field_prefix = "min"
+            op = "$gt"
+        else:
+            field_prefix = "max"
+            op = "$lt"
+
+        value_field = f"{metric_path}__{field_prefix}_value".replace("__", ".")
+        condition = {
+            "$or": [
+                {"$lte": [f"${value_field}", None]},
+                {op: [f"${value_field}", metric_value]},
+            ]
+        }
+        raw_updates[value_field] = {
+            "$cond": [condition, metric_value, f"${value_field}"]
+        }
+
+        value_iteration_field = f"{metric_path}__{field_prefix}_value_iteration".replace(
+            "__", "."
+        )
+        raw_updates[value_iteration_field] = {
+            "$cond": [condition, iter_value, f"${value_iteration_field}"]
+        }
+
+    for metric_key, metric_data in last_scalar_events.items():
+        for variant_key, variant_data in metric_data.items():
+            metric = f"{variant_data.get('metric')}/{variant_data.get('variant')}"
+            if max_values:
+                if len(total_metrics) >= max_values and metric not in total_metrics:
+                    continue
+                total_metrics.add(metric)
+
+            new_metrics.append(metric)
+            path = f"last_metrics__{metric_key}__{variant_key}"
+            for key, value in variant_data.items():
+                if key in ("min_value", "max_value"):
+                    add_last_metric_conditional_update(
+                        metric_path=path,
+                        metric_value=value,
+                        iter_value=variant_data.get(f"{key}_iter", 0),
+                        is_min=(key == "min_value"),
+                    )
+                elif key in ("metric", "variant", "value"):
+                    extra_updates[f"set__{path}__{key}"] = value
+
+    if new_metrics:
+        extra_updates["add_to_set__unique_metrics"] = new_metrics

apiserver/bll/user/__init__.py

@@ -1,7 +1,9 @@
-from apierrors import errors
-from apimodels.users import CreateRequest
-from database.errors import translate_errors_context
-from database.model.user import User
+from datetime import datetime
+
+from apiserver.apierrors import errors
+from apiserver.apimodels.users import CreateRequest
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.user import User
 
 
 class UserBLL:
@@ -12,7 +14,7 @@ class UserBLL:
             if user_id and User.objects(id=user_id).only("id"):
                 raise errors.bad_request.UserIdExists(id=user_id)
 
-            user = User(**request.to_struct())
+            user = User(**request.to_struct(), created=datetime.utcnow())
             user.save(force_insert=True)
 
     @staticmethod

apiserver/bll/util.py

@@ -0,0 +1,92 @@
+import functools
+import itertools
+from concurrent.futures.thread import ThreadPoolExecutor
+from datetime import datetime
+from typing import (
+    Optional,
+    Callable,
+    Iterable,
+    Tuple,
+    Sequence,
+    TypeVar,
+    Union,
+)
+
+from boltons import iterutils
+
+from apiserver.apierrors import APIError
+from apiserver.database.model.project import Project
+from apiserver.database.model.settings import Settings
+
+
+@functools.lru_cache()
+def get_server_uuid() -> Optional[str]:
+    return Settings.get_by_key("server.uuid")
+
+
+def parallel_chunked_decorator(func: Callable = None, chunk_size: int = 100):
+    """
+    Decorates a method for parallel chunked execution. The method should have
+    one positional parameter (that is used for breaking into chunks)
+    and arbitrary number of keyword params. The return value should be iterable
+    The results are concatenated in the same order as the passed params
+    """
+    if func is None:
+        return functools.partial(parallel_chunked_decorator, chunk_size=chunk_size)
+
+    @functools.wraps(func)
+    def wrapper(self, iterable: Iterable, **kwargs):
+        assert iterutils.is_collection(
+            iterable
+        ), "The positional parameter should be an iterable for breaking into chunks"
+
+        func_with_params = functools.partial(func, self, **kwargs)
+        with ThreadPoolExecutor() as pool:
+            return list(
+                itertools.chain.from_iterable(
+                    filter(
+                        None,
+                        pool.map(
+                            func_with_params,
+                            iterutils.chunked_iter(iterable, chunk_size),
+                        ),
+                    )
+                ),
+            )
+
+    return wrapper
+
+
+T = TypeVar("T")
+
+
+def run_batch_operation(
+    func: Callable[[str], T], ids: Sequence[str]
+) -> Tuple[Sequence[Tuple[str, T]], Sequence[dict]]:
+    results = list()
+    failures = list()
+    for _id in ids:
+        try:
+            results.append((_id, func(_id)))
+        except APIError as err:
+            failures.append(
+                {
+                    "id": _id,
+                    "error": {
+                        "codes": [err.code, err.subcode],
+                        "msg": err.msg,
+                        "data": err.error_data,
+                    },
+                }
+            )
+    return results, failures
+
+
+def update_project_time(project_ids: Union[str, Sequence[str]]):
+    if not project_ids:
+        return
+
+    if isinstance(project_ids, str):
+        project_ids = [project_ids]
+
+    return Project.objects(id__in=project_ids).update(last_update=datetime.utcnow())

apiserver/bll/workers/__init__.py

@@ -0,0 +1,579 @@
+import itertools
+from datetime import datetime, timedelta
+from time import time
+from typing import Sequence, Set, Optional
+
+import attr
+import elasticsearch.helpers
+from boltons.iterutils import partition, chunked_iter
+from pyhocon import ConfigTree
+
+from apiserver.es_factory import es_factory
+from apiserver.apierrors import APIError
+from apiserver.apierrors.errors import bad_request, server_error
+from apiserver.apimodels.workers import (
+    IdNameEntry,
+    WorkerEntry,
+    StatusReportRequest,
+    WorkerResponseEntry,
+    QueueEntry,
+    MachineStats,
+)
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.auth import User
+from apiserver.database.model.company import Company
+from apiserver.database.model.project import Project
+from apiserver.database.model.queue import Queue
+from apiserver.database.model.task.task import Task
+from apiserver.redis_manager import redman
+from apiserver.utilities.dicts import nested_get
+from .stats import WorkerStats
+
+log = config.logger(__file__)
+
+
+class WorkerBLL:
+    def __init__(self, es=None, redis=None):
+        self.es_client = es or es_factory.connect("workers")
+        self.config = config.get("services.workers", ConfigTree())
+        self.redis = redis or redman.connection("workers")
+        self._stats = WorkerStats(self.es_client)
+
+    @property
+    def stats(self) -> WorkerStats:
+        return self._stats
+
+    def register_worker(
+        self,
+        company_id: str,
+        user_id: str,
+        worker: str,
+        ip: str = "",
+        queues: Sequence[str] = None,
+        timeout: int = 0,
+        tags: Sequence[str] = None,
+        system_tags: Sequence[str] = None,
+    ) -> WorkerEntry:
+        """
+        Register a worker
+        :param company_id: worker's company ID
+        :param user_id: user ID under which this worker is running
+        :param worker: worker ID
+        :param ip: the real ip of the worker
+        :param queues: queues reported as being monitored by the worker
+        :param timeout: registration expiration timeout in seconds
+        :param tags: a list of tags for this worker
+        :raise bad_request.InvalidUserId: in case the calling user or company does not exist
+        :return: worker entry instance
+        """
+        key = WorkerBLL._get_worker_key(company_id, user_id, worker)
+
+        timeout = timeout or int(self.config.get("default_worker_timeout_sec", 10 * 60))
+        queues = queues or []
+
+        with translate_errors_context():
+            query = dict(id=user_id, company=company_id)
+            user = User.objects(**query).only("id", "name").first()
+            if not user:
+                raise bad_request.InvalidUserId(**query)
+            company = Company.objects(id=company_id).only("id", "name").first()
+            if not company:
+                raise bad_request.InvalidId("invalid company", company=company_id)
+
+            queue_objs = Queue.objects(company=company_id, id__in=queues).only("id")
+            if len(queue_objs) < len(queues):
+                invalid = set(queues).difference(q.id for q in queue_objs)
+                raise bad_request.InvalidQueueId(ids=invalid)
+
+            now = datetime.utcnow()
+            entry = WorkerEntry(
+                key=key,
+                id=worker,
+                user=user.to_proper_dict(),
+                company=company.to_proper_dict(),
+                ip=ip,
+                queues=queues,
+                register_time=now,
+                register_timeout=timeout,
+                last_activity_time=now,
+                tags=tags,
+                system_tags=system_tags,
+            )
+
+            self._save_worker_data(entry)
+
+        return entry
+
+    def unregister_worker(self, company_id: str, user_id: str, worker: str) -> None:
+        """
+        Unregister a worker
+        :param company_id: worker's company ID
+        :param user_id: user ID under which this worker is running
+        :param worker: worker ID
+        :raise bad_request.WorkerNotRegistered: the worker was not previously registered
+        """
+        res = self.redis.delete(
+            company_id, self._get_worker_key(company_id, user_id, worker)
+        )
+        if not res and not config.get("apiserver.workers.auto_unregister", False):
+            raise bad_request.WorkerNotRegistered(worker=worker)
+
+    def status_report(
+        self,
+        company_id: str,
+        user_id: str,
+        ip: str,
+        report: StatusReportRequest,
+        tags: Sequence[str] = None,
+        system_tags: Sequence[str] = None,
+    ) -> None:
+        """
+        Write worker status report
+        :param company_id: worker's company ID
+        :param user_id: user_id ID under which this worker is running
+        :param ip: worker IP
+        :param report: the report itself
+        :param tags: tags for this worker
+        :raise bad_request.InvalidTaskId: the reported task was not found
+        :return: worker entry instance
+        """
+        entry = self._get_worker(company_id, user_id, report.worker)
+
+        try:
+            entry.ip = ip
+
+            if tags is not None:
+                entry.tags = tags
+            if system_tags is not None:
+                entry.system_tags = system_tags
+
+            if report.machine_stats:
+                self.log_stats_to_es(
+                    company_id=company_id,
+                    worker_id=report.worker,
+                    timestamp=report.timestamp,
+                    task=report.task,
+                    machine_stats=report.machine_stats,
+                )
+
+            now = datetime.utcnow()
+            entry.last_activity_time = now
+            entry.queue = report.queue
+
+            if report.queues:
+                entry.queues = report.queues
+
+            if not report.task:
+                entry.task = None
+                entry.project = None
+            else:
+                with translate_errors_context():
+                    query = dict(id=report.task, company=company_id)
+                    update = dict(
+                        last_worker=report.worker,
+                        last_worker_report=now,
+                        last_update=now,
+                        last_change=now,
+                        last_changed_by=user_id,
+                    )
+                    # modify(new=True, ...) returns the modified object
+                    task = Task.objects(**query).modify(new=True, **update)
+                    if not task:
+                        raise bad_request.InvalidTaskId(**query)
+                    entry.task = IdNameEntry(id=task.id, name=task.name)
+
+                    entry.project = None
+                    if task.project:
+                        project = Project.objects(id=task.project).only("name").first()
+                        if project:
+                            entry.project = IdNameEntry(
+                                id=project.id, name=project.name
+                            )
+
+            entry.last_report_time = now
+        except APIError:
+            raise
+        except Exception as e:
+            msg = "Failed processing worker status report"
+            log.exception(msg)
+            raise server_error.DataError(msg, err=e.args[0])
+        finally:
+            self._save_worker(entry)
+
+    def get_count(
+        self,
+        company_id: str,
+        last_seen: Optional[int] = None,
+        tags: Sequence[str] = None,
+        system_tags: Sequence[str] = None,
+    ):
+        if not last_seen:
+            return len(
+                self._get_keys(company_id, user_tags=tags, system_tags=system_tags)
+            )
+
+        return len(
+            self.get_all(
+                company_id, last_seen=last_seen, tags=tags, system_tags=system_tags
+            )
+        )
+
+    def get_all(
+        self,
+        company_id: str,
+        last_seen: Optional[int] = None,
+        tags: Sequence[str] = None,
+        system_tags: Sequence[str] = None,
+    ) -> Sequence[WorkerEntry]:
+        """
+        Get all the company workers that were active during the last_seen period
+        :param company_id: worker's company id
+        :param last_seen: period in seconds to check. Min value is 1 second
+        :return:
+        """
+        try:
+            workers = self._get(company_id, user_tags=tags, system_tags=system_tags)
+        except Exception as e:
+            raise server_error.DataError("failed loading worker entries", err=e.args[0])
+
+        if last_seen:
+            ref_time = datetime.utcnow() - timedelta(seconds=max(1, last_seen))
+            workers = [
+                w
+                for w in workers
+                if w.last_activity_time.replace(tzinfo=None) >= ref_time
+            ]
+
+        return workers
+
+    def get_all_with_projection(
+        self,
+        company_id: str,
+        last_seen: int,
+        tags: Sequence[str] = None,
+        system_tags: Sequence[str] = None,
+    ) -> Sequence[WorkerResponseEntry]:
+        helpers = [
+            WorkerConversionHelper.from_worker_entry(entry)
+            for entry in self.get_all(
+                company_id=company_id,
+                last_seen=last_seen,
+                tags=tags,
+                system_tags=system_tags,
+            )
+        ]
+
+        task_ids = set(filter(None, (helper.task_id for helper in helpers)))
+        all_queues = set(
+            itertools.chain.from_iterable(helper.queue_ids for helper in helpers)
+        )
+
+        queues_info = {}
+        if all_queues:
+            projection = [
+                {"$match": {"_id": {"$in": list(all_queues)}}},
+                {
+                    "$project": {
+                        "name": 1,
+                        "next_entry": {"$arrayElemAt": ["$entries", 0]},
+                        "num_entries": {"$size": "$entries"},
+                    }
+                },
+            ]
+            queues_info = {res["_id"]: res for res in Queue.aggregate(projection)}
+            task_ids = task_ids.union(
+                filter(
+                    None,
+                    (
+                        nested_get(info, ("next_entry", "task"))
+                        for info in queues_info.values()
+                    ),
+                )
+            )
+
+        tasks_info = {}
+        if task_ids:
+            tasks_info = {
+                task.id: task
+                for task in Task.objects(id__in=task_ids).only(
+                    "name", "started", "last_iteration", "active_duration"
+                )
+            }
+
+        def update_queue_entries(*entries):
+            for entry in entries:
+                if not entry:
+                    continue
+                info = queues_info.get(entry.id, None)
+                if not info:
+                    continue
+                entry.name = info.get("name", None)
+                entry.num_tasks = info.get("num_entries", 0)
+                task_id = nested_get(info, ("next_entry", "task"))
+                if task_id:
+                    task = tasks_info.get(task_id, None)
+                    entry.next_task = IdNameEntry(
+                        id=task_id, name=task.name if task else None
+                    )
+
+        for helper in helpers:
+            worker = helper.worker
+            if helper.task_id:
+                task = tasks_info.get(helper.task_id, None)
+                if task:
+                    worker.task.running_time = (task.active_duration or 0) * 1000
+                    worker.task.last_iteration = task.last_iteration
+
+            update_queue_entries(worker.queue)
+            if worker.queues:
+                update_queue_entries(*worker.queues)
+
+        return [helper.worker for helper in helpers]
+
+    @staticmethod
+    def _get_worker_key(company: str, user: str, worker_id: str) -> str:
+        """Build redis key from company, user and worker_id"""
+        return f"worker_{company}_{user}_{worker_id}"
+
+    def _get_worker(self, company_id: str, user_id: str, worker: str) -> WorkerEntry:
+        """
+        Get a worker entry for the provided worker ID. The entry is loaded from Redis
+        if it exists (i.e. worker has already been registered), otherwise the worker
+        is registered and its entry stored into Redis).
+        :param company_id: worker's company ID
+        :param user_id: user ID under which this worker is running
+        :param worker: worker ID
+        :raise bad_request.InvalidWorkerId: in case the worker id was not found
+        :return: worker entry instance
+        """
+        key = self._get_worker_key(company_id, user_id, worker)
+
+        data = self.redis.get(key)
+
+        if data:
+            try:
+                entry = WorkerEntry.from_json(data)
+                if not entry.key:
+                    entry.key = key
+                    self._save_worker(entry)
+                return entry
+            except Exception as e:
+                msg = "Failed parsing worker entry"
+                log.exception(msg)
+                raise server_error.DataError(msg, err=e.args[0])
+
+        # Failed loading worker from Redis
+        if config.get("apiserver.workers.auto_register", False):
+            try:
+                return self.register_worker(company_id, user_id, worker)
+            except Exception:
+                log.error(
+                    "Failed auto registration of {} for company {}".format(
+                        worker, company_id
+                    )
+                )
+
+        raise bad_request.InvalidWorkerId(worker=worker)
+
+    @staticmethod
+    def _get_tagged_workers_key(company: str, tags_field: str, tag: str) -> str:
+        """Build redis key from company, user and worker_id"""
+        return f"workers.{tags_field}_{company}_{tag}"
+
+    @staticmethod
+    def _get_all_workers_key(company: str) -> str:
+        """Build redis key from company, user and worker_id"""
+        return f"workers_{company}"
+
+    def _save_worker_data(self, entry: WorkerEntry):
+        self.redis.setex(
+            entry.key, timedelta(seconds=entry.register_timeout), entry.to_json()
+        )
+        company_id = entry.company.id
+        expiration = int(time()) + entry.register_timeout
+        worker_item = {entry.key: expiration}
+        self.redis.zadd(self._get_all_workers_key(company_id), worker_item)
+        for tags, tags_field in (
+            (entry.tags, "tags"),
+            (entry.system_tags, "systemtags"),
+        ):
+            for tag in tags:
+                name = self._get_tagged_workers_key(company_id, tags_field, tag)
+                self.redis.zadd(name, worker_item)
+
+    def _save_worker(self, entry: WorkerEntry) -> None:
+        """Save worker entry in Redis"""
+        try:
+            self._save_worker_data(entry)
+        except Exception:
+            msg = "Failed saving worker entry"
+            log.exception(msg)
+
+    def _get_keys(
+        self,
+        company: str,
+        user: str = "*",
+        user_tags: Sequence[str] = None,
+        system_tags: Sequence[str] = None,
+    ) -> Sequence[bytes]:
+        if not (user_tags or system_tags):
+            match = self._get_worker_key(company, user, "*")
+            return list(self.redis.scan_iter(match))
+
+        def filter_by_user(in_keys: Set[bytes]) -> Set[bytes]:
+            if user == "*":
+                return in_keys
+            user_bytes = user.encode()
+            return {k for k in in_keys if user_bytes in k}
+
+        worker_keys = set()
+        for tags, tags_field in (
+            (user_tags, "tags"),
+            (system_tags, "systemtags"),
+        ):
+            if not tags:
+                continue
+
+            timestamp = int(time())
+            include, exclude = partition(tags, key=lambda x: x[0] != "-")
+            if include:
+                tagged_workers = set()
+                for tag in include:
+                    tagged_workers_key = self._get_tagged_workers_key(
+                        company, tags_field, tag
+                    )
+                    self.redis.zremrangebyscore(
+                        tagged_workers_key, min=0, max=timestamp
+                    )
+                    tagged_workers.update(self.redis.zrange(tagged_workers_key, 0, -1))
+
+                tagged_workers = filter_by_user(tagged_workers)
+                worker_keys = (
+                    worker_keys.intersection(tagged_workers)
+                    if worker_keys
+                    else tagged_workers
+                )
+                if not worker_keys:
+                    return []
+
+            if exclude:
+                if not worker_keys:
+                    all_workers_key = self._get_all_workers_key(company)
+                    self.redis.zremrangebyscore(all_workers_key, min=0, max=timestamp)
+                    worker_keys.update(self.redis.zrange(all_workers_key, 0, -1))
+                    worker_keys = filter_by_user(worker_keys)
+                    if not worker_keys:
+                        return []
+
+                for tag in exclude:
+                    tagged_workers_key = self._get_tagged_workers_key(
+                        company, tags_field, tag[1:]
+                    )
+                    self.redis.zremrangebyscore(
+                        tagged_workers_key, min=0, max=timestamp
+                    )
+                    worker_keys.difference_update(
+                        self.redis.zrange(tagged_workers_key, 0, -1)
+                    )
+                    if not worker_keys:
+                        return []
+
+        return list(worker_keys)
+
+    def _get(
+        self,
+        company: str,
+        user: str = "*",
+        user_tags: Sequence[str] = None,
+        system_tags: Sequence[str] = None,
+    ) -> Sequence[WorkerEntry]:
+        """Get worker entries matching the company and user, worker patterns"""
+
+        entries = []
+        for keys in chunked_iter(
+            self._get_keys(
+                company, user=user, user_tags=user_tags, system_tags=system_tags
+            ),
+            1000,
+        ):
+            data = self.redis.mget(keys)
+            if data:
+                entries.extend(WorkerEntry.from_json(d) for d in data if d)
+
+        return entries
+
+    @staticmethod
+    def _get_es_index_suffix():
+        """Get the index name suffix for storing current month data"""
+        return datetime.utcnow().strftime("%Y-%m")
+
+    def log_stats_to_es(
+        self,
+        company_id: str,
+        worker_id: str,
+        timestamp: int,
+        task: str,
+        machine_stats: MachineStats,
+    ) -> int:
+        """
+        Actually writing the worker statistics to Elastic
+        :return: The amount of logged documents
+        """
+        es_index = (
+            f"{self._stats.worker_stats_prefix_for_company(company_id)}"
+            f"{self._get_es_index_suffix()}"
+        )
+
+        def make_doc(category, metric, variant, value) -> dict:
+            return dict(
+                _index=es_index,
+                _source=dict(
+                    timestamp=timestamp,
+                    worker=worker_id,
+                    task=task,
+                    category=category,
+                    metric=metric,
+                    variant=variant,
+                    value=float(value),
+                ),
+            )
+
+        actions = []
+        for field, value in machine_stats.to_struct().items():
+            if not value:
+                continue
+            category = field.partition("_")[0]
+            metric = field
+            if not isinstance(value, (list, tuple)):
+                actions.append(make_doc(category, metric, "total", value))
+            else:
+                actions.extend(
+                    make_doc(category, metric, str(i), val)
+                    for i, val in enumerate(value)
+                )
+
+        es_res = elasticsearch.helpers.bulk(self.es_client, actions)
+        added, errors = es_res[:2]
+        return added
+
+
+@attr.s(auto_attribs=True)
+class WorkerConversionHelper:
+    worker: WorkerResponseEntry
+    task_id: str
+    queue_ids: Set[str]
+
+    @classmethod
+    def from_worker_entry(cls, worker: WorkerEntry):
+        data = worker.to_struct()
+        queue = data.pop("queue", None) or None
+        queue_ids = set(data.pop("queues", []))
+        queues = [QueueEntry(id=id) for id in queue_ids]
+        if queue:
+            queue = next((q for q in queues if q.id == queue), None)
+        return cls(
+            worker=WorkerResponseEntry(queues=queues, queue=queue, **data),
+            task_id=worker.task.id if worker.task else None,
+            queue_ids=queue_ids,
+        )

apiserver/bll/workers/stats.py

@@ -0,0 +1,244 @@
+from operator import attrgetter
+from typing import Optional, Sequence
+
+from boltons.iterutils import bucketize
+
+from apiserver.apierrors.errors import bad_request
+from apiserver.apimodels.workers import AggregationType, GetStatsRequest, StatItem
+from apiserver.bll.query import Builder as QueryBuilder
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+
+log = config.logger(__file__)
+
+
+class WorkerStats:
+    def __init__(self, es):
+        self.es = es
+
+    @staticmethod
+    def worker_stats_prefix_for_company(company_id: str) -> str:
+        """Returns the es index prefix for the company"""
+        return f"worker_stats_{company_id.lower()}_"
+
+    def _search_company_stats(self, company_id: str, es_req: dict) -> dict:
+        return self.es.search(
+            index=f"{self.worker_stats_prefix_for_company(company_id)}*",
+            body=es_req,
+        )
+
+    def get_worker_stats_keys(
+        self, company_id: str, worker_ids: Optional[Sequence[str]]
+    ) -> dict:
+        """
+        Get dictionary of metric types grouped by categories
+        :param company_id: company id
+        :param worker_ids: optional list of workers to get metric types from.
+        If not specified them metrics for all the company workers returned
+        :return:
+        """
+        es_req = {
+            "size": 0,
+            "aggs": {
+                "categories": {
+                    "terms": {"field": "category"},
+                    "aggs": {"metrics": {"terms": {"field": "metric"}}},
+                }
+            },
+        }
+        if worker_ids:
+            es_req["query"] = QueryBuilder.terms("worker", worker_ids)
+
+        res = self._search_company_stats(company_id, es_req)
+
+        if not res["hits"]["total"]["value"]:
+            raise bad_request.WorkerStatsNotFound(
+                f"No statistic metrics found for the company {company_id} and workers {worker_ids}"
+            )
+
+        return {
+            category["key"]: [
+                metric["key"] for metric in category["metrics"]["buckets"]
+            ]
+            for category in res["aggregations"]["categories"]["buckets"]
+        }
+
+    def get_worker_stats(self, company_id: str, request: GetStatsRequest) -> dict:
+        """
+        Get statistics for company workers metrics in the specified time range
+        Returned as date histograms for different aggregation types
+        grouped by worker, metric type (and optionally metric variant)
+        Buckets with no metrics are not returned
+        Note: all the statistics are retrieved as one ES query
+        """
+        if request.from_date >= request.to_date:
+            raise bad_request.FieldsValueError("from_date must be less than to_date")
+
+        def get_dates_agg() -> dict:
+            es_to_agg_types = (
+                ("avg", AggregationType.avg.value),
+                ("min", AggregationType.min.value),
+                ("max", AggregationType.max.value),
+            )
+
+            return {
+                "dates": {
+                    "date_histogram": {
+                        "field": "timestamp",
+                        "fixed_interval": f"{request.interval}s",
+                        "min_doc_count": 1,
+                    },
+                    "aggs": {
+                        agg_type: {es_agg: {"field": "value"}}
+                        for es_agg, agg_type in es_to_agg_types
+                    },
+                }
+            }
+
+        def get_variants_agg() -> dict:
+            return {
+                "variants": {"terms": {"field": "variant"}, "aggs": get_dates_agg()}
+            }
+
+        es_req = {
+            "size": 0,
+            "aggs": {
+                "workers": {
+                    "terms": {"field": "worker"},
+                    "aggs": {
+                        "metrics": {
+                            "terms": {"field": "metric"},
+                            "aggs": get_variants_agg()
+                            if request.split_by_variant
+                            else get_dates_agg(),
+                        }
+                    },
+                }
+            },
+        }
+
+        query_terms = [
+            QueryBuilder.dates_range(request.from_date, request.to_date),
+            QueryBuilder.terms("metric", {item.key for item in request.items}),
+        ]
+        if request.worker_ids:
+            query_terms.append(QueryBuilder.terms("worker", request.worker_ids))
+        es_req["query"] = {"bool": {"must": query_terms}}
+
+        with translate_errors_context():
+            data = self._search_company_stats(company_id, es_req)
+
+        return self._extract_results(data, request.items, request.split_by_variant)
+
+    @staticmethod
+    def _extract_results(
+        data: dict, request_items: Sequence[StatItem], split_by_variant: bool
+    ) -> dict:
+        """
+        Clean results returned from elastic search (remove "aggregations", "buckets" etc.),
+        leave only aggregation types requested by the user and return a clean dictionary
+        and return a "clean" dictionary of
+        :param data: aggregation data retrieved from ES
+        :param request_items: aggs types requested by the user
+        :param split_by_variant: if False then aggregate by metric type, otherwise metric type + variant
+        """
+        if "aggregations" not in data:
+            return {}
+
+        items_by_key = bucketize(request_items, key=attrgetter("key"))
+        aggs_per_metric = {
+            key: [item.aggregation for item in items]
+            for key, items in items_by_key.items()
+        }
+
+        def extract_date_stats(date: dict, metric_key) -> dict:
+            return {
+                "date": date["key"],
+                "count": date["doc_count"],
+                **{agg: date[agg]["value"] for agg in aggs_per_metric[metric_key]},
+            }
+
+        def extract_metric_results(
+            metric_or_variant: dict, metric_key: str
+        ) -> Sequence[dict]:
+            return [
+                extract_date_stats(date, metric_key)
+                for date in metric_or_variant["dates"]["buckets"]
+                if date["doc_count"]
+            ]
+
+        def extract_variant_results(metric: dict) -> dict:
+            metric_key = metric["key"]
+            return {
+                variant["key"]: extract_metric_results(variant, metric_key)
+                for variant in metric["variants"]["buckets"]
+            }
+
+        def extract_worker_results(worker: dict) -> dict:
+            return {
+                metric["key"]: extract_variant_results(metric)
+                if split_by_variant
+                else extract_metric_results(metric, metric["key"])
+                for metric in worker["metrics"]["buckets"]
+            }
+
+        return {
+            worker["key"]: extract_worker_results(worker)
+            for worker in data["aggregations"]["workers"]["buckets"]
+        }
+
+    def get_activity_report(
+        self,
+        company_id: str,
+        from_date: float,
+        to_date: float,
+        interval: int,
+        active_only: bool,
+    ) -> Sequence[dict]:
+        """
+        Get statistics for company workers metrics in the specified time range
+        Returned as date histograms for different aggregation types
+        grouped by worker, metric type (and optionally metric variant)
+        Note: all the statistics are retrieved using one ES query
+        """
+        if from_date >= to_date:
+            raise bad_request.FieldsValueError("from_date must be less than to_date")
+
+        must = [QueryBuilder.dates_range(from_date, to_date)]
+        if active_only:
+            must.append({"exists": {"field": "task"}})
+
+        es_req = {
+            "size": 0,
+            "aggs": {
+                "dates": {
+                    "date_histogram": {
+                        "field": "timestamp",
+                        "fixed_interval": f"{interval}s",
+                        "extended_bounds": {
+                          "min": int(from_date) * 1000,
+                          "max": int(to_date) * 1000,
+                        }
+                    },
+                    "aggs": {"workers_count": {"cardinality": {"field": "worker"}}},
+                }
+            },
+            "query": {"bool": {"must": must}},
+        }
+
+        with translate_errors_context():
+            data = self._search_company_stats(company_id, es_req)
+
+        if "aggregations" not in data:
+            return {}
+
+        ret = [
+            dict(date=date["key"], count=date["workers_count"]["value"])
+            for date in data["aggregations"]["dates"]["buckets"]
+        ]
+
+        if ret and ret[-1]["date"] > (to_date - 0.9 * interval):
+            # remove last interval if it's incomplete. Allow 10% tolerance
+            ret.pop()
+
+        return ret

apiserver/config/__init__.py

@@ -0,0 +1 @@
+from .basic import BasicConfig, ConfigurationError

apiserver/config/basic.py

@@ -0,0 +1,238 @@
+import logging
+import logging.config
+import os
+import platform
+from functools import reduce
+from os import getenv
+from os.path import expandvars
+from pathlib import Path
+from typing import List, Any, TypeVar, Sequence, Set
+
+from boltons.iterutils import first
+from pyhocon import ConfigTree, ConfigFactory, ConfigValues
+from pyparsing import (
+    ParseFatalException,
+    ParseException,
+    RecursiveGrammarException,
+    ParseSyntaxException,
+)
+
+from apiserver.utilities import json
+
+EXTRA_CONFIG_PATHS = ("/opt/trains/config", "/opt/clearml/config")
+DEFAULT_PREFIXES = ("clearml", "trains")
+EXTRA_CONFIG_PATH_SEP = ":" if platform.system() != "Windows" else ";"
+
+
+class BasicConfig:
+    NotSet = object()
+
+    extra_config_values_env_key_sep = "__"
+    default_config_dir = "default"
+
+    def __init__(
+        self,
+        folder: str = None,
+        verbose: bool = True,
+        prefix: Sequence[str] = DEFAULT_PREFIXES,
+        exclude_files_from_base_folder: Sequence[str] = None,
+    ):
+        folder = (
+            Path(folder)
+            if folder
+            else Path(__file__).with_name(self.default_config_dir)
+        )
+        if not folder.is_dir():
+            raise ValueError("Invalid configuration folder")
+
+        self.exclude_files_from_base_folder = (
+            set(exclude_files_from_base_folder)
+            if exclude_files_from_base_folder
+            else set()
+        )
+        self.verbose = verbose
+
+        self.extra_config_path_override_var = [
+            f"{p.upper()}_CONFIG_DIR" for p in prefix
+        ]
+
+        self.prefix = prefix[0]
+        self.extra_config_values_env_key_prefix = [
+            f"{p.upper()}{self.extra_config_values_env_key_sep}"
+            for p in reversed(prefix)
+        ]
+
+        self._paths = [folder, *self._get_paths()]
+        self._config = self._reload()
+
+    def __getitem__(self, key):
+        return self._config[key]
+
+    def get(self, key: str, default: Any = NotSet) -> Any:
+        value = self._config.get(key, default)
+        if value is self.NotSet:
+            raise KeyError(
+                f"Unable to find value for key '{key}' and default value was not provided."
+            )
+        return value
+
+    def to_dict(self) -> dict:
+        return self._config.as_plain_ordered_dict()
+
+    def as_json(self) -> str:
+        return json.dumps(self.to_dict(), indent=2)
+
+    def logger(self, name: str) -> logging.Logger:
+        if Path(name).is_file():
+            name = Path(name).stem
+            if name == "__init__" and Path(name).parent.stem:
+                name = Path(name).parent.stem
+        path = ".".join((self.prefix, name))
+        return logging.getLogger(path)
+
+    def _read_extra_env_config_values(self) -> ConfigTree:
+        """Loads extra configuration from environment-injected values"""
+        result = ConfigTree()
+
+        for prefix in self.extra_config_values_env_key_prefix:
+            keys = sorted(k for k in os.environ if k.startswith(prefix))
+            for key in keys:
+                path = (
+                    key[len(prefix) :]
+                    .replace(self.extra_config_values_env_key_sep, ".")
+                    .lower()
+                )
+                result = self._merge_configs(
+                    result, ConfigFactory.parse_string(f"{path}: {os.environ[key]}")
+                )
+
+        return result
+
+    def _get_paths(self) -> List[Path]:
+        default_paths = EXTRA_CONFIG_PATH_SEP.join(EXTRA_CONFIG_PATHS)
+        value = first(map(getenv, self.extra_config_path_override_var), default_paths)
+
+        paths = [
+            Path(expandvars(v)).expanduser() for v in value.split(EXTRA_CONFIG_PATH_SEP)
+        ]
+
+        if value is not default_paths:
+            invalid = [path for path in paths if not path.is_dir()]
+            if invalid:
+                print(
+                    f"WARNING: Invalid paths in {self.extra_config_path_override_var} env var: {' '.join(map(str, invalid))}"
+                )
+
+        return [path for path in paths if path.is_dir()]
+
+    def reload(self):
+        self._config = self._reload()
+
+    def _reload(self) -> ConfigTree:
+        extra_config_values = self._read_extra_env_config_values()
+
+        configs = [
+            self._read_recursive(
+                path,
+                exclude_files=(
+                    self.exclude_files_from_base_folder if idx == 0 else None
+                ),
+            )
+            for idx, path in enumerate(self._paths)
+        ]
+
+        return reduce(
+            lambda last, config: self._merge_configs(last, config, copy_trees=True),
+            configs + [extra_config_values],
+            ConfigTree(),
+        )
+
+    @classmethod
+    def _merge_configs(cls, a, b, copy_trees=False, override_prefix="-"):
+        """Based on pyhocon.ConfigTree.merge_configs, with dict override support using a `-` key prefix"""
+        for key, value in b.items():
+            override = key.startswith(override_prefix)
+            if override:
+                key = key[len(override_prefix) :]
+            # if key is in both a and b and both values are dictionary then merge it otherwise override it
+            if (
+                not override
+                and key in a
+                and isinstance(a[key], ConfigTree)
+                and isinstance(b[key], ConfigTree)
+            ):
+                if copy_trees:
+                    a[key] = a[key].copy()
+                cls._merge_configs(a[key], b[key], copy_trees=copy_trees)
+            else:
+                if isinstance(value, ConfigValues):
+                    value.parent = a
+                    value.key = key
+                    if key in a:
+                        value.overriden_value = a[key]
+                a[key] = value
+                if a.root:
+                    if b.root:
+                        a.history[key] = a.history.get(key, []) + b.history.get(
+                            key, [value]
+                        )
+                    else:
+                        a.history[key] = a.history.get(key, []) + [value]
+
+        return a
+
+    def _read_recursive(self, conf_root, exclude_files: Set[str]) -> ConfigTree:
+        conf = ConfigTree()
+
+        if not conf_root:
+            return conf
+
+        if not conf_root.is_dir():
+            if self.verbose:
+                if not conf_root.exists():
+                    print(f"No config in {conf_root}")
+                else:
+                    print(f"Not a directory: {conf_root}")
+            return conf
+
+        if self.verbose:
+            print(f"Loading config from {conf_root}")
+
+        for file in conf_root.rglob("*.conf"):
+            if exclude_files and file.name in exclude_files:
+                continue
+            key = ".".join(file.relative_to(conf_root).with_suffix("").parts)
+            conf.put(key, self._read_single_file(file))
+
+        return conf
+
+    def _read_single_file(self, file_path):
+        if self.verbose:
+            print(f"Loading config from file {file_path}")
+
+        try:
+            return ConfigFactory.parse_file(file_path)
+        except ParseSyntaxException as ex:
+            msg = f"Failed parsing {file_path} ({ex.__class__.__name__}): (at char {ex.loc}, line:{ex.lineno}, col:{ex.column})"
+            raise ConfigurationError(msg, file_path=file_path) from ex
+        except (ParseException, ParseFatalException, RecursiveGrammarException) as ex:
+            msg = f"Failed parsing {file_path} ({ex.__class__.__name__}): {ex}"
+            raise ConfigurationError(msg) from ex
+        except Exception as ex:
+            print(f"Failed loading {file_path}: {ex}")
+            raise
+
+    def initialize_logging(self):
+        logging_config = self.get("logging", None)
+        if not logging_config:
+            return
+        logging.config.dictConfig(logging_config)
+
+
+class ConfigurationError(Exception):
+    def __init__(self, msg, file_path=None, *args):
+        super().__init__(msg, *args)
+        self.file_path = file_path
+
+
+ConfigType = TypeVar("ConfigType", bound=BasicConfig)

apiserver/config/default/apiserver.conf

@@ -0,0 +1,160 @@
+{
+    watch: false            # Watch for changes (dev only)
+    debug: false            # Debug mode
+    pretty_json: false      # prettify json response
+    return_stack: true      # return stack trace on error
+    return_stack_to_caller: true # top-level control on whether to return stack trace in an API response
+
+    # if 'return_stack' is true and error contains a status code, return stack trace only for these status codes
+    # valid values are:
+    #  - an integer number, specifying a status code
+    #  - a tuple of (code, subcode or list of subcodes)
+    return_stack_on_code: [
+        [500, 0]  # raise on internal server error with no subcode
+    ]
+
+    listen {
+        ip : "0.0.0.0"
+        port: 8008
+    }
+
+    version {
+        required: false
+        default: 1.0
+        # if set then calls to endpoints with the version
+        # greater that the current max version will be rejected
+        check_max_version: false
+    }
+
+    pre_populate {
+        enabled: false
+        zip_files: ["/path/to/export.zip"]
+        fail_on_error: false
+        # artifacts_path: "/mnt/fileserver"
+    }
+
+    # time in seconds to take an exclusive lock to init es and mongodb
+    # not including the pre_populate
+    db_init_timout: 120
+
+    mongo {
+        # controls whether FieldDoesNotExist exception will be raised for any extra attribute existing in stored data
+        # but not declared in a data model
+        strict: false
+    }
+
+    elastic {
+        probing {
+            # settings for inital probing of elastic connection
+            max_retries: 4
+            timeout: 30
+        }
+        upgrade_monitoring {
+            v16_migration_verification: true
+        }
+    }
+
+    auth {
+        # verify user tokens
+        verify_user_tokens: false
+
+        # If set then users that were created from secure credentials or fixed user settings and are no longer in these settings will be deleted on startup
+        delete_missing_autocreated_users: true
+
+        # max token expiration timeout in seconds (1 year)
+        max_expiration_sec: 31536000
+
+        # default token expiration timeout in seconds (30 days)
+        default_expiration_sec: 2592000
+
+        # cookie containing auth token, for requests arriving from a web-browser
+        session_auth_cookie_name: "clearml_token_basic"
+
+        # cookie configuration for authorization cookies generated by auth.login
+        cookies {
+            httponly: true  # allow only http to access the cookies (no JS etc)
+            secure: false   # not using HTTPS
+            domain: null    # Limit to localhost is not supported
+            samesite: Lax
+            max_age: 99999999999
+        }
+
+        # provide a cookie domain override per company
+#        cookies_domain_override {
+#            <company-id>: <domain>
+#        }
+
+#        # A list of fixed users
+#        # Note: password may be bcrypt-hashed (generate using `python -c 'import bcrypt; print(bcrypt.hashpw("password", bcrypt.gensalt()))'`)
+#        fixed_users {
+#            enabled: true
+#            pass_hashed: false
+#            users: [
+#                {
+#                    username: "john"
+#                    password: "123456"
+#                    name: "john doe"
+#                }
+#
+#            ]
+#        }
+    }
+
+    cors {
+        origins: "*"
+
+        # Not supported when origins is "*"
+        supports_credentials: true
+    }
+
+    default_company: "d1bd92a3b039400cbafc60a7a5b1e52b"
+
+    workers {
+        # Auto-register unknown workers on status reports and other calls
+        auto_register: true
+        # Assume unknow workers have unregistered (i.e. do not raise unregistered error)
+        auto_unregister: true
+        # Timeout in seconds on task status update. If exceeded
+        # then task can be stopped without communicating to the worker
+        task_update_timeout: 600
+
+        # Timeout in seconds for worker registration (or status report). If a worker did not report for this long,
+        # it is discarded from the server's table
+        default_timeout: 600
+    }
+
+    check_for_updates {
+        enabled: true
+
+        # Check for updates every 24 hours
+        check_interval_sec: 86400
+
+        url: "https://updates.clear.ml/updates"
+
+        component_name: "clearml-server"
+
+        # GET request timeout
+        request_timeout_sec: 3.0
+    }
+
+    statistics {
+        # Note: statistics are sent ONLY if the user has actively opted-in
+        supported: true
+
+        url: "https://updates.clear.ml/stats"
+
+        report_interval_hours: 24
+        agent_relevant_threshold_days: 30
+
+        max_retries: 5
+        max_backoff_sec: 5
+    }
+
+    getting_started_info {
+        "agentName": "clearml",
+        "configure": "clearml-init",
+        "install": "pip install clearml",
+        "packageName": "clearml"
+    }
+
+}

apiserver/config/default/hosts.conf

@@ -0,0 +1,45 @@
+fileserver = "http://localhost:8081"
+
+elastic {
+  events {
+    hosts: [{host: "127.0.0.1", port: 9200, scheme: http}]
+    args {
+      timeout: 60
+      max_retries: 3
+      retry_on_timeout: true
+    }
+    index_version: "1"
+  }
+
+  workers {
+    hosts: [{host:"127.0.0.1", port:9200, scheme: http}]
+    args {
+      timeout: 60
+      max_retries: 3
+      retry_on_timeout: true
+    }
+    index_version: "1"
+  }
+}
+
+mongo {
+  backend {
+    host: "mongodb://127.0.0.1:27017/backend"
+  }
+  auth {
+    host: "mongodb://127.0.0.1:27017/auth"
+  }
+}
+
+redis {
+  apiserver {
+      host: "127.0.0.1"
+      port: 6379
+      db: 0
+  }
+  workers {
+    host: "127.0.0.1"
+    port: 6379
+    db: 4
+  }
+}

apiserver/config/default/logging.conf

@@ -16,7 +16,7 @@
             backupCount: 3
             maxBytes: 10240000,
             class: "logging.handlers.RotatingFileHandler",
-            filename: "/var/log/trains/apiserver.log"
+            filename: "/var/log/clearml/apiserver.log"
         }
     }
     root {

apiserver/config/default/secure.conf

@@ -13,17 +13,27 @@
     credentials {
         # system credentials as they appear in the auth DB, used for intra-service communications
         apiserver {
+            role: "system"
             user_key: "62T8CP7HGBC6647XF9314C2VY67RJO"
             user_secret: "FhS8VZv_I4%6Mo$8S1BWc$n$=o1dMYSivuiWU-Vguq7qGOKskG-d+b@tn_Iq"
         }
         webserver {
+            role: "system"
             user_key: "EYVQ385RW7Y2QQUH88CZ7DWIQ1WUHP"
             user_secret: "yfc8KQo*GMXb*9p((qcYC7ByFIpF7I&4VH3BfUYXH%o9vX1ZUZQEEw1Inc)S"
+            revoke_in_fixed_mode: true
+        }
+        services_agent {
+            role: "admin"
+            user_key: "P4BMJA7RK3TKBXGSY8OAA1FA8TOD11"
+            user_secret: "9LsgSfa0SYz0zli1_c500ZcLqanre2xkWOpepyt1w-BKK3_DKPHrtoj3JSHvyy8bIi0"
         }
         tests {
+            role: "user"
+            display_name: "Default User"
             user_key: "EGRTCO8JMSIGI6S39GTP43NFWXDQOW"
             user_secret: "x!XTov_G-#vspE*Y(h$Anm&DIc5Ou-F)jsl$PdOyj5wG1&E!Z8"
-
+            revoke_in_fixed_mode: true
         }
     }
 }

apiserver/config/default/services/_mongo.conf

@@ -0,0 +1,9 @@
+max_page_size: 500
+
+# expiration time in seconds for the redis scroll states in get_many family of apis
+scroll_state_expiration_seconds: 600
+
+allow_disk_use {
+    sort: true
+    aggregate: true
+}

apiserver/config/default/services/async_urls_delete.conf

@@ -0,0 +1,12 @@
+# if set to true then on task delete/reset external file urls for known storage types are scheduled for async delete
+# otherwise they are returned to a client for the client side delete
+enabled: true
+max_retries: 3
+retry_timeout_sec: 60
+
+fileserver {
+    # fileserver url prefixes. Evaluated in the order of priority
+    # Can be in the form <schema>://host:port/path or /path
+    url_prefixes: ["https://files.community-master.hosted.allegro.ai/"]
+    timeout_sec: 300
+}

apiserver/config/default/services/auth.conf

@@ -0,0 +1,16 @@
+fixed_users {
+  guest {
+    enabled: false
+
+    default_company: "025315a9321f49f8be07f5ac48fbcf92"
+
+    name: "Guest"
+    username: "guest"
+    password: "guest"
+
+    # Allow access only to the following endpoints when using user/pass credentials
+    allow_endpoints: [
+      "auth.login"
+    ]
+  }
+}

apiserver/config/default/services/events.conf

@@ -0,0 +1,47 @@
+es_index_prefix: "events"
+
+ignore_iteration {
+    metrics: [":monitor:machine", ":monitor:gpu"]
+}
+
+
+events_retrieval {
+    state_expiration_sec: 3600
+
+    # max number of concurrent queries to ES when calculating events metrics
+    # should not exceed the amount of concurrent connections set in the ES driver
+    max_metrics_concurrency: 4
+
+    # If set then max_metrics_count and max_variants_count are calculated dynamically on user data
+    dynamic_metrics_count: true
+
+    # The percentage from the ES aggs limit (10000) to use for the max_metrics and max_variants calculation
+    dynamic_metrics_count_threshold: 80
+
+    # the max amount of metrics to aggregate on
+    max_metrics_count: 100
+
+    # the max amount of variants to aggregate on
+    max_variants_count: 100
+
+    debug_images {
+        # Allow to return the debug images for the variants with uninitialized valid iterations border
+        allow_uninitialized_variants: true
+    }
+
+    max_raw_scalars_size: 200000
+
+    scroll_id_key: "cTN5VEtWEC6QrHvUl0FTx9kNyO0CcCK1p57akxma"
+
+    multi_plots_batch_size: 1000
+}
+
+# if set then plot str will be checked for the valid json on plot add
+# and the result of the check is written to the db
+validate_plot_str: false
+
+# If not 0 then the plots equal or greater to the size will be stored compressed in the DB
+plot_compression_threshold: 100000
+
+# async events delete threshold
+max_async_deleted_events_per_sec: 1000

apiserver/config/default/services/models.conf

@@ -0,0 +1,4 @@
+metadata_values {
+    # cache ttl sec
+    cache_ttl_sec: 86400
+}

apiserver/config/default/services/organization.conf

@@ -0,0 +1,9 @@
+tags_cache {
+  expiration_seconds: 3600
+}
+download {
+    redis_timeout_sec: 300
+    batch_size: 500
+    max_download_items: 50000
+    max_project_name_length: 60
+}

apiserver/config/default/services/projects.conf

@@ -0,0 +1,18 @@
+# Order of featured projects, by name or ID
+featured {
+  order: [
+    #  {id: "<project-id>"}
+    #  OR
+    #  {name: "<project-name>"}
+    #  OR
+    #  {name_regex: "<python-regex>"}
+  ]
+
+  # default featured index for public projects not specified in the order
+  public_default: 9999
+}
+
+sub_projects {
+    # the max sub project depth
+    max_depth: 10
+}

apiserver/config/default/services/queues.conf

@@ -0,0 +1,8 @@
+{
+    metrics_before_from_date: 3600
+    # interval in seconds to update queue metrics. Put 0 to disable
+    metrics_refresh_interval_sec: 300
+    # the queues with these tags will not be returned from get_all/get_all_ex unless id or name specified
+    # or search_hidden is set
+    hidden_tags: [k8s-glue]
+}

apiserver/config/default/services/storage_credentials.conf

@@ -0,0 +1,54 @@
+aws {
+    s3 {
+        # S3 credentials, used for read/write access by various SDK elements
+        # default, used for any bucket not specified below
+        key: ""
+        secret: ""
+        region: ""
+        use_credentials_chain: false
+        # Additional ExtraArgs passed to boto3 when uploading files. Can also be set per-bucket under "credentials".
+        extra_args: {}
+        credentials: [
+            # specifies key/secret credentials to use when handling s3 urls (read or write)
+            # {
+            #     bucket: "my-bucket-name"
+            #     key: "my-access-key"
+            #     secret: "my-secret-key"
+            # },
+            {
+                # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
+                host: "localhost:9000"
+                key: "minioadmin"
+                secret: "minioadmin"
+                # region: my-server
+                multipart: false
+                secure: false
+            }
+        ]
+    }
+}
+google.storage {
+    # Default project and credentials file
+    # Will be used when no bucket configuration is found
+//    project: "clearml"
+//    credentials_json: "/path/to/credentials.json"
+//
+//    # Specific credentials per bucket and sub directory
+//    credentials = [
+//        {
+//            bucket: "my-bucket"
+//            subdir: "path/in/bucket" # Not required
+//            project: "clearml"
+//            credentials_json: "/path/to/credentials.json"
+//        },
+//    ]
+}
+azure.storage {
+    # containers: [
+    #     {
+    #         account_name: "clearml"
+    #         account_key: "secret"
+    #         # container_name:
+    #     }
+    # ]
+}

apiserver/config/default/services/tasks.conf

@@ -0,0 +1,28 @@
+non_responsive_tasks_watchdog {
+    enabled: true
+
+    # In-progress tasks older than this value in seconds will be stopped by the watchdog
+    threshold_sec: 7200
+
+    # Watchdog will sleep for this number of seconds after each cycle
+    watch_interval_sec: 900
+}
+
+multi_task_histogram_limit: 100
+
+hyperparam_values {
+    # max allowed outdate time for the cashed result
+    cache_allowed_outdate_sec: 60
+
+    # cache ttl sec
+    cache_ttl_sec: 86400
+}
+
+# the maximum amount of unique last metrics/variants combinations
+# for which the last values are stored in a task
+max_last_metrics: 2000
+
+# if set then call to tasks.delete/cleanup does not wait for ES events deletion
+async_events_delete: true
+# do not use async_delete if the deleted task has amount of events lower than this threshold
+async_events_delete_threshold: 100000

apiserver/config/info.py

@@ -0,0 +1,51 @@
+from functools import lru_cache
+from os import getenv
+from pathlib import Path
+
+from boltons.iterutils import first
+
+from apiserver.config_repo import config
+from apiserver.version import __version__
+
+root = Path(__file__).parent.parent
+
+
+def _get(prop_name, env_suffix=None, default=""):
+    suffix = env_suffix or prop_name
+    keys = [f"{p}_SERVER_{suffix}" for p in ("CLEARML", "TRAINS")]
+    value = first(map(getenv, keys))
+    if value:
+        return value
+
+    try:
+        return (root / prop_name).read_text().strip()
+    except FileNotFoundError:
+        return default
+
+
+@lru_cache()
+def get_build_number():
+    return _get("BUILD")
+
+
+@lru_cache()
+def get_version():
+    return _get("VERSION", default=__version__)
+
+
+@lru_cache()
+def get_commit_number():
+    return _get("COMMIT")
+
+
+@lru_cache()
+def get_deployment_type() -> str:
+    return _get("DEPLOY", env_suffix="DEPLOYMENT_TYPE", default="manual")
+
+
+def get_default_company():
+    return config.get("apiserver.default_company")
+
+
+missed_es_upgrade = False
+es_connection_error = False

apiserver/config_repo.py

@@ -0,0 +1,4 @@
+from apiserver.config import BasicConfig
+
+config = BasicConfig()
+config.initialize_logging()

apiserver/database/__init__.py

@@ -0,0 +1,137 @@
+from os import getenv
+
+from boltons.iterutils import first
+from furl import furl
+from jsonmodels import models
+from jsonmodels.errors import ValidationError
+from jsonmodels.fields import StringField
+from mongoengine import register_connection
+from mongoengine.connection import get_connection, disconnect
+
+from apiserver.config_repo import config
+from .defs import Database
+from .utils import get_items
+
+log = config.logger("database")
+
+strict = config.get("apiserver.mongo.strict", True)
+
+OVERRIDE_HOST_ENV_KEY = (
+    "CLEARML_MONGODB_SERVICE_HOST",
+    "TRAINS_MONGODB_SERVICE_HOST",
+    "MONGODB_SERVICE_HOST",
+    "MONGODB_SERVICE_SERVICE_HOST",
+)
+OVERRIDE_PORT_ENV_KEY = (
+    "CLEARML_MONGODB_SERVICE_PORT",
+    "TRAINS_MONGODB_SERVICE_PORT",
+    "MONGODB_SERVICE_PORT",
+)
+
+OVERRIDE_CONNECTION_STRING_ENV_KEY = "CLEARML_MONGODB_SERVICE_CONNECTION_STRING"
+OVERRIDE_USERNAME_ENV_KEY = "CLEARML_MONGODB_SERVICE_USERNAME"
+OVERRIDE_PASSWORD_ENV_KEY = "CLEARML_MONGODB_SERVICE_PASSWORD"
+OVERRIDE_QUERY_ENV_KEY = "CLEARML_MONGODB_SERVICE_QUERY"
+
+
+class DatabaseEntry(models.Base):
+    host = StringField(required=True)
+    alias = StringField()
+
+
+class DatabaseFactory:
+    _entries = []
+
+    @classmethod
+    def _create_db_entry(cls, alias: str, settings: dict) -> DatabaseEntry:
+        return DatabaseEntry(alias=alias, **settings)
+
+    @classmethod
+    def initialize(cls):
+        db_entries = config.get("hosts.mongo", {})
+        missing = []
+        log.info("Initializing database connections")
+
+        override_connection_string = getenv(OVERRIDE_CONNECTION_STRING_ENV_KEY)
+        override_hostname = first(map(getenv, OVERRIDE_HOST_ENV_KEY), None)
+        override_port = first(map(getenv, OVERRIDE_PORT_ENV_KEY), None)
+        override_username = getenv(OVERRIDE_USERNAME_ENV_KEY)
+        override_password = getenv(OVERRIDE_PASSWORD_ENV_KEY)
+        override_query = getenv(OVERRIDE_QUERY_ENV_KEY)
+
+        if override_connection_string:
+            log.info(f"Using override mongodb connection string template {override_connection_string}")
+        else:
+            if override_hostname:
+                log.info(f"Using override mongodb host {override_hostname}")
+            if override_port:
+                log.info(f"Using override mongodb port {override_port}")
+            if override_username:
+                log.info(f"Using override mongodb username {override_username}")
+            if override_password:
+                log.info(f"Using override mongodb password ******")
+            if override_query:
+                log.info(f"Using override mongodb query {override_query}")
+
+        for key, alias in get_items(Database).items():
+            if key not in db_entries:
+                missing.append(key)
+                continue
+
+            entry = cls._create_db_entry(alias=alias, settings=db_entries.get(key))
+
+            if override_connection_string:
+                con_str = f"{override_connection_string.rstrip('/')}/{key}"
+                log.info(f"Using override mongodb connection string for {alias}: {con_str}")
+                entry.host = con_str
+            else:
+                if override_hostname:
+                    entry.host = furl(entry.host).set(host=override_hostname).url
+                if override_port:
+                    entry.host = furl(entry.host).set(port=override_port).url
+                if override_username:
+                    entry.host = furl(entry.host).set(username=override_username).url
+                if override_password:
+                    entry.host = furl(entry.host).set(password=override_password).url
+                if override_query:
+                    entry.host = furl(entry.host).set(query=override_query).url
+
+            try:
+                entry.validate()
+                log.info(
+                    "Registering connection to %(alias)s (%(host)s)" % entry.to_struct()
+                )
+                register_connection(**entry.to_struct())
+
+                cls._entries.append(entry)
+            except ValidationError as ex:
+                raise Exception("Invalid database entry `%s`: %s" % (key, ex.args[0]))
+        if missing:
+            raise ValueError(
+                "Missing database configuration for %s" % ", ".join(missing)
+            )
+
+    @classmethod
+    def get_entries(cls):
+        return cls._entries
+
+    @classmethod
+    def get_hosts(cls):
+        return [entry.host for entry in cls.get_entries()]
+
+    @classmethod
+    def get_aliases(cls):
+        return [entry.alias for entry in cls.get_entries()]
+
+    @classmethod
+    def reconnect(cls):
+        for entry in cls.get_entries():
+            # there is bug in the current implementation that prevents
+            # reconnection from work so workaround this
+            # get_connection(entry.alias, reconnect=True)
+            disconnect(entry.alias)
+            register_connection(**entry.to_struct())
+            get_connection(entry.alias)
+
+
+db = DatabaseFactory()

apiserver/database/errors.py

@@ -1,10 +1,11 @@
 import re
 from contextlib import contextmanager
 from functools import wraps
+from textwrap import shorten
 
 import dpath
 from dpath.exceptions import InvalidKeyName
-from elasticsearch import ElasticsearchException
+from elastic_transport import TransportError, ApiError
 from elasticsearch.helpers import BulkIndexError
 from jsonmodels.errors import ValidationError as JsonschemaValidationError
 from mongoengine.errors import (
@@ -15,9 +16,9 @@ from mongoengine.errors import (
     LookUpError,
     InvalidQueryError,
 )
-from pymongo.errors import PyMongoError, NotMasterError
+from pymongo.errors import PyMongoError, NotPrimaryError
 
-from apierrors import errors
+from apiserver.apierrors import errors
 
 
 class MakeGetAllQueryError(Exception):
@@ -33,7 +34,7 @@ class ParseCallError(Exception):
         self.params = kwargs
 
 
-def throws_default_error(err_cls):
+def throws_default_error(err_cls, shorten_width: int = None):
     """
     Used to make functions (Exception, str) -> Optional[str] searching for specialized error messages raise those
     messages in ``err_cls``. If the decorated function does not find a suitable error message,
@@ -45,25 +46,49 @@ def throws_default_error(err_cls):
         @wraps(func)
         def wrapper(self, e, message, **kwargs):
             extra_info = func(self, e, message, **kwargs)
-            raise err_cls(message, err=e, extra_info=extra_info)
+            err = str(e)
+            if shorten_width:
+                err = shorten(err, shorten_width, placeholder="...")
+            raise err_cls(message, err=err, extra_info=extra_info)
 
         return wrapper
 
     return decorator
 
 
+# noinspection RegExpRedundantEscape
 class ElasticErrorsHandler(object):
     @classmethod
-    @throws_default_error(errors.server_error.DataError)
+    def _bulk_meta_error(cls, error):
+        try:
+            _, err_type = next(dpath.search(error, "*/error/type", yielded=True))
+            _, reason = next(dpath.search(error, "*/error/reason", yielded=True))
+            if err_type == "cluster_block_exception":
+                raise errors.server_error.LowDiskSpace(
+                    "metrics, logs and all indexed data is in read-only mode!",
+                    reason=re.sub(r"^index\s\[.*?\]\s", "", reason) if reason else ""
+                )
+            return
+        except StopIteration:
+            pass
+
+    @classmethod
+    @throws_default_error(errors.server_error.DataError, shorten_width=200)
     def bulk_error(cls, e, _, **__):
         if not e.errors:
             return
 
+        # Currently we only handle the first error
+        error = e.errors[0]
+
+        cls._bulk_meta_error(error)
+
         # Else try returning a better error string
         for _, reason in dpath.search(e.errors[0], "*/error/reason", yielded=True):
             return reason
 
 
+# noinspection RegExpRedundantEscape
 class MongoEngineErrorsHandler(object):
     # NotUniqueError
     __not_unique_regex = re.compile(
@@ -81,6 +106,7 @@ class MongoEngineErrorsHandler(object):
     def validation_error(cls, e: ValidationError, message, **_):
         # Thrown when a document is validated. Documents are validated by default on save and on update
         err_dict = e.errors or {e.field_name: e.message}
+        err_dict = {key: str(value) for key, value in err_dict.items()}
         raise errors.bad_request.DataValidationError(message, **err_dict)
 
     @classmethod
@@ -140,7 +166,10 @@ class MongoEngineErrorsHandler(object):
     @classmethod
     @throws_default_error(errors.server_error.InternalError)
     def invalid_query_error(cls, e, message, **_):
-        pass
+        if e.args:
+            inner = e.args[0]
+            if isinstance(inner, LookUpError):
+                cls.lookup_error(inner, message)
 
 
 @contextmanager
@@ -169,7 +198,7 @@ def translate_errors_context(message=None, **kwargs):
         MongoEngineErrorsHandler.invalid_query_error(e, message, **kwargs)
     except PyMongoError as e:
         raise errors.server_error.InternalError(message, err=str(e))
-    except NotMasterError as e:
+    except NotPrimaryError as e:
         raise errors.server_error.InternalError(message, err=str(e))
     except MakeGetAllQueryError as e:
         raise errors.bad_request.ValidationError(e.error, field=e.field)
@@ -181,9 +210,9 @@ def translate_errors_context(message=None, **kwargs):
         raise errors.bad_request.ValidationError(e.args[0])
     except BulkIndexError as e:
         ElasticErrorsHandler.bulk_error(e, message, **kwargs)
-    except ElasticsearchException as e:
+    except (TransportError, ApiError) as e:
         raise errors.server_error.DataError(e, message, **kwargs)
     except InvalidKeyName:
         raise errors.server_error.DataError("invalid empty key encountered in data")
-    except Exception as ex:
+    except Exception:
         raise

apiserver/database/fields.py

@@ -1,5 +1,6 @@
-import re
+from operator import itemgetter
 from sys import maxsize
+from typing import Type, Tuple
 
 import six
 from mongoengine import (
@@ -11,7 +12,11 @@ from mongoengine import (
     SortedListField,
     MapField,
     DictField,
+    DynamicField,
 )
+from mongoengine.fields import key_not_string, key_starts_with_dollar, EmailField
+
+NoneType = type(None)
 
 
 class LengthRangeListField(ListField):
@@ -88,102 +93,22 @@ class CustomFloatField(FloatField):
             self.error("Float value must be greater than %s" % str(self.greater_than))
 
 
-# TODO: bucket name should be at most 63 characters....
-aws_s3_bucket_only_regex = (
-    r"^s3://"
-    r"(?:(?:\w[A-Z0-9\-]+\w)\.)*(?:\w[A-Z0-9\-]+\w)"  # bucket name
-)
+class CanonicEmailField(EmailField):
+    """email field that is always lower cased"""
+    def __set__(self, instance, value: str):
+        if value is not None:
+            try:
+                value = value.lower()
+            except AttributeError:
+                pass
+        super().__set__(instance, value)
 
-aws_s3_url_with_bucket_regex = (
-    r"^s3://"
-    r"(?:(?:\w[A-Z0-9\-]+\w)\.)*(?:\w[A-Z0-9\-]+\w)"  # bucket name
-    r"(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+(?:[A-Z]{2,6}\.?|[A-Z0-9-]{2,}(?<!-)\.?))"  # domain...
-)
-
-non_aws_s3_regex = (
-    r"^s3://"
-    r"(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+(?:[A-Z]{2,6}\.?|[A-Z0-9-]{2,}(?<!-)\.?)|"  # domain...
-    r"localhost|"  # localhost...
-    r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|"  # ...or ipv4
-    r"\[?[A-F0-9]*:[A-F0-9:]+\]?)"  # ...or ipv6
-    r"(?::\d+)?"  # optional port
-    r"(?:/(?:(?:\w[A-Z0-9\-]+\w)\.)*(?:\w[A-Z0-9\-]+\w))"  # bucket name
-)
-
-google_gs_bucket_only_regex = (
-    r"^gs://"
-    r"(?:(?:\w[A-Z0-9\-_]+\w)\.)*(?:\w[A-Z0-9\-_]+\w)"  # bucket name
-)
-
-file_regex = r"^file://"
-
-generic_url_regex = (
-    r"^%s://"  # scheme placeholder
-    r"(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+(?:[A-Z]{2,6}\.?|[A-Z0-9-]{2,}(?<!-)\.?)|"  # domain...
-    r"localhost|"  # localhost...
-    r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|"  # ...or ipv4
-    r"\[?[A-F0-9]*:[A-F0-9:]+\]?)"  # ...or ipv6
-    r"(?::\d+)?"  # optional port
-)
-
-path_suffix = r"(?:/?|[/?]\S+)$"
-file_path_suffix = r"(?:/\S*[^/]+)$"
-
-
-class _RegexURLField(StringField):
-    _regex = []
-
-    def __init__(self, regex, **kwargs):
-        super(_RegexURLField, self).__init__(**kwargs)
-        regex = regex if isinstance(regex, (tuple, list)) else [regex]
-        self._regex = [
-            re.compile(e, re.IGNORECASE) if isinstance(e, six.string_types) else e
-            for e in regex
-        ]
-
-    def validate(self, value):
-        # Check first if the scheme is valid
-        if not any(regex for regex in self._regex if regex.match(value)):
-            self.error("Invalid URL: {}".format(value))
-            return
-
-
-class OutputDestinationField(_RegexURLField):
-    """ A field representing task output URL """
-
-    schemes = ["s3", "gs", "file"]
-    _expressions = (
-        aws_s3_bucket_only_regex + path_suffix,
-        aws_s3_url_with_bucket_regex + path_suffix,
-        non_aws_s3_regex + path_suffix,
-        google_gs_bucket_only_regex + path_suffix,
-        file_regex + path_suffix,
-    )
-
-    def __init__(self, **kwargs):
-        super(OutputDestinationField, self).__init__(self._expressions, **kwargs)
-
-
-class SupportedURLField(_RegexURLField):
-    """ A field representing a model URL """
-
-    schemes = ["s3", "gs", "file", "http", "https"]
-
-    _expressions = tuple(
-        pattern + file_path_suffix
-        for pattern in (
-            aws_s3_bucket_only_regex,
-            aws_s3_url_with_bucket_regex,
-            non_aws_s3_regex,
-            google_gs_bucket_only_regex,
-            file_regex,
-            (generic_url_regex % "http"),
-            (generic_url_regex % "https"),
-        )
-    )
-
-    def __init__(self, **kwargs):
-        super(SupportedURLField, self).__init__(self._expressions, **kwargs)
+    def prepare_query_value(self, op, value):
+        if not isinstance(op, six.string_types):
+            return value
+        if value is not None:
+            value = value.lower()
+        return super().prepare_query_value(op, value)
 
 
 class StrippedStringField(StringField):
@@ -221,17 +146,89 @@ def contains_empty_key(d):
             return True
 
 
-class SafeMapField(MapField):
+class DictValidationMixin:
+    """
+    DictField validation in MongoEngine requires default alias and permissions to access DB version:
+    https://github.com/MongoEngine/mongoengine/issues/2239
+    This is a stripped down implementation that does not require any of the above and implies Mongo ver 3.6+
+    """
+
+    def _safe_validate(self: DictField, value):
+        if not isinstance(value, dict):
+            self.error("Only dictionaries may be used in a DictField")
+
+        if key_not_string(value):
+            msg = "Invalid dictionary key - documents must have only string keys"
+            self.error(msg)
+
+        if key_starts_with_dollar(value):
+            self.error(
+                'Invalid dictionary key name - keys may not startswith "$" characters'
+            )
+        super(DictField, self).validate(value)
+
+
+class SafeMapField(MapField, DictValidationMixin):
     def validate(self, value):
-        super(SafeMapField, self).validate(value)
+        self._safe_validate(value)
 
         if contains_empty_key(value):
             self.error("Empty keys are not allowed in a MapField")
 
 
-class SafeDictField(DictField):
+class NullableStringField(StringField):
     def validate(self, value):
-        super(SafeDictField, self).validate(value)
+        if value is None:
+            return
+        super(NullableStringField, self).validate(value)
+
+
+class SafeDictField(DictField, DictValidationMixin):
+    def validate(self, value):
+        self._safe_validate(value)
 
         if contains_empty_key(value):
             self.error("Empty keys are not allowed in a DictField")
+
+
+class SafeSortedListField(SortedListField):
+    """
+    SortedListField that does not raise an error in case items are not comparable
+    (in which case they will be sorted by their string representation)
+    """
+
+    def to_mongo(self, *args, **kwargs):
+        try:
+            return super(SafeSortedListField, self).to_mongo(*args, **kwargs)
+        except TypeError:
+            return self._safe_to_mongo(*args, **kwargs)
+
+    def _safe_to_mongo(self, value, use_db_field=True, fields=None):
+        value = super(SortedListField, self).to_mongo(value, use_db_field, fields)
+        if self._ordering is not None:
+
+            def key(v):
+                return str(itemgetter(self._ordering)(v))
+
+        else:
+            key = str
+        return sorted(value, key=key, reverse=self._order_reverse)
+
+
+class UnionField(DynamicField):
+    def __init__(self, types, *args, **kwargs):
+        super(UnionField, self).__init__(*args, **kwargs)
+        self.types: Tuple[Type] = tuple(types)
+
+    def validate(self, value, clean=True):
+        if not isinstance(value, self.types):
+            type_names = [t.__name__ for t in self.types]
+            expected = " or ".join(
+                filter(
+                    None,
+                    (", ".join(type_names[:-1]), type_names[-1]))
+            )
+            self.error(
+                f"Expected {expected}, got {type(value).__name__}: {value}"
+            )
+        super(UnionField, self).validate(value, clean)