Version bump to 0.17.0

Move legacy Trains Server docker-compose files to docker/legacy/trains-server

.gitignore

@@ -1,3 +1,4 @@
+syntax: glob
 .idea
 apierrors/errors
 static/build.json
@@ -18,3 +19,4 @@ build
 dist
 code.tar.gz
 server/schema/services/_cache.json
+server/apierrors/errors/*

LICENSE

@@ -1,7 +1,7 @@
                      Server Side Public License
                      VERSION 1, OCTOBER 16, 2018
 
-                    Copyright © 2018 MongoDB, Inc.
+                    Copyright © 2019 allegro.ai, Inc.
 
   Everyone is permitted to copy and distribute verbatim copies of this
   license document, but changing it is not allowed.

README.md

@@ -1,246 +1,230 @@
-# TRAINS Server
+<div align="center">
 
-##  Magic Version Control & Experiment Manager for AI
+<img src="docs/clearml_server_logo.png" width="250px">
 
-## Introduction
+**ClearML - Auto-Magical Suite of tools to streamline your ML workflow 
+</br>Experiment Manager, ML-Ops and Data-Management**
 
-The **trains-server** is the infrastructure for [trains](https://github.com/allegroai/trains).
-It allows multiple users to collaborate and manage their experiments. 
- 
-The **trains-server** contains the following components:
+[![GitHub license](https://img.shields.io/badge/license-SSPL-green.svg)](https://img.shields.io/badge/license-SSPL-green.svg)
+[![Python versions](https://img.shields.io/badge/python-3.6%20%7C%203.7-blue.svg)](https://img.shields.io/badge/python-3.6%20%7C%203.7-blue.svg)
+[![GitHub version](https://img.shields.io/github/release-pre/allegroai/trains-server.svg)](https://img.shields.io/github/release-pre/allegroai/trains-server.svg)
 
-* the Web-App which is a single-page UI for experiment management and browsing
-* a REST interface for: 
-    * documenting and logging experiment information, statistics and results
-    * querying experiments history, logs and results
-* a locally-hosted file server for storing images and models making them easily accessible using the Web-App
+</div>
 
-You can quickly setup your **trains-server** using a pre-built Docker image (see [Installation](#installation)).
+---
+<div align="center">
 
-When new releases are available, you can upgrade your pre-built Docker image (see [Upgrade](#upgrade)). 
+**v0.16 Upgrade Notice**
 
-The **trains-server's** code is freely available [here](https://github.com/allegroai/trains-server).
+</div>
 
-## System diagram
+In v0.16, the Elasticsearch subsystem of ClearML Server has been upgraded from version 5.6 to version 7.6. This change necessitates the migration of the database contents to accommodate the change in index structure across the different versions.  
 
-<pre>
- TRAINS-server
- +--------------------------------------------------------------------+
- |                                                                    |
- |   Server Docker                   Elastic Docker     Mongo Docker  |
- |  +-------------------------+     +---------------+  +------------+ |
- |  |     Pythonic Server     |     |               |  |            | |
- |  |   +-----------------+   |     | ElasticSearch |  |  MongoDB   | |
- |  |   |   WEB server    |   |     |               |  |            | |
- |  |   |   Port 8080     |   |     |               |  |            | |
- |  |   +--------+--------+   |     |               |  |            | |
- |  |            |            |     |               |  |            | |
- |  |   +--------+--------+   |     |               |  |            | |
- |  |   |   API server    +----------------------------+            | |
- |  |   |   Port 8008     +---------+               |  |            | |
- |  |   +-----------------+   |     +-------+-------+  +-----+------+ |
- |  |                         |             |                |        |
- |  |   +-----------------+   |         +---+----------------+------+ |
- |  |   |   File Server   +-------+     |    Host Storage           | |
- |  |   |   Port 8081     |   |   +-----+                           | |
- |  |   +-----------------+   |         +---------------------------+ |
- |  +------------+------------+                                       |
- +---------------|----------------------------------------------------+
-                 |HTTP
-                 +--------+
- GPU Machine              |
- +------------------------|-------------------------------------------+
- |     +------------------|--------------+                            |
- |     |  Training        |              |    +---------------------+ |
- |     |  Code        +---+------------+ |    | trains configuration| |
- |     |              | TRAINS         | |    | ~/trains.conf       | |
- |     |              |                +------+                     | |
- |     |              +----------------+ |    +---------------------+ |
- |     +---------------------------------+                            |
- +--------------------------------------------------------------------+
-</pre>
+Follow [this procedure](https://allegro.ai/docs/deploying_trains/trains_server_es7_migration/) to migrate existing data.
 
-## Installation
+---
 
-This section contains the instructions to setup and launch a pre-built Docker image for the **trains-server**.
+### ClearML Server
+#### *Formerly known as Trains Server*
 
-**Note**: This Docker image was tested with Linux, only. For Windows users, we recommend running the server
-on a Linux virtual machine.
+The **ClearML Server** is the backend service infrastructure for [ClearML](https://github.com/allegroai/clearml).
+It allows multiple users to collaborate and manage their experiments.
+By default, **ClearML** is set up to work with the **ClearML** demo server, which is open to anyone and resets periodically.
+In order to host your own server, you will need to launch the **ClearML Server** and point **ClearML** to it.
+
+The **ClearML Server** contains the following components:
+
+* The **ClearML** Web-App, a single-page UI for experiment management and browsing
+* RESTful API for:
+    * Documenting and logging experiment information, statistics and results
+    * Querying experiments history, logs and results
+* Locally-hosted file server for storing images and models making them easily accessible using the Web-App
+
+You can quickly [deploy](#launching-the-clearml-server)  your **ClearML Server** using Docker, AWS EC2 AMI, or Kubernetes. 
+
+## System design
+
+
+![Alt Text](https://allegro.ai/clearml/docs/_images/ClearML_Server_Diagram.png)
+
+The **ClearML Server** has two supported configurations:
+- Single IP (domain) with the following open ports
+    - Web application on port 8080
+    - API service on port 8008
+    - File storage service on port 8081
+
+- Sub-Domain configuration with default http/s ports (80 or 443)
+    - Web application on sub-domain: app.\*.\*
+    - API service on sub-domain: api.\*.\*
+    - File storage service on sub-domain: files.\*.\*
+    
+## Launching The ClearML Server
 
 ### Prerequisites
 
-You must be logged in as a user with sudo privileges.
- 
-### Setup
+The ports 8080/8081/8008 must be available for the **ClearML Server** services.
+   
+For example, to see if port `8080` is in use:
 
-#### Step 1. Install Docker CE
+* Linux or macOS: 
+   
+        sudo lsof -Pn -i4 | grep :8080 | grep LISTEN
 
-You must install Docker to run the pre-packaged **trains-server**.
+* Windows:
 
-* For [Ubuntu](https://docs.docker.com/install/linux/docker-ce/ubuntu/) / Mint (x86_64/amd64):
+        netstat -an |find /i "8080"
+   
+### Launching   
+    
+Launch The **ClearML Server** in any of the following formats:
 
-```bash
-sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common
-curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
-. /etc/os-release
-sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $UBUNTU_CODENAME stable"
-sudo apt-get update
-sudo apt-get install -y docker-ce
-```
+- Pre-built [AWS EC2 AMI](https://allegro.ai/docs/deploying_trains/trains_server_aws_ec2_ami/)
+- Pre-built [GCP Custom Image](https://allegro.ai/docs/deploying_trains/trains_server_gcp/)
+- Pre-built Docker Image
+    - [Linux](https://allegro.ai/docs/deploying_trains/trains_server_linux_mac/)
+    - [macOS](https://allegro.ai/docs/deploying_trains/trains_server_linux_mac/)
+    - [Windows 10](https://allegro.ai/docs/deploying_trains/trains_server_win/)
+- Kubernetes    
+    - [Kubernetes Helm](https://allegro.ai/docs/deploying_trains/trains_server_kubernetes_helm/)
+    - Manual [Kubernetes installation](https://allegro.ai/docs/deploying_trains/trains_server_kubernetes/)
 
-* For other operating systems, see [Supported platforms](https://docs.docker.com/install//#support) in the Docker documentation for instructions.
+## Connecting ClearML to your ClearML Server
 
-#### Step 2. Setup the Docker daemon
+By default, the **ClearML** client is set up to work with the [**ClearML** demo server](https://demoapp.demo.clear.ml/).  
+To have the **ClearML** client use your **ClearML Server** instead:
+- Run the `clearml-init` command for an interactive setup.
+- Or manually edit `~/clearml.conf` file, making sure the server settings (`api_server`, `web_server`, `file_server`) are configured correctly, for example:
 
-To run the ElasticSearch Docker container, you must setup the Docker daemon by modifing the default 
-values required by Elastic in your Docker configuration file
-that are used by the **trains-server**. We provide instructions for the most common Docker configuration files.
+        api {
+            # API server on port 8008
+            api_server: "http://localhost:8008"
 
-You must edit or create a Docker configuration file:
+            # web_server on port 8080
+            web_server: "http://localhost:8080"
 
-* If your Docker configuration file is `/etc/sysconfig/docker`, edit it. 
-
-    Add the options in quotes to the available arguments in the `OPTIONS` section:
-
-```bash
-OPTIONS="--default-ulimit nofile=1024:65536 --default-ulimit memlock=-1:-1"
-```
-
-* Otherwise, edit `/etc/docker/daemon.json` (if it exists) or create it (if it does not exist).
-
-    Add or modify the `defaults-ulimits` section as shown below. Be sure your configuration file contains the `nofile` and `memlock` sub-sections and values shown. 
-
-    **Note**: Your configuration file may contain other sections. If so, confirm that the sections are separated by commas. For more information about Docker configuration files, see an [Daemon configuration file](https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file) in the Docker documentation.
-
-    The **trains-server** required defaults values are:
-
-```json
-{
-    "default-ulimits": {
-        "nofile": {
-            "name": "nofile",
-            "hard": 65536,
-            "soft": 1024
-        },
-        "memlock":
-        {
-            "name": "memlock",
-            "soft": -1,
-            "hard": -1
+            # file server on port 8081
+            files_server: "http://localhost:8081"
         }
-    }
-}
-```
 
-#### Step 3. Restart the Docker daemon
+**Note**: If you have set up your **ClearML Server** in a sub-domain configuration, then there is no need to specify a port number,
+it will be inferred from the http/s scheme.
 
-You must restart the Docker daemon after modifying the configuration file:
+After launching the **ClearML Server** and configuring the **ClearML** client to use the **ClearML Server**,
+you can [use](https://github.com/allegroai/clearml) **ClearML** in your experiments and view them in your **ClearML Server** web server,
+for example http://localhost:8080.  
+For more information about the ClearML client, see [**ClearML**](https://github.com/allegroai/clearml).
 
-```bash
-sudo service docker stop
-sudo service docker start
-```
+## ClearML-Agent Services  <a name="services"></a> 
 
-#### Step 4. Set the Maximum Number of Memory Map Areas
+As of version 0.15 of **ClearML Server**, dockerized deployment includes a **ClearML-Agent Services** container running as 
+part of the docker container collection.
 
-The maximum number of memory map areas a process can use is defined
-using the `vm.max_map_count` kernel setting. 
+ClearML-Agent Services is an extension of ClearML-Agent that provides the ability to launch long-lasting jobs 
+that previously had to be executed on local / dedicated machines. It allows a single agent to 
+launch multiple dockers (Tasks) for different use cases. To name a few use cases, auto-scaler service (spinning instances 
+when the need arises and the budget allows), Controllers (Implementing pipelines and more sophisticated DevOps logic),
+Optimizer (such as Hyper-parameter Optimization or sweeping), and Application (such as interactive Bokeh apps for 
+increased data transparency)
 
-Elastic requires that `vm.max_map_count` to be at least 262144.
+ClearML-Agent Services container will spin **any** task enqueued into the dedicated `services` queue. 
+Every task launched by ClearML-Agent Services  will be registered as a new node in the system, 
+providing tracking and transparency capabilities.  
+You can also run the ClearML-Agent Services manually, see details in [ClearML-agent services mode](https://github.com/allegroai/clearml-agent#clearml-agent-services-mode-)
 
-* For CentOS 7, Ubuntu 16.04, Mint 18.3, Ubuntu 18.04 and Mint 19 users, we tested the following commands to set
-`vm.max_map_count`:
+**Note**: It is the user's responsibility to make sure the proper tasks are pushed into the `services` queue. 
+Do not enqueue training / inference tasks into the `services` queue, as it will put unnecessary load on the server.
 
-```bash
-sudo echo "vm.max_map_count=262144" > /tmp/99-trains.conf
-sudo mv /tmp/99-trains.conf /etc/sysctl.d/99-trains.conf
-sudo sysctl -w vm.max_map_count=262144
-```
+## Advanced Functionality
 
-* For information about setting this parameter on other systems, see the [elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-prod-mode) documentation.
+The **ClearML Server** provides a few additional useful features, which can be manually enabled:
+ 
+* [Web login authentication](https://allegro.ai/clearml/docs/deploying_clearml/clearml_server_config/#web-login-authentication)
+* [Non-responsive experiments watchdog](https://allegro.ai/clearml/docs/deploying_clearml/clearml_server_config/#task_watchdog)  
 
-#### Step 5. Choose a Data Directory
+## Restarting ClearML Server
 
-You must choose a directory on your system in which all data maintained by the **trains-server** is stored,
-create that directory, and set its permissions. The data stored in that directory includes the database, uploaded files and logs.
+To restart the **ClearML Server**, you must first stop the containers, and then restart them.
 
-For example, if your data directory is `/opt/trains`, then use the following command:
+   ```bash
+   docker-compose down
+   docker-compose -f docker-compose.yml up
+   ```
 
-```bash
-sudo mkdir -p /opt/trains/data/elastic && sudo chown -R 1000:1000 /opt/trains
-```
+## Upgrading <a name="upgrade"></a>
 
-### Launching Docker Containers
+**ClearML Server** releases are also reflected in the [docker compose configuration file](https://github.com/allegroai/trains-server/blob/master/docker/docker-compose.yml).  
+We strongly encourage you to keep your **ClearML Server** up to date, by keeping up with the current release.
 
-Launch the Docker containers. For example, if your data directory is `\opt\trains`,
-then use the following commands:
+**Note**: The following upgrade instructions use the Linux OS as an example.
 
-```bash
-sudo docker run -d --restart="always" --name="trains-elastic" -e "ES_JAVA_OPTS=-Xms2g -Xmx2g" -e "bootstrap.memory_lock=true" -e "cluster.name=trains" -e "discovery.zen.minimum_master_nodes=1" -e "node.name=trains" -e "script.inline=true" -e "script.update=true" -e "thread_pool.bulk.queue_size=2000" -e "thread_pool.search.queue_size=10000" -e "xpack.security.enabled=false" -e "xpack.monitoring.enabled=false" -e "cluster.routing.allocation.node_initial_primaries_recoveries=500" -e "node.ingest=true" -e "http.compression_level=7" -e "reindex.remote.whitelist=*.*" -e "script.painless.regex.enabled=true" --network="host" -v /opt/trains/data/elastic:/usr/share/elasticsearch/data docker.elastic.co/elasticsearch/elasticsearch:5.6.16
-```
+To upgrade your existing **ClearML Server** deployment:
 
-```bash
-sudo docker run -d --restart="always" --name="trains-mongo" -v /opt/trains/data/mongo/db:/data/db -v /opt/trains/data/mongo/configdb:/data/configdb --network="host" mongo:3.6.5
-```
+1. Shut down the docker containers
+   ```bash
+   docker-compose down
+   ```
 
-```bash
-sudo docker run -d --restart="always" --name="trains-fileserver" --network="host" -v /opt/trains/logs:/var/log/trains -v /opt/trains/data/fileserver:/mnt/fileserver allegroai/trains:latest fileserver
-```
+1. We highly recommend backing up your data directory before upgrading.
 
-```bash
-sudo docker run -d --restart="always" --name="trains-apiserver" --network="host" -v /opt/trains/logs:/var/log/trains allegroai/trains:latest apiserver
-```
+   Assuming your data directory is `/opt/clearml`, to archive all data into `~/clearml_backup.tgz` execute:
 
-```bash
-sudo docker run -d --restart="always" --name="trains-webserver" --network="host" -v /opt/trains/logs:/var/log/trains allegroai/trains:latest webserver
-```
+   ```bash
+   sudo tar czvf ~/clearml_backup.tgz /opt/clearml/data
+   ```    
 
-After the **trains-server** Dockers are up, the following are available:
+   <details>
+   <summary>Restore instructions:</summary>
 
-* API server on port `8008`
-* Web server on port `8080`
-* File server on port `8081`
+   To restore this example backup, execute:
+   ```bash
+   sudo rm -R /opt/clearml/data
+   sudo tar -xzf ~/clearml_backup.tgz -C /opt/clearml/data
+   ```
+   </details>
 
-## Upgrade
+1. Download the latest `docker-compose.yml` file.
 
-We are constantly updating, improving and adding to the **trains-server**.
-New releases will include new pre-built Docker images.
-When we release a new version and include a new pre-built Docker image for it, upgrade as follows:
+   ```bash
+   curl https://raw.githubusercontent.com/allegroai/trains-server/master/docker/docker-compose.yml -o docker-compose.yml 
+   ```
 
-1. Shut down and remove each of your Docker instances using the following commands:
+1. Configure the ClearML-Agent Services (not supported on Windows installation). 
+   If `TRAINS_HOST_IP` is not provided, ClearML-Agent Services will use the external 
+   public address of the **ClearML Server**. If `TRAINS_AGENT_GIT_USER` / `TRAINS_AGENT_GIT_PASS` are not provided, 
+   the ClearML-Agent Services will not be able to access any private repositories for running service tasks.
+   
+   ```bash
+   export TRAINS_HOST_IP=server_host_ip_here
+   export TRAINS_AGENT_GIT_USER=git_username_here
+   export TRAINS_AGENT_GIT_PASS=git_password_here
+   ```
 
-        sudo docker stop <docker-name>
-        sudo docker rm -v <docker-name>
-    
-    The Docker names are (see [Launching Docker images](##launching-docker-images)):
-    
-       * `trains-elastic`
-       * `trains-mongo`
-       * `trains-fileserver`
-       * `trains-apiserver`
-       * `trains-webserver`
+1. Spin up the docker containers, it will automatically pull the latest **ClearML Server** build    
+   ```bash
+   docker-compose -f docker-compose.yml pull
+   docker-compose -f docker-compose.yml up
+   ```
 
-2. We highly recommend backing up your data directory!. A simple way to do that is using `tar`:
+**\* If something went wrong along the way, check our FAQ: [Common Docker Upgrade Errors](https://allegro.ai/clearml/docs/docs/faq/faq.html).**
 
-    For example, if your data directory is `/opt/trains`, use the following command:
-    
-        sudo tar czvf ~/trains_backup.tgz /opt/trains/data
-    
-    This back ups all data to an archive in your home directory. 
-    
-    To restore this example backup, use the following command:
-    
-        sudo rm -R /opt/trains/data
-        sudo tar -xzf ~/trains_backup.tgz -C /opt/trains/data
 
-3. Launch the newly released Docker image (see [Launching Docker images](#Launching-docker-images)).
+## Community & Support
+
+If you have any questions, look to the ClearML [FAQ](https://allegro.ai/clearml/docs/docs/faq/faq.html), or
+tag your questions on [stackoverflow](https://stackoverflow.com/questions/tagged/trains) with '**trains**' tag.
+
+For feature requests or bug reports, please use [GitHub issues](https://github.com/allegroai/trains-server/issues).
+
+Additionally, you can always find us at *clearml@allegro.ai*
 
 ## License
 
 [Server Side Public License v1.0](https://github.com/mongodb/mongo/blob/master/LICENSE-Community.txt)
 
-**trains-server** relies *heavily* on both [MongoDB](https://github.com/mongodb/mongo) and [ElasticSearch](https://github.com/elastic/elasticsearch).
-With the recent changes in both MongoDB's and ElasticSearch's OSS license, we feel it is our job as a community to support the projects we love and cherish.
-We feel the cause for the license change in both cases is more than just, and chose [SSPL](https://www.mongodb.com/licensing/server-side-public-license) because it is the more general and flexible of the two.
+The **ClearML Server** relies on both [MongoDB](https://github.com/mongodb/mongo) and [ElasticSearch](https://github.com/elastic/elasticsearch).
+With the recent changes in both MongoDB's and ElasticSearch's OSS license, we feel it is our responsibility as a
+member of the community to support the projects we love and cherish.
+We believe the cause for the license change in both cases is more than just,
+and chose [SSPL](https://www.mongodb.com/licensing/server-side-public-license) because it is the more general and flexible of the two licenses.
 
 This is our way to say - we support you guys!

apiserver/LICENSE

@@ -0,0 +1,557 @@
+                     Server Side Public License
+                     VERSION 1, OCTOBER 16, 2018
+
+                    Copyright © 2019 allegro.ai, Inc.
+
+  Everyone is permitted to copy and distribute verbatim copies of this
+  license document, but changing it is not allowed.
+
+                       TERMS AND CONDITIONS
+
+  0. Definitions.
+  
+  “This License” refers to Server Side Public License.
+
+  “Copyright” also means copyright-like laws that apply to other kinds of
+  works, such as semiconductor masks.
+
+  “The Program” refers to any copyrightable work licensed under this
+  License.  Each licensee is addressed as “you”. “Licensees” and
+  “recipients” may be individuals or organizations.
+
+  To “modify” a work means to copy from or adapt all or part of the work in
+  a fashion requiring copyright permission, other than the making of an
+  exact copy. The resulting work is called a “modified version” of the
+  earlier work or a work “based on” the earlier work.
+
+  A “covered work” means either the unmodified Program or a work based on
+  the Program.
+
+  To “propagate” a work means to do anything with it that, without
+  permission, would make you directly or secondarily liable for
+  infringement under applicable copyright law, except executing it on a
+  computer or modifying a private copy. Propagation includes copying,
+  distribution (with or without modification), making available to the
+  public, and in some countries other activities as well.
+
+  To “convey” a work means any kind of propagation that enables other
+  parties to make or receive copies. Mere interaction with a user through a
+  computer network, with no transfer of a copy, is not conveying.
+
+  An interactive user interface displays “Appropriate Legal Notices” to the
+  extent that it includes a convenient and prominently visible feature that
+  (1) displays an appropriate copyright notice, and (2) tells the user that
+  there is no warranty for the work (except to the extent that warranties
+  are provided), that licensees may convey the work under this License, and
+  how to view a copy of this License. If the interface presents a list of
+  user commands or options, such as a menu, a prominent item in the list
+  meets this criterion.
+
+  1. Source Code.
+
+  The “source code” for a work means the preferred form of the work for
+  making modifications to it. “Object code” means any non-source form of a
+  work.
+
+  A “Standard Interface” means an interface that either is an official
+  standard defined by a recognized standards body, or, in the case of
+  interfaces specified for a particular programming language, one that is
+  widely used among developers working in that language.  The “System
+  Libraries” of an executable work include anything, other than the work as
+  a whole, that (a) is included in the normal form of packaging a Major
+  Component, but which is not part of that Major Component, and (b) serves
+  only to enable use of the work with that Major Component, or to implement
+  a Standard Interface for which an implementation is available to the
+  public in source code form. A “Major Component”, in this context, means a
+  major essential component (kernel, window system, and so on) of the
+  specific operating system (if any) on which the executable work runs, or
+  a compiler used to produce the work, or an object code interpreter used
+  to run it.
+
+  The “Corresponding Source” for a work in object code form means all the
+  source code needed to generate, install, and (for an executable work) run
+  the object code and to modify the work, including scripts to control
+  those activities. However, it does not include the work's System
+  Libraries, or general-purpose tools or generally available free programs
+  which are used unmodified in performing those activities but which are
+  not part of the work. For example, Corresponding Source includes
+  interface definition files associated with source files for the work, and
+  the source code for shared libraries and dynamically linked subprograms
+  that the work is specifically designed to require, such as by intimate
+  data communication or control flow between those subprograms and other
+  parts of the work.
+
+  The Corresponding Source need not include anything that users can
+  regenerate automatically from other parts of the Corresponding Source.
+
+  The Corresponding Source for a work in source code form is that same work.
+
+  2. Basic Permissions.
+
+  All rights granted under this License are granted for the term of
+  copyright on the Program, and are irrevocable provided the stated
+  conditions are met. This License explicitly affirms your unlimited
+  permission to run the unmodified Program, subject to section 13. The
+  output from running a covered work is covered by this License only if the
+  output, given its content, constitutes a covered work. This License
+  acknowledges your rights of fair use or other equivalent, as provided by
+  copyright law.  Subject to section 13, you may make, run and propagate
+  covered works that you do not convey, without conditions so long as your
+  license otherwise remains in force. You may convey covered works to
+  others for the sole purpose of having them make modifications exclusively
+  for you, or provide you with facilities for running those works, provided
+  that you comply with the terms of this License in conveying all
+  material for which you do not control copyright. Those thus making or
+  running the covered works for you must do so exclusively on your
+  behalf, under your direction and control, on terms that prohibit them
+  from making any copies of your copyrighted material outside their
+  relationship with you.
+
+  Conveying under any other circumstances is permitted solely under the
+  conditions stated below. Sublicensing is not allowed; section 10 makes it
+  unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+  measure under any applicable law fulfilling obligations under article 11
+  of the WIPO copyright treaty adopted on 20 December 1996, or similar laws
+  prohibiting or restricting circumvention of such measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+  circumvention of technological measures to the extent such circumvention is
+  effected by exercising rights under this License with respect to the
+  covered work, and you disclaim any intention to limit operation or
+  modification of the work as a means of enforcing, against the work's users,
+  your or third parties' legal rights to forbid circumvention of
+  technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+  receive it, in any medium, provided that you conspicuously and
+  appropriately publish on each copy an appropriate copyright notice; keep
+  intact all notices stating that this License and any non-permissive terms
+  added in accord with section 7 apply to the code; keep intact all notices
+  of the absence of any warranty; and give all recipients a copy of this
+  License along with the Program.  You may charge any price or no price for
+  each copy that you convey, and you may offer support or warranty
+  protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+  produce it from the Program, in the form of source code under the terms
+  of section 4, provided that you also meet all of these conditions:
+
+    a) The work must carry prominent notices stating that you modified it,
+    and giving a relevant date.
+
+    b) The work must carry prominent notices stating that it is released
+    under this License and any conditions added under section 7. This
+    requirement modifies the requirement in section 4 to “keep intact all
+    notices”.
+
+    c) You must license the entire work, as a whole, under this License to
+    anyone who comes into possession of a copy. This License will therefore
+    apply, along with any applicable section 7 additional terms, to the
+    whole of the work, and all its parts, regardless of how they are
+    packaged. This License gives no permission to license the work in any
+    other way, but it does not invalidate such permission if you have
+    separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your work
+    need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+  works, which are not by their nature extensions of the covered work, and
+  which are not combined with it such as to form a larger program, in or on
+  a volume of a storage or distribution medium, is called an “aggregate” if
+  the compilation and its resulting copyright are not used to limit the
+  access or legal rights of the compilation's users beyond what the
+  individual works permit. Inclusion of a covered work in an aggregate does
+  not cause this License to apply to the other parts of the aggregate.
+  
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms of
+  sections 4 and 5, provided that you also convey the machine-readable
+  Corresponding Source under the terms of this License, in one of these
+  ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium customarily
+    used for software interchange.
+   
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a written
+    offer, valid for at least three years and valid for as long as you
+    offer spare parts or customer support for that product model, to give
+    anyone who possesses the object code either (1) a copy of the
+    Corresponding Source for all the software in the product that is
+    covered by this License, on a durable physical medium customarily used
+    for software interchange, for a price no more than your reasonable cost
+    of physically performing this conveying of source, or (2) access to
+    copy the Corresponding Source from a network server at no charge.
+   
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source. This alternative is
+    allowed only occasionally and noncommercially, and only if you received
+    the object code with such an offer, in accord with subsection 6b.
+   
+    d) Convey the object code by offering access from a designated place
+    (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge. You need not require recipients to copy the
+    Corresponding Source along with the object code. If the place to copy
+    the object code is a network server, the Corresponding Source may be on
+    a different server (operated by you or a third party) that supports
+    equivalent copying facilities, provided you maintain clear directions
+    next to the object code saying where to find the Corresponding Source.
+    Regardless of what server hosts the Corresponding Source, you remain
+    obligated to ensure that it is available for as long as needed to
+    satisfy these requirements.
+   
+    e) Convey the object code using peer-to-peer transmission, provided you
+    inform other peers where the object code and Corresponding Source of
+    the work are being offered to the general public at no charge under
+    subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+  from the Corresponding Source as a System Library, need not be included
+  in conveying the object code work.
+
+  A “User Product” is either (1) a “consumer product”, which means any
+  tangible personal property which is normally used for personal, family,
+  or household purposes, or (2) anything designed or sold for incorporation
+  into a dwelling. In determining whether a product is a consumer product,
+  doubtful cases shall be resolved in favor of coverage. For a particular
+  product received by a particular user, “normally used” refers to a
+  typical or common use of that class of product, regardless of the status
+  of the particular user or of the way in which the particular user
+  actually uses, or expects or is expected to use, the product. A product
+  is a consumer product regardless of whether the product has substantial
+  commercial, industrial or non-consumer uses, unless such uses represent
+  the only significant mode of use of the product.
+
+  “Installation Information” for a User Product means any methods,
+  procedures, authorization keys, or other information required to install
+  and execute modified versions of a covered work in that User Product from
+  a modified version of its Corresponding Source. The information must
+  suffice to ensure that the continued functioning of the modified object
+  code is in no case prevented or interfered with solely because
+  modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+  specifically for use in, a User Product, and the conveying occurs as part
+  of a transaction in which the right of possession and use of the User
+  Product is transferred to the recipient in perpetuity or for a fixed term
+  (regardless of how the transaction is characterized), the Corresponding
+  Source conveyed under this section must be accompanied by the
+  Installation Information. But this requirement does not apply if neither
+  you nor any third party retains the ability to install modified object
+  code on the User Product (for example, the work has been installed in
+  ROM).
+
+  The requirement to provide Installation Information does not include a
+  requirement to continue to provide support service, warranty, or updates
+  for a work that has been modified or installed by the recipient, or for
+  the User Product in which it has been modified or installed. Access
+  to a network may be denied when the modification itself materially
+  and adversely affects the operation of the network or violates the
+  rules and protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided, in
+  accord with this section must be in a format that is publicly documented
+  (and with an implementation available to the public in source code form),
+  and must require no special password or key for unpacking, reading or
+  copying.
+
+  7. Additional Terms.
+
+  “Additional permissions” are terms that supplement the terms of this
+  License by making exceptions from one or more of its conditions.
+  Additional permissions that are applicable to the entire Program shall be
+  treated as though they were included in this License, to the extent that
+  they are valid under applicable law. If additional permissions apply only
+  to part of the Program, that part may be used separately under those
+  permissions, but the entire Program remains governed by this License
+  without regard to the additional permissions.  When you convey a copy of
+  a covered work, you may at your option remove any additional permissions
+  from that copy, or from any part of it. (Additional permissions may be
+  written to require their own removal in certain cases when you modify the
+  work.) You may place additional permissions on material, added by you to
+  a covered work, for which you have or can give appropriate copyright
+  permission.
+
+  Notwithstanding any other provision of this License, for material you add
+  to a covered work, you may (if authorized by the copyright holders of
+  that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some trade
+    names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that material
+    by anyone who conveys the material (or modified versions of it) with
+    contractual assumptions of liability to the recipient, for any
+    liability that these contractual assumptions directly impose on those
+    licensors and authors.
+
+  All other non-permissive additional terms are considered “further
+  restrictions” within the meaning of section 10. If the Program as you
+  received it, or any part of it, contains a notice stating that it is
+  governed by this License along with a term that is a further restriction,
+  you may remove that term. If a license document contains a further
+  restriction but permits relicensing or conveying under this License, you
+  may add to a covered work material governed by the terms of that license
+  document, provided that the further restriction does not survive such
+  relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you must
+  place, in the relevant source files, a statement of the additional terms
+  that apply to those files, or a notice indicating where to find the
+  applicable terms.  Additional terms, permissive or non-permissive, may be
+  stated in the form of a separately written license, or stated as
+  exceptions; the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+  provided under this License. Any attempt otherwise to propagate or modify
+  it is void, and will automatically terminate your rights under this
+  License (including any patent licenses granted under the third paragraph
+  of section 11).
+
+  However, if you cease all violation of this License, then your license
+  from a particular copyright holder is reinstated (a) provisionally,
+  unless and until the copyright holder explicitly and finally terminates
+  your license, and (b) permanently, if the copyright holder fails to
+  notify you of the violation by some reasonable means prior to 60 days
+  after the cessation.
+
+  Moreover, your license from a particular copyright holder is reinstated
+  permanently if the copyright holder notifies you of the violation by some
+  reasonable means, this is the first time you have received notice of
+  violation of this License (for any work) from that copyright holder, and
+  you cure the violation prior to 30 days after your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+  licenses of parties who have received copies or rights from you under
+  this License. If your rights have been terminated and not permanently
+  reinstated, you do not qualify to receive new licenses for the same
+  material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or run a
+  copy of the Program. Ancillary propagation of a covered work occurring
+  solely as a consequence of using peer-to-peer transmission to receive a
+  copy likewise does not require acceptance. However, nothing other than
+  this License grants you permission to propagate or modify any covered
+  work. These actions infringe copyright if you do not accept this License.
+  Therefore, by modifying or propagating a covered work, you indicate your
+  acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically receives
+  a license from the original licensors, to run, modify and propagate that
+  work, subject to this License. You are not responsible for enforcing
+  compliance by third parties with this License.
+
+  An “entity transaction” is a transaction transferring control of an
+  organization, or substantially all assets of one, or subdividing an
+  organization, or merging organizations. If propagation of a covered work
+  results from an entity transaction, each party to that transaction who
+  receives a copy of the work also receives whatever licenses to the work
+  the party's predecessor in interest had or could give under the previous
+  paragraph, plus a right to possession of the Corresponding Source of the
+  work from the predecessor in interest, if the predecessor has it or can
+  get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the rights
+  granted or affirmed under this License. For example, you may not impose a
+  license fee, royalty, or other charge for exercise of rights granted
+  under this License, and you may not initiate litigation (including a
+  cross-claim or counterclaim in a lawsuit) alleging that any patent claim
+  is infringed by making, using, selling, offering for sale, or importing
+  the Program or any portion of it.
+
+  11. Patents.
+
+  A “contributor” is a copyright holder who authorizes use under this
+  License of the Program or a work on which the Program is based. The work
+  thus licensed is called the contributor's “contributor version”.
+
+  A contributor's “essential patent claims” are all patent claims owned or
+  controlled by the contributor, whether already acquired or hereafter
+  acquired, that would be infringed by some manner, permitted by this
+  License, of making, using, or selling its contributor version, but do not
+  include claims that would be infringed only as a consequence of further
+  modification of the contributor version. For purposes of this definition,
+  “control” includes the right to grant patent sublicenses in a manner
+  consistent with the requirements of this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+  patent license under the contributor's essential patent claims, to make,
+  use, sell, offer for sale, import and otherwise run, modify and propagate
+  the contents of its contributor version.
+
+  In the following three paragraphs, a “patent license” is any express
+  agreement or commitment, however denominated, not to enforce a patent
+  (such as an express permission to practice a patent or covenant not to
+  sue for patent infringement). To “grant” such a patent license to a party
+  means to make such an agreement or commitment not to enforce a patent
+  against the party.
+
+  If you convey a covered work, knowingly relying on a patent license, and
+  the Corresponding Source of the work is not available for anyone to copy,
+  free of charge and under the terms of this License, through a publicly
+  available network server or other readily accessible means, then you must
+  either (1) cause the Corresponding Source to be so available, or (2)
+  arrange to deprive yourself of the benefit of the patent license for this
+  particular work, or (3) arrange, in a manner consistent with the
+  requirements of this License, to extend the patent license to downstream
+  recipients. “Knowingly relying” means you have actual knowledge that, but
+  for the patent license, your conveying the covered work in a country, or
+  your recipient's use of the covered work in a country, would infringe
+  one or more identifiable patents in that country that you have reason
+  to believe are valid.
+
+  If, pursuant to or in connection with a single transaction or
+  arrangement, you convey, or propagate by procuring conveyance of, a
+  covered work, and grant a patent license to some of the parties receiving
+  the covered work authorizing them to use, propagate, modify or convey a
+  specific copy of the covered work, then the patent license you grant is
+  automatically extended to all recipients of the covered work and works
+  based on it.
+
+  A patent license is “discriminatory” if it does not include within the
+  scope of its coverage, prohibits the exercise of, or is conditioned on
+  the non-exercise of one or more of the rights that are specifically
+  granted under this License. You may not convey a covered work if you are
+  a party to an arrangement with a third party that is in the business of
+  distributing software, under which you make payment to the third party
+  based on the extent of your activity of conveying the work, and under
+  which the third party grants, to any of the parties who would receive the
+  covered work from you, a discriminatory patent license (a) in connection
+  with copies of the covered work conveyed by you (or copies made from
+  those copies), or (b) primarily for and in connection with specific
+  products or compilations that contain the covered work, unless you
+  entered into that arrangement, or that patent license was granted, prior
+  to 28 March 2007.
+
+  Nothing in this License shall be construed as excluding or limiting any
+  implied license or other defenses to infringement that may otherwise be
+  available to you under applicable patent law.
+
+  12. No Surrender of Others' Freedom.
+
+  If conditions are imposed on you (whether by court order, agreement or
+  otherwise) that contradict the conditions of this License, they do not
+  excuse you from the conditions of this License. If you cannot use,
+  propagate or convey a covered work so as to satisfy simultaneously your
+  obligations under this License and any other pertinent obligations, then
+  as a consequence you may not use, propagate or convey it at all. For
+  example, if you agree to terms that obligate you to collect a royalty for
+  further conveying from those to whom you convey the Program, the only way
+  you could satisfy both those terms and this License would be to refrain
+  entirely from conveying the Program.
+
+  13. Offering the Program as a Service.
+
+  If you make the functionality of the Program or a modified version
+  available to third parties as a service, you must make the Service Source
+  Code available via network download to everyone at no charge, under the
+  terms of this License. Making the functionality of the Program or
+  modified version available to third parties as a service includes,
+  without limitation, enabling third parties to interact with the
+  functionality of the Program or modified version remotely through a
+  computer network, offering a service the value of which entirely or
+  primarily derives from the value of the Program or modified version, or
+  offering a service that accomplishes for users the primary purpose of the
+  Program or modified version.
+
+  “Service Source Code” means the Corresponding Source for the Program or
+  the modified version, and the Corresponding Source for all programs that
+  you use to make the Program or modified version available as a service,
+  including, without limitation, management software, user interfaces,
+  application program interfaces, automation software, monitoring software,
+  backup software, storage software and hosting software, all such that a
+  user could run an instance of the service using the Service Source Code
+  you make available.  
+
+  14. Revised Versions of this License.
+
+  MongoDB, Inc. may publish revised and/or new versions of the Server Side
+  Public License from time to time. Such new versions will be similar in
+  spirit to the present version, but may differ in detail to address new
+  problems or concerns.
+
+  Each version is given a distinguishing version number. If the Program
+  specifies that a certain numbered version of the Server Side Public
+  License “or any later version” applies to it, you have the option of
+  following the terms and conditions either of that numbered version or of
+  any later version published by MongoDB, Inc. If the Program does not
+  specify a version number of the Server Side Public License, you may
+  choose any version ever published by MongoDB, Inc.
+
+  If the Program specifies that a proxy can decide which future versions of
+  the Server Side Public License can be used, that proxy's public statement
+  of acceptance of a version permanently authorizes you to choose that
+  version for the Program.
+
+  Later license versions may give you additional or different permissions.
+  However, no additional obligations are imposed on any author or copyright
+  holder as a result of your choosing to follow a later version.
+
+  15. Disclaimer of Warranty.
+
+  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+  APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+  HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM “AS IS” WITHOUT WARRANTY
+  OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+  THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+  PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+  IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+  ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+  
+  16. Limitation of Liability.
+  
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+  WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+  THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING
+  ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF
+  THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO
+  LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU
+  OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
+  PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
+  POSSIBILITY OF SUCH DAMAGES.
+  
+  17. Interpretation of Sections 15 and 16.
+
+  If the disclaimer of warranty and limitation of liability provided above
+  cannot be given local legal effect according to their terms, reviewing
+  courts shall apply local law that most closely approximates an absolute
+  waiver of all civil liability in connection with the Program, unless a
+  warranty or assumption of liability accompanies a copy of the Program in
+  return for a fee.
+  
+                        END OF TERMS AND CONDITIONS

apiserver/apierrors/__init__.py

@@ -0,0 +1,6 @@
+from .apierror import APIError
+from .base import BaseError
+
+from apiserver.apierrors_generator import ErrorsGenerator
+
+ErrorsGenerator.generate_python_files()

apiserver/apierrors/apierror.py

@@ -1,9 +1,10 @@
 class APIError(Exception):
-    def __init__(self, msg, code=500, subcode=0, **_):
+    def __init__(self, msg, code=500, subcode=0, error_data=None, **_):
         super(APIError, self).__init__()
         self._msg = msg
         self._code = code
         self._subcode = subcode
+        self._error_data = error_data or {}
 
     @property
     def msg(self):
@@ -17,5 +18,9 @@ class APIError(Exception):
     def subcode(self):
         return self._subcode
 
+    @property
+    def error_data(self):
+        return self._error_data
+
     def __str__(self):
         return self.msg

apiserver/apierrors/base.py

@@ -1,9 +1,13 @@
-import six
-from boltons.typeutils import classproperty
 from typing import Tuple
 
+import six
+from boltons.iterutils import is_collection, remap
+from boltons.typeutils import classproperty
+
 from .apierror import APIError
 
+jsonable_types = (dict, list, tuple, str, int, float, bool, type(None))
+
 
 class BaseError(APIError):
     _default_code = 500
@@ -19,15 +23,26 @@ class BaseError(APIError):
                 f"{k}={self._format_kwarg(v)}" for k, v in kwargs.items()
             )
             message += f": {kwargs_msg}"
-        params = kwargs.copy()
-        params.update(
-            code=self._default_code, subcode=self._default_subcode, msg=message
+
+        super(BaseError, self).__init__(
+            code=self._default_code,
+            subcode=self._default_subcode,
+            msg=message,
+            error_data=self._to_safe_json_types(kwargs),
         )
-        super(BaseError, self).__init__(**params)
+
+    @staticmethod
+    def _to_safe_json_types(data):
+        def visit(_, k, v):
+            if not isinstance(v, jsonable_types):
+                v = str(v)
+            return k, v
+
+        return remap(data, visit=visit)
 
     @staticmethod
     def _format_kwarg(value):
-        if isinstance(value, (tuple, list)):
+        if is_collection(value):
             return f'({", ".join(str(v) for v in value)})'
         elif isinstance(value, six.string_types):
             return value

apiserver/apierrors/errors.conf

@@ -0,0 +1,129 @@
+400 {
+    _: "bad_request"
+    1: ["not_supported", "endpoint is not supported"]
+    2: ["request_path_has_invalid_version", "request path has invalid version"]
+    5: ["invalid_headers", "invalid headers"]
+    6: ["impersonation_error", "impersonation error"]
+
+    10: ["invalid_id", "invalid object id"]
+    11: ["missing_required_fields", "missing required fields"]
+    12: ["validation_error", "validation error"]
+    13: ["fields_not_allowed_for_role", "fields not allowed for role"]
+    14: ["invalid fields", "fields not defined for object"]
+    15: ["fields_conflict", "conflicting fields"]
+    16: ["fields_value_error", "invalid value for fields"]
+    17: ["batch_contains_no_items", "batch request contains no items"]
+    18: ["batch_validation_error", "batch request validation error"]
+    19: ["invalid_lucene_syntax", "malformed lucene query"]
+    20: ["fields_type_error", "invalid type for fields"]
+    21: ["invalid_regex_error", "malformed regular expression"]
+    22: ["invalid_email_address", "malformed email address"]
+    23: ["invalid_domain_name", "malformed domain name"]
+    24: ["not_public_object", "object is not public"]
+
+    # Tasks
+    100: ["task_error", "general task error"]
+    101: ["invalid_task_id", "invalid task id"]
+    102: ["task_validation_error", "task validation error"]
+    110: ["invalid_task_status", "invalid task status"]
+    111: ["task_not_started", "task not started (invalid task status)"]
+    112: ["task_in_progress", "task in progress (invalid task status)"]
+    113: ["task_published", "task published (invalid task status)"]
+    114: ["task_status_unknown", "task unknown (invalid task status)"]
+    120: ["invalid_task_execution_progress", "invalid task execution progress"]
+    121: ["failed_changing_task_status", "failed changing task status. probably someone changed it before you"]
+    122: ["missing_task_fields", "task is missing expected fields"]
+    123: ["task_cannot_be_deleted", "task cannot be deleted"]
+    125: ["task_has_jobs_running", "task has jobs that haven't completed yet"]
+    126: ["invalid_task_type", "invalid task type for this operations"]
+    127: ["invalid_task_input", "invalid task output"]
+    128: ["invalid_task_output", "invalid task output"]
+    129: ["task_publish_in_progress", "Task publish in progress"]
+    130: ["task_not_found", "task not found"]
+    131: ["events_not_added", "events not added"]
+
+    # Models
+    200: ["model_error", "general task error"]
+    201: ["invalid_model_id", "invalid model id"]
+    202: ["model_not_ready", "model is not ready"]
+    203: ["model_is_ready", "model is ready"]
+    204: ["invalid_model_uri", "invalid model URI"]
+    205: ["model_in_use", "model is used by tasks"]
+    206: ["model_creating_task_exists", "task that created this model exists"]
+
+    # Users
+    300: ["invalid_user", "invalid user"]
+    301: ["invalid_user_id", "invalid user id"]
+    302: ["user_id_exists", "user id already exists"]
+    305: ["invalid_preferences_update", "Malformed key and/or value"]
+
+    # Projects
+    401: ["invalid_project_id", "invalid project id"]
+    402: ["project_has_tasks", "project has associated tasks"]
+    403: ["project_not_found", "project not found"]
+    405: ["project_has_models", "project has associated models"]
+
+    # Queues
+    701: ["invalid_queue_id", "invalid queue id"]
+    702: ["queue_not_empty", "queue is not empty"]
+    703: ["invalid_queue_or_task_not_queued", "invalid queue id or task not in queue"]
+    704: ["removed_during_reposition", "task was removed by another party during reposition"]
+    705: ["failed_adding_during_reposition", "failed adding task back to queue during reposition"]
+    706: ["task_already_queued", "failed adding task to queue since task is already queued"]
+    707: ["no_default_queue", "no queue is tagged as the default queue for this company"]
+    708: ["multiple_default_queues", "more than one queue is tagged as the default queue for this company"]
+
+    # Database
+    800: ["data_validation_error", "data validation error"]
+    801: ["expected_unique_data", "value combination already exists"]
+
+    # Workers
+    1001: ["invalid_worker_id", "invalid worker id"]
+    1002: ["worker_registration_failed", "worker registration failed"]
+    1003: ["worker_registered", "worker is already registered"]
+    1004: ["worker_not_registered", "worker is not registered"]
+    1005: ["worker_stats_not_found", "worker stats not found"]
+
+    1104: ["invalid_scroll_id", "Invalid scroll id"]
+}
+
+401 {
+    _:  "unauthorized"
+    1:  ["not_authorized", "unauthorized (not authorized for endpoint)"]
+    2:  ["entity_not_allowed", "unauthorized (entity not allowed)"]
+    10: ["bad_auth_type", "unauthorized (bad authentication header type)"]
+    20: ["no_credentials", "unauthorized (missing credentials)"]
+    21: ["bad_credentials", "unauthorized (malformed credentials)"]
+    22: ["invalid_credentials", "unauthorized (invalid credentials)"]
+    30: ["invalid_token", "invalid token"]
+    31: ["blocked_token", "token is blocked"]
+    40: ["invalid_fixed_user", "fixed user ID was not found"]
+}
+
+403: {
+    _:  "forbidden"
+    10: ["routing_error", "forbidden (routing error)"]
+    12: ["blocked_internal_endpoint", "forbidden (blocked internal endpoint)"]
+    20: ["role_not_allowed", "forbidden (not allowed for role)"]
+    21: ["no_write_permission", "forbidden (modification not allowed)"]
+}
+
+500 {
+    _:   "server_error"
+    0:   ["general_error", "general server error"]
+    1:   ["internal_error", "internal server error"]
+    2:   ["config_error", "configuration error"]
+    3:   ["build_info_error", "build info unavailable or corrupted"]
+    4:   ["low_disk_space", "Critical server error! Server reports low or insufficient disk space. Please resolve immediately by allocating additional disk space or freeing up storage space."]
+    10:  ["transaction_error", "a transaction call has returned with an error"]
+    # Database-related issues
+    100: ["data_error", "general data error"]
+    101: ["inconsistent_data", "inconsistent data encountered in document"]
+    102: ["database_unavailable", "database is temporarily unavailable"]
+    110: ["update_failed", "update failed"]
+
+    # Index-related issues
+    201: ["missing_index", "missing internal index"]
+
+    9999: ["not_implemented", "action is not yet implemented"]
+}

apiserver/apierrors_generator/__init__.py

@@ -0,0 +1 @@
+from .errors_generator import ErrorsGenerator

apiserver/apierrors_generator/__main__.py

@@ -0,0 +1,4 @@
+from .errors_generator import ErrorsGenerator
+
+if __name__ == '__main__':
+    ErrorsGenerator.generate_python_files()

apiserver/apierrors_generator/errors_generator.py

@@ -0,0 +1,31 @@
+from functools import reduce
+from pathlib import Path
+from typing import Union
+
+from pyhocon import ConfigFactory, ConfigTree
+
+from .generator import Generator
+
+
+class ErrorsGenerator:
+    _apierrors_path = Path(__file__).parents[1] / "apierrors"
+    _files = [_apierrors_path / "errors.conf"]
+
+    @classmethod
+    def _get_codes(cls):
+        return {
+            (k, v.pop("_")): v
+            for k, v in reduce(
+                ConfigTree.merge_configs, map(ConfigFactory.parse_file, cls._files),
+            ).items()
+        }
+
+    @classmethod
+    def add_errors_file(cls, path: Union[Path, str]):
+        cls._files.append(path)
+
+    @classmethod
+    def generate_python_files(cls):
+        Generator(cls._apierrors_path / "errors", format_pep8=False).make_errors(
+            cls._get_codes()
+        )

apiserver/apierrors_generator/generator.py

@@ -8,9 +8,12 @@ from pathlib import Path
 
 env = jinja2.Environment(
     loader=jinja2.FileSystemLoader(str(Path(__file__).parent)),
-    autoescape=jinja2.select_autoescape(disabled_extensions=('py',), default_for_string=False),
+    autoescape=jinja2.select_autoescape(
+        disabled_extensions=("py",), default_for_string=False
+    ),
     trim_blocks=True,
-    lstrip_blocks=True)
+    lstrip_blocks=True,
+)
 
 
 def env_filter(name=None):
@@ -19,14 +22,14 @@ def env_filter(name=None):
 
 @env_filter()
 def cls_name(name):
-    delims = list(map(re.escape, (' ', '_')))
-    parts = re.split('|'.join(delims), name)
-    return ''.join(x.capitalize() for x in parts)
+    delims = list(map(re.escape, (" ", "_")))
+    parts = re.split("|".join(delims), name)
+    return "".join(x.capitalize() for x in parts)
 
 
 class Generator(object):
-    _base_class_name = 'BaseError'
-    _base_class_module = 'apierrors.base'
+    _base_class_name = "BaseError"
+    _base_class_module = "apiserver.apierrors.base"
 
     def __init__(self, path, format_pep8=True, use_md5=True):
         self._use_md5 = use_md5
@@ -35,29 +38,37 @@ class Generator(object):
         self._path.mkdir(parents=True, exist_ok=True)
 
     def _make_init_file(self, path):
-        (self._path / path / '__init__.py').write_bytes('')
+        (self._path / path / "__init__.py").write_bytes(b"")
 
     def _do_render(self, file, template, context):
-        with file.open('w') as f:
+        with file.open("w") as f:
             result = template.render(
                 base_class_name=self._base_class_name,
                 base_class_module=self._base_class_module,
-                **context)
+                **context
+            )
             if self._format_pep8:
-                result = autopep8.fix_code(result, options={'aggressive': 1, 'verbose': 0, 'max_line_length': 120})
+                import autopep8
+
+                result = autopep8.fix_code(
+                    result,
+                    options={"aggressive": 1, "verbose": 0, "max_line_length": 120},
+                )
             f.write(result)
 
     def _make_section(self, name, code, subcodes):
         self._do_render(
-            file=(self._path / name).with_suffix('.py'),
-            template=env.get_template('templates/section.jinja2'),
-            context=dict(code=code, subcodes=list(subcodes.items()),))
+            file=(self._path / name).with_suffix(".py"),
+            template=env.get_template("templates/section.jinja2"),
+            context=dict(code=code, subcodes=list(subcodes.items()),),
+        )
 
     def _make_init(self, sections):
         self._do_render(
-            file=(self._path / '__init__.py'),
-            template=env.get_template('templates/init.jinja2'),
-            context=dict(sections=sections,))
+            file=(self._path / "__init__.py"),
+            template=env.get_template("templates/init.jinja2"),
+            context=dict(sections=sections,),
+        )
 
     def _key_to_str(self, data):
         if isinstance(data, dict):
@@ -66,11 +77,11 @@ class Generator(object):
 
     def _calc_digest(self, data):
         data = json.dumps(self._key_to_str(data), sort_keys=True)
-        return hashlib.md5(data.encode('utf8')).hexdigest()
+        return hashlib.md5(data.encode("utf8")).hexdigest()
 
     def make_errors(self, errors):
         digest = None
-        digest_file = self._path / 'digest.md5'
+        digest_file = self._path / "digest.md5"
         if self._use_md5:
             digest = self._calc_digest(errors)
             if digest_file.is_file():
@@ -79,7 +90,7 @@ class Generator(object):
 
         self._make_init(errors)
         for (code, section_name), subcodes in errors.items():
-            self._make_section(section_name, code, subcodes)
+            self._make_section(section_name, int(code), subcodes)
 
         if self._use_md5:
             digest_file.write_text(digest)

apiserver/apierrors_generator/templates/section.jinja2

@@ -5,5 +5,5 @@ from {{ base_class_module }} import {{ base_class_name }}
 {% for subcode, (name, msg) in subcodes %}
 
 
-{{ error_class(name|cls_name, msg, code, subcode) -}}
+{{ error_class(name|cls_name, msg, code, subcode|int) -}}
 {% endfor %}

apiserver/apimodels/__init__.py

@@ -0,0 +1,303 @@
+from enum import Enum
+from typing import Union, Type, Iterable
+
+import jsonmodels.errors
+import six
+from jsonmodels import fields
+from jsonmodels.fields import _LazyType, NotSet
+from jsonmodels.models import Base as ModelBase
+from jsonmodels.validators import Enum as EnumValidator
+from mongoengine.base import BaseDocument
+from validators import email as email_validator, domain as domain_validator
+
+from apiserver.apierrors import errors
+from apiserver.utilities.json import loads, dumps
+
+
+class EmailField(fields.StringField):
+    def validate(self, value):
+        super().validate(value)
+        if value is None:
+            return
+        if email_validator(value) is not True:
+            raise errors.bad_request.InvalidEmailAddress()
+
+
+class DomainField(fields.StringField):
+    def validate(self, value):
+        super().validate(value)
+        if value is None:
+            return
+        if domain_validator(value) is not True:
+            raise errors.bad_request.InvalidDomainName()
+
+
+def make_default(field_cls, default_value):
+    class _FieldWithDefault(field_cls):
+        def get_default_value(self):
+            return default_value
+
+    return _FieldWithDefault
+
+
+class ListField(fields.ListField):
+    def __init__(self, items_types=None, *args, default=NotSet, **kwargs):
+        if default is not NotSet and callable(default):
+            default = default()
+
+        super(ListField, self).__init__(items_types, *args, default=default, **kwargs)
+
+    def _cast_value(self, value):
+        try:
+            return super(ListField, self)._cast_value(value)
+        except TypeError:
+            if len(self.items_types) == 1 and issubclass(self.items_types[0], Enum):
+                return self.items_types[0](value)
+            return value
+
+    def validate_single_value(self, item):
+        super(ListField, self).validate_single_value(item)
+        if isinstance(item, ModelBase):
+            item.validate()
+
+
+# since there is no distinction between None and empty DictField
+# this value can be used as sentinel in order to distinguish
+# between not set and empty DictField
+DictFieldNotSet = {}
+
+
+class DictField(fields.BaseField):
+    types = (dict,)
+
+    def __init__(self, value_types=None, *args, **kwargs):
+        self.value_types = self._assign_types(value_types)
+        super(DictField, self).__init__(*args, **kwargs)
+
+    def get_default_value(self):
+        default = super(DictField, self).get_default_value()
+        if default is None and not self.required:
+            return {}
+        return default
+
+    @staticmethod
+    def _assign_types(value_types):
+        if value_types:
+            try:
+                value_types = tuple(value_types)
+            except TypeError:
+                value_types = (value_types,)
+        else:
+            value_types = tuple()
+
+        return tuple(
+            _LazyType(type_) if isinstance(type_, six.string_types) else type_
+            for type_ in value_types
+        )
+
+    def parse_value(self, values):
+        """Cast value to proper collection."""
+        result = self.get_default_value()
+
+        if values is None:
+            return result
+
+        if not self.value_types or not isinstance(values, dict):
+            return values
+
+        return {key: self._cast_value(value) for key, value in values.items()}
+
+    def _cast_value(self, value):
+        if isinstance(value, self.value_types):
+            return value
+        else:
+            if len(self.value_types) != 1:
+                tpl = 'Cannot decide which type to choose from "{types}".'
+                raise jsonmodels.errors.ValidationError(
+                    tpl.format(
+                        types=', '.join([t.__name__ for t in self.value_types])
+                    )
+                )
+            return self.value_types[0](**value)
+
+    def validate(self, value):
+        super(DictField, self).validate(value)
+
+        if not self.value_types:
+            return
+
+        if not value:
+            return
+
+        for item in value.values():
+            self.validate_single_value(item)
+
+    def validate_single_value(self, item):
+        if not self.value_types:
+            return
+
+        if not isinstance(item, self.value_types):
+            raise jsonmodels.errors.ValidationError(
+                "All items must be instances "
+                'of "{types}", and not "{type}".'.format(
+                    types=", ".join([t.__name__ for t in self.value_types]),
+                    type=type(item).__name__,
+                )
+            )
+
+    def _elem_to_struct(self, value):
+        try:
+            return value.to_struct()
+        except AttributeError:
+            return value
+
+    def to_struct(self, values):
+        return {k: self._elem_to_struct(v) for k, v in values.items()}
+
+
+class IntField(fields.IntField):
+    def parse_value(self, value):
+        try:
+            return super(IntField, self).parse_value(value)
+        except (ValueError, TypeError):
+            return value
+
+
+class NullableEnumValidator(EnumValidator):
+    """Validator for enums that allows a None value."""
+
+    def validate(self, value):
+        if value is not None:
+            super(NullableEnumValidator, self).validate(value)
+
+
+class EnumField(fields.StringField):
+    def __init__(
+        self,
+        values_or_type: Union[Iterable, Type[Enum]],
+        *args,
+        required=False,
+        default=None,
+        **kwargs
+    ):
+        choices = list(map(self.parse_value, values_or_type))
+        validator_cls = EnumValidator if required else NullableEnumValidator
+        kwargs.setdefault("validators", []).append(validator_cls(*choices))
+        super().__init__(
+            default=self.parse_value(default), required=required, *args, **kwargs
+        )
+
+    def parse_value(self, value):
+        if isinstance(value, Enum):
+            return str(value.value)
+        return super().parse_value(value)
+
+
+class ActualEnumField(fields.StringField):
+    def __init__(
+        self,
+        enum_class: Type[Enum],
+        *args,
+        validators=None,
+        required=False,
+        default=None,
+        **kwargs
+    ):
+        self.__enum = enum_class
+        self.types = (enum_class,)
+        # noinspection PyTypeChecker
+        choices = list(enum_class)
+        validator_cls = EnumValidator if required else NullableEnumValidator
+        validators = [*(validators or []), validator_cls(*choices)]
+        super().__init__(
+            default=self.parse_value(default) if default else NotSet,
+            *args,
+            required=required,
+            validators=validators,
+            **kwargs
+        )
+
+    def parse_value(self, value):
+        if value is None and not self.required:
+            return self.get_default_value()
+        try:
+            # noinspection PyArgumentList
+            return self.__enum(value)
+        except ValueError:
+            return value
+
+    def to_struct(self, value):
+        return super().to_struct(value.value)
+
+
+class JsonSerializableMixin:
+    def to_json(self: ModelBase):
+        return dumps(self.to_struct())
+
+    @classmethod
+    def from_json(cls: Type[ModelBase], s):
+        return cls(**loads(s))
+
+
+def callable_default(cls: Type[fields.BaseField]) -> Type[fields.BaseField]:
+    class _Wrapped(cls):
+        _callable_default = None
+
+        def get_default_value(self):
+            if self._callable_default:
+                return self._callable_default()
+            return super(_Wrapped, self).get_default_value()
+
+        def __init__(self, *args, default=None, **kwargs):
+            if default and callable(default):
+                self._callable_default = default
+                default = default()
+            super(_Wrapped, self).__init__(*args, default=default, **kwargs)
+
+    return _Wrapped
+
+
+class MongoengineFieldsDict(DictField):
+    """
+    DictField representing mongoengine field names/value mapping.
+    Used to convert mongoengine-style field/subfield notation to user-presentable syntax, including handling update
+        operators.
+    """
+
+    mongoengine_update_operators = (
+        "inc",
+        "dec",
+        "push",
+        "push_all",
+        "pop",
+        "pull",
+        "pull_all",
+        "add_to_set",
+    )
+
+    @staticmethod
+    def _normalize_mongo_value(value):
+        if isinstance(value, BaseDocument):
+            return value.to_mongo()
+        return value
+
+    @classmethod
+    def _normalize_mongo_field_path(cls, path, value):
+        parts = path.split("__")
+        if len(parts) > 1:
+            if parts[0] == "set":
+                parts = parts[1:]
+            elif parts[0] == "unset":
+                parts = parts[1:]
+                value = None
+            elif parts[0] in cls.mongoengine_update_operators:
+                return None, None
+        return ".".join(parts), cls._normalize_mongo_value(value)
+
+    def parse_value(self, value):
+        value = super(MongoengineFieldsDict, self).parse_value(value)
+        return {
+            k: v
+            for k, v in (self._normalize_mongo_field_path(*p) for p in value.items())
+            if k is not None
+        }

apiserver/apimodels/auth.py

@@ -1,11 +1,11 @@
-from jsonmodels.fields import IntField, StringField, BoolField, EmbeddedField
+from jsonmodels.fields import IntField, StringField, BoolField, EmbeddedField, DateTimeField
 from jsonmodels.models import Base
 from jsonmodels.validators import Max, Enum
 
-from apimodels import ListField, EnumField
-from config import config
-from database.model.auth import Role
-from database.utils import get_options
+from apiserver.apimodels import ListField, EnumField
+from apiserver.config_repo import config
+from apiserver.database.model.auth import Role
+from apiserver.database.utils import get_options
 
 
 class GetTokenRequest(Base):
@@ -79,6 +79,7 @@ class Credentials(Base):
 
 class CredentialsResponse(Credentials):
     secret_key = StringField()
+    last_used = DateTimeField(default=None)
 
 
 class CreateCredentialsResponse(Base):

apiserver/apimodels/base.py

@@ -0,0 +1,28 @@
+from jsonmodels import models, fields
+from jsonmodels.validators import Length
+
+from apiserver.apimodels import MongoengineFieldsDict, ListField
+
+
+class UpdateResponse(models.Base):
+    updated = fields.IntField(required=True)
+    fields = MongoengineFieldsDict()
+
+
+class PagedRequest(models.Base):
+    page = fields.IntField()
+    page_size = fields.IntField()
+
+
+class IdResponse(models.Base):
+    id = fields.StringField(required=True)
+
+
+class MakePublicRequest(models.Base):
+    ids = ListField(items_types=str, validators=[Length(minimum_value=1)])
+
+
+class MoveRequest(models.Base):
+    ids = ListField([str], validators=Length(minimum_value=1))
+    project = fields.StringField()
+    project_name = fields.StringField()

apiserver/apimodels/custom_validators/__init__.py

@@ -0,0 +1,34 @@
+import validators
+from jsonmodels.errors import ValidationError
+
+
+class ForEach(object):
+    def __init__(self, validator):
+        self.validator = validator
+
+    def validate(self, values):
+        for value in values:
+            self.validator.validate(value)
+
+    def modify_schema(self, field_schema):
+        return self.validator.modify_schema(field_schema)
+
+
+class Hostname(object):
+
+    def validate(self, value):
+        if validators.domain(value) is not True:
+            raise ValidationError(f"Value '{value}' is not a valid hostname")
+
+    def modify_schema(self, field_schema):
+        field_schema["format"] = "hostname"
+
+
+class Email(object):
+
+    def validate(self, value):
+        if validators.email(value) is not True:
+            raise ValidationError(f"Value '{value}' is not a valid email address")
+
+    def modify_schema(self, field_schema):
+        field_schema["format"] = "email"

apiserver/apimodels/events.py

@@ -0,0 +1,105 @@
+from enum import auto
+from typing import Sequence, Optional
+
+from jsonmodels import validators
+from jsonmodels.fields import StringField, BoolField
+from jsonmodels.models import Base
+from jsonmodels.validators import Length, Min, Max
+
+from apiserver.apimodels import ListField, IntField, ActualEnumField
+from apiserver.bll.event.event_common import EventType
+from apiserver.bll.event.scalar_key import ScalarKeyEnum
+from apiserver.config_repo import config
+from apiserver.utilities.stringenum import StringEnum
+
+
+class HistogramRequestBase(Base):
+    samples: int = IntField(default=6000, validators=[Min(1), Max(6000)])
+    key: ScalarKeyEnum = ActualEnumField(ScalarKeyEnum, default=ScalarKeyEnum.iter)
+
+
+class ScalarMetricsIterHistogramRequest(HistogramRequestBase):
+    task: str = StringField(required=True)
+
+
+class MultiTaskScalarMetricsIterHistogramRequest(HistogramRequestBase):
+    tasks: Sequence[str] = ListField(
+        items_types=str,
+        validators=[
+            Length(
+                minimum_value=1,
+                maximum_value=config.get(
+                    "services.tasks.multi_task_histogram_limit", 10
+                ),
+            )
+        ],
+    )
+
+
+class TaskMetric(Base):
+    task: str = StringField(required=True)
+    metric: str = StringField(required=True)
+
+
+class DebugImagesRequest(Base):
+    metrics: Sequence[TaskMetric] = ListField(
+        items_types=TaskMetric, validators=[Length(minimum_value=1)]
+    )
+    iters: int = IntField(default=1, validators=validators.Min(1))
+    navigate_earlier: bool = BoolField(default=True)
+    refresh: bool = BoolField(default=False)
+    scroll_id: str = StringField()
+
+
+class TaskMetricVariant(Base):
+    task: str = StringField(required=True)
+    metric: str = StringField(required=True)
+    variant: str = StringField(required=True)
+
+
+class GetDebugImageSampleRequest(TaskMetricVariant):
+    iteration: Optional[int] = IntField()
+    scroll_id: Optional[str] = StringField()
+    refresh: bool = BoolField(default=False)
+
+
+class NextDebugImageSampleRequest(Base):
+    task: str = StringField(required=True)
+    scroll_id: Optional[str] = StringField()
+    navigate_earlier: bool = BoolField(default=True)
+
+
+class LogOrderEnum(StringEnum):
+    asc = auto()
+    desc = auto()
+
+
+class LogEventsRequest(Base):
+    task: str = StringField(required=True)
+    batch_size: int = IntField(default=500)
+    navigate_earlier: bool = BoolField(default=True)
+    from_timestamp: Optional[int] = IntField()
+    order: Optional[str] = ActualEnumField(LogOrderEnum)
+
+
+class IterationEvents(Base):
+    iter: int = IntField()
+    events: Sequence[dict] = ListField(items_types=dict)
+
+
+class MetricEvents(Base):
+    task: str = StringField()
+    metric: str = StringField()
+    iterations: Sequence[IterationEvents] = ListField(items_types=IterationEvents)
+
+
+class DebugImageResponse(Base):
+    metrics: Sequence[MetricEvents] = ListField(items_types=MetricEvents)
+    scroll_id: str = StringField()
+
+
+class TaskMetricsRequest(Base):
+    tasks: Sequence[str] = ListField(
+        items_types=str, validators=[Length(minimum_value=1)]
+    )
+    event_type: EventType = ActualEnumField(EventType, required=True)

apiserver/apimodels/login.py

@@ -0,0 +1,33 @@
+from jsonmodels.fields import StringField, BoolField, EmbeddedField, ListField
+from jsonmodels.models import Base
+
+from apiserver.apimodels import DictField, callable_default
+
+
+class GetSupportedModesRequest(Base):
+    state = StringField(help_text="ASCII base64 encoded application state")
+    callback_url_prefix = StringField()
+
+
+class BasicGuestMode(Base):
+    enabled = BoolField(default=False)
+    name = StringField()
+    username = StringField()
+    password = StringField()
+
+
+class BasicMode(Base):
+    enabled = BoolField(default=False)
+    guest = callable_default(EmbeddedField)(BasicGuestMode, default=BasicGuestMode)
+
+
+class ServerErrors(Base):
+    missed_es_upgrade = BoolField(default=False)
+    es_connection_error = BoolField(default=False)
+
+
+class GetSupportedModesResponse(Base):
+    basic = EmbeddedField(BasicMode)
+    server_errors = EmbeddedField(ServerErrors)
+    sso = DictField([str, type(None)])
+    sso_providers = ListField([dict])

apiserver/apimodels/models.py

@@ -1,16 +1,21 @@
 from jsonmodels import models, fields
 from six import string_types
 
-from apimodels import ListField, DictField
-from apimodels.base import UpdateResponse
-from apimodels.tasks import PublishResponse as TaskPublishResponse
+from apiserver.apimodels import ListField, DictField
+from apiserver.apimodels.base import UpdateResponse
+from apiserver.apimodels.tasks import PublishResponse as TaskPublishResponse
+
+
+class GetFrameworksRequest(models.Base):
+    projects = fields.ListField(items_types=[str])
 
 
 class CreateModelRequest(models.Base):
     name = fields.StringField(required=True)
     uri = fields.StringField(required=True)
-    labels = DictField(value_types=string_types+(int,), required=True)
+    labels = DictField(value_types=string_types+(int,))
     tags = ListField(items_types=string_types)
+    system_tags = ListField(items_types=string_types)
     comment = fields.StringField()
     public = fields.BoolField(default=False)
     project = fields.StringField()

apiserver/apimodels/organization.py

@@ -0,0 +1,11 @@
+from jsonmodels import fields, models
+
+
+class Filter(models.Base):
+    tags = fields.ListField([str])
+    system_tags = fields.ListField([str])
+
+
+class TagsRequest(models.Base):
+    include_system = fields.BoolField(default=False)
+    filter = fields.EmbeddedField(Filter)

apiserver/apimodels/projects.py

@@ -0,0 +1,24 @@
+from jsonmodels import models, fields
+
+from apiserver.apimodels import ListField, ActualEnumField
+from apiserver.apimodels.organization import TagsRequest
+from apiserver.database.model import EntityVisibility
+
+
+class ProjectReq(models.Base):
+    project = fields.StringField()
+
+
+class GetHyperParamReq(ProjectReq):
+    page = fields.IntField(default=0)
+    page_size = fields.IntField(default=500)
+
+
+class ProjectTagsRequest(TagsRequest):
+    projects = ListField(str)
+
+
+class ProjectTaskParentsRequest(ProjectReq):
+    projects = ListField(str)
+    tasks_state = ActualEnumField(EntityVisibility)
+

apiserver/apimodels/queues.py

@@ -0,0 +1,60 @@
+from jsonmodels import validators
+from jsonmodels.fields import StringField, IntField, BoolField, FloatField
+from jsonmodels.models import Base
+
+from apiserver.apimodels import ListField
+
+
+class GetDefaultResp(Base):
+    id = StringField(required=True)
+    name = StringField(required=True)
+
+
+class CreateRequest(Base):
+    name = StringField(required=True)
+    tags = ListField(items_types=[str])
+    system_tags = ListField(items_types=[str])
+
+
+class QueueRequest(Base):
+    queue = StringField(required=True)
+
+
+class DeleteRequest(QueueRequest):
+    force = BoolField(default=False)
+
+
+class UpdateRequest(QueueRequest):
+    name = StringField()
+    tags = ListField(items_types=[str])
+    system_tags = ListField(items_types=[str])
+
+
+class TaskRequest(QueueRequest):
+    task = StringField(required=True)
+
+
+class MoveTaskRequest(TaskRequest):
+    count = IntField(default=1)
+
+
+class MoveTaskResponse(Base):
+    position = IntField()
+
+
+class GetMetricsRequest(Base):
+    queue_ids = ListField([str])
+    from_date = FloatField(required=True, validators=validators.Min(0))
+    to_date = FloatField(required=True, validators=validators.Min(0))
+    interval = IntField(required=True, validators=validators.Min(1))
+
+
+class QueueMetrics(Base):
+    queue = StringField()
+    dates = ListField(int)
+    avg_waiting_times = ListField([float, int])
+    queue_lengths = ListField(int)
+
+
+class GetMetricsResponse(Base):
+    queues = ListField(QueueMetrics)

apiserver/apimodels/server.py

@@ -0,0 +1,15 @@
+from jsonmodels.fields import BoolField, DateTimeField, StringField
+from jsonmodels.models import Base
+
+
+class ReportStatsOptionRequest(Base):
+    enabled = BoolField(default=None, nullable=True)
+
+
+class ReportStatsOptionResponse(Base):
+    supported = BoolField(default=True)
+    enabled = BoolField()
+    enabled_time = DateTimeField(nullable=True)
+    enabled_version = StringField(nullable=True)
+    enabled_user = StringField(nullable=True)
+    current_version = StringField()

apiserver/apimodels/tasks.py

@@ -0,0 +1,221 @@
+from typing import Sequence
+
+import six
+from jsonmodels import models
+from jsonmodels.fields import StringField, BoolField, IntField, EmbeddedField
+from jsonmodels.validators import Enum, Length
+
+from apiserver.apimodels import DictField, ListField
+from apiserver.apimodels.base import UpdateResponse
+from apiserver.database.model.task.task import (
+    TaskType,
+    ArtifactModes,
+    DEFAULT_ARTIFACT_MODE,
+)
+from apiserver.database.utils import get_options
+
+
+class ArtifactTypeData(models.Base):
+    preview = StringField()
+    content_type = StringField()
+    data_hash = StringField()
+
+
+class Artifact(models.Base):
+    key = StringField(required=True)
+    type = StringField(required=True)
+    mode = StringField(
+        validators=Enum(*get_options(ArtifactModes)), default=DEFAULT_ARTIFACT_MODE
+    )
+    uri = StringField()
+    hash = StringField()
+    content_size = IntField()
+    timestamp = IntField()
+    type_data = EmbeddedField(ArtifactTypeData)
+    display_data = ListField([list])
+
+
+class StartedResponse(UpdateResponse):
+    started = IntField()
+
+
+class EnqueueResponse(UpdateResponse):
+    queued = IntField()
+
+
+class DequeueResponse(UpdateResponse):
+    dequeued = IntField()
+
+
+class ResetResponse(UpdateResponse):
+    deleted_indices = ListField(items_types=six.string_types)
+    dequeued = DictField()
+    frames = DictField()
+    events = DictField()
+    model_deleted = IntField()
+
+
+class TaskRequest(models.Base):
+    task = StringField(required=True)
+
+
+class UpdateRequest(TaskRequest):
+    status_reason = StringField(default="")
+    status_message = StringField(default="")
+    force = BoolField(default=False)
+
+
+class EnqueueRequest(UpdateRequest):
+    queue = StringField()
+
+
+class DeleteRequest(UpdateRequest):
+    move_to_trash = BoolField(default=True)
+
+
+class SetRequirementsRequest(TaskRequest):
+    requirements = DictField(required=True)
+
+
+class PublishRequest(UpdateRequest):
+    publish_model = BoolField(default=True)
+
+
+class PublishResponse(UpdateResponse):
+    pass
+
+
+class TaskData(models.Base):
+    """
+    This is a partial description of task can be updated incrementally
+    """
+
+
+class CreateRequest(TaskData):
+    name = StringField(required=True)
+    type = StringField(required=True, validators=Enum(*get_options(TaskType)))
+
+
+class PingRequest(TaskRequest):
+    pass
+
+
+class GetTypesRequest(models.Base):
+    projects = ListField(items_types=[str])
+
+
+class CloneRequest(TaskRequest):
+    new_task_name = StringField()
+    new_task_comment = StringField()
+    new_task_tags = ListField([str])
+    new_task_system_tags = ListField([str])
+    new_task_parent = StringField()
+    new_task_project = StringField()
+    new_task_hyperparams = DictField()
+    new_task_configuration = DictField()
+    execution_overrides = DictField()
+    validate_references = BoolField(default=False)
+    new_project_name = StringField()
+
+
+class AddOrUpdateArtifactsRequest(TaskRequest):
+    artifacts = ListField([Artifact], validators=Length(minimum_value=1))
+    force = BoolField(default=False)
+
+
+class ArtifactId(models.Base):
+    key = StringField(required=True)
+    mode = StringField(
+        validators=Enum(*get_options(ArtifactModes)), default=DEFAULT_ARTIFACT_MODE
+    )
+
+
+class DeleteArtifactsRequest(TaskRequest):
+    artifacts = ListField([ArtifactId], validators=Length(minimum_value=1))
+    force = BoolField(default=False)
+
+
+class ResetRequest(UpdateRequest):
+    clear_all = BoolField(default=False)
+
+
+class MultiTaskRequest(models.Base):
+    tasks = ListField([str], validators=Length(minimum_value=1))
+
+
+class GetHyperParamsRequest(MultiTaskRequest):
+    pass
+
+
+class HyperParamItem(models.Base):
+    section = StringField(required=True, validators=Length(minimum_value=1))
+    name = StringField(required=True, validators=Length(minimum_value=1))
+    value = StringField(required=True)
+    type = StringField()
+    description = StringField()
+
+
+class ReplaceHyperparams(object):
+    none = "none"
+    section = "section"
+    all = "all"
+
+
+class EditHyperParamsRequest(TaskRequest):
+    hyperparams: Sequence[HyperParamItem] = ListField(
+        [HyperParamItem], validators=Length(minimum_value=1)
+    )
+    replace_hyperparams = StringField(
+        validators=Enum(*get_options(ReplaceHyperparams)),
+        default=ReplaceHyperparams.none,
+    )
+    force = BoolField(default=False)
+
+
+class HyperParamKey(models.Base):
+    section = StringField(required=True, validators=Length(minimum_value=1))
+    name = StringField(nullable=True)
+
+
+class DeleteHyperParamsRequest(TaskRequest):
+    hyperparams: Sequence[HyperParamKey] = ListField(
+        [HyperParamKey], validators=Length(minimum_value=1)
+    )
+    force = BoolField(default=False)
+
+
+class GetConfigurationsRequest(MultiTaskRequest):
+    names = ListField([str])
+
+
+class GetConfigurationNamesRequest(MultiTaskRequest):
+    pass
+
+
+class Configuration(models.Base):
+    name = StringField(required=True, validators=Length(minimum_value=1))
+    value = StringField(required=True)
+    type = StringField()
+    description = StringField()
+
+
+class EditConfigurationRequest(TaskRequest):
+    configuration: Sequence[Configuration] = ListField(
+        [Configuration], validators=Length(minimum_value=1)
+    )
+    replace_configuration = BoolField(default=False)
+    force = BoolField(default=False)
+
+
+class DeleteConfigurationRequest(TaskRequest):
+    configuration: Sequence[str] = ListField([str], validators=Length(minimum_value=1))
+    force = BoolField(default=False)
+
+
+class ArchiveRequest(MultiTaskRequest):
+    status_reason = StringField(default="")
+    status_message = StringField(default="")
+
+
+class ArchiveResponse(models.Base):
+    archived = IntField()

apiserver/apimodels/users.py

@@ -1,7 +1,7 @@
 from jsonmodels.fields import StringField
 from jsonmodels.models import Base
 
-from apimodels import DictField
+from apiserver.apimodels import DictField
 
 
 class CreateRequest(Base):

apiserver/apimodels/workers.py

@@ -0,0 +1,178 @@
+from enum import Enum
+
+import six
+from jsonmodels import validators
+from jsonmodels.fields import (
+    StringField,
+    EmbeddedField,
+    DateTimeField,
+    IntField,
+    FloatField,
+    BoolField,
+)
+from jsonmodels.models import Base
+
+from apiserver.apimodels import make_default, ListField, EnumField, JsonSerializableMixin
+
+DEFAULT_TIMEOUT = 10 * 60
+
+
+class WorkerRequest(Base):
+    worker = StringField(required=True)
+    tags = ListField(str)
+
+
+class RegisterRequest(WorkerRequest):
+    timeout = make_default(
+        IntField, DEFAULT_TIMEOUT
+    )()  # registration timeout in seconds (default is 10min)
+    queues = ListField(six.string_types)  # list of queues this worker listens to
+
+
+class MachineStats(Base):
+    cpu_usage = ListField(six.integer_types + (float,))
+    cpu_temperature = ListField(six.integer_types + (float,))
+    gpu_usage = ListField(six.integer_types + (float,))
+    gpu_temperature = ListField(six.integer_types + (float,))
+    gpu_memory_free = ListField(six.integer_types + (float,))
+    gpu_memory_used = ListField(six.integer_types + (float,))
+    memory_used = FloatField()
+    memory_free = FloatField()
+    network_tx = FloatField()
+    network_rx = FloatField()
+    disk_free_home = FloatField()
+    disk_free_temp = FloatField()
+    disk_read = FloatField()
+    disk_write = FloatField()
+
+
+class StatusReportRequest(WorkerRequest):
+    task = StringField()  # task the worker is running on
+    queue = StringField()  # queue from which task was taken
+    queues = ListField(
+        str
+    )  # list of queues this worker listens to. if None, this will not update the worker's queues list.
+    timestamp = IntField(required=True)
+    machine_stats = EmbeddedField(MachineStats)
+
+
+class IdNameEntry(Base):
+    id = StringField(required=True)
+    name = StringField()
+
+
+class WorkerEntry(Base, JsonSerializableMixin):
+    key = StringField()  # not required due to migration issues
+    id = StringField(required=True)
+    user = EmbeddedField(IdNameEntry)
+    company = EmbeddedField(IdNameEntry)
+    ip = StringField()
+    task = EmbeddedField(IdNameEntry)
+    project = EmbeddedField(IdNameEntry)
+    queue = StringField()  # queue from which current task was taken
+    queues = ListField(str)  # list of queues this worker listens to
+    register_time = DateTimeField(required=True)
+    register_timeout = IntField(required=True)
+    last_activity_time = DateTimeField(required=True)
+    last_report_time = DateTimeField()
+    tags = ListField(str)
+
+
+class CurrentTaskEntry(IdNameEntry):
+    running_time = IntField()
+    last_iteration = IntField()
+
+
+class QueueEntry(IdNameEntry):
+    next_task = EmbeddedField(IdNameEntry)
+    num_tasks = IntField()
+
+
+class WorkerResponseEntry(WorkerEntry):
+    task = EmbeddedField(CurrentTaskEntry)
+    queue = EmbeddedField(QueueEntry)
+    queues = ListField(QueueEntry)
+
+
+class GetAllRequest(Base):
+    last_seen = IntField(default=3600)
+
+
+class GetAllResponse(Base):
+    workers = ListField(WorkerResponseEntry)
+
+
+class StatsBase(Base):
+    worker_ids = ListField(str)
+
+
+class StatsReportBase(StatsBase):
+    from_date = FloatField(required=True, validators=validators.Min(0))
+    to_date = FloatField(required=True, validators=validators.Min(0))
+    interval = IntField(required=True, validators=validators.Min(1))
+
+
+class AggregationType(Enum):
+    avg = "avg"
+    min = "min"
+    max = "max"
+
+
+class StatItem(Base):
+    key = StringField(required=True)
+    aggregation = EnumField(AggregationType, default=AggregationType.avg)
+
+
+class GetStatsRequest(StatsReportBase):
+    items = ListField(
+        StatItem, required=True, validators=validators.Length(minimum_value=1)
+    )
+    split_by_variant = BoolField(default=False)
+
+
+class AggregationStats(Base):
+    aggregation = EnumField(AggregationType)
+    values = ListField(float)
+
+
+class MetricStats(Base):
+    metric = StringField()
+    variant = StringField()
+    dates = ListField(int)
+    stats = ListField(AggregationStats)
+
+
+class WorkerStatistics(Base):
+    worker = StringField()
+    metrics = ListField(MetricStats)
+
+
+class GetStatsResponse(Base):
+    workers = ListField(WorkerStatistics)
+
+
+class GetMetricKeysRequest(StatsBase):
+    pass
+
+
+class MetricCategory(Base):
+    name = StringField()
+    metric_keys = ListField(str)
+
+
+class GetMetricKeysResponse(Base):
+    categories = ListField(MetricCategory)
+
+
+class GetActivityReportRequest(StatsReportBase):
+    pass
+
+
+class ActivityReportSeries(Base):
+    dates = ListField(int)
+    counts = ListField(int)
+
+
+class GetActivityReportResponse(Base):
+    total = EmbeddedField(ActivityReportSeries)
+    active = EmbeddedField(ActivityReportSeries)

apiserver/bll/auth/__init__.py

@@ -1,25 +1,17 @@
 from datetime import datetime
 
-import database
-from apierrors import errors
-from apimodels.auth import (
-    GetTokenResponse,
-    CreateUserRequest,
-    Credentials as CredModel,
-)
-from apimodels.users import CreateRequest as Users_CreateRequest
-from bll.user import UserBLL
-from config import config
-from database.errors import translate_errors_context
-from database.model.auth import User, Role, Credentials
-from database.model.company import Company
-from service_repo import APICall
-from service_repo.auth import (
-    Identity,
-    Token,
-    get_client_id,
-    get_secret_key,
-)
+from apiserver import database
+from apiserver.apierrors import errors
+from apiserver.apimodels.auth import GetTokenResponse, CreateUserRequest, Credentials as CredModel
+from apiserver.apimodels.users import CreateRequest as Users_CreateRequest
+from apiserver.bll.user import UserBLL
+from apiserver.config_repo import config
+from apiserver.config.info import get_version, get_build_number
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.auth import User, Role, Credentials
+from apiserver.database.model.company import Company
+from apiserver.service_repo import APICall, ServiceRepo
+from apiserver.service_repo.auth import Identity, Token, get_client_id, get_secret_key
 
 log = config.logger("AuthBLL")
 
@@ -62,6 +54,9 @@ class AuthBLL:
                 identity=identity,
                 entities=entities,
                 expiration_sec=expiration_sec,
+                api_version=str(ServiceRepo.max_endpoint_version()),
+                server_version=str(get_version()),
+                server_build=str(get_build_number()),
             )
 
             return GetTokenResponse(token=token.decode("ascii"))

apiserver/bll/event/debug_images_iterator.py

@@ -0,0 +1,476 @@
+from collections import defaultdict
+from concurrent.futures.thread import ThreadPoolExecutor
+from functools import partial
+from itertools import chain
+from operator import attrgetter, itemgetter
+from typing import Sequence, Tuple, Optional, Mapping
+
+import attr
+import dpath
+from boltons.iterutils import bucketize
+from elasticsearch import Elasticsearch
+from jsonmodels.fields import StringField, ListField, IntField
+from jsonmodels.models import Base
+from redis import StrictRedis
+
+from apiserver.apierrors import errors
+from apiserver.apimodels import JsonSerializableMixin
+from apiserver.bll.event.event_common import (
+    EventSettings,
+    check_empty_data,
+    search_company_events,
+    EventType,
+)
+from apiserver.bll.redis_cache_manager import RedisCacheManager
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.task.metrics import MetricEventStats
+from apiserver.database.model.task.task import Task
+from apiserver.timing_context import TimingContext
+
+
+class VariantScrollState(Base):
+    name: str = StringField(required=True)
+    recycle_url_marker: str = StringField()
+    last_invalid_iteration: int = IntField()
+
+
+class MetricScrollState(Base):
+    task: str = StringField(required=True)
+    name: str = StringField(required=True)
+    last_min_iter: Optional[int] = IntField()
+    last_max_iter: Optional[int] = IntField()
+    timestamp: int = IntField(default=0)
+    variants: Sequence[VariantScrollState] = ListField([VariantScrollState])
+
+    def reset(self):
+        """Reset the scrolling state for the metric"""
+        self.last_min_iter = self.last_max_iter = None
+
+
+class DebugImageEventsScrollState(Base, JsonSerializableMixin):
+    id: str = StringField(required=True)
+    metrics: Sequence[MetricScrollState] = ListField([MetricScrollState])
+    warning: str = StringField()
+
+
+@attr.s(auto_attribs=True)
+class DebugImagesResult(object):
+    metric_events: Sequence[tuple] = []
+    next_scroll_id: str = None
+
+
+class DebugImagesIterator:
+    EVENT_TYPE = EventType.metrics_image
+
+    def __init__(self, redis: StrictRedis, es: Elasticsearch):
+        self.es = es
+        self.cache_manager = RedisCacheManager(
+            state_class=DebugImageEventsScrollState,
+            redis=redis,
+            expiration_interval=EventSettings.state_expiration_sec,
+        )
+
+    def get_task_events(
+        self,
+        company_id: str,
+        metrics: Sequence[Tuple[str, str]],
+        iter_count: int,
+        navigate_earlier: bool = True,
+        refresh: bool = False,
+        state_id: str = None,
+    ) -> DebugImagesResult:
+        if check_empty_data(self.es, company_id, self.EVENT_TYPE):
+            return DebugImagesResult()
+
+        def init_state(state_: DebugImageEventsScrollState):
+            unique_metrics = set(metrics)
+            state_.metrics = self._init_metric_states(company_id, list(unique_metrics))
+
+        def validate_state(state_: DebugImageEventsScrollState):
+            """
+            Validate that the metrics stored in the state are the same
+            as requested in the current call.
+            Refresh the state if requested
+            """
+            state_metrics = set((m.task, m.name) for m in state_.metrics)
+            if state_metrics != set(metrics):
+                raise errors.bad_request.InvalidScrollId(
+                    "Task metrics stored in the state do not match the passed ones",
+                    scroll_id=state_.id,
+                )
+            if refresh:
+                self._reinit_outdated_metric_states(company_id, state_)
+                for metric_state in state_.metrics:
+                    metric_state.reset()
+
+        with self.cache_manager.get_or_create_state(
+            state_id=state_id, init_state=init_state, validate_state=validate_state
+        ) as state:
+            res = DebugImagesResult(next_scroll_id=state.id)
+            with ThreadPoolExecutor(EventSettings.max_workers) as pool:
+                res.metric_events = list(
+                    pool.map(
+                        partial(
+                            self._get_task_metric_events,
+                            company_id=company_id,
+                            iter_count=iter_count,
+                            navigate_earlier=navigate_earlier,
+                        ),
+                        state.metrics,
+                    )
+                )
+
+            return res
+
+    def _reinit_outdated_metric_states(
+        self, company_id, state: DebugImageEventsScrollState
+    ):
+        """
+        Determines the metrics for which new debug image events were added
+        since their states were initialized and reinits these states
+        """
+        task_ids = set(metric.task for metric in state.metrics)
+        tasks = Task.objects(id__in=list(task_ids), company=company_id).only(
+            "id", "metric_stats"
+        )
+
+        def get_last_update_times_for_task_metrics(task: Task) -> Sequence[Tuple]:
+            """For metrics that reported debug image events get tuples of task_id/metric_name and last update times"""
+            metric_stats: Mapping[str, MetricEventStats] = task.metric_stats
+            if not metric_stats:
+                return []
+
+            return [
+                (
+                    (task.id, stats.metric),
+                    stats.event_stats_by_type[self.EVENT_TYPE.value].last_update,
+                )
+                for stats in metric_stats.values()
+                if self.EVENT_TYPE.value in stats.event_stats_by_type
+            ]
+
+        update_times = dict(
+            chain.from_iterable(
+                get_last_update_times_for_task_metrics(task) for task in tasks
+            )
+        )
+        outdated_metrics = [
+            metric
+            for metric in state.metrics
+            if (metric.task, metric.name) in update_times
+            and update_times[metric.task, metric.name] > metric.timestamp
+        ]
+        state.metrics = [
+            *(metric for metric in state.metrics if metric not in outdated_metrics),
+            *(
+                self._init_metric_states(
+                    company_id,
+                    [(metric.task, metric.name) for metric in outdated_metrics],
+                )
+            ),
+        ]
+
+    def _init_metric_states(
+        self, company_id: str, metrics: Sequence[Tuple[str, str]]
+    ) -> Sequence[MetricScrollState]:
+        """
+        Returned initialized metric scroll stated for the requested task metrics
+        """
+        tasks = defaultdict(list)
+        for (task, metric) in metrics:
+            tasks[task].append(metric)
+
+        with ThreadPoolExecutor(EventSettings.max_workers) as pool:
+            return list(
+                chain.from_iterable(
+                    pool.map(
+                        partial(
+                            self._init_metric_states_for_task, company_id=company_id
+                        ),
+                        tasks.items(),
+                    )
+                )
+            )
+
+    def _init_metric_states_for_task(
+        self, task_metrics: Tuple[str, Sequence[str]], company_id: str
+    ) -> Sequence[MetricScrollState]:
+        """
+        Return metric scroll states for the task filled with the variant states
+        for the variants that reported any debug images
+        """
+        task, metrics = task_metrics
+        es_req: dict = {
+            "size": 0,
+            "query": {
+                "bool": {
+                    "must": [
+                        {"term": {"task": task}},
+                        {"terms": {"metric": metrics}},
+                        {"exists": {"field": "url"}},
+                    ]
+                }
+            },
+            "aggs": {
+                "metrics": {
+                    "terms": {
+                        "field": "metric",
+                        "size": EventSettings.max_metrics_count,
+                        "order": {"_key": "asc"},
+                    },
+                    "aggs": {
+                        "last_event_timestamp": {"max": {"field": "timestamp"}},
+                        "variants": {
+                            "terms": {
+                                "field": "variant",
+                                "size": EventSettings.max_variants_count,
+                                "order": {"_key": "asc"},
+                            },
+                            "aggs": {
+                                "urls": {
+                                    "terms": {
+                                        "field": "url",
+                                        "order": {"max_iter": "desc"},
+                                        "size": 1,  # we need only one url from the most recent iteration
+                                    },
+                                    "aggs": {
+                                        "max_iter": {"max": {"field": "iter"}},
+                                        "iters": {
+                                            "top_hits": {
+                                                "sort": {"iter": {"order": "desc"}},
+                                                "size": 2,  # need two last iterations so that we can take
+                                                # the second one as invalid
+                                                "_source": "iter",
+                                            }
+                                        },
+                                    },
+                                }
+                            },
+                        },
+                    },
+                }
+            },
+        }
+
+        with translate_errors_context(), TimingContext("es", "_init_metric_states"):
+            es_res = search_company_events(
+                self.es,
+                company_id=company_id,
+                event_type=self.EVENT_TYPE,
+                body=es_req,
+            )
+        if "aggregations" not in es_res:
+            return []
+
+        def init_variant_scroll_state(variant: dict):
+            """
+            Return new variant scroll state for the passed variant bucket
+            If the image urls get recycled then fill the last_invalid_iteration field
+            """
+            state = VariantScrollState(name=variant["key"])
+            top_iter_url = dpath.get(variant, "urls/buckets")[0]
+            iters = dpath.get(top_iter_url, "iters/hits/hits")
+            if len(iters) > 1:
+                state.last_invalid_iteration = dpath.get(iters[1], "_source/iter")
+            return state
+
+        return [
+            MetricScrollState(
+                task=task,
+                name=metric["key"],
+                variants=[
+                    init_variant_scroll_state(variant)
+                    for variant in dpath.get(metric, "variants/buckets")
+                ],
+                timestamp=dpath.get(metric, "last_event_timestamp/value"),
+            )
+            for metric in dpath.get(es_res, "aggregations/metrics/buckets")
+        ]
+
+    def _get_task_metric_events(
+        self,
+        metric: MetricScrollState,
+        company_id: str,
+        iter_count: int,
+        navigate_earlier: bool,
+    ) -> Tuple:
+        """
+        Return task metric events grouped by iterations
+        Update metric scroll state
+        """
+        if metric.last_max_iter is None:
+            # the first fetch is always from the latest iteration to the earlier ones
+            navigate_earlier = True
+
+        must_conditions = [
+            {"term": {"task": metric.task}},
+            {"term": {"metric": metric.name}},
+            {"exists": {"field": "url"}},
+        ]
+        must_not_conditions = []
+
+        range_condition = None
+        if navigate_earlier and metric.last_min_iter is not None:
+            range_condition = {"lt": metric.last_min_iter}
+        elif not navigate_earlier and metric.last_max_iter is not None:
+            range_condition = {"gt": metric.last_max_iter}
+        if range_condition:
+            must_conditions.append({"range": {"iter": range_condition}})
+
+        if navigate_earlier:
+            """
+            When navigating to earlier iterations consider only
+            variants whose invalid iterations border is lower than
+            our starting iteration. For these variants make sure
+            that only events from the valid iterations are returned 
+            """
+            if not metric.last_min_iter:
+                variants = metric.variants
+            else:
+                variants = list(
+                    v
+                    for v in metric.variants
+                    if v.last_invalid_iteration is None
+                    or v.last_invalid_iteration < metric.last_min_iter
+                )
+                if not variants:
+                    return metric.task, metric.name, []
+                must_conditions.append(
+                    {"terms": {"variant": list(v.name for v in variants)}}
+                )
+        else:
+            """
+            When navigating to later iterations all variants may be relevant.
+            For the variants whose invalid border is higher than our starting 
+            iteration make sure that only events from valid iterations are returned 
+            """
+            variants = list(
+                v
+                for v in metric.variants
+                if v.last_invalid_iteration is not None
+                and v.last_invalid_iteration > metric.last_max_iter
+            )
+
+        variants_conditions = [
+            {
+                "bool": {
+                    "must": [
+                        {"term": {"variant": v.name}},
+                        {"range": {"iter": {"lte": v.last_invalid_iteration}}},
+                    ]
+                }
+            }
+            for v in variants
+            if v.last_invalid_iteration is not None
+        ]
+        if variants_conditions:
+            must_not_conditions.append({"bool": {"should": variants_conditions}})
+
+        es_req = {
+            "size": 0,
+            "query": {
+                "bool": {"must": must_conditions, "must_not": must_not_conditions}
+            },
+            "aggs": {
+                "iters": {
+                    "terms": {
+                        "field": "iter",
+                        "size": iter_count,
+                        "order": {"_key": "desc" if navigate_earlier else "asc"},
+                    },
+                    "aggs": {
+                        "variants": {
+                            "terms": {
+                                "field": "variant",
+                                "size": EventSettings.max_variants_count,
+                                "order": {"_key": "asc"},
+                            },
+                            "aggs": {
+                                "events": {
+                                    "top_hits": {"sort": {"url": {"order": "desc"}}}
+                                }
+                            },
+                        }
+                    },
+                }
+            },
+        }
+        with translate_errors_context(), TimingContext("es", "get_debug_image_events"):
+            es_res = search_company_events(
+                self.es,
+                company_id=company_id,
+                event_type=self.EVENT_TYPE,
+                body=es_req,
+            )
+        if "aggregations" not in es_res:
+            return metric.task, metric.name, []
+
+        def get_iteration_events(variant_buckets: Sequence[dict]) -> Sequence:
+            return [
+                ev["_source"]
+                for v in variant_buckets
+                for ev in dpath.get(v, "events/hits/hits")
+            ]
+
+        iterations = [
+            {
+                "iter": it["key"],
+                "events": get_iteration_events(dpath.get(it, "variants/buckets")),
+            }
+            for it in dpath.get(es_res, "aggregations/iters/buckets")
+        ]
+        if not navigate_earlier:
+            iterations.sort(key=itemgetter("iter"), reverse=True)
+        if iterations:
+            metric.last_max_iter = iterations[0]["iter"]
+            metric.last_min_iter = iterations[-1]["iter"]
+
+        # Commented for now since the last invalid iteration is calculated in the beginning
+        # if navigate_earlier and any(
+        #     variant.last_invalid_iteration is None for variant in variants
+        # ):
+        #     """
+        #     Variants validation flags due to recycling can
+        #     be set only on navigation to earlier frames
+        #     """
+        #     iterations = self._update_variants_invalid_iterations(variants, iterations)
+
+        return metric.task, metric.name, iterations
+
+    @staticmethod
+    def _update_variants_invalid_iterations(
+        variants: Sequence[VariantScrollState], iterations: Sequence[dict]
+    ) -> Sequence[dict]:
+        """
+        This code is currently not in used since the invalid iterations
+        are calculated during MetricState initialization
+        For variants that do not have recycle url marker set it from the
+        first event
+        For variants that do not have last_invalid_iteration set check if the
+        recycle marker was reached on a certain iteration and set it to the
+        corresponding iteration
+        For variants that have a newly set last_invalid_iteration remove
+        events from the invalid iterations
+        Return the updated iterations list
+        """
+        variants_lookup = bucketize(variants, attrgetter("name"))
+        for it in iterations:
+            iteration = it["iter"]
+            events_to_remove = []
+            for event in it["events"]:
+                variant = variants_lookup[event["variant"]][0]
+                if (
+                    variant.last_invalid_iteration
+                    and variant.last_invalid_iteration >= iteration
+                ):
+                    events_to_remove.append(event)
+                    continue
+                event_url = event.get("url")
+                if not variant.recycle_url_marker:
+                    variant.recycle_url_marker = event_url
+                elif variant.recycle_url_marker == event_url:
+                    variant.last_invalid_iteration = iteration
+                    events_to_remove.append(event)
+            if events_to_remove:
+                it["events"] = [ev for ev in it["events"] if ev not in events_to_remove]
+        return [it for it in iterations if it["events"]]

apiserver/bll/event/debug_sample_history.py

@@ -0,0 +1,375 @@
+import operator
+from typing import Sequence, Tuple, Optional
+
+import attr
+from boltons.iterutils import first
+from elasticsearch import Elasticsearch
+from jsonmodels.fields import StringField, ListField, IntField, BoolField
+from jsonmodels.models import Base
+from redis import StrictRedis
+
+from apiserver.apierrors import errors
+from apiserver.apimodels import JsonSerializableMixin
+from apiserver.bll.event.event_common import (
+    EventSettings,
+    EventType,
+    check_empty_data,
+    search_company_events,
+)
+from apiserver.bll.redis_cache_manager import RedisCacheManager
+from apiserver.database.errors import translate_errors_context
+from apiserver.timing_context import TimingContext
+from apiserver.utilities.dicts import nested_get
+
+
+class VariantState(Base):
+    name: str = StringField(required=True)
+    min_iteration: int = IntField()
+    max_iteration: int = IntField()
+
+
+class DebugSampleHistoryState(Base, JsonSerializableMixin):
+    id: str = StringField(required=True)
+    iteration: int = IntField()
+    variant: str = StringField()
+    task: str = StringField()
+    metric: str = StringField()
+    reached_first: bool = BoolField()
+    reached_last: bool = BoolField()
+    variant_states: Sequence[VariantState] = ListField([VariantState])
+    warning: str = StringField()
+
+
+@attr.s(auto_attribs=True)
+class DebugSampleHistoryResult(object):
+    scroll_id: str = None
+    event: dict = None
+    min_iteration: int = None
+    max_iteration: int = None
+
+
+class DebugSampleHistory:
+    EVENT_TYPE = EventType.metrics_image
+
+    def __init__(self, redis: StrictRedis, es: Elasticsearch):
+        self.es = es
+        self.cache_manager = RedisCacheManager(
+            state_class=DebugSampleHistoryState,
+            redis=redis,
+            expiration_interval=EventSettings.state_expiration_sec,
+        )
+
+    def get_next_debug_image(
+        self, company_id: str, task: str, state_id: str, navigate_earlier: bool
+    ) -> DebugSampleHistoryResult:
+        """
+        Get the debug image for next/prev variant on the current iteration
+        If does not exist then try getting image for the first/last variant from next/prev iteration
+        """
+        res = DebugSampleHistoryResult(scroll_id=state_id)
+        state = self.cache_manager.get_state(state_id)
+        if not state or state.task != task:
+            raise errors.bad_request.InvalidScrollId(scroll_id=state_id)
+
+        if check_empty_data(self.es, company_id=company_id, event_type=self.EVENT_TYPE):
+            return res
+
+        image = self._get_next_for_current_iteration(
+            company_id=company_id, navigate_earlier=navigate_earlier, state=state
+        ) or self._get_next_for_another_iteration(
+            company_id=company_id, navigate_earlier=navigate_earlier, state=state
+        )
+        if not image:
+            return res
+
+        self._fill_res_and_update_state(image=image, res=res, state=state)
+        self.cache_manager.set_state(state=state)
+        return res
+
+    def _fill_res_and_update_state(
+        self, image: dict, res: DebugSampleHistoryResult, state: DebugSampleHistoryState
+    ):
+        state.variant = image["variant"]
+        state.iteration = image["iter"]
+        res.event = image
+        var_state = first(s for s in state.variant_states if s.name == state.variant)
+        if var_state:
+            res.min_iteration = var_state.min_iteration
+            res.max_iteration = var_state.max_iteration
+
+    def _get_next_for_current_iteration(
+        self, company_id: str, navigate_earlier: bool, state: DebugSampleHistoryState
+    ) -> Optional[dict]:
+        """
+        Get the image for next (if navigated earlier is False) or previous variant sorted by name for the same iteration
+        Only variants for which the iteration falls into their valid range are considered
+        Return None if no such variant or image is found
+        """
+        cmp = operator.lt if navigate_earlier else operator.gt
+        variants = [
+            var_state
+            for var_state in state.variant_states
+            if cmp(var_state.name, state.variant)
+            and var_state.min_iteration <= state.iteration
+        ]
+        if not variants:
+            return
+
+        must_conditions = [
+            {"term": {"task": state.task}},
+            {"term": {"metric": state.metric}},
+            {"terms": {"variant": [v.name for v in variants]}},
+            {"term": {"iter": state.iteration}},
+            {"exists": {"field": "url"}},
+        ]
+        es_req = {
+            "size": 1,
+            "sort": {"variant": "desc" if navigate_earlier else "asc"},
+            "query": {"bool": {"must": must_conditions}},
+        }
+
+        with translate_errors_context(), TimingContext(
+            "es", "get_next_for_current_iteration"
+        ):
+            es_res = search_company_events(
+                self.es, company_id=company_id, event_type=self.EVENT_TYPE, body=es_req
+            )
+
+        hits = nested_get(es_res, ("hits", "hits"))
+        if not hits:
+            return
+
+        return hits[0]["_source"]
+
+    def _get_next_for_another_iteration(
+        self, company_id: str, navigate_earlier: bool, state: DebugSampleHistoryState
+    ) -> Optional[dict]:
+        """
+        Get the image for the first variant for the next iteration (if navigate_earlier is set to False)
+        or from the last variant for the previous iteration (otherwise)
+        The variants for which the image falls in invalid range are discarded
+        If no suitable image is found then None is returned
+        """
+
+        must_conditions = [
+            {"term": {"task": state.task}},
+            {"term": {"metric": state.metric}},
+            {"exists": {"field": "url"}},
+        ]
+
+        if navigate_earlier:
+            range_operator = "lt"
+            order = "desc"
+            variants = [
+                var_state
+                for var_state in state.variant_states
+                if var_state.min_iteration < state.iteration
+            ]
+        else:
+            range_operator = "gt"
+            order = "asc"
+            variants = state.variant_states
+
+        if not variants:
+            return
+
+        variants_conditions = [
+            {
+                "bool": {
+                    "must": [
+                        {"term": {"variant": v.name}},
+                        {"range": {"iter": {"gte": v.min_iteration}}},
+                    ]
+                }
+            }
+            for v in variants
+        ]
+        must_conditions.append({"bool": {"should": variants_conditions}})
+        must_conditions.append({"range": {"iter": {range_operator: state.iteration}}},)
+
+        es_req = {
+            "size": 1,
+            "sort": [{"iter": order}, {"variant": order}],
+            "query": {"bool": {"must": must_conditions}},
+        }
+        with translate_errors_context(), TimingContext(
+            "es", "get_next_for_another_iteration"
+        ):
+            es_res = search_company_events(
+                self.es, company_id=company_id, event_type=self.EVENT_TYPE, body=es_req
+            )
+
+        hits = nested_get(es_res, ("hits", "hits"))
+        if not hits:
+            return
+
+        return hits[0]["_source"]
+
+    def get_debug_image_for_variant(
+        self,
+        company_id: str,
+        task: str,
+        metric: str,
+        variant: str,
+        iteration: Optional[int] = None,
+        refresh: bool = False,
+        state_id: str = None,
+    ) -> DebugSampleHistoryResult:
+        """
+        Get the debug image for the requested iteration or the latest before it
+        If the iteration is not passed then get the latest event
+        """
+        res = DebugSampleHistoryResult()
+        if check_empty_data(self.es, company_id=company_id, event_type=self.EVENT_TYPE):
+            return res
+
+        def init_state(state_: DebugSampleHistoryState):
+            state_.task = task
+            state_.metric = metric
+            self._reset_variant_states(company_id=company_id, state=state_)
+
+        def validate_state(state_: DebugSampleHistoryState):
+            if state_.task != task or state_.metric != metric:
+                raise errors.bad_request.InvalidScrollId(
+                    "Task and metric stored in the state do not match the passed ones",
+                    scroll_id=state_.id,
+                )
+            if refresh:
+                self._reset_variant_states(company_id=company_id, state=state_)
+
+        state: DebugSampleHistoryState
+        with self.cache_manager.get_or_create_state(
+            state_id=state_id, init_state=init_state, validate_state=validate_state,
+        ) as state:
+            res.scroll_id = state.id
+
+            var_state = first(s for s in state.variant_states if s.name == variant)
+            if not var_state:
+                return res
+
+            res.min_iteration = var_state.min_iteration
+            res.max_iteration = var_state.max_iteration
+
+            must_conditions = [
+                {"term": {"task": task}},
+                {"term": {"metric": metric}},
+                {"term": {"variant": variant}},
+                {"exists": {"field": "url"}},
+            ]
+            if iteration is not None:
+                must_conditions.append(
+                    {
+                        "range": {
+                            "iter": {"lte": iteration, "gte": var_state.min_iteration}
+                        }
+                    }
+                )
+            else:
+                must_conditions.append(
+                    {"range": {"iter": {"gte": var_state.min_iteration}}}
+                )
+
+            es_req = {
+                "size": 1,
+                "sort": {"iter": "desc"},
+                "query": {"bool": {"must": must_conditions}},
+            }
+
+            with translate_errors_context(), TimingContext(
+                "es", "get_debug_image_for_variant"
+            ):
+                es_res = search_company_events(
+                    self.es,
+                    company_id=company_id,
+                    event_type=self.EVENT_TYPE,
+                    body=es_req,
+                )
+
+            hits = nested_get(es_res, ("hits", "hits"))
+            if not hits:
+                return res
+
+            self._fill_res_and_update_state(
+                image=hits[0]["_source"], res=res, state=state
+            )
+            return res
+
+    def _reset_variant_states(self, company_id: str, state: DebugSampleHistoryState):
+        variant_iterations = self._get_variant_iterations(
+            company_id=company_id, task=state.task, metric=state.metric
+        )
+        state.variant_states = [
+            VariantState(name=var_name, min_iteration=min_iter, max_iteration=max_iter)
+            for var_name, min_iter, max_iter in variant_iterations
+        ]
+
+    def _get_variant_iterations(
+        self,
+        company_id: str,
+        task: str,
+        metric: str,
+        variants: Optional[Sequence[str]] = None,
+    ) -> Sequence[Tuple[str, int, int]]:
+        """
+        Return valid min and max iterations that the task reported images
+        The min iteration is the lowest iteration that contains non-recycled image url
+        """
+        must = [
+            {"term": {"task": task}},
+            {"term": {"metric": metric}},
+            {"exists": {"field": "url"}},
+        ]
+        if variants:
+            must.append({"terms": {"variant": variants}})
+
+        es_req: dict = {
+            "size": 0,
+            "query": {"bool": {"must": must}},
+            "aggs": {
+                "variants": {
+                    # all variants that sent debug images
+                    "terms": {
+                        "field": "variant",
+                        "size": EventSettings.max_variants_count,
+                        "order": {"_key": "asc"},
+                    },
+                    "aggs": {
+                        "last_iter": {"max": {"field": "iter"}},
+                        "urls": {
+                            # group by urls and choose the minimal iteration
+                            # from all the maximal iterations per url
+                            "terms": {
+                                "field": "url",
+                                "order": {"max_iter": "asc"},
+                                "size": 1,
+                            },
+                            "aggs": {
+                                # find max iteration for each url
+                                "max_iter": {"max": {"field": "iter"}}
+                            },
+                        },
+                    },
+                }
+            },
+        }
+
+        with translate_errors_context(), TimingContext(
+            "es", "get_debug_image_iterations"
+        ):
+            es_res = search_company_events(
+                self.es, company_id=company_id, event_type=self.EVENT_TYPE, body=es_req
+            )
+
+        def get_variant_data(variant_bucket: dict) -> Tuple[str, int, int]:
+            variant = variant_bucket["key"]
+            urls = nested_get(variant_bucket, ("urls", "buckets"))
+            min_iter = int(urls[0]["max_iter"]["value"])
+            max_iter = int(variant_bucket["last_iter"]["value"])
+            return variant, min_iter, max_iter
+
+        return [
+            get_variant_data(variant_bucket)
+            for variant_bucket in nested_get(
+                es_res, ("aggregations", "variants", "buckets")
+            )
+        ]

apiserver/bll/event/event_bll.py

@@ -0,0 +1,894 @@
+import base64
+import hashlib
+import zlib
+from collections import defaultdict
+from contextlib import closing
+from datetime import datetime
+from operator import attrgetter
+from typing import Sequence, Set, Tuple, Optional, Dict
+
+import six
+from elasticsearch import helpers
+from mongoengine import Q
+from nested_dict import nested_dict
+
+from apiserver.bll.event.debug_sample_history import DebugSampleHistory
+from apiserver.bll.event.event_common import (
+    EventType,
+    EventSettings,
+    get_index_name,
+    check_empty_data,
+    search_company_events,
+    delete_company_events,
+)
+from apiserver.bll.util import parallel_chunked_decorator
+from apiserver.database import utils as dbutils
+from apiserver.es_factory import es_factory
+from apiserver.apierrors import errors
+from apiserver.bll.event.debug_images_iterator import DebugImagesIterator
+from apiserver.bll.event.event_metrics import EventMetrics
+from apiserver.bll.event.log_events_iterator import LogEventsIterator, TaskEventsResult
+from apiserver.bll.task import TaskBLL
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.task.task import Task, TaskStatus
+from apiserver.redis_manager import redman
+from apiserver.timing_context import TimingContext
+from apiserver.tools import safe_get
+from apiserver.utilities.dicts import flatten_nested_items
+
+# noinspection PyTypeChecker
+from apiserver.utilities.json import loads
+
+EVENT_TYPES = set(map(attrgetter("value"), EventType))
+LOCKED_TASK_STATUSES = (TaskStatus.publishing, TaskStatus.published)
+
+
+class PlotFields:
+    valid_plot = "valid_plot"
+    plot_len = "plot_len"
+    plot_str = "plot_str"
+    plot_data = "plot_data"
+
+
+class EventBLL(object):
+    id_fields = ("task", "iter", "metric", "variant", "key")
+    empty_scroll = "FFFF"
+
+    def __init__(self, events_es=None, redis=None):
+        self.es = events_es or es_factory.connect("events")
+        self._metrics = EventMetrics(self.es)
+        self._skip_iteration_for_metric = set(
+            config.get("services.events.ignore_iteration.metrics", [])
+        )
+        self.redis = redis or redman.connection("apiserver")
+        self.debug_images_iterator = DebugImagesIterator(es=self.es, redis=self.redis)
+        self.debug_sample_history = DebugSampleHistory(es=self.es, redis=self.redis)
+        self.log_events_iterator = LogEventsIterator(es=self.es)
+
+    @property
+    def metrics(self) -> EventMetrics:
+        return self._metrics
+
+    @staticmethod
+    def _get_valid_tasks(company_id, task_ids: Set, allow_locked_tasks=False) -> Set:
+        """Verify that task exists and can be updated"""
+        if not task_ids:
+            return set()
+
+        with translate_errors_context(), TimingContext("mongo", "task_by_ids"):
+            query = Q(id__in=task_ids, company=company_id)
+            if not allow_locked_tasks:
+                query &= Q(status__nin=LOCKED_TASK_STATUSES)
+            res = Task.objects(query).only("id")
+            return {r.id for r in res}
+
+    def add_events(
+        self, company_id, events, worker, allow_locked_tasks=False
+    ) -> Tuple[int, int, dict]:
+        actions = []
+        task_ids = set()
+        task_iteration = defaultdict(lambda: 0)
+        task_last_scalar_events = nested_dict(
+            3, dict
+        )  # task_id -> metric_hash -> variant_hash -> MetricEvent
+        task_last_events = nested_dict(
+            3, dict
+        )  # task_id -> metric_hash -> event_type -> MetricEvent
+        errors_per_type = defaultdict(int)
+        valid_tasks = self._get_valid_tasks(
+            company_id,
+            task_ids={
+                event["task"] for event in events if event.get("task") is not None
+            },
+            allow_locked_tasks=allow_locked_tasks,
+        )
+
+        for event in events:
+            # remove spaces from event type
+            event_type = event.get("type")
+            if event_type is None:
+                errors_per_type["Event must have a 'type' field"] += 1
+                continue
+
+            event_type = event_type.replace(" ", "_")
+            if event_type not in EVENT_TYPES:
+                errors_per_type[f"Invalid event type {event_type}"] += 1
+                continue
+
+            task_id = event.get("task")
+            if task_id is None:
+                errors_per_type["Event must have a 'task' field"] += 1
+                continue
+
+            if task_id not in valid_tasks:
+                errors_per_type["Invalid task id"] += 1
+                continue
+
+            event["type"] = event_type
+
+            # @timestamp indicates the time the event is written, not when it happened
+            event["@timestamp"] = es_factory.get_es_timestamp_str()
+
+            # for backward bomba-tavili-tea
+            if "ts" in event:
+                event["timestamp"] = event.pop("ts")
+
+            # set timestamp and worker if not sent
+            if "timestamp" not in event:
+                event["timestamp"] = es_factory.get_timestamp_millis()
+
+            if "worker" not in event:
+                event["worker"] = worker
+
+            # force iter to be a long int
+            iter = event.get("iter")
+            if iter is not None:
+                iter = int(iter)
+                event["iter"] = iter
+
+            # used to have "values" to indicate array. no need anymore
+            if "values" in event:
+                event["value"] = event["values"]
+                del event["values"]
+
+            event["metric"] = event.get("metric") or ""
+            event["variant"] = event.get("variant") or ""
+
+            index_name = get_index_name(company_id, event_type)
+            es_action = {
+                "_op_type": "index",  # overwrite if exists with same ID
+                "_index": index_name,
+                "_source": event,
+            }
+
+            # for "log" events, don't assing custom _id - whatever is sent, is written (not overwritten)
+            if event_type != EventType.task_log.value:
+                es_action["_id"] = self._get_event_id(event)
+            else:
+                es_action["_id"] = dbutils.id()
+
+            task_ids.add(task_id)
+            if (
+                iter is not None
+                and event.get("metric") not in self._skip_iteration_for_metric
+            ):
+                task_iteration[task_id] = max(iter, task_iteration[task_id])
+
+            self._update_last_metric_events_for_task(
+                last_events=task_last_events[task_id], event=event,
+            )
+            if event_type == EventType.metrics_scalar.value:
+                self._update_last_scalar_events_for_task(
+                    last_events=task_last_scalar_events[task_id], event=event
+                )
+
+            actions.append(es_action)
+
+        action: Dict[dict]
+        plot_actions = [
+            action["_source"]
+            for action in actions
+            if action["_source"]["type"] == EventType.metrics_plot.value
+        ]
+        if plot_actions:
+            self.validate_and_compress_plots(
+                plot_actions,
+                validate_json=config.get("services.events.validate_plot_str", False),
+                compression_threshold=config.get(
+                    "services.events.plot_compression_threshold", 100_000
+                ),
+            )
+
+        added = 0
+        if actions:
+            chunk_size = 500
+            with translate_errors_context(), TimingContext("es", "events_add_batch"):
+                # TODO: replace it with helpers.parallel_bulk in the future once the parallel pool leak is fixed
+                with closing(
+                    helpers.streaming_bulk(
+                        self.es,
+                        actions,
+                        chunk_size=chunk_size,
+                        # thread_count=8,
+                        refresh=True,
+                    )
+                ) as it:
+                    for success, info in it:
+                        if success:
+                            added += 1
+                        else:
+                            errors_per_type["Error when indexing events batch"] += 1
+
+                remaining_tasks = set()
+                now = datetime.utcnow()
+                for task_id in task_ids:
+                    # Update related tasks. For reasons of performance, we prefer to update
+                    # all of them and not only those who's events were successful
+                    updated = self._update_task(
+                        company_id=company_id,
+                        task_id=task_id,
+                        now=now,
+                        iter_max=task_iteration.get(task_id),
+                        last_scalar_events=task_last_scalar_events.get(task_id),
+                        last_events=task_last_events.get(task_id),
+                    )
+
+                    if not updated:
+                        remaining_tasks.add(task_id)
+                        continue
+
+                if remaining_tasks:
+                    TaskBLL.set_last_update(
+                        remaining_tasks, company_id, last_update=now
+                    )
+
+        if not added:
+            raise errors.bad_request.EventsNotAdded(**errors_per_type)
+
+        errors_count = sum(errors_per_type.values())
+        return added, errors_count, errors_per_type
+
+    @parallel_chunked_decorator(chunk_size=10)
+    def validate_and_compress_plots(
+        self,
+        plot_events: Sequence[dict],
+        validate_json: bool,
+        compression_threshold: int,
+    ):
+        for event in plot_events:
+            validate = validate_json and not event.pop("skip_validation", False)
+            plot_str = event.get(PlotFields.plot_str)
+            if not plot_str:
+                event[PlotFields.plot_len] = 0
+                if validate:
+                    event[PlotFields.valid_plot] = False
+                continue
+
+            plot_len = len(plot_str)
+            event[PlotFields.plot_len] = plot_len
+            if validate:
+                event[PlotFields.valid_plot] = self._is_valid_json(plot_str)
+            if compression_threshold and plot_len >= compression_threshold:
+                event[PlotFields.plot_data] = base64.encodebytes(
+                    zlib.compress(plot_str.encode(), level=1)
+                ).decode("ascii")
+                event.pop(PlotFields.plot_str, None)
+
+    @parallel_chunked_decorator(chunk_size=10)
+    def uncompress_plots(self, plot_events: Sequence[dict]):
+        for event in plot_events:
+            plot_data = event.pop(PlotFields.plot_data, None)
+            if plot_data and event.get(PlotFields.plot_str) is None:
+                event[PlotFields.plot_str] = zlib.decompress(
+                    base64.b64decode(plot_data)
+                ).decode()
+
+    @staticmethod
+    def _is_valid_json(text: str) -> bool:
+        """Check str for valid json"""
+        if not text:
+            return False
+        try:
+            loads(text)
+        except Exception:
+            return False
+        return True
+
+    def _update_last_scalar_events_for_task(self, last_events, event):
+        """
+        Update last_events structure with the provided event details if this event is more
+        recent than the currently stored event for its metric/variant combination.
+
+        last_events contains [hashed_metric_name -> hashed_variant_name -> event]. Keys are hashed to avoid mongodb
+        key conflicts due to invalid characters and/or long field names.
+        """
+        metric = event.get("metric")
+        variant = event.get("variant")
+        if not (metric and variant):
+            return
+
+        metric_hash = dbutils.hash_field_name(metric)
+        variant_hash = dbutils.hash_field_name(variant)
+
+        last_event = last_events[metric_hash][variant_hash]
+        event_iter = event.get("iter", 0)
+        event_timestamp = event.get("timestamp", 0)
+        value = event.get("value")
+        if value is not None and (
+            (event_iter, event_timestamp)
+            >= (
+                last_event.get("iter", event_iter),
+                last_event.get("timestamp", event_timestamp),
+            )
+        ):
+            event_data = {
+                k: event[k]
+                for k in ("value", "metric", "variant", "iter", "timestamp")
+                if k in event
+            }
+            event_data["min_value"] = min(value, last_event.get("min_value", value))
+            event_data["max_value"] = max(value, last_event.get("max_value", value))
+            last_events[metric_hash][variant_hash] = event_data
+
+    def _update_last_metric_events_for_task(self, last_events, event):
+        """
+        Update last_events structure with the provided event details if this event is more
+        recent than the currently stored event for its metric/event_type combination.
+        last_events contains [metric_name -> event_type -> event]
+        """
+        metric = event.get("metric")
+        event_type = event.get("type")
+        if not (metric and event_type):
+            return
+
+        timestamp = last_events[metric][event_type].get("timestamp", None)
+        if timestamp is None or timestamp < event["timestamp"]:
+            last_events[metric][event_type] = event
+
+    def _update_task(
+        self,
+        company_id,
+        task_id,
+        now,
+        iter_max=None,
+        last_scalar_events=None,
+        last_events=None,
+    ):
+        """
+        Update task information in DB with aggregated results after handling event(s) related to this task.
+
+        This updates the task with the highest iteration value encountered during the last events update, as well
+        as the latest metric/variant scalar values reported (according to the report timestamp) and the task's last
+        update time.
+        """
+        fields = {}
+
+        if iter_max is not None:
+            fields["last_iteration_max"] = iter_max
+
+        if last_scalar_events:
+            fields["last_scalar_values"] = list(
+                flatten_nested_items(
+                    last_scalar_events,
+                    nesting=2,
+                    include_leaves=[
+                        "value",
+                        "min_value",
+                        "max_value",
+                        "metric",
+                        "variant",
+                    ],
+                )
+            )
+
+        if last_events:
+            fields["last_events"] = last_events
+
+        if not fields:
+            return False
+
+        return TaskBLL.update_statistics(task_id, company_id, last_update=now, **fields)
+
+    def _get_event_id(self, event):
+        id_values = (str(event[field]) for field in self.id_fields if field in event)
+        return hashlib.md5("-".join(id_values).encode()).hexdigest()
+
+    def scroll_task_events(
+        self,
+        company_id: str,
+        task_id: str,
+        order: str,
+        event_type: EventType,
+        batch_size=10000,
+        scroll_id=None,
+    ):
+        if scroll_id == self.empty_scroll:
+            return [], scroll_id, 0
+
+        if scroll_id:
+            with translate_errors_context(), TimingContext("es", "task_log_events"):
+                es_res = self.es.scroll(scroll_id=scroll_id, scroll="1h")
+        else:
+            size = min(batch_size, 10000)
+            if check_empty_data(self.es, company_id=company_id, event_type=event_type):
+                return [], None, 0
+
+            es_req = {
+                "size": size,
+                "sort": {"timestamp": {"order": order}},
+                "query": {"bool": {"must": [{"term": {"task": task_id}}]}},
+            }
+
+            with translate_errors_context(), TimingContext("es", "scroll_task_events"):
+                es_res = search_company_events(
+                    self.es,
+                    company_id=company_id,
+                    event_type=event_type,
+                    body=es_req,
+                    scroll="1h",
+                )
+
+        events, total_events, next_scroll_id = self._get_events_from_es_res(es_res)
+        return events, next_scroll_id, total_events
+
+    def get_last_iterations_per_event_metric_variant(
+        self,
+        company_id: str,
+        task_id: str,
+        num_last_iterations: int,
+        event_type: EventType,
+    ):
+        if check_empty_data(self.es, company_id=company_id, event_type=event_type):
+            return []
+
+        es_req: dict = {
+            "size": 0,
+            "aggs": {
+                "metrics": {
+                    "terms": {
+                        "field": "metric",
+                        "size": EventSettings.max_metrics_count,
+                        "order": {"_key": "asc"},
+                    },
+                    "aggs": {
+                        "variants": {
+                            "terms": {
+                                "field": "variant",
+                                "size": EventSettings.max_variants_count,
+                                "order": {"_key": "asc"},
+                            },
+                            "aggs": {
+                                "iters": {
+                                    "terms": {
+                                        "field": "iter",
+                                        "size": num_last_iterations,
+                                        "order": {"_key": "desc"},
+                                    }
+                                }
+                            },
+                        }
+                    },
+                }
+            },
+            "query": {"bool": {"must": [{"term": {"task": task_id}}]}},
+        }
+
+        with translate_errors_context(), TimingContext(
+            "es", "task_last_iter_metric_variant"
+        ):
+            es_res = search_company_events(
+                self.es, company_id=company_id, event_type=event_type, body=es_req
+            )
+
+        if "aggregations" not in es_res:
+            return []
+
+        return [
+            (metric["key"], variant["key"], iter["key"])
+            for metric in es_res["aggregations"]["metrics"]["buckets"]
+            for variant in metric["variants"]["buckets"]
+            for iter in variant["iters"]["buckets"]
+        ]
+
+    def get_task_plots(
+        self,
+        company_id: str,
+        tasks: Sequence[str],
+        last_iterations_per_plot: int = None,
+        sort=None,
+        size: int = 500,
+        scroll_id: str = None,
+    ):
+        if scroll_id == self.empty_scroll:
+            return [], scroll_id, 0
+
+        if scroll_id:
+            with translate_errors_context(), TimingContext("es", "get_task_events"):
+                es_res = self.es.scroll(scroll_id=scroll_id, scroll="1h")
+        else:
+            event_type = EventType.metrics_plot
+            if check_empty_data(self.es, company_id=company_id, event_type=event_type):
+                return TaskEventsResult()
+
+            plot_valid_condition = {
+                "bool": {
+                    "should": [
+                        {"term": {PlotFields.valid_plot: True}},
+                        {
+                            "bool": {
+                                "must_not": {"exists": {"field": PlotFields.valid_plot}}
+                            }
+                        },
+                    ]
+                }
+            }
+            must = [plot_valid_condition]
+
+            if last_iterations_per_plot is None:
+                must.append({"terms": {"task": tasks}})
+            else:
+                should = []
+                for i, task_id in enumerate(tasks):
+                    last_iters = self.get_last_iterations_per_event_metric_variant(
+                        company_id=company_id,
+                        task_id=task_id,
+                        num_last_iterations=last_iterations_per_plot,
+                        event_type=event_type,
+                    )
+                    if not last_iters:
+                        continue
+
+                    for metric, variant, iter in last_iters:
+                        should.append(
+                            {
+                                "bool": {
+                                    "must": [
+                                        {"term": {"task": task_id}},
+                                        {"term": {"metric": metric}},
+                                        {"term": {"variant": variant}},
+                                        {"term": {"iter": iter}},
+                                    ]
+                                }
+                            }
+                        )
+                if not should:
+                    return TaskEventsResult()
+                must.append({"bool": {"should": should}})
+
+            if sort is None:
+                sort = [{"timestamp": {"order": "asc"}}]
+
+            es_req = {
+                "sort": sort,
+                "size": min(size, 10000),
+                "query": {"bool": {"must": must}},
+            }
+
+            with translate_errors_context(), TimingContext("es", "get_task_plots"):
+                es_res = search_company_events(
+                    self.es,
+                    company_id=company_id,
+                    event_type=event_type,
+                    body=es_req,
+                    ignore=404,
+                    scroll="1h",
+                )
+
+        events, total_events, next_scroll_id = self._get_events_from_es_res(es_res)
+        self.uncompress_plots(events)
+        return TaskEventsResult(
+            events=events, next_scroll_id=next_scroll_id, total_events=total_events
+        )
+
+    def _get_events_from_es_res(self, es_res: dict) -> Tuple[list, int, Optional[str]]:
+        """
+        Return events and next scroll id from the scrolled query
+        Release the scroll once it is exhausted
+        """
+        total_events = safe_get(es_res, "hits/total/value", default=0)
+        events = [doc["_source"] for doc in safe_get(es_res, "hits/hits", default=[])]
+        next_scroll_id = es_res.get("_scroll_id")
+        if next_scroll_id and not events:
+            self.es.clear_scroll(scroll_id=next_scroll_id)
+            next_scroll_id = self.empty_scroll
+
+        return events, total_events, next_scroll_id
+
+    def get_task_events(
+        self,
+        company_id: str,
+        task_id: str,
+        event_type: EventType,
+        metric=None,
+        variant=None,
+        last_iter_count=None,
+        sort=None,
+        size=500,
+        scroll_id=None,
+    ):
+        if scroll_id == self.empty_scroll:
+            return [], scroll_id, 0
+
+        if scroll_id:
+            with translate_errors_context(), TimingContext("es", "get_task_events"):
+                es_res = self.es.scroll(scroll_id=scroll_id, scroll="1h")
+        else:
+            task_ids = [task_id] if isinstance(task_id, six.string_types) else task_id
+
+            if check_empty_data(self.es, company_id=company_id, event_type=event_type):
+                return TaskEventsResult()
+
+            must = []
+            if metric:
+                must.append({"term": {"metric": metric}})
+            if variant:
+                must.append({"term": {"variant": variant}})
+
+            if last_iter_count is None:
+                must.append({"terms": {"task": task_ids}})
+            else:
+                should = []
+                for i, task_id in enumerate(task_ids):
+                    last_iters = self.get_last_iters(
+                        company_id=company_id,
+                        event_type=event_type,
+                        task_id=task_id,
+                        iters=last_iter_count,
+                    )
+                    if not last_iters:
+                        continue
+                    should.append(
+                        {
+                            "bool": {
+                                "must": [
+                                    {"term": {"task": task_id}},
+                                    {"terms": {"iter": last_iters}},
+                                ]
+                            }
+                        }
+                    )
+                if not should:
+                    return TaskEventsResult()
+                must.append({"bool": {"should": should}})
+
+            if sort is None:
+                sort = [{"timestamp": {"order": "asc"}}]
+
+            es_req = {
+                "sort": sort,
+                "size": min(size, 10000),
+                "query": {"bool": {"must": must}},
+            }
+
+            with translate_errors_context(), TimingContext("es", "get_task_events"):
+                es_res = search_company_events(
+                    self.es,
+                    company_id=company_id,
+                    event_type=event_type,
+                    body=es_req,
+                    ignore=404,
+                    scroll="1h",
+                )
+
+        events, total_events, next_scroll_id = self._get_events_from_es_res(es_res)
+        return TaskEventsResult(
+            events=events, next_scroll_id=next_scroll_id, total_events=total_events
+        )
+
+    def get_metrics_and_variants(
+        self, company_id: str, task_id: str, event_type: EventType
+    ):
+        if check_empty_data(self.es, company_id=company_id, event_type=event_type):
+            return {}
+
+        es_req = {
+            "size": 0,
+            "aggs": {
+                "metrics": {
+                    "terms": {
+                        "field": "metric",
+                        "size": EventSettings.max_metrics_count,
+                        "order": {"_key": "asc"},
+                    },
+                    "aggs": {
+                        "variants": {
+                            "terms": {
+                                "field": "variant",
+                                "size": EventSettings.max_variants_count,
+                                "order": {"_key": "asc"},
+                            }
+                        }
+                    },
+                }
+            },
+            "query": {"bool": {"must": [{"term": {"task": task_id}}]}},
+        }
+
+        with translate_errors_context(), TimingContext(
+            "es", "events_get_metrics_and_variants"
+        ):
+            es_res = search_company_events(
+                self.es, company_id=company_id, event_type=event_type, body=es_req
+            )
+
+        metrics = {}
+        for metric_bucket in es_res["aggregations"]["metrics"].get("buckets"):
+            metric = metric_bucket["key"]
+            metrics[metric] = [
+                b["key"] for b in metric_bucket["variants"].get("buckets")
+            ]
+
+        return metrics
+
+    def get_task_latest_scalar_values(self, company_id: str, task_id: str):
+        event_type = EventType.metrics_scalar
+        if check_empty_data(self.es, company_id=company_id, event_type=event_type):
+            return {}
+
+        es_req = {
+            "size": 0,
+            "query": {
+                "bool": {
+                    "must": [
+                        {"query_string": {"query": "value:>0"}},
+                        {"term": {"task": task_id}},
+                    ]
+                }
+            },
+            "aggs": {
+                "metrics": {
+                    "terms": {
+                        "field": "metric",
+                        "size": EventSettings.max_metrics_count,
+                        "order": {"_key": "asc"},
+                    },
+                    "aggs": {
+                        "variants": {
+                            "terms": {
+                                "field": "variant",
+                                "size": EventSettings.max_variants_count,
+                                "order": {"_key": "asc"},
+                            },
+                            "aggs": {
+                                "last_value": {
+                                    "top_hits": {
+                                        "docvalue_fields": ["value"],
+                                        "_source": "value",
+                                        "size": 1,
+                                        "sort": [{"iter": {"order": "desc"}}],
+                                    }
+                                },
+                                "last_timestamp": {"max": {"field": "@timestamp"}},
+                                "last_10_value": {
+                                    "top_hits": {
+                                        "docvalue_fields": ["value"],
+                                        "_source": "value",
+                                        "size": 10,
+                                        "sort": [{"iter": {"order": "desc"}}],
+                                    }
+                                },
+                            },
+                        }
+                    },
+                }
+            },
+            "_source": {"excludes": []},
+        }
+        with translate_errors_context(), TimingContext(
+            "es", "events_get_metrics_and_variants"
+        ):
+            es_res = search_company_events(
+                self.es, company_id=company_id, event_type=event_type, body=es_req
+            )
+
+        metrics = []
+        max_timestamp = 0
+        for metric_bucket in es_res["aggregations"]["metrics"].get("buckets"):
+            metric_summary = dict(name=metric_bucket["key"], variants=[])
+            for variant_bucket in metric_bucket["variants"].get("buckets"):
+                variant_name = variant_bucket["key"]
+                last_value = variant_bucket["last_value"]["hits"]["hits"][0]["fields"][
+                    "value"
+                ][0]
+                last_10_value = variant_bucket["last_10_value"]["hits"]["hits"][0][
+                    "fields"
+                ]["value"][0]
+                timestamp = variant_bucket["last_timestamp"]["value"]
+                max_timestamp = max(timestamp, max_timestamp)
+                metric_summary["variants"].append(
+                    dict(
+                        name=variant_name,
+                        last_value=last_value,
+                        last_10_value=last_10_value,
+                    )
+                )
+            metrics.append(metric_summary)
+        return metrics, max_timestamp
+
+    def get_vector_metrics_per_iter(self, company_id, task_id, metric, variant):
+        event_type = EventType.metrics_vector
+        if check_empty_data(self.es, company_id=company_id, event_type=event_type):
+            return [], []
+
+        es_req = {
+            "size": 10000,
+            "query": {
+                "bool": {
+                    "must": [
+                        {"term": {"task": task_id}},
+                        {"term": {"metric": metric}},
+                        {"term": {"variant": variant}},
+                    ]
+                }
+            },
+            "_source": ["iter", "value"],
+            "sort": ["iter"],
+        }
+        with translate_errors_context(), TimingContext("es", "task_stats_vector"):
+            es_res = search_company_events(
+                self.es, company_id=company_id, event_type=event_type, body=es_req
+            )
+
+        vectors = []
+        iterations = []
+        for hit in es_res["hits"]["hits"]:
+            vectors.append(hit["_source"]["value"])
+            iterations.append(hit["_source"]["iter"])
+
+        return iterations, vectors
+
+    def get_last_iters(
+        self, company_id: str, event_type: EventType, task_id: str, iters: int
+    ):
+        if check_empty_data(self.es, company_id=company_id, event_type=event_type):
+            return []
+
+        es_req: dict = {
+            "size": 0,
+            "aggs": {
+                "iters": {
+                    "terms": {
+                        "field": "iter",
+                        "size": iters,
+                        "order": {"_key": "desc"},
+                    }
+                }
+            },
+            "query": {"bool": {"must": [{"term": {"task": task_id}}]}},
+        }
+
+        with translate_errors_context(), TimingContext("es", "task_last_iter"):
+            es_res = search_company_events(
+                self.es, company_id=company_id, event_type=event_type, body=es_req
+            )
+
+        if "aggregations" not in es_res:
+            return []
+
+        return [b["key"] for b in es_res["aggregations"]["iters"]["buckets"]]
+
+    def delete_task_events(self, company_id, task_id, allow_locked=False):
+        with translate_errors_context():
+            extra_msg = None
+            query = Q(id=task_id, company=company_id)
+            if not allow_locked:
+                query &= Q(status__nin=LOCKED_TASK_STATUSES)
+                extra_msg = "or task published"
+            res = Task.objects(query).only("id").first()
+            if not res:
+                raise errors.bad_request.InvalidTaskId(
+                    extra_msg, company=company_id, id=task_id
+                )
+
+        es_req = {"query": {"term": {"task": task_id}}}
+        with translate_errors_context(), TimingContext("es", "delete_task_events"):
+            es_res = delete_company_events(
+                es=self.es,
+                company_id=company_id,
+                event_type=EventType.all,
+                body=es_req,
+                refresh=True,
+            )
+
+        return es_res.get("deleted", 0)

apiserver/bll/event/event_common.py

@@ -0,0 +1,66 @@
+from enum import Enum
+from typing import Union, Sequence
+
+from boltons.typeutils import classproperty
+from elasticsearch import Elasticsearch
+
+from apiserver.config_repo import config
+
+
+class EventType(Enum):
+    metrics_scalar = "training_stats_scalar"
+    metrics_vector = "training_stats_vector"
+    metrics_image = "training_debug_image"
+    metrics_plot = "plot"
+    task_log = "log"
+    all = "*"
+
+
+class EventSettings:
+    @classproperty
+    def max_workers(self):
+        return config.get("services.events.events_retrieval.max_metrics_concurrency", 4)
+
+    @classproperty
+    def state_expiration_sec(self):
+        return config.get(
+            f"services.events.events_retrieval.state_expiration_sec", 3600
+        )
+
+    @classproperty
+    def max_metrics_count(self):
+        return config.get("services.events.events_retrieval.max_metrics_count", 100)
+
+    @classproperty
+    def max_variants_count(self):
+        return config.get("services.events.events_retrieval.max_variants_count", 100)
+
+
+def get_index_name(company_id: str, event_type: str):
+    event_type = event_type.lower().replace(" ", "_")
+    return f"events-{event_type}-{company_id}"
+
+
+def check_empty_data(es: Elasticsearch, company_id: str, event_type: EventType) -> bool:
+    es_index = get_index_name(company_id, event_type.value)
+    if not es.indices.exists(es_index):
+        return True
+    return False
+
+
+def search_company_events(
+    es: Elasticsearch,
+    company_id: Union[str, Sequence[str]],
+    event_type: EventType,
+    body: dict,
+    **kwargs,
+) -> dict:
+    es_index = get_index_name(company_id, event_type.value)
+    return es.search(index=es_index, body=body, **kwargs)
+
+
+def delete_company_events(
+    es: Elasticsearch, company_id: str, event_type: EventType, body: dict, **kwargs
+) -> dict:
+    es_index = get_index_name(company_id, event_type.value)
+    return es.delete_by_query(index=es_index, body=body, **kwargs)

apiserver/bll/event/event_metrics.py

@@ -0,0 +1,429 @@
+import itertools
+import math
+from collections import defaultdict
+from concurrent.futures.thread import ThreadPoolExecutor
+from functools import partial
+from operator import itemgetter
+from typing import Sequence, Tuple
+
+from elasticsearch import Elasticsearch
+from mongoengine import Q
+
+from apiserver.apierrors import errors
+from apiserver.bll.event.event_common import (
+    EventType,
+    EventSettings,
+    search_company_events,
+    check_empty_data,
+)
+from apiserver.bll.event.scalar_key import ScalarKey, ScalarKeyEnum
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.task.task import Task
+from apiserver.timing_context import TimingContext
+from apiserver.tools import safe_get
+
+log = config.logger(__file__)
+
+
+class EventMetrics:
+    MAX_AGGS_ELEMENTS_COUNT = 50
+    MAX_SAMPLE_BUCKETS = 6000
+
+    def __init__(self, es: Elasticsearch):
+        self.es = es
+
+    def get_scalar_metrics_average_per_iter(
+        self, company_id: str, task_id: str, samples: int, key: ScalarKeyEnum
+    ) -> dict:
+        """
+        Get scalar metric histogram per metric and variant
+        The amount of points in each histogram should not exceed
+        the requested samples
+        """
+        event_type = EventType.metrics_scalar
+        if check_empty_data(self.es, company_id=company_id, event_type=event_type):
+            return {}
+
+        return self._get_scalar_average_per_iter_core(
+            task_id, company_id, event_type, samples, ScalarKey.resolve(key)
+        )
+
+    def _get_scalar_average_per_iter_core(
+        self,
+        task_id: str,
+        company_id: str,
+        event_type: EventType,
+        samples: int,
+        key: ScalarKey,
+        run_parallel: bool = True,
+    ) -> dict:
+        intervals = self._get_task_metric_intervals(
+            company_id=company_id,
+            event_type=event_type,
+            task_id=task_id,
+            samples=samples,
+            field=key.field,
+        )
+        if not intervals:
+            return {}
+        interval_groups = self._group_task_metric_intervals(intervals)
+
+        get_scalar_average = partial(
+            self._get_scalar_average,
+            task_id=task_id,
+            company_id=company_id,
+            event_type=event_type,
+            key=key,
+        )
+        if run_parallel:
+            with ThreadPoolExecutor(max_workers=EventSettings.max_workers) as pool:
+                metrics = itertools.chain.from_iterable(
+                    pool.map(get_scalar_average, interval_groups)
+                )
+        else:
+            metrics = itertools.chain.from_iterable(
+                get_scalar_average(group) for group in interval_groups
+            )
+
+        ret = defaultdict(dict)
+        for metric_key, metric_values in metrics:
+            ret[metric_key].update(metric_values)
+
+        return ret
+
+    def compare_scalar_metrics_average_per_iter(
+        self,
+        company_id,
+        task_ids: Sequence[str],
+        samples,
+        key: ScalarKeyEnum,
+        allow_public=True,
+    ):
+        """
+        Compare scalar metrics for different tasks per metric and variant
+        The amount of points in each histogram should not exceed the requested samples
+        """
+        task_name_by_id = {}
+        with translate_errors_context():
+            task_objs = Task.get_many(
+                company=company_id,
+                query=Q(id__in=task_ids),
+                allow_public=allow_public,
+                override_projection=("id", "name", "company", "company_origin"),
+                return_dicts=False,
+            )
+            if len(task_objs) < len(task_ids):
+                invalid = tuple(set(task_ids) - set(r.id for r in task_objs))
+                raise errors.bad_request.InvalidTaskId(company=company_id, ids=invalid)
+            task_name_by_id = {t.id: t.name for t in task_objs}
+
+        companies = {t.get_index_company() for t in task_objs}
+        if len(companies) > 1:
+            raise errors.bad_request.InvalidTaskId(
+                "only tasks from the same company are supported"
+            )
+
+        event_type = EventType.metrics_scalar
+        company_id = next(iter(companies))
+        if check_empty_data(self.es, company_id=company_id, event_type=event_type):
+            return {}
+
+        get_scalar_average_per_iter = partial(
+            self._get_scalar_average_per_iter_core,
+            company_id=company_id,
+            event_type=event_type,
+            samples=samples,
+            key=ScalarKey.resolve(key),
+            run_parallel=False,
+        )
+        with ThreadPoolExecutor(max_workers=EventSettings.max_workers) as pool:
+            task_metrics = zip(
+                task_ids, pool.map(get_scalar_average_per_iter, task_ids)
+            )
+
+        res = defaultdict(lambda: defaultdict(dict))
+        for task_id, task_data in task_metrics:
+            task_name = task_name_by_id[task_id]
+            for metric_key, metric_data in task_data.items():
+                for variant_key, variant_data in metric_data.items():
+                    variant_data["name"] = task_name
+                    res[metric_key][variant_key][task_id] = variant_data
+
+        return res
+
+    MetricInterval = Tuple[str, str, int, int]
+    MetricIntervalGroup = Tuple[int, Sequence[Tuple[str, str]]]
+
+    @classmethod
+    def _group_task_metric_intervals(
+        cls, intervals: Sequence[MetricInterval]
+    ) -> Sequence[MetricIntervalGroup]:
+        """
+        Group task metric intervals so that the following conditions are meat:
+            - All the metrics in the same group have the same interval (with 10% rounding)
+            - The amount of metrics in the group does not exceed MAX_AGGS_ELEMENTS_COUNT
+            - The total count of samples in the group does not exceed MAX_SAMPLE_BUCKETS
+        """
+        metric_interval_groups = []
+        interval_group = []
+        group_interval_upper_bound = 0
+        group_max_interval = 0
+        group_samples = 0
+        for metric, variant, interval, size in sorted(intervals, key=itemgetter(2)):
+            if (
+                interval > group_interval_upper_bound
+                or (group_samples + size) > cls.MAX_SAMPLE_BUCKETS
+                or len(interval_group) >= cls.MAX_AGGS_ELEMENTS_COUNT
+            ):
+                if interval_group:
+                    metric_interval_groups.append((group_max_interval, interval_group))
+                    interval_group = []
+                group_max_interval = interval
+                group_interval_upper_bound = interval + int(interval * 0.1)
+                group_samples = 0
+            interval_group.append((metric, variant))
+            group_samples += size
+            group_max_interval = max(group_max_interval, interval)
+        if interval_group:
+            metric_interval_groups.append((group_max_interval, interval_group))
+
+        return metric_interval_groups
+
+    def _get_task_metric_intervals(
+        self,
+        company_id: str,
+        event_type: EventType,
+        task_id: str,
+        samples: int,
+        field: str = "iter",
+    ) -> Sequence[MetricInterval]:
+        """
+        Calculate interval per task metric variant so that the resulting
+        amount of points does not exceed sample.
+        Return the list og metric variant intervals as the following tuple:
+        (metric, variant, interval, samples)
+        """
+        es_req = {
+            "size": 0,
+            "query": {"term": {"task": task_id}},
+            "aggs": {
+                "metrics": {
+                    "terms": {
+                        "field": "metric",
+                        "size": EventSettings.max_metrics_count,
+                        "order": {"_key": "asc"},
+                    },
+                    "aggs": {
+                        "variants": {
+                            "terms": {
+                                "field": "variant",
+                                "size": EventSettings.max_variants_count,
+                                "order": {"_key": "asc"},
+                            },
+                            "aggs": {
+                                "count": {"value_count": {"field": field}},
+                                "min_index": {"min": {"field": field}},
+                                "max_index": {"max": {"field": field}},
+                            },
+                        }
+                    },
+                }
+            },
+        }
+
+        with translate_errors_context(), TimingContext("es", "task_stats_get_interval"):
+            es_res = search_company_events(
+                self.es, company_id=company_id, event_type=event_type, body=es_req,
+            )
+
+        aggs_result = es_res.get("aggregations")
+        if not aggs_result:
+            return []
+
+        return [
+            self._build_metric_interval(metric["key"], variant["key"], variant, samples)
+            for metric in aggs_result["metrics"]["buckets"]
+            for variant in metric["variants"]["buckets"]
+        ]
+
+    @staticmethod
+    def _build_metric_interval(
+        metric: str, variant: str, data: dict, samples: int
+    ) -> Tuple[str, str, int, int]:
+        """
+        Calculate index interval per metric_variant variant so that the
+        total amount of intervals does not exceeds the samples
+        Return the interval and resulting amount of intervals
+        """
+        count = safe_get(data, "count/value", default=0)
+        if count < samples:
+            return metric, variant, 1, count
+
+        min_index = safe_get(data, "min_index/value", default=0)
+        max_index = safe_get(data, "max_index/value", default=min_index)
+        index_range = max_index - min_index + 1
+        interval = max(1, math.ceil(float(index_range) / samples))
+        max_samples = math.ceil(float(index_range) / interval)
+        return (
+            metric,
+            variant,
+            interval,
+            max_samples,
+        )
+
+    MetricData = Tuple[str, dict]
+
+    def _get_scalar_average(
+        self,
+        metrics_interval: MetricIntervalGroup,
+        task_id: str,
+        company_id: str,
+        event_type: EventType,
+        key: ScalarKey,
+    ) -> Sequence[MetricData]:
+        """
+        Retrieve scalar histograms per several metric variants that share the same interval
+        """
+        interval, metrics = metrics_interval
+        aggregation = self._add_aggregation_average(key.get_aggregation(interval))
+        aggs = {
+            "metrics": {
+                "terms": {
+                    "field": "metric",
+                    "size": EventSettings.max_metrics_count,
+                    "order": {"_key": "asc"},
+                },
+                "aggs": {
+                    "variants": {
+                        "terms": {
+                            "field": "variant",
+                            "size": EventSettings.max_variants_count,
+                            "order": {"_key": "asc"},
+                        },
+                        "aggs": aggregation,
+                    }
+                },
+            }
+        }
+        aggs_result = self._query_aggregation_for_task_metrics(
+            company_id=company_id,
+            event_type=event_type,
+            aggs=aggs,
+            task_id=task_id,
+            metrics=metrics,
+        )
+
+        if not aggs_result:
+            return {}
+
+        metrics = [
+            (
+                metric["key"],
+                {
+                    variant["key"]: {
+                        "name": variant["key"],
+                        **key.get_iterations_data(variant),
+                    }
+                    for variant in metric["variants"]["buckets"]
+                },
+            )
+            for metric in aggs_result["metrics"]["buckets"]
+        ]
+        return metrics
+
+    @staticmethod
+    def _add_aggregation_average(aggregation):
+        average_agg = {"avg_val": {"avg": {"field": "value"}}}
+        return {
+            key: {**value, "aggs": {**value.get("aggs", {}), **average_agg}}
+            for key, value in aggregation.items()
+        }
+
+    def _query_aggregation_for_task_metrics(
+        self,
+        company_id: str,
+        event_type: EventType,
+        aggs: dict,
+        task_id: str,
+        metrics: Sequence[Tuple[str, str]],
+    ) -> dict:
+        """
+        Return the result of elastic search query for the given aggregation filtered
+        by the given task_ids and metrics
+        """
+        must = [{"term": {"task": task_id}}]
+        if metrics:
+            should = [
+                {
+                    "bool": {
+                        "must": [
+                            {"term": {"metric": metric}},
+                            {"term": {"variant": variant}},
+                        ]
+                    }
+                }
+                for metric, variant in metrics
+            ]
+            must.append({"bool": {"should": should}})
+
+        es_req = {
+            "size": 0,
+            "query": {"bool": {"must": must}},
+            "aggs": aggs,
+        }
+
+        with translate_errors_context(), TimingContext("es", "task_stats_scalar"):
+            es_res = search_company_events(
+                self.es, company_id=company_id, event_type=event_type, body=es_req,
+            )
+
+        return es_res.get("aggregations")
+
+    def get_tasks_metrics(
+        self, company_id, task_ids: Sequence, event_type: EventType
+    ) -> Sequence:
+        """
+        For the requested tasks return all the metrics that
+        reported events of the requested types
+        """
+        if check_empty_data(self.es, company_id, event_type):
+            return {}
+
+        with ThreadPoolExecutor(EventSettings.max_workers) as pool:
+            res = pool.map(
+                partial(
+                    self._get_task_metrics,
+                    company_id=company_id,
+                    event_type=event_type,
+                ),
+                task_ids,
+            )
+        return list(zip(task_ids, res))
+
+    def _get_task_metrics(
+        self, task_id: str, company_id: str, event_type: EventType
+    ) -> Sequence:
+        es_req = {
+            "size": 0,
+            "query": {"bool": {"must": [{"term": {"task": task_id}}]}},
+            "aggs": {
+                "metrics": {
+                    "terms": {
+                        "field": "metric",
+                        "size": EventSettings.max_metrics_count,
+                        "order": {"_key": "asc"},
+                    }
+                }
+            },
+        }
+
+        with translate_errors_context(), TimingContext("es", "_get_task_metrics"):
+            es_res = search_company_events(
+                self.es, company_id=company_id, event_type=event_type, body=es_req
+            )
+
+        return [
+            metric["key"]
+            for metric in safe_get(es_res, "aggregations/metrics/buckets", default=[])
+        ]

apiserver/bll/event/log_events_iterator.py

@@ -0,0 +1,127 @@
+from typing import Optional, Tuple, Sequence
+
+import attr
+from elasticsearch import Elasticsearch
+
+from apiserver.bll.event.event_common import (
+    check_empty_data,
+    search_company_events,
+    EventType,
+)
+from apiserver.database.errors import translate_errors_context
+from apiserver.timing_context import TimingContext
+
+
+@attr.s(auto_attribs=True)
+class TaskEventsResult:
+    total_events: int = 0
+    next_scroll_id: str = None
+    events: list = attr.Factory(list)
+
+
+class LogEventsIterator:
+    EVENT_TYPE = EventType.task_log
+
+    def __init__(self, es: Elasticsearch):
+        self.es = es
+
+    def get_task_events(
+        self,
+        company_id: str,
+        task_id: str,
+        batch_size: int,
+        navigate_earlier: bool = True,
+        from_timestamp: Optional[int] = None,
+    ) -> TaskEventsResult:
+        if check_empty_data(self.es, company_id, self.EVENT_TYPE):
+            return TaskEventsResult()
+
+        res = TaskEventsResult()
+        res.events, res.total_events = self._get_events(
+            company_id=company_id,
+            task_id=task_id,
+            batch_size=batch_size,
+            navigate_earlier=navigate_earlier,
+            from_timestamp=from_timestamp,
+        )
+        return res
+
+    def _get_events(
+        self,
+        company_id: str,
+        task_id: str,
+        batch_size: int,
+        navigate_earlier: bool,
+        from_timestamp: Optional[int],
+    ) -> Tuple[Sequence[dict], int]:
+        """
+        Return up to 'batch size' events starting from the previous timestamp either in the
+        direction of earlier events (navigate_earlier=True) or in the direction of later events.
+        If last_min_timestamp and last_max_timestamp are not set then start either from latest or earliest.
+        For the last timestamp all the events are brought (even if the resulting size
+        exceeds batch_size) so that this timestamp events will not be lost between the calls.
+        In case any events were received update 'last_min_timestamp' and 'last_max_timestamp'
+        """
+
+        # retrieve the next batch of events
+        es_req = {
+            "size": batch_size,
+            "query": {"term": {"task": task_id}},
+            "sort": {"timestamp": "desc" if navigate_earlier else "asc"},
+        }
+
+        if from_timestamp:
+            es_req["search_after"] = [from_timestamp]
+
+        with translate_errors_context(), TimingContext("es", "get_task_events"):
+            es_result = search_company_events(
+                self.es,
+                company_id=company_id,
+                event_type=self.EVENT_TYPE,
+                body=es_req,
+            )
+            hits = es_result["hits"]["hits"]
+            hits_total = es_result["hits"]["total"]["value"]
+            if not hits:
+                return [], hits_total
+
+            events = [hit["_source"] for hit in hits]
+
+            # retrieve the events that match the last event timestamp
+            # but did not make it into the previous call due to batch_size limitation
+            es_req = {
+                "size": 10000,
+                "query": {
+                    "bool": {
+                        "must": [
+                            {"term": {"task": task_id}},
+                            {"term": {"timestamp": events[-1]["timestamp"]}},
+                        ]
+                    }
+                },
+            }
+            es_result = search_company_events(
+                self.es,
+                company_id=company_id,
+                event_type=self.EVENT_TYPE,
+                body=es_req,
+            )
+            last_second_hits = es_result["hits"]["hits"]
+            if not last_second_hits or len(last_second_hits) < 2:
+                # if only one element is returned for the last timestamp
+                # then it is already present in the events
+                return events, hits_total
+
+            already_present_ids = set(hit["_id"] for hit in hits)
+            last_second_events = [
+                hit["_source"]
+                for hit in last_second_hits
+                if hit["_id"] not in already_present_ids
+            ]
+
+            # return the list merged from original query results +
+            # leftovers from the last timestamp
+            return (
+                [*events, *last_second_events],
+                hits_total,
+            )

apiserver/bll/event/scalar_key.py

@@ -0,0 +1,161 @@
+"""
+Module for polymorphism over different types of X axes in scalar aggregations
+"""
+from abc import ABC, abstractmethod
+from enum import auto
+
+from apiserver.utilities.stringenum import StringEnum
+from apiserver.bll.util import extract_properties_to_lists
+from apiserver.config_repo import config
+
+log = config.logger(__file__)
+
+
+class ScalarKeyEnum(StringEnum):
+    """
+    String enum representing X axes key
+    """
+
+    iter = auto()
+    timestamp = auto()
+    iso_time = auto()
+
+
+class ScalarKey(ABC):
+    """
+    Abstract scalar key
+    """
+
+    _enum_to_key = {}
+    bucket_key_key = "key"
+
+    @property
+    @abstractmethod
+    def enum_value(self) -> ScalarKeyEnum:
+        """
+        Enum value accepted in API requests
+        """
+        pass
+
+    @property
+    @abstractmethod
+    def name(self) -> str:
+        """
+        Key name. Used as arbitrary internal key in elasticsearch queries
+        """
+        pass
+
+    @property
+    @abstractmethod
+    def field(self) -> str:
+        """
+        Event key to aggregate by
+        """
+        pass
+
+    @abstractmethod
+    def get_aggregation(self, interval: int) -> dict:
+        """
+        Get aggregation for this type of key
+        :param interval: elasticsearch aggregation interval
+        """
+        pass
+
+    def __init_subclass__(cls, **kwargs):
+        """
+        Save a mapping from enum values to key class
+        """
+        if cls.enum_value not in ScalarKeyEnum:
+            raise ValueError(f"{cls.enum_value!r} not in {ScalarKeyEnum.__name__}")
+        if cls.enum_value in cls._enum_to_key:
+            log.warning(
+                f"'{cls.enum_value.value}' is already registered to {ScalarKey.__name__}"
+            )
+        cls._enum_to_key[cls.enum_value] = cls
+
+    @classmethod
+    def resolve(cls, key: ScalarKeyEnum):
+        """
+        Create a key instance from enum instance
+        """
+        return cls._enum_to_key[key]()
+
+    def get_iterations_data(self, iter_buckets: dict) -> dict:
+        """
+        Convert a list of bucket entries to `x`s array and `y`s array
+        """
+        return extract_properties_to_lists(
+            ("x", "y"),
+            iter_buckets[self.name]["buckets"],
+            self._get_iterations_data_single,
+        )
+
+    def _get_iterations_data_single(self, iter_data):
+        """
+        Extract x value and y value from a single bucket item
+        """
+        return int(iter_data[self.bucket_key_key]), iter_data["avg_val"]["value"]
+
+
+class TimestampKey(ScalarKey):
+    """
+    Aggregate by timestamp in milliseconds since epoch
+    """
+
+    name = "timestamp"
+    field = "timestamp"
+    enum_value = ScalarKeyEnum.timestamp
+
+    def get_aggregation(self, interval: int) -> dict:
+        return {
+            self.name: {
+                "date_histogram": {
+                    "field": "timestamp",
+                    "fixed_interval": f"{interval}ms",
+                    "min_doc_count": 1,
+                }
+            }
+        }
+
+
+class IterKey(ScalarKey):
+    """
+    Aggregate by iteration number
+    """
+
+    name = "iters"
+    field = "iter"
+    enum_value = ScalarKeyEnum.iter
+
+    def get_aggregation(self, interval: int) -> dict:
+        return {
+            self.name: {
+                "histogram": {"field": "iter", "interval": interval, "min_doc_count": 1}
+            }
+        }
+
+
+class ISOTimeKey(ScalarKey):
+    """
+    Aggregate by time formatted as ISO strings
+    """
+
+    name = "iso_time"
+    field = "timestamp"
+    enum_value = ScalarKeyEnum.iso_time
+    bucket_key_key = "key_as_string"
+
+    def get_aggregation(self, interval: int) -> dict:
+        return {
+            self.name: {
+                "date_histogram": {
+                    "field": "timestamp",
+                    "fixed_interval": f"{interval}ms",
+                    "min_doc_count": 1,
+                    "format": "strict_date_time",
+                }
+            }
+        }
+
+    def _get_iterations_data_single(self, iter_data):
+        return iter_data[self.bucket_key_key], iter_data["avg_val"]["value"]

apiserver/bll/model/__init__.py

@@ -0,0 +1,18 @@
+from typing import Optional, Sequence
+
+from mongoengine import Q
+
+from apiserver.database.model.model import Model
+from apiserver.database.utils import get_company_or_none_constraint
+
+
+class ModelBLL:
+    def get_frameworks(self, company, project_ids: Optional[Sequence]) -> Sequence:
+        """
+        Return the list of unique frameworks used by company and public models
+        If project ids passed then only models from these projects are considered
+        """
+        query = get_company_or_none_constraint(company)
+        if project_ids:
+            query &= Q(project__in=project_ids)
+        return Model.objects(query).distinct(field="framework")

apiserver/bll/organization/__init__.py

@@ -0,0 +1,98 @@
+from collections import defaultdict
+from enum import Enum
+from operator import itemgetter
+from typing import Sequence, Dict, Optional
+
+from mongoengine import Q
+
+from apiserver.config_repo import config
+from apiserver.database.model import EntityVisibility
+from apiserver.database.model.model import Model
+from apiserver.database.model.task.task import Task
+from apiserver.redis_manager import redman
+from .tags_cache import _TagsCache
+
+log = config.logger(__file__)
+
+
+class Tags(Enum):
+    Task = "task"
+    Model = "model"
+
+
+class OrgBLL:
+    def __init__(self, redis=None):
+        self.redis = redis or redman.connection("apiserver")
+        self._task_tags = _TagsCache(Task, self.redis)
+        self._model_tags = _TagsCache(Model, self.redis)
+
+    def get_tags(
+        self,
+        company_id: str,
+        entity: Tags,
+        include_system: bool = False,
+        filter_: Dict[str, Sequence[str]] = None,
+        projects: Sequence[str] = None,
+    ) -> dict:
+        tags_cache = self._get_tags_cache_for_entity(entity)
+        if not projects:
+            return tags_cache.get_tags(
+                company_id, include_system=include_system, filter_=filter_
+            )
+
+        ret = defaultdict(set)
+        for project in projects:
+            project_tags = tags_cache.get_tags(
+                company_id,
+                include_system=include_system,
+                filter_=filter_,
+                project=project,
+            )
+            for field, tags in project_tags.items():
+                ret[field] |= tags
+
+        return ret
+
+    def update_tags(
+        self, company_id: str, entity: Tags, project: str, tags=None, system_tags=None,
+    ):
+        tags_cache = self._get_tags_cache_for_entity(entity)
+        tags_cache.update_tags(company_id, project, tags, system_tags)
+
+    def reset_tags(self, company_id: str, entity: Tags, projects: Sequence[str]):
+        tags_cache = self._get_tags_cache_for_entity(entity)
+        tags_cache.reset_tags(company_id, projects=projects)
+
+    def _get_tags_cache_for_entity(self, entity: Tags) -> _TagsCache:
+        return self._task_tags if entity == Tags.Task else self._model_tags
+
+    @classmethod
+    def get_parent_tasks(
+        cls,
+        company_id: str,
+        projects: Sequence[str],
+        state: Optional[EntityVisibility] = None,
+    ) -> Sequence[dict]:
+        """
+        Get list of unique parent tasks sorted by task name for the passed company projects
+        If projects is None or empty then get parents for all the company tasks
+        """
+        query = Q(company=company_id)
+        if projects:
+            query &= Q(project__in=projects)
+        if state == EntityVisibility.archived:
+            query &= Q(system_tags__in=[EntityVisibility.archived.value])
+        elif state == EntityVisibility.active:
+            query &= Q(system_tags__nin=[EntityVisibility.archived.value])
+
+        parent_ids = set(Task.objects(query).distinct("parent"))
+        if not parent_ids:
+            return []
+
+        parents = Task.get_many_with_join(
+            company_id,
+            query=Q(id__in=parent_ids),
+            allow_public=True,
+            override_projection=("id", "name", "project.name"),
+        )
+        return sorted(parents, key=itemgetter("name"))

apiserver/bll/organization/tags_cache.py

@@ -0,0 +1,144 @@
+from itertools import chain
+from typing import Sequence, Union, Type, Dict
+
+from mongoengine import Q
+from redis import Redis
+
+from apiserver.config_repo import config
+from apiserver.database.model.base import GetMixin
+from apiserver.database.model.model import Model
+from apiserver.database.model.task.task import Task
+
+log = config.logger(__file__)
+_settings_prefix = "services.organization"
+
+
+class _TagsCache:
+    _tags_field = "tags"
+    _system_tags_field = "system_tags"
+    _dummy_tag = "__dummy__"
+    # prepend our list in redis with this tag since empty lists are auto deleted
+
+    def __init__(self, db_cls: Union[Type[Model], Type[Task]], redis: Redis):
+        self.db_cls = db_cls
+        self.redis = redis
+
+    @property
+    def _tags_cache_expiration_seconds(self):
+        return config.get(f"{_settings_prefix}.tags_cache.expiration_seconds", 3600)
+
+    def _get_tags_from_db(
+        self,
+        company_id: str,
+        field: str,
+        project: str = None,
+        filter_: Dict[str, Sequence[str]] = None,
+    ) -> set:
+        query = Q(company=company_id)
+        if filter_:
+            for name, vals in filter_.items():
+                if vals:
+                    query &= GetMixin.get_list_field_query(name, vals)
+        if project:
+            query &= Q(project=project)
+
+        return self.db_cls.objects(query).distinct(field)
+
+    def _get_tags_cache_key(
+        self,
+        company_id: str,
+        field: str,
+        project: str = None,
+        filter_: Dict[str, Sequence[str]] = None,
+    ):
+        """
+        Project None means 'from all company projects'
+        The key is built in the way that scanning company keys for 'all company projects'
+        will not return the keys related to the particular company projects and vice versa.
+        So that we can have a fine grain control on what redis keys to invalidate
+        """
+        filter_str = None
+        if filter_:
+            filter_str = "_".join(
+                ["filter", *chain.from_iterable([f, *v] for f, v in filter_.items())]
+            )
+        key_parts = [field, company_id, project, self.db_cls.__name__, filter_str]
+        return "_".join(filter(None, key_parts))
+
+    def get_tags(
+        self,
+        company_id: str,
+        include_system: bool = False,
+        filter_: Dict[str, Sequence[str]] = None,
+        project: str = None,
+    ) -> dict:
+        """
+        Get tags and optionally system tags for the company
+        Return the dictionary of tags per tags field name
+        The function retrieves both cached values from Redis in one call
+        and re calculates any of them if missing in Redis
+        """
+        fields = [self._tags_field]
+        if include_system:
+            fields.append(self._system_tags_field)
+
+        ret = {}
+        for field in fields:
+            redis_key = self._get_tags_cache_key(
+                company_id, field=field, project=project, filter_=filter_
+            )
+            cached_tags = self.redis.lrange(redis_key, 0, -1)
+            if cached_tags:
+                tags = [c.decode() for c in cached_tags[1:]]
+            else:
+                tags = list(
+                    self._get_tags_from_db(
+                        company_id, field=field, project=project, filter_=filter_
+                    )
+                )
+                self.redis.rpush(redis_key, self._dummy_tag, *tags)
+                self.redis.expire(redis_key, self._tags_cache_expiration_seconds)
+
+            ret[field] = set(tags)
+
+        return ret
+
+    def update_tags(self, company_id: str, project: str, tags=None, system_tags=None):
+        """
+        Updates tags. If reset is set then both tags and system_tags
+        are recalculated. Otherwise only those that are not 'None'
+        """
+        fields = [
+            field
+            for field, update in (
+                (self._tags_field, tags),
+                (self._system_tags_field, system_tags),
+            )
+            if update is not None
+        ]
+        if not fields:
+            return
+
+        self._delete_redis_keys(company_id, projects=[project], fields=fields)
+
+    def reset_tags(self, company_id: str, projects: Sequence[str]):
+        self._delete_redis_keys(
+            company_id,
+            projects=projects,
+            fields=(self._tags_field, self._system_tags_field),
+        )
+
+    def _delete_redis_keys(
+        self, company_id: str, projects: [Sequence[str]], fields: Sequence[str]
+    ):
+        redis_keys = list(
+            chain.from_iterable(
+                self.redis.keys(
+                    self._get_tags_cache_key(company_id, field=f, project=p) + "*"
+                )
+                for f in fields
+                for p in set(projects) | {None}
+            )
+        )
+        if redis_keys:
+            self.redis.delete(*redis_keys)

apiserver/bll/project/__init__.py

@@ -0,0 +1 @@
+from .project_bll import ProjectBLL

apiserver/bll/project/project_bll.py

@@ -0,0 +1,137 @@
+from datetime import datetime
+from typing import Sequence, Optional, Type
+
+from mongoengine import Q, Document
+
+from apiserver import database
+from apiserver.apierrors import errors
+from apiserver.config_repo import config
+from apiserver.database.model.model import Model
+from apiserver.database.model.project import Project
+from apiserver.database.model.task.task import Task
+from apiserver.timing_context import TimingContext
+
+log = config.logger(__file__)
+
+
+class ProjectBLL:
+    @classmethod
+    def get_active_users(
+        cls, company, project_ids: Sequence, user_ids: Optional[Sequence] = None
+    ) -> set:
+        """
+        Get the set of user ids that created tasks/models in the given projects
+        If project_ids is empty then all projects are examined
+        If user_ids are passed then only subset of these users is returned
+        """
+        with TimingContext("mongo", "active_users_in_projects"):
+            res = set()
+            query = Q(company=company)
+            if project_ids:
+                query &= Q(project__in=project_ids)
+            if user_ids:
+                query &= Q(user__in=user_ids)
+            for cls_ in (Task, Model):
+                res |= set(cls_.objects(query).distinct(field="user"))
+
+            return res
+
+    @classmethod
+    def create(
+        cls,
+        user: str,
+        company: str,
+        name: str,
+        description: str,
+        tags: Sequence[str] = None,
+        system_tags: Sequence[str] = None,
+        default_output_destination: str = None,
+    ) -> str:
+        """
+        Create a new project.
+        Returns project ID
+        """
+        now = datetime.utcnow()
+        project = Project(
+            id=database.utils.id(),
+            user=user,
+            company=company,
+            name=name,
+            description=description,
+            tags=tags,
+            system_tags=system_tags,
+            default_output_destination=default_output_destination,
+            created=now,
+            last_update=now,
+        )
+        project.save()
+        return project.id
+
+    @classmethod
+    def find_or_create(
+        cls,
+        user: str,
+        company: str,
+        project_name: str,
+        description: str,
+        project_id: str = None,
+        tags: Sequence[str] = None,
+        system_tags: Sequence[str] = None,
+        default_output_destination: str = None,
+    ) -> str:
+        """
+        Find a project named `project_name` or create a new one.
+        Returns project ID
+        """
+        if not project_id and not project_name:
+            raise ValueError("project id or name required")
+
+        if project_id:
+            project = Project.objects(company=company, id=project_id).only("id").first()
+            if not project:
+                raise errors.bad_request.InvalidProjectId(id=project_id)
+            return project_id
+
+        project = Project.objects(company=company, name=project_name).only("id").first()
+        if project:
+            return project.id
+
+        return cls.create(
+            user=user,
+            company=company,
+            name=project_name,
+            description=description,
+            tags=tags,
+            system_tags=system_tags,
+            default_output_destination=default_output_destination,
+        )
+
+    @classmethod
+    def move_under_project(
+        cls,
+        entity_cls: Type[Document],
+        user: str,
+        company: str,
+        ids: Sequence[str],
+        project: str = None,
+        project_name: str = None,
+    ):
+        """
+        Move a batch of entities to `project` or a project named `project_name` (create if does not exist)
+        """
+        with TimingContext("mongo", "move_under_project"):
+            project = cls.find_or_create(
+                user=user,
+                company=company,
+                project_id=project,
+                project_name=project_name,
+                description="Auto-generated during move",
+            )
+            extra = (
+                {"set__last_change": datetime.utcnow()}
+                if hasattr(entity_cls, "last_change")
+                else {}
+            )
+            entity_cls.objects(company=company, id__in=ids).update(set__project=project, **extra)
+
+            return project

apiserver/bll/query/__init__.py

@@ -0,0 +1 @@
+from .builder import Builder

apiserver/bll/query/builder.py

@@ -0,0 +1,36 @@
+from typing import Optional, Sequence, Iterable, Union
+
+from apiserver.config_repo import config
+
+log = config.logger(__file__)
+
+RANGE_IGNORE_VALUE = -1
+
+
+class Builder:
+    @staticmethod
+    def dates_range(from_date: Union[int, float], to_date: Union[int, float]) -> dict:
+        return {
+            "range": {
+                "timestamp": {
+                    "gte": int(from_date),
+                    "lte": int(to_date),
+                    "format": "epoch_second",
+                }
+            }
+        }
+
+    @staticmethod
+    def terms(field: str, values: Iterable[str]) -> dict:
+        return {"terms": {field: list(values)}}
+
+    @staticmethod
+    def normalize_range(
+        range_: Sequence[Union[int, float]],
+        ignore_value: Union[int, float] = RANGE_IGNORE_VALUE,
+    ) -> Optional[Sequence[Union[int, float]]]:
+        if not range_ or set(range_) == {ignore_value}:
+            return None
+        if len(range_) < 2:
+            return [range_[0]] * 2
+        return range_

apiserver/bll/queue/__init__.py

@@ -0,0 +1 @@
+from .queue_bll import QueueBLL

apiserver/bll/queue/queue_bll.py

@@ -0,0 +1,270 @@
+from collections import defaultdict
+from datetime import datetime
+from typing import Callable, Sequence, Optional, Tuple
+
+from elasticsearch import Elasticsearch
+
+from apiserver import database
+from apiserver.es_factory import es_factory
+from apiserver.apierrors import errors
+from apiserver.bll.queue.queue_metrics import QueueMetrics
+from apiserver.bll.workers import WorkerBLL
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.queue import Queue, Entry
+
+log = config.logger(__file__)
+
+
+class QueueBLL(object):
+    def __init__(self, worker_bll: WorkerBLL = None, es: Elasticsearch = None):
+        self.worker_bll = worker_bll or WorkerBLL()
+        self.es = es or es_factory.connect("workers")
+        self._metrics = QueueMetrics(self.es)
+
+    @property
+    def metrics(self) -> QueueMetrics:
+        return self._metrics
+
+    @staticmethod
+    def create(
+        company_id: str,
+        name: str,
+        tags: Optional[Sequence[str]] = None,
+        system_tags: Optional[Sequence[str]] = None,
+    ) -> Queue:
+        """Creates a queue"""
+        with translate_errors_context():
+            now = datetime.utcnow()
+            queue = Queue(
+                id=database.utils.id(),
+                company=company_id,
+                created=now,
+                name=name,
+                tags=tags or [],
+                system_tags=system_tags or [],
+                last_update=now,
+            )
+            queue.save()
+            return queue
+
+    def get_by_id(
+        self, company_id: str, queue_id: str, only: Optional[Sequence[str]] = None
+    ) -> Queue:
+        """
+        Get queue by id
+        :raise errors.bad_request.InvalidQueueId: if the queue is not found
+        """
+        with translate_errors_context():
+            query = dict(id=queue_id, company=company_id)
+            qs = Queue.objects(**query)
+            if only:
+                qs = qs.only(*only)
+            queue = qs.first()
+            if not queue:
+                raise errors.bad_request.InvalidQueueId(**query)
+
+            return queue
+
+    @classmethod
+    def get_queue_with_task(cls, company_id: str, queue_id: str, task_id: str) -> Queue:
+        with translate_errors_context():
+            query = dict(id=queue_id, company=company_id)
+            queue = Queue.objects(entries__task=task_id, **query).first()
+            if not queue:
+                raise errors.bad_request.InvalidQueueOrTaskNotQueued(
+                    task=task_id, **query
+                )
+
+            return queue
+
+    def get_default(self, company_id: str) -> Queue:
+        """
+        Get the default queue
+        :raise errors.bad_request.NoDefaultQueue: if the default queue not found
+        :raise errors.bad_request.MultipleDefaultQueues: if more than one default queue is found
+        """
+        with translate_errors_context():
+            res = Queue.objects(company=company_id, system_tags="default").only(
+                "id", "name"
+            )
+            if not res:
+                raise errors.bad_request.NoDefaultQueue()
+            if len(res) > 1:
+                raise errors.bad_request.MultipleDefaultQueues(
+                    queues=tuple(r.id for r in res)
+                )
+
+            return res.first()
+
+    def update(
+        self, company_id: str, queue_id: str, **update_fields
+    ) -> Tuple[int, dict]:
+        """
+        Partial update of the queue from update_fields
+        :raise errors.bad_request.InvalidQueueId: if the queue is not found
+        :return: number of updated objects and updated fields dictionary
+        """
+        with translate_errors_context():
+            # validate the queue exists
+            self.get_by_id(company_id=company_id, queue_id=queue_id, only=("id",))
+            return Queue.safe_update(company_id, queue_id, update_fields)
+
+    def delete(self, company_id: str, queue_id: str, force: bool) -> None:
+        """
+        Delete the queue
+        :raise errors.bad_request.InvalidQueueId: if the queue is not found
+        :raise errors.bad_request.QueueNotEmpty: if the queue is not empty and 'force' not set
+        """
+        with translate_errors_context():
+            queue = self.get_by_id(company_id=company_id, queue_id=queue_id)
+            if queue.entries and not force:
+                raise errors.bad_request.QueueNotEmpty(
+                    "use force=true to delete", id=queue_id
+                )
+            queue.delete()
+
+    def get_all(self, company_id: str, query_dict: dict) -> Sequence[dict]:
+        """Get all the queues according to the query"""
+        with translate_errors_context():
+            return Queue.get_many(
+                company=company_id, parameters=query_dict, query_dict=query_dict
+            )
+
+    def get_queue_infos(self, company_id: str, query_dict: dict) -> Sequence[dict]:
+        """
+        Get infos on all the company queues, including queue tasks and workers
+        """
+        projection = Queue.get_extra_projection("entries.task.name")
+        with translate_errors_context():
+            res = Queue.get_many_with_join(
+                company=company_id,
+                query_dict=query_dict,
+                override_projection=projection,
+            )
+
+            queue_workers = defaultdict(list)
+            for worker in self.worker_bll.get_all(company_id):
+                for queue in worker.queues:
+                    queue_workers[queue].append(worker)
+
+            for item in res:
+                item["workers"] = [
+                    {
+                        "name": w.id,
+                        "ip": w.ip,
+                        "task": w.task.to_struct() if w.task else None,
+                    }
+                    for w in queue_workers.get(item["id"], [])
+                ]
+
+        return res
+
+    def add_task(self, company_id: str, queue_id: str, task_id: str) -> dict:
+        """
+        Add the task to the queue and return the queue update results
+        :raise errors.bad_request.TaskAlreadyQueued: if the task is already in the queue
+        :raise errors.bad_request.InvalidQueueOrTaskNotQueued: if the queue update operation failed
+        """
+        with translate_errors_context():
+            queue = self.get_by_id(company_id=company_id, queue_id=queue_id)
+            if any(e.task == task_id for e in queue.entries):
+                raise errors.bad_request.TaskAlreadyQueued(task=task_id)
+
+            self.metrics.log_queue_metrics_to_es(company_id=company_id, queues=[queue])
+
+            entry = Entry(added=datetime.utcnow(), task=task_id)
+            query = dict(id=queue_id, company=company_id)
+            res = Queue.objects(entries__task__ne=task_id, **query).update_one(
+                push__entries=entry, last_update=datetime.utcnow(), upsert=False
+            )
+            if not res:
+                raise errors.bad_request.InvalidQueueOrTaskNotQueued(
+                    task=task_id, **query
+                )
+
+            return res
+
+    def get_next_task(self, company_id: str, queue_id: str) -> Optional[Entry]:
+        """
+        Atomically pop and return the first task from the queue (or None)
+        :raise errors.bad_request.InvalidQueueId: if the queue does not exist
+        """
+        with translate_errors_context():
+            query = dict(id=queue_id, company=company_id)
+            queue = Queue.objects(**query).modify(pop__entries=-1, upsert=False)
+            if not queue:
+                raise errors.bad_request.InvalidQueueId(**query)
+
+            self.metrics.log_queue_metrics_to_es(company_id, queues=[queue])
+
+            if not queue.entries:
+                return
+
+            try:
+                Queue.objects(**query).update(last_update=datetime.utcnow())
+            except Exception:
+                log.exception("Error while updating Queue.last_update")
+
+            return queue.entries[0]
+
+    def remove_task(self, company_id: str, queue_id: str, task_id: str) -> int:
+        """
+        Removes the task from the queue and returns the number of removed items
+        :raise errors.bad_request.InvalidQueueOrTaskNotQueued: if the task is not found in the queue
+        """
+        with translate_errors_context():
+            queue = self.get_queue_with_task(
+                company_id=company_id, queue_id=queue_id, task_id=task_id
+            )
+            self.metrics.log_queue_metrics_to_es(company_id, queues=[queue])
+
+            entries_to_remove = [e for e in queue.entries if e.task == task_id]
+            query = dict(id=queue_id, company=company_id)
+            res = Queue.objects(entries__task=task_id, **query).update_one(
+                pull_all__entries=entries_to_remove, last_update=datetime.utcnow()
+            )
+
+            return len(entries_to_remove) if res else 0
+
+    def reposition_task(
+        self,
+        company_id: str,
+        queue_id: str,
+        task_id: str,
+        pos_func: Callable[[int], int],
+    ) -> int:
+        """
+        Moves the task in the queue to the position calculated by pos_func
+        Returns the updated task position in the queue
+        """
+        with translate_errors_context():
+            queue = self.get_queue_with_task(
+                company_id=company_id, queue_id=queue_id, task_id=task_id
+            )
+
+            position = next(i for i, e in enumerate(queue.entries) if e.task == task_id)
+            new_position = pos_func(position)
+
+            if new_position != position:
+                entry = queue.entries[position]
+                query = dict(id=queue_id, company=company_id)
+                updated = Queue.objects(entries__task=task_id, **query).update_one(
+                    pull__entries=entry, last_update=datetime.utcnow()
+                )
+                if not updated:
+                    raise errors.bad_request.RemovedDuringReposition(
+                        task=task_id, **query
+                    )
+                inst = {"$push": {"entries": {"$each": [entry.to_proper_dict()]}}}
+                if new_position >= 0:
+                    inst["$push"]["entries"]["$position"] = new_position
+                res = Queue.objects(entries__task__ne=task_id, **query).update_one(
+                    __raw__=inst
+                )
+                if not res:
+                    raise errors.bad_request.FailedAddingDuringReposition(
+                        task=task_id, **query
+                    )
+
+            return new_position

apiserver/bll/queue/queue_metrics.py

@@ -0,0 +1,262 @@
+from collections import defaultdict
+from datetime import datetime
+from typing import Sequence
+
+import elasticsearch.helpers
+from elasticsearch import Elasticsearch
+
+from apiserver.es_factory import es_factory
+from apiserver.apierrors.errors import bad_request
+from apiserver.bll.query import Builder as QueryBuilder
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.queue import Queue, Entry
+from apiserver.timing_context import TimingContext
+
+log = config.logger(__file__)
+
+
+class QueueMetrics:
+    class EsKeys:
+        WAITING_TIME_FIELD = "average_waiting_time"
+        QUEUE_LENGTH_FIELD = "queue_length"
+        TIMESTAMP_FIELD = "timestamp"
+        QUEUE_FIELD = "queue"
+
+    def __init__(self, es: Elasticsearch):
+        self.es = es
+
+    @staticmethod
+    def _queue_metrics_prefix_for_company(company_id: str) -> str:
+        """Returns the es index prefix for the company"""
+        return f"queue_metrics_{company_id}_"
+
+    @staticmethod
+    def _get_es_index_suffix():
+        """Get the index name suffix for storing current month data"""
+        return datetime.utcnow().strftime("%Y-%m")
+
+    @staticmethod
+    def _calc_avg_waiting_time(entries: Sequence[Entry]) -> float:
+        """
+        Calculate avg waiting time for the given tasks.
+        Return 0 if the list is empty
+        """
+        if not entries:
+            return 0
+
+        now = datetime.utcnow()
+        total_waiting_in_secs = sum((now - e.added).total_seconds() for e in entries)
+        return total_waiting_in_secs / len(entries)
+
+    def log_queue_metrics_to_es(self, company_id: str, queues: Sequence[Queue]) -> bool:
+        """
+        Calculate and write queue statistics (avg waiting time and queue length) to Elastic
+        :return: True if the write to es was successful, false otherwise
+        """
+        es_index = (
+            self._queue_metrics_prefix_for_company(company_id)
+            + self._get_es_index_suffix()
+        )
+
+        timestamp = es_factory.get_timestamp_millis()
+
+        def make_doc(queue: Queue) -> dict:
+            entries = [e for e in queue.entries if e.added]
+            return dict(
+                _index=es_index,
+                _source={
+                    self.EsKeys.TIMESTAMP_FIELD: timestamp,
+                    self.EsKeys.QUEUE_FIELD: queue.id,
+                    self.EsKeys.WAITING_TIME_FIELD: self._calc_avg_waiting_time(
+                        entries
+                    ),
+                    self.EsKeys.QUEUE_LENGTH_FIELD: len(entries),
+                },
+            )
+
+        actions = list(map(make_doc, queues))
+
+        es_res = elasticsearch.helpers.bulk(self.es, actions)
+        added, errors = es_res[:2]
+        return (added == len(actions)) and not errors
+
+    def _log_current_metrics(self, company_id: str, queue_ids=Sequence[str]):
+        query = dict(company=company_id)
+        if queue_ids:
+            query["id__in"] = list(queue_ids)
+        queues = Queue.objects(**query)
+        self.log_queue_metrics_to_es(company_id, queues=list(queues))
+
+    def _search_company_metrics(self, company_id: str, es_req: dict) -> dict:
+        return self.es.search(
+            index=f"{self._queue_metrics_prefix_for_company(company_id)}*",
+            body=es_req,
+        )
+
+    @classmethod
+    def _get_dates_agg(cls, interval) -> dict:
+        """
+        Aggregation for building date histogram with internal grouping per queue.
+        We are grouping by queue inside date histogram and not vice versa so that
+        it will be easy to average between queue metrics inside each date bucket.
+        Ignore empty buckets.
+        """
+        return {
+            "dates": {
+                "date_histogram": {
+                    "field": cls.EsKeys.TIMESTAMP_FIELD,
+                    "fixed_interval": f"{interval}s",
+                    "min_doc_count": 1,
+                },
+                "aggs": {
+                    "queues": {
+                        "terms": {"field": cls.EsKeys.QUEUE_FIELD},
+                        "aggs": cls._get_top_waiting_agg(),
+                    }
+                },
+            }
+        }
+
+    @classmethod
+    def _get_top_waiting_agg(cls) -> dict:
+        """
+        Aggregation for getting max waiting time and the corresponding queue length
+        inside each date->queue bucket
+        """
+        return {
+            "top_avg_waiting": {
+                "top_hits": {
+                    "sort": [
+                        {cls.EsKeys.WAITING_TIME_FIELD: {"order": "desc"}},
+                        {cls.EsKeys.QUEUE_LENGTH_FIELD: {"order": "desc"}},
+                    ],
+                    "_source": {
+                        "includes": [
+                            cls.EsKeys.WAITING_TIME_FIELD,
+                            cls.EsKeys.QUEUE_LENGTH_FIELD,
+                        ]
+                    },
+                    "size": 1,
+                }
+            }
+        }
+
+    def get_queue_metrics(
+        self,
+        company_id: str,
+        from_date: float,
+        to_date: float,
+        interval: int,
+        queue_ids: Sequence[str],
+    ) -> dict:
+        """
+        Get the company queue metrics in the specified time range.
+        Returned as date histograms of average values per queue and metric type.
+        The from_date is extended by 'metrics_before_from_date' seconds from
+        queues.conf due to possibly small amount of points. The default extension is 3600s
+        In case no queue ids are specified the avg across all the
+        company queues is calculated for each metric
+        """
+        # self._log_current_metrics(company, queue_ids=queue_ids)
+
+        if from_date >= to_date:
+            raise bad_request.FieldsValueError("from_date must be less than to_date")
+
+        seconds_before = config.get("services.queues.metrics_before_from_date", 3600)
+        must_terms = [QueryBuilder.dates_range(from_date - seconds_before, to_date)]
+        if queue_ids:
+            must_terms.append(QueryBuilder.terms("queue", queue_ids))
+
+        es_req = {
+            "size": 0,
+            "query": {"bool": {"must": must_terms}},
+            "aggs": self._get_dates_agg(interval),
+        }
+
+        with translate_errors_context(), TimingContext("es", "get_queue_metrics"):
+            res = self._search_company_metrics(company_id, es_req)
+
+        if "aggregations" not in res:
+            return {}
+
+        date_metrics = [
+            dict(
+                timestamp=d["key"],
+                queue_metrics=self._extract_queue_metrics(d["queues"]["buckets"]),
+            )
+            for d in res["aggregations"]["dates"]["buckets"]
+            if d["doc_count"] > 0
+        ]
+        if queue_ids:
+            return self._datetime_histogram_per_queue(date_metrics)
+
+        return self._average_datetime_histogram(date_metrics)
+
+    @classmethod
+    def _datetime_histogram_per_queue(cls, date_metrics: Sequence[dict]) -> dict:
+        """
+        Build datetime histogram per queue from datetime histogram where every
+        bucket contains all the queues metrics
+        """
+        queues_data = defaultdict(list)
+        for date_data in date_metrics:
+            timestamp = date_data["timestamp"]
+            for queue, metrics in date_data["queue_metrics"].items():
+                queues_data[queue].append({"date": timestamp, **metrics})
+
+        return queues_data
+
+    @classmethod
+    def _average_datetime_histogram(cls, date_metrics: Sequence[dict]) -> dict:
+        """
+        Calculate weighted averages and total count for each bucket of date_metrics histogram.
+        If for any queue the data is missing then take it from the previous bucket
+        The result is returned as a dictionary with one key 'total'
+        """
+        queues_total = []
+        last_values = {}
+        for date_data in date_metrics:
+            date_metrics = date_data["queue_metrics"]
+            queue_metrics = {
+                **date_metrics,
+                **{k: v for k, v in last_values.items() if k not in date_metrics},
+            }
+
+            total_length = sum(m["queue_length"] for m in queue_metrics.values())
+            if total_length:
+                total_average = sum(
+                    m["avg_waiting_time"] * m["queue_length"] / total_length
+                    for m in queue_metrics.values()
+                )
+            else:
+                total_average = 0
+
+            queues_total.append(
+                dict(
+                    date=date_data["timestamp"],
+                    avg_waiting_time=total_average,
+                    queue_length=total_length,
+                )
+            )
+
+            for k, v in date_metrics.items():
+                last_values[k] = v
+
+        return dict(total=queues_total)
+
+    @classmethod
+    def _extract_queue_metrics(cls, queue_buckets: Sequence[dict]) -> dict:
+        """
+        Extract ES data for single date and queue bucket
+        """
+        queue_metrics = dict()
+        for queue_data in queue_buckets:
+            if not queue_data["doc_count"]:
+                continue
+            res = queue_data["top_avg_waiting"]["hits"]["hits"][0]["_source"]
+            queue_metrics[queue_data["key"]] = {
+                "queue_length": res[cls.EsKeys.QUEUE_LENGTH_FIELD],
+                "avg_waiting_time": res[cls.EsKeys.WAITING_TIME_FIELD],
+            }
+        return queue_metrics

apiserver/bll/redis_cache_manager.py

@@ -0,0 +1,79 @@
+from contextlib import contextmanager
+from typing import Optional, TypeVar, Generic, Type, Callable
+
+from redis import StrictRedis
+
+from apiserver import database
+from apiserver.timing_context import TimingContext
+
+T = TypeVar("T")
+
+
+def _do_nothing(_: T):
+    return
+
+
+class RedisCacheManager(Generic[T]):
+    """
+    Class for store/retrieve of state objects from redis
+
+    self.state_class - class of the state
+    self.redis - instance of redis
+    self.expiration_interval - expiration interval in seconds
+    """
+
+    def __init__(
+        self, state_class: Type[T], redis: StrictRedis, expiration_interval: int
+    ):
+        self.state_class = state_class
+        self.redis = redis
+        self.expiration_interval = expiration_interval
+
+    def set_state(self, state: T) -> None:
+        redis_key = self._get_redis_key(state.id)
+        with TimingContext("redis", "cache_set_state"):
+            self.redis.set(redis_key, state.to_json())
+            self.redis.expire(redis_key, self.expiration_interval)
+
+    def get_state(self, state_id) -> Optional[T]:
+        redis_key = self._get_redis_key(state_id)
+        with TimingContext("redis", "cache_get_state"):
+            response = self.redis.get(redis_key)
+        if response:
+            return self.state_class.from_json(response)
+
+    def delete_state(self, state_id) -> None:
+        with TimingContext("redis", "cache_delete_state"):
+            self.redis.delete(self._get_redis_key(state_id))
+
+    def _get_redis_key(self, state_id):
+        return f"{self.state_class}/{state_id}"
+
+    @contextmanager
+    def get_or_create_state(
+        self,
+        state_id=None,
+        init_state: Callable[[T], None] = _do_nothing,
+        validate_state: Callable[[T], None] = _do_nothing,
+    ):
+        """
+        Try to retrieve state with the given id from the Redis cache if yes then validates it
+        If no then create a new one with randomly generated id
+        Yield the state and write it back to redis once the user code block exits
+        :param state_id: id of the state to retrieve
+        :param init_state: user callback to init the newly created state
+        If not passed then no init except for the id generation is done
+        :param validate_state: user callback to validate the state if retrieved from cache
+        Should throw an exception if the state is not valid. If not passed then no validation is done
+        """
+        state = self.get_state(state_id) if state_id else None
+        if state:
+            validate_state(state)
+        else:
+            state = self.state_class(id=database.utils.id())
+            init_state(state)
+
+        try:
+            yield state
+        finally:
+            self.set_state(state)

apiserver/bll/statistics/resource_monitor.py

@@ -0,0 +1,90 @@
+from datetime import datetime
+import operator
+from threading import Thread, Lock
+from time import sleep
+
+import attr
+import psutil
+
+from apiserver.utilities.threads_manager import ThreadsManager
+
+
+class ResourceMonitor(Thread):
+    @attr.s(auto_attribs=True)
+    class Sample:
+        cpu_usage: float = 0.0
+        mem_used_gb: float = 0
+        mem_free_gb: float = 0
+
+        @classmethod
+        def _apply(cls, op, *samples):
+            return cls(
+                **{
+                    field: op(*(getattr(sample, field) for sample in samples))
+                    for field in attr.fields_dict(cls)
+                }
+            )
+
+        def min(self, sample):
+            return self._apply(min, self, sample)
+
+        def max(self, sample):
+            return self._apply(max, self, sample)
+
+        def avg(self, sample, count):
+            res = self._apply(lambda x: x * count, self)
+            res = self._apply(operator.add, res, sample)
+            res = self._apply(lambda x: x / (count + 1), res)
+            return res
+
+    def __init__(self, sample_interval_sec=5):
+        super(ResourceMonitor, self).__init__(daemon=True)
+        self.sample_interval_sec = sample_interval_sec
+        self._lock = Lock()
+        self._clear()
+
+    def _clear(self):
+        sample = self._get_sample()
+        self._avg = sample
+        self._min = sample
+        self._max = sample
+        self._clear_time = datetime.utcnow()
+        self._count = 1
+
+    @classmethod
+    def _get_sample(cls) -> Sample:
+        return cls.Sample(
+            cpu_usage=psutil.cpu_percent(),
+            mem_used_gb=psutil.virtual_memory().used / (1024 ** 3),
+            mem_free_gb=psutil.virtual_memory().free / (1024 ** 3),
+        )
+
+    def run(self):
+        while not ThreadsManager.terminating:
+            sleep(self.sample_interval_sec)
+
+            sample = self._get_sample()
+
+            with self._lock:
+                self._min = self._min.min(sample)
+                self._max = self._max.max(sample)
+                self._avg = self._avg.avg(sample, self._count)
+                self._count += 1
+
+    def get_stats(self) -> dict:
+        """ Returns current resource statistics and clears internal resource statistics """
+        with self._lock:
+            min_ = attr.asdict(self._min)
+            max_ = attr.asdict(self._max)
+            avg = attr.asdict(self._avg)
+            interval = datetime.utcnow() - self._clear_time
+            self._clear()
+
+        return {
+            "interval_sec": interval.total_seconds(),
+            "num_cores": psutil.cpu_count(),
+            **{
+                k: {"min": v, "max": max_[k], "avg": avg[k]}
+                for k, v in min_.items()
+            }
+        }

apiserver/bll/statistics/stats_reporter.py

@@ -0,0 +1,304 @@
+import logging
+import queue
+import random
+import time
+from datetime import timedelta, datetime
+from time import sleep
+from typing import Sequence, Optional
+
+import dpath
+import requests
+from requests.adapters import HTTPAdapter
+from requests.packages.urllib3.util.retry import Retry
+
+from apiserver.bll.query import Builder as QueryBuilder
+from apiserver.bll.util import get_server_uuid
+from apiserver.bll.workers import WorkerStats, WorkerBLL
+from apiserver.config_repo import config
+from apiserver.config.info import get_deployment_type
+from apiserver.database.model import Company, User
+from apiserver.database.model.queue import Queue
+from apiserver.database.model.task.task import Task
+from apiserver.tools import safe_get
+from apiserver.utilities.json import dumps
+from apiserver.utilities.threads_manager import ThreadsManager
+from apiserver.version import __version__ as current_version
+from .resource_monitor import ResourceMonitor
+
+log = config.logger(__file__)
+
+worker_bll = WorkerBLL()
+
+
+class StatisticsReporter:
+    threads = ThreadsManager("Statistics", resource_monitor=ResourceMonitor)
+    send_queue = queue.Queue()
+    supported = config.get("apiserver.statistics.supported", True)
+
+    @classmethod
+    def start(cls):
+        cls.start_sender()
+        cls.start_reporter()
+
+    @classmethod
+    @threads.register("reporter", daemon=True)
+    def start_reporter(cls):
+        """
+        Periodically send statistics reports for companies who have opted in.
+        Note: in trains we usually have only a single company
+        """
+        if not cls.supported:
+            return
+
+        report_interval = timedelta(
+            hours=config.get("apiserver.statistics.report_interval_hours", 24)
+        )
+        sleep(report_interval.total_seconds())
+        while not ThreadsManager.terminating:
+            try:
+                for company in Company.objects(
+                    defaults__stats_option__enabled=True
+                ).only("id"):
+                    stats = cls.get_statistics(company.id)
+                    cls.send_queue.put(stats)
+
+            except Exception as ex:
+                log.exception(f"Failed collecting stats: {str(ex)}")
+
+            sleep(report_interval.total_seconds())
+
+    @classmethod
+    @threads.register("sender", daemon=True)
+    def start_sender(cls):
+        if not cls.supported:
+            return
+
+        url = config.get("apiserver.statistics.url")
+
+        retries = config.get("apiserver.statistics.max_retries", 5)
+        max_backoff = config.get("apiserver.statistics.max_backoff_sec", 5)
+        session = requests.Session()
+        adapter = HTTPAdapter(max_retries=Retry(retries))
+        session.mount("http://", adapter)
+        session.mount("https://", adapter)
+        session.headers["Content-type"] = "application/json"
+
+        WarningFilter.attach()
+
+        while not ThreadsManager.terminating:
+            try:
+                report = cls.send_queue.get()
+
+                # Set a random backoff factor each time we send a report
+                adapter.max_retries.backoff_factor = random.random() * max_backoff
+
+                session.post(url, data=dumps(report))
+
+            except Exception as ex:
+                pass
+
+    @classmethod
+    def get_statistics(cls, company_id: str) -> dict:
+        """
+        Returns a statistics report per company
+        """
+        return {
+            "time": datetime.utcnow(),
+            "company_id": company_id,
+            "server": {
+                "version": current_version,
+                "deployment": get_deployment_type(),
+                "uuid": get_server_uuid(),
+                "queues": {"count": Queue.objects(company=company_id).count()},
+                "users": {"count": User.objects(company=company_id).count()},
+                "resources": cls.threads.resource_monitor.get_stats(),
+                "experiments": next(
+                    iter(cls._get_experiments_stats(company_id).values()), {}
+                ),
+            },
+            "agents": cls._get_agents_statistics(company_id),
+        }
+
+    @classmethod
+    def _get_agents_statistics(cls, company_id: str) -> Sequence[dict]:
+        result = cls._get_resource_stats_per_agent(company_id, key="resources")
+        dpath.merge(
+            result, cls._get_experiments_stats_per_agent(company_id, key="experiments")
+        )
+        return [{"uuid": agent_id, **data} for agent_id, data in result.items()]
+
+    @classmethod
+    def _get_resource_stats_per_agent(cls, company_id: str, key: str) -> dict:
+        agent_resource_threshold_sec = timedelta(
+            hours=config.get("apiserver.statistics.report_interval_hours", 24)
+        ).total_seconds()
+        to_timestamp = int(time.time())
+        from_timestamp = to_timestamp - int(agent_resource_threshold_sec)
+        es_req = {
+            "size": 0,
+            "query": QueryBuilder.dates_range(from_timestamp, to_timestamp),
+            "aggs": {
+                "workers": {
+                    "terms": {"field": "worker"},
+                    "aggs": {
+                        "categories": {
+                            "terms": {"field": "category"},
+                            "aggs": {"count": {"cardinality": {"field": "variant"}}},
+                        },
+                        "metrics": {
+                            "terms": {"field": "metric"},
+                            "aggs": {
+                                "min": {"min": {"field": "value"}},
+                                "max": {"max": {"field": "value"}},
+                                "avg": {"avg": {"field": "value"}},
+                            },
+                        },
+                    },
+                }
+            },
+        }
+        res = cls._run_worker_stats_query(company_id, es_req)
+
+        def _get_cardinality_fields(categories: Sequence[dict]) -> dict:
+            names = {"cpu": "num_cores"}
+            return {
+                names[c["key"]]: safe_get(c, "count/value")
+                for c in categories
+                if c["key"] in names
+            }
+
+        def _get_metric_fields(metrics: Sequence[dict]) -> dict:
+            names = {
+                "cpu_usage": "cpu_usage",
+                "memory_used": "mem_used_gb",
+                "memory_free": "mem_free_gb",
+            }
+            return {
+                names[m["key"]]: {
+                    "min": safe_get(m, "min/value"),
+                    "max": safe_get(m, "max/value"),
+                    "avg": safe_get(m, "avg/value"),
+                }
+                for m in metrics
+                if m["key"] in names
+            }
+
+        buckets = safe_get(res, "aggregations/workers/buckets", default=[])
+        return {
+            b["key"]: {
+                key: {
+                    "interval_sec": agent_resource_threshold_sec,
+                    **_get_cardinality_fields(safe_get(b, "categories/buckets", [])),
+                    **_get_metric_fields(safe_get(b, "metrics/buckets", [])),
+                }
+            }
+            for b in buckets
+        }
+
+    @classmethod
+    def _get_experiments_stats_per_agent(cls, company_id: str, key: str) -> dict:
+        agent_relevant_threshold = timedelta(
+            days=config.get("apiserver.statistics.agent_relevant_threshold_days", 30)
+        )
+        to_timestamp = int(time.time())
+        from_timestamp = to_timestamp - int(agent_relevant_threshold.total_seconds())
+        workers = cls._get_active_workers(company_id, from_timestamp, to_timestamp)
+        if not workers:
+            return {}
+
+        stats = cls._get_experiments_stats(company_id, list(workers.keys()))
+        return {
+            worker_id: {key: {**workers[worker_id], **stat}}
+            for worker_id, stat in stats.items()
+        }
+
+    @classmethod
+    def _get_active_workers(
+        cls, company_id, from_timestamp: int, to_timestamp: int
+    ) -> dict:
+        es_req = {
+            "size": 0,
+            "query": QueryBuilder.dates_range(from_timestamp, to_timestamp),
+            "aggs": {
+                "workers": {
+                    "terms": {"field": "worker"},
+                    "aggs": {"last_activity_time": {"max": {"field": "timestamp"}}},
+                }
+            },
+        }
+        res = cls._run_worker_stats_query(company_id, es_req)
+        buckets = safe_get(res, "aggregations/workers/buckets", default=[])
+        return {
+            b["key"]: {"last_activity_time": b["last_activity_time"]["value"]}
+            for b in buckets
+        }
+
+    @classmethod
+    def _run_worker_stats_query(cls, company_id, es_req) -> dict:
+        return worker_bll.es_client.search(
+            index=f"{WorkerStats.worker_stats_prefix_for_company(company_id)}*",
+            body=es_req,
+        )
+
+    @classmethod
+    def _get_experiments_stats(
+        cls, company_id, workers: Optional[Sequence] = None
+    ) -> dict:
+        pipeline = [
+            {
+                "$match": {
+                    "company": company_id,
+                    "started": {"$exists": True, "$ne": None},
+                    "last_update": {"$exists": True, "$ne": None},
+                    "status": {"$nin": ["created", "queued"]},
+                    **({"last_worker": {"$in": workers}} if workers else {}),
+                }
+            },
+            {
+                "$group": {
+                    "_id": "$last_worker" if workers else None,
+                    "count": {"$sum": 1},
+                    "avg_run_time_sec": {
+                        "$avg": {
+                            "$divide": [
+                                {"$subtract": ["$last_update", "$started"]},
+                                1000,
+                            ]
+                        }
+                    },
+                    "avg_iterations": {"$avg": "$last_iteration"},
+                }
+            },
+            {
+                "$project": {
+                    "count": 1,
+                    "avg_run_time_sec": {"$trunc": "$avg_run_time_sec"},
+                    "avg_iterations": {"$trunc": "$avg_iterations"},
+                }
+            },
+        ]
+        return {
+            group["_id"]: {k: v for k, v in group.items() if k != "_id"}
+            for group in Task.aggregate(pipeline)
+        }
+
+
+class WarningFilter(logging.Filter):
+    @classmethod
+    def attach(cls):
+        from urllib3.connectionpool import (
+            ConnectionPool,
+        )  # required to make sure the logger is created
+
+        assert ConnectionPool  # make sure import is not optimized out
+
+        logging.getLogger("urllib3.connectionpool").addFilter(cls())
+
+    def filter(self, record):
+        if (
+            record.levelno == logging.WARNING
+            and len(record.args) > 2
+            and record.args[2] == "/stats"
+        ):
+            return False
+        return True

apiserver/bll/task/artifacts.py

@@ -0,0 +1,97 @@
+from hashlib import md5
+from operator import itemgetter
+from typing import Sequence
+
+from apiserver.apimodels.tasks import Artifact as ApiArtifact, ArtifactId
+from apiserver.bll.task.utils import get_task_for_update, update_task
+from apiserver.database.model.task.task import DEFAULT_ARTIFACT_MODE, Artifact
+from apiserver.timing_context import TimingContext
+from apiserver.utilities.dicts import nested_get, nested_set
+from apiserver.utilities.parameter_key_escaper import mongoengine_safe
+
+
+def get_artifact_id(artifact: dict):
+    """
+    Calculate id from 'key' and 'mode' fields
+    Return hash on on the id so that it will not contain mongo illegal characters
+    """
+    key_hash: str = md5(artifact["key"].encode()).hexdigest()
+    mode: str = artifact.get("mode", DEFAULT_ARTIFACT_MODE)
+    return f"{key_hash}_{mode}"
+
+
+def artifacts_prepare_for_save(fields: dict):
+    artifacts_field = ("execution", "artifacts")
+    artifacts = nested_get(fields, artifacts_field)
+    if artifacts is None:
+        return
+
+    nested_set(
+        fields, artifacts_field, value={get_artifact_id(a): a for a in artifacts}
+    )
+
+
+def artifacts_unprepare_from_saved(fields):
+    artifacts_field = ("execution", "artifacts")
+    artifacts = nested_get(fields, artifacts_field)
+    if artifacts is None:
+        return
+
+    nested_set(
+        fields,
+        artifacts_field,
+        value=sorted(artifacts.values(), key=itemgetter("key", "mode")),
+    )
+
+
+class Artifacts:
+    @classmethod
+    def add_or_update_artifacts(
+        cls,
+        company_id: str,
+        task_id: str,
+        artifacts: Sequence[ApiArtifact],
+        force: bool,
+    ) -> int:
+        with TimingContext("mongo", "update_artifacts"):
+            task = get_task_for_update(
+                company_id=company_id,
+                task_id=task_id,
+                force=force,
+            )
+
+            artifacts = {
+                get_artifact_id(a): Artifact(**a)
+                for a in (api_artifact.to_struct() for api_artifact in artifacts)
+            }
+
+            update_cmds = {
+                f"set__execution__artifacts__{mongoengine_safe(name)}": value
+                for name, value in artifacts.items()
+            }
+            return update_task(task, update_cmds=update_cmds)
+
+    @classmethod
+    def delete_artifacts(
+        cls,
+        company_id: str,
+        task_id: str,
+        artifact_ids: Sequence[ArtifactId],
+        force: bool,
+    ) -> int:
+        with TimingContext("mongo", "delete_artifacts"):
+            task = get_task_for_update(
+                company_id=company_id,
+                task_id=task_id,
+                force=force,
+            )
+
+            artifact_ids = [
+                get_artifact_id(a)
+                for a in (artifact_id.to_struct() for artifact_id in artifact_ids)
+            ]
+            delete_cmds = {
+                f"unset__execution__artifacts__{id_}": 1 for id_ in set(artifact_ids)
+            }
+
+            return update_task(task, update_cmds=delete_cmds)

apiserver/bll/task/hyperparams.py

@@ -0,0 +1,245 @@
+from itertools import chain
+from operator import attrgetter
+from typing import Sequence, Dict
+
+from boltons import iterutils
+
+from apiserver.apierrors import errors
+from apiserver.apimodels.tasks import (
+    HyperParamKey,
+    HyperParamItem,
+    ReplaceHyperparams,
+    Configuration,
+)
+from apiserver.bll.task import TaskBLL
+from apiserver.bll.task.utils import get_task_for_update, update_task
+from apiserver.config_repo import config
+from apiserver.database.model.task.task import ParamsItem, Task, ConfigurationItem
+from apiserver.timing_context import TimingContext
+from apiserver.utilities.parameter_key_escaper import (
+    ParameterKeyEscaper,
+    mongoengine_safe,
+)
+
+log = config.logger(__file__)
+task_bll = TaskBLL()
+
+
+class HyperParams:
+    _properties_section = "properties"
+
+    @classmethod
+    def get_params(cls, company_id: str, task_ids: Sequence[str]) -> Dict[str, dict]:
+        only = ("id", "hyperparams")
+        tasks = task_bll.assert_exists(
+            company_id=company_id, task_ids=task_ids, only=only, allow_public=True,
+        )
+
+        return {
+            task.id: {"hyperparams": cls._get_params_list(items=task.hyperparams)}
+            for task in tasks
+        }
+
+    @classmethod
+    def _get_params_list(
+        cls, items: Dict[str, Dict[str, ParamsItem]]
+    ) -> Sequence[dict]:
+        ret = list(chain.from_iterable(v.values() for v in items.values()))
+        return [
+            p.to_proper_dict() for p in sorted(ret, key=attrgetter("section", "name"))
+        ]
+
+    @classmethod
+    def _normalize_params(cls, params: Sequence) -> bool:
+        """
+        Lower case properties section and return True if it is the only section
+        """
+        for p in params:
+            if p.section.lower() == cls._properties_section:
+                p.section = cls._properties_section
+
+        return all(p.section == cls._properties_section for p in params)
+
+    @classmethod
+    def delete_params(
+        cls,
+        company_id: str,
+        task_id: str,
+        hyperparams: Sequence[HyperParamKey],
+        force: bool,
+    ) -> int:
+        with TimingContext("mongo", "delete_hyperparams"):
+            properties_only = cls._normalize_params(hyperparams)
+            task = get_task_for_update(
+                company_id=company_id,
+                task_id=task_id,
+                allow_all_statuses=properties_only,
+                force=force,
+            )
+
+            with_param, without_param = iterutils.partition(
+                hyperparams, key=lambda p: bool(p.name)
+            )
+            sections_to_delete = {p.section for p in without_param}
+            delete_cmds = {
+                f"unset__hyperparams__{ParameterKeyEscaper.escape(section)}": 1
+                for section in sections_to_delete
+            }
+
+            for item in with_param:
+                section = ParameterKeyEscaper.escape(item.section)
+                if item.section in sections_to_delete:
+                    raise errors.bad_request.FieldsConflict(
+                        "Cannot delete section field if the whole section was scheduled for deletion"
+                    )
+                name = ParameterKeyEscaper.escape(item.name)
+                delete_cmds[f"unset__hyperparams__{section}__{name}"] = 1
+
+            return update_task(
+                task, update_cmds=delete_cmds, set_last_update=not properties_only
+            )
+
+    @classmethod
+    def edit_params(
+        cls,
+        company_id: str,
+        task_id: str,
+        hyperparams: Sequence[HyperParamItem],
+        replace_hyperparams: str,
+        force: bool,
+    ) -> int:
+        with TimingContext("mongo", "edit_hyperparams"):
+            properties_only = cls._normalize_params(hyperparams)
+            task = get_task_for_update(
+                company_id=company_id,
+                task_id=task_id,
+                allow_all_statuses=properties_only,
+                force=force,
+            )
+
+            update_cmds = dict()
+            hyperparams = cls._db_dicts_from_list(hyperparams)
+            if replace_hyperparams == ReplaceHyperparams.all:
+                update_cmds["set__hyperparams"] = hyperparams
+            elif replace_hyperparams == ReplaceHyperparams.section:
+                for section, value in hyperparams.items():
+                    update_cmds[
+                        f"set__hyperparams__{mongoengine_safe(section)}"
+                    ] = value
+            else:
+                for section, section_params in hyperparams.items():
+                    for name, value in section_params.items():
+                        update_cmds[
+                            f"set__hyperparams__{section}__{mongoengine_safe(name)}"
+                        ] = value
+
+            return update_task(
+                task, update_cmds=update_cmds, set_last_update=not properties_only
+            )
+
+    @classmethod
+    def _db_dicts_from_list(cls, items: Sequence[HyperParamItem]) -> Dict[str, dict]:
+        sections = iterutils.bucketize(items, key=attrgetter("section"))
+        return {
+            ParameterKeyEscaper.escape(section): {
+                ParameterKeyEscaper.escape(param.name): ParamsItem(**param.to_struct())
+                for param in params
+            }
+            for section, params in sections.items()
+        }
+
+    @classmethod
+    def get_configurations(
+        cls, company_id: str, task_ids: Sequence[str], names: Sequence[str]
+    ) -> Dict[str, dict]:
+        only = ["id"]
+        if names:
+            only.extend(
+                f"configuration.{ParameterKeyEscaper.escape(name)}" for name in names
+            )
+        else:
+            only.append("configuration")
+        tasks = task_bll.assert_exists(
+            company_id=company_id, task_ids=task_ids, only=only, allow_public=True,
+        )
+
+        return {
+            task.id: {
+                "configuration": [
+                    c.to_proper_dict()
+                    for c in sorted(task.configuration.values(), key=attrgetter("name"))
+                ]
+            }
+            for task in tasks
+        }
+
+    @classmethod
+    def get_configuration_names(
+        cls, company_id: str, task_ids: Sequence[str]
+    ) -> Dict[str, list]:
+        with TimingContext("mongo", "get_configuration_names"):
+            pipeline = [
+                {
+                    "$match": {
+                        "company": {"$in": [None, "", company_id]},
+                        "_id": {"$in": task_ids},
+                    }
+                },
+                {"$project": {"items": {"$objectToArray": "$configuration"}}},
+                {"$unwind": "$items"},
+                {"$group": {"_id": "$_id", "names": {"$addToSet": "$items.k"}}},
+            ]
+
+            tasks = Task.aggregate(pipeline)
+
+            return {
+                task["_id"]: {
+                    "names": sorted(
+                        ParameterKeyEscaper.unescape(name) for name in task["names"]
+                    )
+                }
+                for task in tasks
+            }
+
+    @classmethod
+    def edit_configuration(
+        cls,
+        company_id: str,
+        task_id: str,
+        configuration: Sequence[Configuration],
+        replace_configuration: bool,
+        force: bool,
+    ) -> int:
+        with TimingContext("mongo", "edit_configuration"):
+            task = get_task_for_update(
+                company_id=company_id, task_id=task_id, force=force
+            )
+
+            update_cmds = dict()
+            configuration = {
+                ParameterKeyEscaper.escape(c.name): ConfigurationItem(**c.to_struct())
+                for c in configuration
+            }
+            if replace_configuration:
+                update_cmds["set__configuration"] = configuration
+            else:
+                for name, value in configuration.items():
+                    update_cmds[f"set__configuration__{mongoengine_safe(name)}"] = value
+
+            return update_task(task, update_cmds=update_cmds)
+
+    @classmethod
+    def delete_configuration(
+        cls, company_id: str, task_id: str, configuration: Sequence[str], force: bool
+    ) -> int:
+        with TimingContext("mongo", "delete_configuration"):
+            task = get_task_for_update(
+                company_id=company_id, task_id=task_id, force=force
+            )
+
+            delete_cmds = {
+                f"unset__configuration__{ParameterKeyEscaper.escape(name)}": 1
+                for name in set(configuration)
+            }
+
+            return update_task(task, update_cmds=delete_cmds)

apiserver/bll/task/non_responsive_tasks_watchdog.py

@@ -0,0 +1,98 @@
+from datetime import timedelta, datetime
+from time import sleep
+
+from apiserver.bll.task import update_project_time
+from apiserver.config_repo import config
+from apiserver.database.model.task.task import TaskStatus, Task
+from apiserver.utilities.threads_manager import ThreadsManager
+
+log = config.logger(__file__)
+
+
+class NonResponsiveTasksWatchdog:
+    threads = ThreadsManager()
+
+    class _Settings:
+        """
+        Retrieves watchdog settings from the config file
+        The properties are not cached so that the updates in
+        the config file are reflected
+        """
+
+        _prefix = "services.tasks.non_responsive_tasks_watchdog"
+
+        @property
+        def enabled(self):
+            return config.get(f"{self._prefix}.enabled", True)
+
+        @property
+        def watch_interval_sec(self):
+            return config.get(f"{self._prefix}.watch_interval_sec", 900)
+
+        @property
+        def threshold_sec(self):
+            return config.get(f"{self._prefix}.threshold_sec", 7200)
+
+    settings = _Settings()
+
+    @classmethod
+    @threads.register("non_responsive_tasks_watchdog", daemon=True)
+    def start(cls):
+        sleep(cls.settings.watch_interval_sec)
+        while not ThreadsManager.terminating:
+            watch_interval = cls.settings.watch_interval_sec
+            if cls.settings.enabled:
+                try:
+                    stopped = cls.cleanup_tasks(
+                        threshold_sec=cls.settings.threshold_sec
+                    )
+                    log.info(f"{stopped} non-responsive tasks stopped")
+                except Exception as ex:
+                    log.exception(f"Failed stopping tasks: {str(ex)}")
+            sleep(watch_interval)
+
+    @classmethod
+    def cleanup_tasks(cls, threshold_sec):
+        relevant_status = (TaskStatus.in_progress,)
+        threshold = timedelta(seconds=threshold_sec)
+        ref_time = datetime.utcnow() - threshold
+        log.info(
+            f"Starting cleanup cycle for running tasks last updated before {ref_time}"
+        )
+
+        tasks = list(
+            Task.objects(status__in=relevant_status, last_update__lt=ref_time).only(
+                "id", "name", "status", "project", "last_update"
+            )
+        )
+        log.info(f"{len(tasks)} non-responsive tasks found")
+        if not tasks:
+            return 0
+
+        err_count = 0
+        project_ids = set()
+        now = datetime.utcnow()
+        for task in tasks:
+            log.info(
+                f"Stopping {task.id} ({task.name}), last updated at {task.last_update}"
+            )
+            # noinspection PyBroadException
+            try:
+                updated = Task.objects(id=task.id, status=task.status).update(
+                    status=TaskStatus.stopped,
+                    status_reason="Forced stop (non-responsive)",
+                    status_message="Forced stop (non-responsive)",
+                    status_changed=now,
+                    last_update=now,
+                    last_change=now,
+                )
+                if updated:
+                    project_ids.add(task.project)
+                else:
+                    err_count += 1
+            except Exception as ex:
+                log.error("Failed setting status: %s", str(ex))
+
+        update_project_time(list(project_ids))
+
+        return len(tasks) - err_count

apiserver/bll/task/param_utils.py

@@ -0,0 +1,201 @@
+import itertools
+from typing import Sequence, Tuple
+
+import dpath
+
+from apiserver.apierrors import errors
+from apiserver.database.model.task.task import Task
+from apiserver.tools import safe_get
+from apiserver.utilities.parameter_key_escaper import ParameterKeyEscaper
+
+
+hyperparams_default_section = "Args"
+hyperparams_legacy_type = "legacy"
+tf_define_section = "TF_DEFINE"
+
+
+def split_param_name(full_name: str, default_section: str) -> Tuple[str, str]:
+    """
+    Return parameter section and name. The section is either TF_DEFINE or the default one
+    """
+    if default_section is None:
+        return None, full_name
+
+    section, _, name = full_name.partition("/")
+    if section != tf_define_section:
+        return default_section, full_name
+
+    if not name:
+        raise errors.bad_request.ValidationError("Parameter name cannot be empty")
+    return section, name
+
+
+def _get_full_param_name(param: dict) -> str:
+    section = param.get("section")
+    if section != tf_define_section:
+        return param["name"]
+
+    return "/".join((section, param["name"]))
+
+
+def _remove_legacy_params(data: dict, with_sections: bool = False) -> int:
+    """
+    Remove the legacy params from the data dict and return the number of removed params
+    If the path not found then return 0
+    """
+    removed = 0
+    if not data:
+        return removed
+
+    if with_sections:
+        for section, section_data in list(data.items()):
+            removed += _remove_legacy_params(section_data)
+            if not section_data:
+                """If section is empty after removing legacy params then delete it"""
+                del data[section]
+    else:
+        for key, param in list(data.items()):
+            if param.get("type") == hyperparams_legacy_type:
+                removed += 1
+                del data[key]
+
+    return removed
+
+
+def _get_legacy_params(data: dict, with_sections: bool = False) -> Sequence[str]:
+    """
+    Remove the legacy params from the data dict and return the number of removed params
+    If the path not found then return 0
+    """
+    if not data:
+        return []
+
+    if with_sections:
+        return itertools.chain.from_iterable(
+            _get_legacy_params(section_data) for section_data in data.values()
+        )
+
+    return [
+        param for param in data.values() if param.get("type") == hyperparams_legacy_type
+    ]
+
+
+def params_prepare_for_save(fields: dict, previous_task: Task = None):
+    """
+    If legacy hyper params or configuration is passed then replace the corresponding section in the new structure
+    Escape all the section and param names for hyper params and configuration to make it mongo sage
+    """
+    for old_params_field, new_params_field, default_section in (
+        ("execution/parameters", "hyperparams", hyperparams_default_section),
+        ("execution/model_desc", "configuration", None),
+    ):
+        legacy_params = safe_get(fields, old_params_field)
+        if legacy_params is None:
+            continue
+
+        if (
+            not safe_get(fields, new_params_field)
+            and previous_task
+            and previous_task[new_params_field]
+        ):
+            previous_data = previous_task.to_proper_dict().get(new_params_field)
+            removed = _remove_legacy_params(
+                previous_data, with_sections=default_section is not None
+            )
+            if not legacy_params and not removed:
+                # if we only need to delete legacy fields from the db
+                # but they are not there then there is no point to proceed
+                continue
+
+            fields_update = {new_params_field: previous_data}
+            params_unprepare_from_saved(fields_update)
+            fields.update(fields_update)
+
+        for full_name, value in legacy_params.items():
+            section, name = split_param_name(full_name, default_section)
+            new_path = list(filter(None, (new_params_field, section, name)))
+            new_param = dict(name=name, type=hyperparams_legacy_type, value=str(value))
+            if section is not None:
+                new_param["section"] = section
+            dpath.new(fields, new_path, new_param)
+        dpath.delete(fields, old_params_field)
+
+    for param_field in ("hyperparams", "configuration"):
+        params = safe_get(fields, param_field)
+        if params:
+            escaped_params = {
+                ParameterKeyEscaper.escape(key): {
+                    ParameterKeyEscaper.escape(k): v for k, v in value.items()
+                }
+                if isinstance(value, dict)
+                else value
+                for key, value in params.items()
+            }
+            dpath.set(fields, param_field, escaped_params)
+
+
+def params_unprepare_from_saved(fields, copy_to_legacy=False):
+    """
+    Unescape all section and param names for hyper params and configuration
+    If copy_to_legacy is set then copy hyperparams and configuration data to the legacy location for the old clients
+    """
+    for param_field in ("hyperparams", "configuration"):
+        params = safe_get(fields, param_field)
+        if params:
+            unescaped_params = {
+                ParameterKeyEscaper.unescape(key): {
+                    ParameterKeyEscaper.unescape(k): v for k, v in value.items()
+                }
+                if isinstance(value, dict)
+                else value
+                for key, value in params.items()
+            }
+            dpath.set(fields, param_field, unescaped_params)
+
+    if copy_to_legacy:
+        for new_params_field, old_params_field, use_sections in (
+            (f"hyperparams", "execution/parameters", True),
+            (f"configuration", "execution/model_desc", False),
+        ):
+            legacy_params = _get_legacy_params(
+                safe_get(fields, new_params_field), with_sections=use_sections
+            )
+            if legacy_params:
+                dpath.new(
+                    fields,
+                    old_params_field,
+                    {_get_full_param_name(p): p["value"] for p in legacy_params},
+                )
+
+
+def _process_path(path: str):
+    """
+    Frontend does a partial escaping on the path so the all '.' in section and key names are escaped
+    Need to unescape and apply a full mongo escaping
+    """
+    parts = path.split(".")
+    if len(parts) < 2 or len(parts) > 3:
+        raise errors.bad_request.ValidationError("invalid task field", path=path)
+    return ".".join(
+        ParameterKeyEscaper.escape(ParameterKeyEscaper.unescape(p)) for p in parts
+    )
+
+
+def escape_paths(paths: Sequence[str]) -> Sequence[str]:
+    for old_prefix, new_prefix in (
+        ("execution.parameters", f"hyperparams.{hyperparams_default_section}"),
+        ("execution.model_desc", f"configuration"),
+    ):
+        path: str
+        paths = [path.replace(old_prefix, new_prefix) for path in paths]
+
+    for prefix in (
+        "hyperparams.",
+        "-hyperparams.",
+        "configuration.",
+        "-configuration.",
+    ):
+        paths = [
+            _process_path(path) if path.startswith(prefix) else path for path in paths
+        ]
+    return paths

apiserver/bll/task/task_bll.py

@@ -0,0 +1,715 @@
+from collections import OrderedDict
+from datetime import datetime
+from typing import Collection, Sequence, Tuple, Any, Optional, Dict
+
+import dpath
+import six
+from mongoengine import Q
+from six import string_types
+
+import apiserver.database.utils as dbutils
+from apiserver.apierrors import errors
+from apiserver.bll.queue import QueueBLL
+from apiserver.bll.organization import OrgBLL, Tags
+from apiserver.bll.project import ProjectBLL
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.model import Model
+from apiserver.database.model.project import Project
+from apiserver.database.model.task.metrics import EventStats, MetricEventStats
+from apiserver.database.model.task.output import Output
+from apiserver.database.model.task.task import (
+    Task,
+    TaskStatus,
+    TaskStatusMessage,
+    TaskSystemTags,
+    ArtifactModes,
+    external_task_types,
+)
+from apiserver.database.model import EntityVisibility
+from apiserver.database.utils import get_company_or_none_constraint, id as create_id
+from apiserver.es_factory import es_factory
+from apiserver.service_repo import APICall
+from apiserver.services.utils import validate_tags
+from apiserver.timing_context import TimingContext
+from apiserver.utilities.parameter_key_escaper import ParameterKeyEscaper
+from .artifacts import artifacts_prepare_for_save
+from .param_utils import params_prepare_for_save
+from .utils import ChangeStatusRequest, validate_status_change, update_project_time
+
+log = config.logger(__file__)
+org_bll = OrgBLL()
+queue_bll = QueueBLL()
+project_bll = ProjectBLL()
+
+
+class TaskBLL:
+    def __init__(self, events_es=None):
+        self.events_es = (
+            events_es if events_es is not None else es_factory.connect("events")
+        )
+
+    @classmethod
+    def get_types(cls, company, project_ids: Optional[Sequence]) -> set:
+        """
+        Return the list of unique task types used by company and public tasks
+        If project ids passed then only tasks from these projects are considered
+        """
+        query = get_company_or_none_constraint(company)
+        if project_ids:
+            query &= Q(project__in=project_ids)
+        res = Task.objects(query).distinct(field="type")
+        return set(res).intersection(external_task_types)
+
+    @staticmethod
+    def get_task_with_access(
+        task_id, company_id, only=None, allow_public=False, requires_write_access=False
+    ) -> Task:
+        """
+        Gets a task that has a required write access
+        :except errors.bad_request.InvalidTaskId: if the task is not found
+        :except errors.forbidden.NoWritePermission: if write_access was required and the task cannot be modified
+        """
+        with translate_errors_context():
+            query = dict(id=task_id, company=company_id)
+            with TimingContext("mongo", "task_with_access"):
+                if requires_write_access:
+                    task = Task.get_for_writing(_only=only, **query)
+                else:
+                    task = Task.get(_only=only, **query, include_public=allow_public)
+
+            if not task:
+                raise errors.bad_request.InvalidTaskId(**query)
+
+            return task
+
+    @staticmethod
+    def get_by_id(
+        company_id, task_id, required_status=None, only_fields=None, allow_public=False,
+    ):
+        if only_fields:
+            if isinstance(only_fields, string_types):
+                only_fields = [only_fields]
+            else:
+                only_fields = list(only_fields)
+            only_fields = only_fields + ["status"]
+
+        with TimingContext("mongo", "task_by_id_all"):
+            tasks = Task.get_many(
+                company=company_id,
+                query=Q(id=task_id),
+                allow_public=allow_public,
+                override_projection=only_fields,
+                return_dicts=False,
+            )
+            task = None if not tasks else tasks[0]
+
+        if not task:
+            raise errors.bad_request.InvalidTaskId(id=task_id)
+
+        if required_status and not task.status == required_status:
+            raise errors.bad_request.InvalidTaskStatus(expected=required_status)
+
+        return task
+
+    @staticmethod
+    def assert_exists(
+        company_id, task_ids, only=None, allow_public=False, return_tasks=True
+    ) -> Optional[Sequence[Task]]:
+        task_ids = [task_ids] if isinstance(task_ids, six.string_types) else task_ids
+        with translate_errors_context(), TimingContext("mongo", "task_exists"):
+            ids = set(task_ids)
+            q = Task.get_many(
+                company=company_id,
+                query=Q(id__in=ids),
+                allow_public=allow_public,
+                return_dicts=False,
+            )
+            if only:
+                # Make sure to reset fields filters (some fields are excluded by default) since this
+                # is an internal call and specific fields were requested.
+                q = q.all_fields().only(*only)
+
+            if q.count() != len(ids):
+                raise errors.bad_request.InvalidTaskId(ids=task_ids)
+
+            if return_tasks:
+                return list(q)
+
+    @staticmethod
+    def create(call: APICall, fields: dict):
+        identity = call.identity
+        now = datetime.utcnow()
+        return Task(
+            id=create_id(),
+            user=identity.user,
+            company=identity.company,
+            created=now,
+            last_update=now,
+            last_change=now,
+            **fields,
+        )
+
+    @staticmethod
+    def validate_execution_model(task, allow_only_public=False):
+        if not task.execution or not task.execution.model:
+            return
+
+        company = None if allow_only_public else task.company
+        model_id = task.execution.model
+        model = Model.objects(
+            Q(id=model_id) & get_company_or_none_constraint(company)
+        ).first()
+        if not model:
+            raise errors.bad_request.InvalidModelId(model=model_id)
+
+        return model
+
+    @classmethod
+    def clone_task(
+        cls,
+        company_id: str,
+        user_id: str,
+        task_id: str,
+        name: Optional[str] = None,
+        comment: Optional[str] = None,
+        parent: Optional[str] = None,
+        project: Optional[str] = None,
+        tags: Optional[Sequence[str]] = None,
+        system_tags: Optional[Sequence[str]] = None,
+        hyperparams: Optional[dict] = None,
+        configuration: Optional[dict] = None,
+        execution_overrides: Optional[dict] = None,
+        validate_references: bool = False,
+        new_project_name: str = None,
+    ) -> Tuple[Task, dict]:
+        validate_tags(tags, system_tags)
+        params_dict = {
+            field: value
+            for field, value in (
+                ("hyperparams", hyperparams),
+                ("configuration", configuration),
+            )
+            if value is not None
+        }
+
+        task = cls.get_by_id(company_id=company_id, task_id=task_id, allow_public=True)
+
+        execution_dict = task.execution.to_proper_dict() if task.execution else {}
+        execution_model_overriden = False
+        if execution_overrides:
+            execution_model_overriden = execution_overrides.get("model") is not None
+            artifacts_prepare_for_save({"execution": execution_overrides})
+
+            params_dict["execution"] = {}
+            for legacy_param in ("parameters", "configuration"):
+                legacy_value = execution_overrides.pop(legacy_param, None)
+                if legacy_value is not None:
+                    params_dict["execution"] = legacy_value
+
+            execution_dict.update(execution_overrides)
+
+        params_prepare_for_save(params_dict, previous_task=task)
+
+        artifacts = execution_dict.get("artifacts")
+        if artifacts:
+            execution_dict["artifacts"] = {
+                k: a
+                for k, a in artifacts.items()
+                if a.get("mode") != ArtifactModes.output
+            }
+        execution_dict.pop("queue", None)
+
+        new_project_data = None
+        if not project and new_project_name:
+            # Use a project with the provided name, or create a new project
+            project = ProjectBLL.find_or_create(
+                project_name=new_project_name,
+                user=user_id,
+                company=company_id,
+                description="Auto-generated while cloning",
+            )
+            new_project_data = {"id": project, "name": new_project_name}
+
+        now = datetime.utcnow()
+
+        def clean_system_tags(input_tags: Sequence[str]) -> Sequence[str]:
+            if not input_tags:
+                return input_tags
+
+            return [
+                tag
+                for tag in input_tags
+                if tag not in [TaskSystemTags.development, EntityVisibility.archived.value]
+            ]
+
+        with TimingContext("mongo", "clone task"):
+            new_task = Task(
+                id=create_id(),
+                user=user_id,
+                company=company_id,
+                created=now,
+                last_update=now,
+                last_change=now,
+                name=name or task.name,
+                comment=comment or task.comment,
+                parent=parent or task.parent,
+                project=project or task.project,
+                tags=tags or task.tags,
+                system_tags=system_tags or clean_system_tags(task.system_tags),
+                type=task.type,
+                script=task.script,
+                output=Output(destination=task.output.destination)
+                if task.output
+                else None,
+                execution=execution_dict,
+                configuration=params_dict.get("configuration") or task.configuration,
+                hyperparams=params_dict.get("hyperparams") or task.hyperparams,
+            )
+            cls.validate(
+                new_task,
+                validate_model=validate_references or execution_model_overriden,
+                validate_parent=validate_references or parent,
+                validate_project=validate_references or project,
+            )
+            new_task.save()
+
+            if task.project == new_task.project:
+                updated_tags = tags
+                updated_system_tags = system_tags
+            else:
+                updated_tags = new_task.tags
+                updated_system_tags = new_task.system_tags
+            org_bll.update_tags(
+                company_id,
+                Tags.Task,
+                project=new_task.project,
+                tags=updated_tags,
+                system_tags=updated_system_tags,
+            )
+            update_project_time(new_task.project)
+
+        return new_task, new_project_data
+
+    @classmethod
+    def validate(
+        cls,
+        task: Task,
+        validate_model=True,
+        validate_parent=True,
+        validate_project=True,
+    ):
+        """
+        Validate task properties according to the flag
+        Task project is always checked for being writable
+        in order to disable the modification of public projects
+        """
+        if (
+            validate_parent
+            and task.parent
+            and not Task.get(
+                company=task.company, id=task.parent, _only=("id",), include_public=True
+            )
+        ):
+            raise errors.bad_request.InvalidTaskId("invalid parent", parent=task.parent)
+
+        if task.project:
+            project = Project.get_for_writing(company=task.company, id=task.project)
+            if validate_project and not project:
+                raise errors.bad_request.InvalidProjectId(id=task.project)
+
+        if validate_model:
+            cls.validate_execution_model(task)
+
+    @staticmethod
+    def get_unique_metric_variants(company_id, project_ids=None):
+        pipeline = [
+            {
+                "$match": dict(
+                    company={"$in": [None, "", company_id]},
+                    **({"project": {"$in": project_ids}} if project_ids else {}),
+                )
+            },
+            {"$project": {"metrics": {"$objectToArray": "$last_metrics"}}},
+            {"$unwind": "$metrics"},
+            {
+                "$project": {
+                    "metric": "$metrics.k",
+                    "variants": {"$objectToArray": "$metrics.v"},
+                }
+            },
+            {"$unwind": "$variants"},
+            {
+                "$group": {
+                    "_id": {
+                        "metric": "$variants.v.metric",
+                        "variant": "$variants.v.variant",
+                    },
+                    "metrics": {
+                        "$addToSet": {
+                            "metric": "$variants.v.metric",
+                            "metric_hash": "$metric",
+                            "variant": "$variants.v.variant",
+                            "variant_hash": "$variants.k",
+                        }
+                    },
+                }
+            },
+            {"$sort": OrderedDict({"_id.metric": 1, "_id.variant": 1})},
+        ]
+
+        with translate_errors_context():
+            result = Task.aggregate(pipeline)
+            return [r["metrics"][0] for r in result]
+
+    @staticmethod
+    def set_last_update(
+        task_ids: Collection[str],
+        company_id: str,
+        last_update: datetime,
+        **extra_updates,
+    ):
+        tasks = Task.objects(id__in=task_ids, company=company_id).only(
+            "status", "started"
+        )
+        for task in tasks:
+            updates = extra_updates
+            if task.status == TaskStatus.in_progress and task.started:
+                updates = {
+                    "active_duration": (
+                        datetime.utcnow() - task.started
+                    ).total_seconds(),
+                    **extra_updates,
+                }
+            Task.objects(id=task.id, company=company_id).update(
+                upsert=False,
+                last_update=last_update,
+                last_change=last_update,
+                **updates,
+            )
+
+    @staticmethod
+    def update_statistics(
+        task_id: str,
+        company_id: str,
+        last_update: datetime = None,
+        last_iteration: int = None,
+        last_iteration_max: int = None,
+        last_scalar_values: Sequence[Tuple[Tuple[str, ...], Any]] = None,
+        last_events: Dict[str, Dict[str, dict]] = None,
+        **extra_updates,
+    ):
+        """
+        Update task statistics
+        :param task_id: Task's ID.
+        :param company_id: Task's company ID.
+        :param last_update: Last update time. If not provided, defaults to datetime.utcnow().
+        :param last_iteration: Last reported iteration. Use this to set a value regardless of current
+            task's last iteration value.
+        :param last_iteration_max: Last reported iteration. Use this to conditionally set a value only
+            if the current task's last iteration value is smaller than the provided value.
+        :param last_scalar_values: Last reported metrics summary for scalar events (value, metric, variant).
+        :param last_events: Last reported metrics summary (value, metric, event type).
+        :param extra_updates: Extra task updates to include in this update call.
+        :return:
+        """
+        last_update = last_update or datetime.utcnow()
+
+        if last_iteration is not None:
+            extra_updates.update(last_iteration=last_iteration)
+        elif last_iteration_max is not None:
+            extra_updates.update(max__last_iteration=last_iteration_max)
+
+        if last_scalar_values is not None:
+
+            def op_path(op, *path):
+                return "__".join((op, "last_metrics") + path)
+
+            for path, value in last_scalar_values:
+                if path[-1] == "min_value":
+                    extra_updates[op_path("min", *path[:-1], "min_value")] = value
+                elif path[-1] == "max_value":
+                    extra_updates[op_path("max", *path[:-1], "max_value")] = value
+                else:
+                    extra_updates[op_path("set", *path)] = value
+
+        if last_events is not None:
+
+            def events_per_type(metric_data: Dict[str, dict]) -> Dict[str, EventStats]:
+                return {
+                    event_type: EventStats(last_update=event["timestamp"])
+                    for event_type, event in metric_data.items()
+                }
+
+            metric_stats = {
+                dbutils.hash_field_name(metric_key): MetricEventStats(
+                    metric=metric_key, event_stats_by_type=events_per_type(metric_data)
+                )
+                for metric_key, metric_data in last_events.items()
+            }
+            extra_updates["metric_stats"] = metric_stats
+
+        TaskBLL.set_last_update(
+            task_ids=[task_id],
+            company_id=company_id,
+            last_update=last_update,
+            **extra_updates,
+        )
+
+    @classmethod
+    def model_set_ready(
+        cls,
+        model_id: str,
+        company_id: str,
+        publish_task: bool,
+        force_publish_task: bool = False,
+    ) -> tuple:
+        with translate_errors_context():
+            query = dict(id=model_id, company=company_id)
+            model = Model.objects(**query).first()
+            if not model:
+                raise errors.bad_request.InvalidModelId(**query)
+            elif model.ready:
+                raise errors.bad_request.ModelIsReady(**query)
+
+            published_task_data = {}
+            if model.task and publish_task:
+                task = (
+                    Task.objects(id=model.task, company=company_id)
+                    .only("id", "status")
+                    .first()
+                )
+                if task and task.status != TaskStatus.published:
+                    published_task_data["data"] = cls.publish_task(
+                        task_id=model.task,
+                        company_id=company_id,
+                        publish_model=False,
+                        force=force_publish_task,
+                    )
+                    published_task_data["id"] = model.task
+
+            updated = model.update(upsert=False, ready=True)
+            return updated, published_task_data
+
+    @classmethod
+    def publish_task(
+        cls,
+        task_id: str,
+        company_id: str,
+        publish_model: bool,
+        force: bool,
+        status_reason: str = "",
+        status_message: str = "",
+    ) -> dict:
+        task = cls.get_task_with_access(
+            task_id, company_id=company_id, requires_write_access=True
+        )
+        if not force:
+            validate_status_change(task.status, TaskStatus.published)
+
+        previous_task_status = task.status
+        output = task.output or Output()
+        publish_failed = False
+
+        try:
+            # set state to publishing
+            task.status = TaskStatus.publishing
+            task.save()
+
+            # publish task models
+            if task.output.model and publish_model:
+                output_model = (
+                    Model.objects(id=task.output.model)
+                    .only("id", "task", "ready")
+                    .first()
+                )
+                if output_model and not output_model.ready:
+                    cls.model_set_ready(
+                        model_id=task.output.model,
+                        company_id=company_id,
+                        publish_task=False,
+                    )
+
+            # set task status to published, and update (or set) it's new output (view and models)
+            return ChangeStatusRequest(
+                task=task,
+                new_status=TaskStatus.published,
+                force=force,
+                status_reason=status_reason,
+                status_message=status_message,
+            ).execute(published=datetime.utcnow(), output=output)
+
+        except Exception as ex:
+            publish_failed = True
+            raise ex
+        finally:
+            if publish_failed:
+                task.status = previous_task_status
+                task.save()
+
+    @classmethod
+    def stop_task(
+        cls,
+        task_id: str,
+        company_id: str,
+        user_name: str,
+        status_reason: str,
+        force: bool,
+    ) -> dict:
+        """
+        Stop a running task. Requires task status 'in_progress' and
+        execution_progress 'running', or force=True. Development task or
+        task that has no associated worker is stopped immediately.
+        For a non-development task with worker only the status message
+        is set to 'stopping' to allow the worker to stop the task and report by itself
+        :return: updated task fields
+        """
+
+        task = cls.get_task_with_access(
+            task_id,
+            company_id=company_id,
+            only=(
+                "status",
+                "project",
+                "tags",
+                "system_tags",
+                "last_worker",
+                "last_update",
+            ),
+            requires_write_access=True,
+        )
+
+        def is_run_by_worker(t: Task) -> bool:
+            """Checks if there is an active worker running the task"""
+            update_timeout = config.get("apiserver.workers.task_update_timeout", 600)
+            return (
+                t.last_worker
+                and t.last_update
+                and (datetime.utcnow() - t.last_update).total_seconds() < update_timeout
+            )
+
+        if TaskSystemTags.development in task.system_tags or not is_run_by_worker(task):
+            new_status = TaskStatus.stopped
+            status_message = f"Stopped by {user_name}"
+        else:
+            new_status = task.status
+            status_message = TaskStatusMessage.stopping
+
+        return ChangeStatusRequest(
+            task=task,
+            new_status=new_status,
+            status_reason=status_reason,
+            status_message=status_message,
+            force=force,
+        ).execute()
+
+    @staticmethod
+    def get_aggregated_project_parameters(
+        company_id,
+        project_ids: Sequence[str] = None,
+        page: int = 0,
+        page_size: int = 500,
+    ) -> Tuple[int, int, Sequence[dict]]:
+
+        page = max(0, page)
+        page_size = max(1, page_size)
+        pipeline = [
+            {
+                "$match": {
+                    "company": {"$in": [None, "", company_id]},
+                    "hyperparams": {"$exists": True, "$gt": {}},
+                    **({"project": {"$in": project_ids}} if project_ids else {}),
+                }
+            },
+            {"$project": {"sections": {"$objectToArray": "$hyperparams"}}},
+            {"$unwind": "$sections"},
+            {
+                "$project": {
+                    "section": "$sections.k",
+                    "names": {"$objectToArray": "$sections.v"},
+                }
+            },
+            {"$unwind": "$names"},
+            {"$group": {"_id": {"section": "$section", "name": "$names.k"}}},
+            {"$sort": OrderedDict({"_id.section": 1, "_id.name": 1})},
+            {
+                "$group": {
+                    "_id": 1,
+                    "total": {"$sum": 1},
+                    "results": {"$push": "$$ROOT"},
+                }
+            },
+            {
+                "$project": {
+                    "total": 1,
+                    "results": {"$slice": ["$results", page * page_size, page_size]},
+                }
+            },
+        ]
+
+        with translate_errors_context():
+            result = next(Task.aggregate(pipeline), None)
+
+        total = 0
+        remaining = 0
+        results = []
+
+        if result:
+            total = int(result.get("total", -1))
+            results = [
+                {
+                    "section": ParameterKeyEscaper.unescape(
+                        dpath.get(r, "_id/section")
+                    ),
+                    "name": ParameterKeyEscaper.unescape(dpath.get(r, "_id/name")),
+                }
+                for r in result.get("results", [])
+            ]
+            remaining = max(0, total - (len(results) + page * page_size))
+
+        return total, remaining, results
+
+    @classmethod
+    def dequeue_and_change_status(
+        cls, task: Task, company_id: str, status_message: str, status_reason: str,
+    ):
+        cls.dequeue(task, company_id)
+
+        return ChangeStatusRequest(
+            task=task,
+            new_status=TaskStatus.created,
+            status_reason=status_reason,
+            status_message=status_message,
+        ).execute(unset__execution__queue=1)
+
+    @classmethod
+    def dequeue(cls, task: Task, company_id: str, silent_fail=False):
+        """
+        Dequeue the task from the queue
+        :param task: task to dequeue
+        :param company_id: task's company ID.
+        :param silent_fail: do not throw exceptions. APIError is still thrown
+        :raise errors.bad_request.InvalidTaskId: if the task's status is not queued
+        :raise errors.bad_request.MissingRequiredFields: if the task is not queued
+        :raise APIError or errors.server_error.TransactionError: if internal call to queues.remove_task fails
+        :return: the result of queues.remove_task call. None in case of silent failure
+        """
+        if task.status not in (TaskStatus.queued,):
+            if silent_fail:
+                return
+            raise errors.bad_request.InvalidTaskId(
+                status=task.status, expected=TaskStatus.queued
+            )
+
+        if not task.execution or not task.execution.queue:
+            if silent_fail:
+                return
+            raise errors.bad_request.MissingRequiredFields(
+                "task has no queue value", field="execution.queue"
+            )
+
+        return {
+            "removed": queue_bll.remove_task(
+                company_id=company_id, queue_id=task.execution.queue, task_id=task.id
+            )
+        }

apiserver/bll/task/utils.py

@@ -1,16 +1,16 @@
 from datetime import datetime
-from typing import TypeVar, Callable, Tuple, Sequence
+from typing import TypeVar, Callable, Tuple, Sequence, Union
 
 import attr
 import six
 
-from apierrors import errors
-from database.errors import translate_errors_context
-from database.model.project import Project
-from database.model.task.task import Task, TaskStatus
-from database.utils import get_options
-from timing_context import TimingContext
-from utilities.attrs import typed_attrs
+from apiserver.apierrors import errors
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.project import Project
+from apiserver.database.model.task.task import Task, TaskStatus, TaskSystemTags
+from apiserver.database.utils import get_options
+from apiserver.timing_context import TimingContext
+from apiserver.utilities.attrs import typed_attrs
 
 valid_statuses = get_options(TaskStatus)
 
@@ -25,9 +25,10 @@ class ChangeStatusRequest(object):
     status_message = attr.ib(type=six.string_types, default="")
     force = attr.ib(type=bool, default=False)
     allow_same_state_transition = attr.ib(type=bool, default=True)
+    current_status_override = attr.ib(default=None)
 
     def execute(self, **kwargs):
-        current_status = self.task.status
+        current_status = self.current_status_override or self.task.status
         project_id = self.task.project
 
         # Verify new status is allowed from current status (will throw exception if not valid)
@@ -42,8 +43,12 @@ class ChangeStatusRequest(object):
             status_message=self.status_message,
             status_changed=now,
             last_update=now,
+            last_change=now,
         )
 
+        if self.new_status == TaskStatus.queued:
+            fields["pull__system_tags"] = TaskSystemTags.development
+
         def safe_mongoengine_key(key):
             return f"__{key}" if key in control else key
 
@@ -66,6 +71,10 @@ class ChangeStatusRequest(object):
             )
 
         update_project_time(project_id)
+
+        # make sure that _raw_ queries are not returned back to the client
+        fields.pop("__raw__", None)
+
         return dict(updated=updated, fields=fields)
 
     def validate_transition(self, current_status):
@@ -95,8 +104,14 @@ def validate_status_change(current_status, new_status):
 
 
 state_machine = {
-    TaskStatus.created: {TaskStatus.in_progress},
-    TaskStatus.in_progress: {TaskStatus.stopped, TaskStatus.failed, TaskStatus.created},
+    TaskStatus.created: {TaskStatus.queued, TaskStatus.in_progress},
+    TaskStatus.queued: {TaskStatus.created, TaskStatus.in_progress},
+    TaskStatus.in_progress: {
+        TaskStatus.stopped,
+        TaskStatus.failed,
+        TaskStatus.created,
+        TaskStatus.completed,
+    },
     TaskStatus.stopped: {
         TaskStatus.closed,
         TaskStatus.created,
@@ -104,6 +119,7 @@ state_machine = {
         TaskStatus.in_progress,
         TaskStatus.published,
         TaskStatus.publishing,
+        TaskStatus.completed,
     },
     TaskStatus.closed: {
         TaskStatus.created,
@@ -115,6 +131,11 @@ state_machine = {
     TaskStatus.failed: {TaskStatus.created, TaskStatus.stopped, TaskStatus.published},
     TaskStatus.publishing: {TaskStatus.published},
     TaskStatus.published: set(),
+    TaskStatus.completed: {
+        TaskStatus.published,
+        TaskStatus.in_progress,
+        TaskStatus.created,
+    },
 }
 
 
@@ -124,15 +145,22 @@ def get_possible_status_changes(current_status):
     :return possible states from current state
     """
     possible = state_machine.get(current_status)
-    assert (
-        possible is not None
-    ), f"Current status {current_status} not supported by state machine"
+    if possible is None:
+        raise errors.server_error.InternalError(
+            f"Current status {current_status} not supported by state machine"
+        )
+
     return possible
 
 
-def update_project_time(project_id):
-    if project_id:
-        Project.objects(id=project_id).update(last_update=datetime.utcnow())
+def update_project_time(project_ids: Union[str, Sequence[str]]):
+    if not project_ids:
+        return
+
+    if isinstance(project_ids, str):
+        project_ids = [project_ids]
+
+    return Project.objects(id__in=project_ids).update(last_update=datetime.utcnow())
 
 
 T = TypeVar("T")
@@ -149,3 +177,34 @@ def split_by(
         [item for cond, item in applied if cond],
         [item for cond, item in applied if not cond],
     )
+
+
+def get_task_for_update(
+    company_id: str, task_id: str, allow_all_statuses: bool = False, force: bool = False
+) -> Task:
+    """
+    Loads only task id and return the task only if it is updatable (status == 'created')
+    """
+    task = Task.get_for_writing(company=company_id, id=task_id, _only=("id", "status"))
+    if not task:
+        raise errors.bad_request.InvalidTaskId(id=task_id)
+
+    if allow_all_statuses:
+        return task
+
+    allowed_statuses = (
+        [TaskStatus.created, TaskStatus.in_progress] if force else [TaskStatus.created]
+    )
+    if task.status not in allowed_statuses:
+        raise errors.bad_request.InvalidTaskStatus(
+            expected=TaskStatus.created, status=task.status
+        )
+    return task
+
+
+def update_task(task: Task, update_cmds: dict, set_last_update: bool = True):
+    now = datetime.utcnow()
+    last_updates = dict(last_change=now)
+    if set_last_update:
+        last_updates.update(last_update=now)
+    return task.update(**update_cmds, **last_updates)

apiserver/bll/user/__init__.py

@@ -1,7 +1,7 @@
-from apierrors import errors
-from apimodels.users import CreateRequest
-from database.errors import translate_errors_context
-from database.model.user import User
+from apiserver.apierrors import errors
+from apiserver.apimodels.users import CreateRequest
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.user import User
 
 
 class UserBLL:

apiserver/bll/util.py

@@ -0,0 +1,117 @@
+import functools
+import itertools
+from concurrent.futures.thread import ThreadPoolExecutor
+from operator import itemgetter
+from typing import Sequence, Optional, Callable, Tuple, Dict, Any, Set, Iterable
+
+from boltons import iterutils
+
+from apiserver.database.model import AttributedDocument
+from apiserver.database.model.settings import Settings
+
+
+def extract_properties_to_lists(
+    key_names: Sequence[str],
+    data: Sequence[dict],
+    extract_func: Optional[Callable[[dict], Tuple]] = None,
+) -> dict:
+    """
+    Given a list of dictionaries and names of dictionary keys
+    builds a dictionary with the requested keys and values lists
+    :param key_names: names of the keys in the resulting dictionary
+    :param data: sequence of dictionaries to extract values from
+    :param extract_func: the optional callable that extracts properties
+    from a dictionary and put them in a tuple in the order corresponding to
+    key_names. If not specified then properties are extracted according to key_names
+    """
+    value_sequences = zip(*map(extract_func or itemgetter(*key_names), data))
+    return dict(zip(key_names, map(list, value_sequences)))
+
+
+class SetFieldsResolver:
+    """
+    The class receives set fields dictionary
+    and for the set fields that require 'min' or 'max'
+    operation replace them with a simple set in case the
+    DB document does not have these fields set
+    """
+
+    SET_MODIFIERS = ("min", "max")
+
+    def __init__(self, set_fields: Dict[str, Any]):
+        self.orig_fields = {}
+        self.fields = {}
+        self.add_fields(**set_fields)
+
+    def add_fields(self, **set_fields: Any):
+        self.orig_fields.update(set_fields)
+        self.fields.update(
+            {
+                f: fname
+                for f, modifier, dunder, fname in (
+                    (f,) + f.partition("__") for f in set_fields.keys()
+                )
+                if dunder and modifier in self.SET_MODIFIERS
+            }
+        )
+
+    def _get_updated_name(self, doc: AttributedDocument, name: str) -> str:
+        if name in self.fields and doc.get_field_value(self.fields[name]) is None:
+            return self.fields[name]
+        return name
+
+    def get_fields(self, doc: AttributedDocument):
+        """
+        For the given document return the set fields instructions
+        with min/max operations replaced with a single set in case
+        the document does not have the field set
+        """
+        return {
+            self._get_updated_name(doc, name): value
+            for name, value in self.orig_fields.items()
+        }
+
+    def get_names(self) -> Set[str]:
+        """
+        Returns the names of the fields that had min/max modifiers
+        in the format suitable for projection (dot separated)
+        """
+        return set(name.replace("__", ".") for name in self.fields.values())
+
+
+@functools.lru_cache()
+def get_server_uuid() -> Optional[str]:
+    return Settings.get_by_key("server.uuid")
+
+
+def parallel_chunked_decorator(func: Callable = None, chunk_size: int = 100):
+    """
+    Decorates a method for parallel chunked execution. The method should have
+    one positional parameter (that is used for breaking into chunks)
+    and arbitrary number of keyword params. The return value should be iterable
+    The results are concatenated in the same order as the passed params
+    """
+    if func is None:
+        return functools.partial(parallel_chunked_decorator, chunk_size=chunk_size)
+
+    @functools.wraps(func)
+    def wrapper(self, iterable: Iterable, **kwargs):
+        assert iterutils.is_collection(
+            iterable
+        ), "The positional parameter should be an iterable for breaking into chunks"
+
+        func_with_params = functools.partial(func, self, **kwargs)
+        with ThreadPoolExecutor() as pool:
+            return list(
+                itertools.chain.from_iterable(
+                    filter(
+                        None,
+                        pool.map(
+                            func_with_params,
+                            iterutils.chunked_iter(iterable, chunk_size),
+                        ),
+                    )
+                ),
+            )
+
+    return wrapper

apiserver/bll/workers/__init__.py

@@ -0,0 +1,439 @@
+import itertools
+from datetime import datetime, timedelta
+from typing import Sequence, Set, Optional
+
+import attr
+import elasticsearch.helpers
+
+from apiserver.es_factory import es_factory
+from apiserver.apierrors import APIError
+from apiserver.apierrors.errors import bad_request, server_error
+from apiserver.apimodels.workers import (
+    DEFAULT_TIMEOUT,
+    IdNameEntry,
+    WorkerEntry,
+    StatusReportRequest,
+    WorkerResponseEntry,
+    QueueEntry,
+    MachineStats,
+)
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.auth import User
+from apiserver.database.model.company import Company
+from apiserver.database.model.project import Project
+from apiserver.database.model.queue import Queue
+from apiserver.database.model.task.task import Task
+from apiserver.redis_manager import redman
+from apiserver.timing_context import TimingContext
+from apiserver.tools import safe_get
+from .stats import WorkerStats
+
+log = config.logger(__file__)
+
+
+class WorkerBLL:
+    def __init__(self, es=None, redis=None):
+        self.es_client = es or es_factory.connect("workers")
+        self.redis = redis or redman.connection("workers")
+        self._stats = WorkerStats(self.es_client)
+
+    @property
+    def stats(self) -> WorkerStats:
+        return self._stats
+
+    def register_worker(
+        self,
+        company_id: str,
+        user_id: str,
+        worker: str,
+        ip: str = "",
+        queues: Sequence[str] = None,
+        timeout: int = 0,
+        tags: Sequence[str] = None,
+    ) -> WorkerEntry:
+        """
+        Register a worker
+        :param company_id: worker's company ID
+        :param user_id: user ID under which this worker is running
+        :param worker: worker ID
+        :param ip: the real ip of the worker
+        :param queues: queues reported as being monitored by the worker
+        :param timeout: registration expiration timeout in seconds
+        :param tags: a list of tags for this worker
+        :raise bad_request.InvalidUserId: in case the calling user or company does not exist
+        :return: worker entry instance
+        """
+        key = WorkerBLL._get_worker_key(company_id, user_id, worker)
+
+        timeout = timeout or DEFAULT_TIMEOUT
+        queues = queues or []
+
+        with translate_errors_context():
+            query = dict(id=user_id, company=company_id)
+            user = User.objects(**query).only("id", "name").first()
+            if not user:
+                raise bad_request.InvalidUserId(**query)
+            company = Company.objects(id=company_id).only("id", "name").first()
+            if not company:
+                raise server_error.InternalError("invalid company", company=company_id)
+
+            queue_objs = Queue.objects(company=company_id, id__in=queues).only("id")
+            if len(queue_objs) < len(queues):
+                invalid = set(queues).difference(q.id for q in queue_objs)
+                raise bad_request.InvalidQueueId(ids=invalid)
+
+            now = datetime.utcnow()
+            entry = WorkerEntry(
+                key=key,
+                id=worker,
+                user=user.to_proper_dict(),
+                company=company.to_proper_dict(),
+                ip=ip,
+                queues=queues,
+                register_time=now,
+                register_timeout=timeout,
+                last_activity_time=now,
+                tags=tags,
+            )
+
+            self.redis.setex(key, timedelta(seconds=timeout), entry.to_json())
+
+        return entry
+
+    def unregister_worker(self, company_id: str, user_id: str, worker: str) -> None:
+        """
+        Unregister a worker
+        :param company_id: worker's company ID
+        :param user_id: user ID under which this worker is running
+        :param worker: worker ID
+        :raise bad_request.WorkerNotRegistered: the worker was not previously registered
+        """
+        with TimingContext("redis", "workers_unregister"):
+            res = self.redis.delete(
+                company_id, self._get_worker_key(company_id, user_id, worker)
+            )
+        if not res:
+            raise bad_request.WorkerNotRegistered(worker=worker)
+
+    def status_report(
+        self, company_id: str, user_id: str, ip: str, report: StatusReportRequest, tags: Sequence[str] = None,
+    ) -> None:
+        """
+        Write worker status report
+        :param company_id: worker's company ID
+        :param user_id: user_id ID under which this worker is running
+        :param ip: worker IP
+        :param report: the report itself
+        :param tags: tags for this worker
+        :raise bad_request.InvalidTaskId: the reported task was not found
+        :return: worker entry instance
+        """
+        entry = self._get_worker(company_id, user_id, report.worker)
+
+        try:
+            entry.ip = ip
+            now = datetime.utcnow()
+            entry.last_activity_time = now
+
+            if tags is not None:
+                entry.tags = tags
+
+            if report.machine_stats:
+                self._log_stats_to_es(
+                    company_id=company_id,
+                    company_name=entry.company.name,
+                    worker=report.worker,
+                    timestamp=report.timestamp,
+                    task=report.task,
+                    machine_stats=report.machine_stats,
+                )
+
+            entry.queue = report.queue
+
+            if report.queues:
+                entry.queues = report.queues
+
+            if not report.task:
+                entry.task = None
+                entry.project = None
+            else:
+                with translate_errors_context():
+                    query = dict(id=report.task, company=company_id)
+                    update = dict(
+                        last_worker=report.worker,
+                        last_worker_report=now,
+                        last_update=now,
+                        last_change=now,
+                    )
+                    # modify(new=True, ...) returns the modified object
+                    task = Task.objects(**query).modify(new=True, **update)
+                    if not task:
+                        raise bad_request.InvalidTaskId(**query)
+                    entry.task = IdNameEntry(id=task.id, name=task.name)
+
+                    entry.project = None
+                    if task.project:
+                        project = Project.objects(id=task.project).only("name").first()
+                        if project:
+                            entry.project = IdNameEntry(id=project.id, name=project.name)
+
+            entry.last_report_time = now
+        except APIError:
+            raise
+        except Exception as e:
+            msg = "Failed processing worker status report"
+            log.exception(msg)
+            raise server_error.DataError(msg, err=e.args[0])
+        finally:
+            self._save_worker(entry)
+
+    def get_all(
+        self, company_id: str, last_seen: Optional[int] = None
+    ) -> Sequence[WorkerEntry]:
+        """
+        Get all the company workers that were active during the last_seen period
+        :param company_id: worker's company id
+        :param last_seen: period in seconds to check. Min value is 1 second
+        :return:
+        """
+        try:
+            workers = self._get(company_id)
+        except Exception as e:
+            raise server_error.DataError("failed loading worker entries", err=e.args[0])
+
+        if last_seen:
+            ref_time = datetime.utcnow() - timedelta(seconds=max(1, last_seen))
+            workers = [
+                w
+                for w in workers
+                if w.last_activity_time.replace(tzinfo=None) >= ref_time
+            ]
+
+        return workers
+
+    def get_all_with_projection(
+        self, company_id: str, last_seen: int
+    ) -> Sequence[WorkerResponseEntry]:
+
+        helpers = list(
+            map(
+                WorkerConversionHelper.from_worker_entry,
+                self.get_all(company_id=company_id, last_seen=last_seen),
+            )
+        )
+
+        task_ids = set(filter(None, (helper.task_id for helper in helpers)))
+        all_queues = set(
+            itertools.chain.from_iterable(helper.queue_ids for helper in helpers)
+        )
+
+        queues_info = {}
+        if all_queues:
+            projection = [
+                {"$match": {"_id": {"$in": list(all_queues)}}},
+                {
+                    "$project": {
+                        "name": 1,
+                        "next_entry": {"$arrayElemAt": ["$entries", 0]},
+                        "num_entries": {"$size": "$entries"},
+                    }
+                },
+            ]
+            queues_info = {
+                res["_id"]: res for res in Queue.objects.aggregate(projection)
+            }
+            task_ids = task_ids.union(
+                filter(
+                    None,
+                    (
+                        safe_get(info, "next_entry/task")
+                        for info in queues_info.values()
+                    ),
+                )
+            )
+
+        tasks_info = {}
+        if task_ids:
+            tasks_info = {
+                task.id: task
+                for task in Task.objects(id__in=task_ids).only(
+                    "name", "started", "last_iteration"
+                )
+            }
+
+        def update_queue_entries(*entries):
+            for entry in entries:
+                if not entry:
+                    continue
+                info = queues_info.get(entry.id, None)
+                if not info:
+                    continue
+                entry.name = info.get("name", None)
+                entry.num_tasks = info.get("num_entries", 0)
+                task_id = safe_get(info, "next_entry/task")
+                if task_id:
+                    task = tasks_info.get(task_id, None)
+                    entry.next_task = IdNameEntry(
+                        id=task_id, name=task.name if task else None
+                    )
+
+        for helper in helpers:
+            worker = helper.worker
+            if helper.task_id:
+                task = tasks_info.get(helper.task_id, None)
+                if task:
+                    worker.task.running_time = (
+                        int((datetime.utcnow() - task.started).total_seconds() * 1000)
+                        if task.started
+                        else 0
+                    )
+                    worker.task.last_iteration = task.last_iteration
+
+            update_queue_entries(worker.queue)
+            if worker.queues:
+                update_queue_entries(*worker.queues)
+
+        return [helper.worker for helper in helpers]
+
+    @staticmethod
+    def _get_worker_key(company: str, user: str, worker_id: str) -> str:
+        """Build redis key from company, user and worker_id"""
+        return f"worker_{company}_{user}_{worker_id}"
+
+    def _get_worker(self, company_id: str, user_id: str, worker: str) -> WorkerEntry:
+        """
+        Get a worker entry for the provided worker ID. The entry is loaded from Redis
+        if it exists (i.e. worker has already been registered), otherwise the worker
+        is registered and its entry stored into Redis).
+        :param company_id: worker's company ID
+        :param user_id: user ID under which this worker is running
+        :param worker: worker ID
+        :raise bad_request.InvalidWorkerId: in case the worker id was not found
+        :return: worker entry instance
+        """
+        key = self._get_worker_key(company_id, user_id, worker)
+
+        with TimingContext("redis", "get_worker"):
+            data = self.redis.get(key)
+
+        if data:
+            try:
+                entry = WorkerEntry.from_json(data)
+                if not entry.key:
+                    entry.key = key
+                    self._save_worker(entry)
+                return entry
+            except Exception as e:
+                msg = "Failed parsing worker entry"
+                log.exception(msg)
+                raise server_error.DataError(msg, err=e.args[0])
+
+        # Failed loading worker from Redis
+        if config.get("apiserver.workers.auto_register", False):
+            try:
+                return self.register_worker(company_id, user_id, worker)
+            except Exception:
+                log.error(
+                    "Failed auto registration of {} for company {}".format(
+                        worker, company_id
+                    )
+                )
+
+        raise bad_request.InvalidWorkerId(worker=worker)
+
+    def _save_worker(self, entry: WorkerEntry) -> None:
+        """Save worker entry in Redis"""
+        try:
+            self.redis.setex(
+                entry.key, timedelta(seconds=entry.register_timeout), entry.to_json()
+            )
+        except Exception:
+            msg = "Failed saving worker entry"
+            log.exception(msg)
+
+    def _get(
+        self, company: str, user: str = "*", worker_id: str = "*"
+    ) -> Sequence[WorkerEntry]:
+        """Get worker entries matching the company and user, worker patterns"""
+        match = self._get_worker_key(company, user, worker_id)
+        with TimingContext("redis", "workers_get_all"):
+            res = self.redis.scan_iter(match)
+        return [WorkerEntry.from_json(self.redis.get(r)) for r in res]
+
+    @staticmethod
+    def _get_es_index_suffix():
+        """Get the index name suffix for storing current month data"""
+        return datetime.utcnow().strftime("%Y-%m")
+
+    def _log_stats_to_es(
+        self,
+        company_id: str,
+        company_name: str,
+        worker: str,
+        timestamp: int,
+        task: str,
+        machine_stats: MachineStats,
+    ) -> bool:
+        """
+        Actually writing the worker statistics to Elastic
+        :return: True if successful, False otherwise
+        """
+        es_index = (
+            f"{self._stats.worker_stats_prefix_for_company(company_id)}"
+            f"{self._get_es_index_suffix()}"
+        )
+
+        def make_doc(category, metric, variant, value) -> dict:
+            return dict(
+                _index=es_index,
+                _source=dict(
+                    timestamp=timestamp,
+                    worker=worker,
+                    company=company_name,
+                    task=task,
+                    category=category,
+                    metric=metric,
+                    variant=variant,
+                    value=float(value),
+                ),
+            )
+
+        actions = []
+        for field, value in machine_stats.to_struct().items():
+            if not value:
+                continue
+            category = field.partition("_")[0]
+            metric = field
+            if not isinstance(value, (list, tuple)):
+                actions.append(make_doc(category, metric, "total", value))
+            else:
+                actions.extend(
+                    make_doc(category, metric, str(i), val)
+                    for i, val in enumerate(value)
+                )
+
+        es_res = elasticsearch.helpers.bulk(self.es_client, actions)
+        added, errors = es_res[:2]
+        return (added == len(actions)) and not errors
+
+
+@attr.s(auto_attribs=True)
+class WorkerConversionHelper:
+    worker: WorkerResponseEntry
+    task_id: str
+    queue_ids: Set[str]
+
+    @classmethod
+    def from_worker_entry(cls, worker: WorkerEntry):
+        data = worker.to_struct()
+        queue = data.pop("queue", None) or None
+        queue_ids = set(data.pop("queues", []))
+        queues = [QueueEntry(id=id) for id in queue_ids]
+        if queue:
+            queue = next((q for q in queues if q.id == queue), None)
+        return cls(
+            worker=WorkerResponseEntry(queues=queues, queue=queue, **data),
+            task_id=worker.task.id if worker.task else None,
+            queue_ids=queue_ids,
+        )

apiserver/bll/workers/stats.py

@@ -0,0 +1,243 @@
+from operator import attrgetter
+from typing import Optional, Sequence
+
+from boltons.iterutils import bucketize
+
+from apiserver.apierrors.errors import bad_request
+from apiserver.apimodels.workers import AggregationType, GetStatsRequest, StatItem
+from apiserver.bll.query import Builder as QueryBuilder
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.timing_context import TimingContext
+
+log = config.logger(__file__)
+
+
+class WorkerStats:
+    def __init__(self, es):
+        self.es = es
+
+    @staticmethod
+    def worker_stats_prefix_for_company(company_id: str) -> str:
+        """Returns the es index prefix for the company"""
+        return f"worker_stats_{company_id}_"
+
+    def _search_company_stats(self, company_id: str, es_req: dict) -> dict:
+        return self.es.search(
+            index=f"{self.worker_stats_prefix_for_company(company_id)}*",
+            body=es_req,
+        )
+
+    def get_worker_stats_keys(
+        self, company_id: str, worker_ids: Optional[Sequence[str]]
+    ) -> dict:
+        """
+        Get dictionary of metric types grouped by categories
+        :param company_id: company id
+        :param worker_ids: optional list of workers to get metric types from.
+        If not specified them metrics for all the company workers returned
+        :return:
+        """
+        es_req = {
+            "size": 0,
+            "aggs": {
+                "categories": {
+                    "terms": {"field": "category"},
+                    "aggs": {"metrics": {"terms": {"field": "metric"}}},
+                }
+            },
+        }
+        if worker_ids:
+            es_req["query"] = QueryBuilder.terms("worker", worker_ids)
+
+        res = self._search_company_stats(company_id, es_req)
+
+        if not res["hits"]["total"]["value"]:
+            raise bad_request.WorkerStatsNotFound(
+                f"No statistic metrics found for the company {company_id} and workers {worker_ids}"
+            )
+
+        return {
+            category["key"]: [
+                metric["key"] for metric in category["metrics"]["buckets"]
+            ]
+            for category in res["aggregations"]["categories"]["buckets"]
+        }
+
+    def get_worker_stats(self, company_id: str, request: GetStatsRequest) -> dict:
+        """
+        Get statistics for company workers metrics in the specified time range
+        Returned as date histograms for different aggregation types
+        grouped by worker, metric type (and optionally metric variant)
+        Buckets with no metrics are not returned
+        Note: all the statistics are retrieved as one ES query
+        """
+        if request.from_date >= request.to_date:
+            raise bad_request.FieldsValueError("from_date must be less than to_date")
+
+        def get_dates_agg() -> dict:
+            es_to_agg_types = (
+                ("avg", AggregationType.avg.value),
+                ("min", AggregationType.min.value),
+                ("max", AggregationType.max.value),
+            )
+
+            return {
+                "dates": {
+                    "date_histogram": {
+                        "field": "timestamp",
+                        "fixed_interval": f"{request.interval}s",
+                        "min_doc_count": 1,
+                    },
+                    "aggs": {
+                        agg_type: {es_agg: {"field": "value"}}
+                        for es_agg, agg_type in es_to_agg_types
+                    },
+                }
+            }
+
+        def get_variants_agg() -> dict:
+            return {
+                "variants": {"terms": {"field": "variant"}, "aggs": get_dates_agg()}
+            }
+
+        es_req = {
+            "size": 0,
+            "aggs": {
+                "workers": {
+                    "terms": {"field": "worker"},
+                    "aggs": {
+                        "metrics": {
+                            "terms": {"field": "metric"},
+                            "aggs": get_variants_agg()
+                            if request.split_by_variant
+                            else get_dates_agg(),
+                        }
+                    },
+                }
+            },
+        }
+
+        query_terms = [
+            QueryBuilder.dates_range(request.from_date, request.to_date),
+            QueryBuilder.terms("metric", {item.key for item in request.items}),
+        ]
+        if request.worker_ids:
+            query_terms.append(QueryBuilder.terms("worker", request.worker_ids))
+        es_req["query"] = {"bool": {"must": query_terms}}
+
+        with translate_errors_context(), TimingContext("es", "get_worker_stats"):
+            data = self._search_company_stats(company_id, es_req)
+
+        return self._extract_results(data, request.items, request.split_by_variant)
+
+    @staticmethod
+    def _extract_results(
+        data: dict, request_items: Sequence[StatItem], split_by_variant: bool
+    ) -> dict:
+        """
+        Clean results returned from elastic search (remove "aggregations", "buckets" etc.),
+        leave only aggregation types requested by the user and return a clean dictionary
+        and return a "clean" dictionary of
+        :param data: aggregation data retrieved from ES
+        :param request_items: aggs types requested by the user
+        :param split_by_variant: if False then aggregate by metric type, otherwise metric type + variant
+        """
+        if "aggregations" not in data:
+            return {}
+
+        items_by_key = bucketize(request_items, key=attrgetter("key"))
+        aggs_per_metric = {
+            key: [item.aggregation for item in items]
+            for key, items in items_by_key.items()
+        }
+
+        def extract_date_stats(date: dict, metric_key) -> dict:
+            return {
+                "date": date["key"],
+                "count": date["doc_count"],
+                **{agg: date[agg]["value"] for agg in aggs_per_metric[metric_key]},
+            }
+
+        def extract_metric_results(
+            metric_or_variant: dict, metric_key: str
+        ) -> Sequence[dict]:
+            return [
+                extract_date_stats(date, metric_key)
+                for date in metric_or_variant["dates"]["buckets"]
+                if date["doc_count"]
+            ]
+
+        def extract_variant_results(metric: dict) -> dict:
+            metric_key = metric["key"]
+            return {
+                variant["key"]: extract_metric_results(variant, metric_key)
+                for variant in metric["variants"]["buckets"]
+            }
+
+        def extract_worker_results(worker: dict) -> dict:
+            return {
+                metric["key"]: extract_variant_results(metric)
+                if split_by_variant
+                else extract_metric_results(metric, metric["key"])
+                for metric in worker["metrics"]["buckets"]
+            }
+
+        return {
+            worker["key"]: extract_worker_results(worker)
+            for worker in data["aggregations"]["workers"]["buckets"]
+        }
+
+    def get_activity_report(
+        self,
+        company_id: str,
+        from_date: float,
+        to_date: float,
+        interval: int,
+        active_only: bool,
+    ) -> Sequence[dict]:
+        """
+        Get statistics for company workers metrics in the specified time range
+        Returned as date histograms for different aggregation types
+        grouped by worker, metric type (and optionally metric variant)
+        Note: all the statistics are retrieved using one ES query
+        """
+        if from_date >= to_date:
+            raise bad_request.FieldsValueError("from_date must be less than to_date")
+
+        must = [QueryBuilder.dates_range(from_date, to_date)]
+        if active_only:
+            must.append({"exists": {"field": "task"}})
+
+        es_req = {
+            "size": 0,
+            "aggs": {
+                "dates": {
+                    "date_histogram": {
+                        "field": "timestamp",
+                        "fixed_interval": f"{interval}s",
+                    },
+                    "aggs": {"workers_count": {"cardinality": {"field": "worker"}}},
+                }
+            },
+            "query": {"bool": {"must": must}},
+        }
+
+        with translate_errors_context(), TimingContext(
+            "es", "get_worker_activity_report"
+        ):
+            data = self._search_company_stats(company_id, es_req)
+
+        if "aggregations" not in data:
+            return {}
+
+        ret = [
+            dict(date=date["key"], count=date["workers_count"]["value"])
+            for date in data["aggregations"]["dates"]["buckets"]
+        ]
+
+        if ret and ret[-1]["date"] > (to_date - 0.9 * interval):
+            # remove last interval if it's incomplete. Allow 10% tolerance
+            ret.pop()
+
+        return ret

apiserver/config/__init__.py

@@ -0,0 +1 @@
+from .basic import BasicConfig, ConfigurationError

apiserver/config/basic.py

@@ -0,0 +1,176 @@
+import logging
+import logging.config
+import os
+import platform
+from functools import reduce
+from os import getenv
+from os.path import expandvars
+from pathlib import Path
+from typing import List, Any, TypeVar
+
+from pyhocon import ConfigTree, ConfigFactory
+from pyparsing import (
+    ParseFatalException,
+    ParseException,
+    RecursiveGrammarException,
+    ParseSyntaxException,
+)
+
+from apiserver.utilities import json
+
+EXTRA_CONFIG_PATHS = ("/opt/trains/config",)
+EXTRA_CONFIG_PATH_OVERRIDE_VAR = "TRAINS_CONFIG_DIR"
+EXTRA_CONFIG_PATH_SEP = ":" if platform.system() != "Windows" else ";"
+
+
+class BasicConfig:
+    NotSet = object()
+
+    extra_config_values_env_key_sep = "__"
+    default_config_dir = "default"
+
+    def __init__(
+        self, folder: str = None, verbose: bool = True, prefix: str = "trains"
+    ):
+        folder = (
+            Path(folder)
+            if folder
+            else Path(__file__).with_name(self.default_config_dir)
+        )
+        if not folder.is_dir():
+            raise ValueError("Invalid configuration folder")
+
+        self.verbose = verbose
+        self.prefix = prefix
+        self.extra_config_values_env_key_prefix = f"{self.prefix.upper()}__"
+
+        self._paths = [folder, *self._get_paths()]
+        self._config = self._reload()
+
+    def __getitem__(self, key):
+        return self._config[key]
+
+    def get(self, key: str, default: Any = NotSet) -> Any:
+        value = self._config.get(key, default)
+        if value is self.NotSet:
+            raise KeyError(
+                f"Unable to find value for key '{key}' and default value was not provided."
+            )
+        return value
+
+    def to_dict(self) -> dict:
+        return self._config.as_plain_ordered_dict()
+
+    def as_json(self) -> str:
+        return json.dumps(self.to_dict(), indent=2)
+
+    def logger(self, name: str) -> logging.Logger:
+        if Path(name).is_file():
+            name = Path(name).stem
+        path = ".".join((self.prefix, name))
+        return logging.getLogger(path)
+
+    def _read_extra_env_config_values(self) -> ConfigTree:
+        """ Loads extra configuration from environment-injected values """
+        result = ConfigTree()
+        prefix = self.extra_config_values_env_key_prefix
+
+        keys = sorted(k for k in os.environ if k.startswith(prefix))
+        for key in keys:
+            path = (
+                key[len(prefix) :]
+                .replace(self.extra_config_values_env_key_sep, ".")
+                .lower()
+            )
+            result = ConfigTree.merge_configs(
+                result, ConfigFactory.parse_string(f"{path}: {os.environ[key]}")
+            )
+
+        return result
+
+    def _get_paths(self) -> List[Path]:
+        default_paths = EXTRA_CONFIG_PATH_SEP.join(EXTRA_CONFIG_PATHS)
+        value = getenv(EXTRA_CONFIG_PATH_OVERRIDE_VAR, default_paths)
+
+        paths = [
+            Path(expandvars(v)).expanduser() for v in value.split(EXTRA_CONFIG_PATH_SEP)
+        ]
+
+        if value is not default_paths:
+            invalid = [path for path in paths if not path.is_dir()]
+            if invalid:
+                print(
+                    f"WARNING: Invalid paths in {EXTRA_CONFIG_PATH_OVERRIDE_VAR} env var: {' '.join(map(str, invalid))}"
+                )
+
+        return [path for path in paths if path.is_dir()]
+
+    def reload(self):
+        self._config = self._reload()
+
+    def _reload(self) -> ConfigTree:
+        extra_config_values = self._read_extra_env_config_values()
+
+        configs = [self._read_recursive(path) for path in self._paths]
+
+        return reduce(
+            lambda last, config: ConfigTree.merge_configs(
+                last, config, copy_trees=True
+            ),
+            configs + [extra_config_values],
+            ConfigTree(),
+        )
+
+    def _read_recursive(self, conf_root) -> ConfigTree:
+        conf = ConfigTree()
+
+        if not conf_root:
+            return conf
+
+        if not conf_root.is_dir():
+            if self.verbose:
+                if not conf_root.exists():
+                    print(f"No config in {conf_root}")
+                else:
+                    print(f"Not a directory: {conf_root}")
+            return conf
+
+        if self.verbose:
+            print(f"Loading config from {conf_root}")
+
+        for file in conf_root.rglob("*.conf"):
+            key = ".".join(file.relative_to(conf_root).with_suffix("").parts)
+            conf.put(key, self._read_single_file(file))
+
+        return conf
+
+    def _read_single_file(self, file_path):
+        if self.verbose:
+            print(f"Loading config from file {file_path}")
+
+        try:
+            return ConfigFactory.parse_file(file_path)
+        except ParseSyntaxException as ex:
+            msg = f"Failed parsing {file_path} ({ex.__class__.__name__}): (at char {ex.loc}, line:{ex.lineno}, col:{ex.column})"
+            raise ConfigurationError(msg, file_path=file_path) from ex
+        except (ParseException, ParseFatalException, RecursiveGrammarException) as ex:
+            msg = f"Failed parsing {file_path} ({ex.__class__.__name__}): {ex}"
+            raise ConfigurationError(msg) from ex
+        except Exception as ex:
+            print(f"Failed loading {file_path}: {ex}")
+            raise
+
+    def initialize_logging(self):
+        logging_config = self.get("logging", None)
+        if not logging_config:
+            return
+        logging.config.dictConfig(logging_config)
+
+
+class ConfigurationError(Exception):
+    def __init__(self, msg, file_path=None, *args):
+        super().__init__(msg, *args)
+        self.file_path = file_path
+
+
+ConfigType = TypeVar("ConfigType", bound=BasicConfig)

apiserver/config/default/apiserver.conf

@@ -0,0 +1,140 @@
+{
+    watch: false            # Watch for changes (dev only)
+    debug: false            # Debug mode
+    pretty_json: false      # prettify json response
+    return_stack: true      # return stack trace on error
+    log_calls: true         # Log API Calls
+
+    # if 'return_stack' is true and error contains a status code, return stack trace only for these status codes
+    # valid values are:
+    #  - an integer number, specifying a status code
+    #  - a tuple of (code, subcode or list of subcodes)
+    return_stack_on_code: [
+        [500, 0]  # raise on internal server error with no subcode
+    ]
+
+    listen {
+        ip : "0.0.0.0"
+        port: 8008
+    }
+
+    version {
+        required: false
+        default: 1.0
+        # if set then calls to endpoints with the version
+        # greater that the current max version will be rejected
+        check_max_version: false
+    }
+
+    pre_populate {
+        enabled: false
+        zip_files: ["/path/to/export.zip"]
+        fail_on_error: false
+        # artifacts_path: "/mnt/fileserver"
+    }
+
+    # time in seconds to take an exclusive lock to init es and mongodb
+    # not including the pre_populate
+    db_init_timout: 120
+
+    mongo {
+        # controls whether FieldDoesNotExist exception will be raised for any extra attribute existing in stored data
+        # but not declared in a data model
+        strict: false
+
+        aggregate {
+            allow_disk_use: true
+        }
+    }
+
+    elastic {
+        probing {
+            # settings for inital probing of elastic connection
+            max_retries: 4
+            timeout: 30
+        }
+        upgrade_monitoring {
+            v16_migration_verification: true
+        }
+    }
+
+    auth {
+        # verify user tokens
+        verify_user_tokens: false
+
+        # max token expiration timeout in seconds (1 year)
+        max_expiration_sec: 31536000
+
+        # default token expiration timeout in seconds (30 days)
+        default_expiration_sec: 2592000
+
+        # cookie containing auth token, for requests arriving from a web-browser
+        session_auth_cookie_name: "trains_token_basic"
+
+        # cookie configuration for authorization cookies generated by auth.login
+        cookies {
+            httponly: true  # allow only http to access the cookies (no JS etc)
+            secure: false   # not using HTTPS
+            domain: null    # Limit to localhost is not supported
+            max_age: 99999999999
+        }
+
+#        # A list of fixed users
+#        fixed_users {
+#            enabled: true
+#            users: [
+#                {
+#                    username: "john"
+#                    password: "123456"
+#                    name: "john doe"
+#                }
+#
+#            ]
+#        }
+    }
+
+    cors {
+        origins: "*"
+
+        # Not supported when origins is "*"
+        supports_credentials: true
+    }
+
+    default_company: "d1bd92a3b039400cbafc60a7a5b1e52b"
+
+    workers {
+        # Auto-register unknown workers on status reports and other calls
+        auto_register: true
+        # Timeout in seconds on task status update. If exceeded
+        # then task can be stopped without communicating to the worker
+        task_update_timeout: 600
+    }
+
+    check_for_updates {
+        enabled: true
+
+        # Check for updates every 24 hours
+        check_interval_sec: 86400
+
+        url: "https://updates.trains.allegro.ai/updates"
+
+        component_name: "trains-server"
+
+        # GET request timeout
+        request_timeout_sec: 3.0
+    }
+
+    statistics {
+        # Note: statistics are sent ONLY if the user has actively opted-in
+        supported: true
+
+        url: "https://updates.trains.allegro.ai/stats"
+
+        report_interval_hours: 24
+        agent_relevant_threshold_days: 30
+
+        max_retries: 5
+        max_backoff_sec: 5
+    }
+
+}

apiserver/config/default/hosts.conf

@@ -0,0 +1,45 @@
+elastic {
+  events {
+    hosts: [{host: "127.0.0.1", port: 9200}]
+    args {
+      timeout: 60
+      dead_timeout: 10
+      max_retries: 3
+      retry_on_timeout: true
+    }
+    index_version: "1"
+  }
+
+  workers {
+    hosts: [{host:"127.0.0.1", port:9200}]
+    args {
+      timeout: 60
+      dead_timeout: 10
+      max_retries: 3
+      retry_on_timeout: true
+    }
+    index_version: "1"
+  }
+}
+
+mongo {
+  backend {
+    host: "mongodb://127.0.0.1:27017/backend"
+  }
+  auth {
+    host: "mongodb://127.0.0.1:27017/auth"
+  }
+}
+
+redis {
+  apiserver {
+      host: "127.0.0.1"
+      port: 6379
+      db: 0
+  }
+  workers {
+    host: "127.0.0.1"
+    port: 6379
+    db: 4
+  }
+}

apiserver/config/default/secure.conf

@@ -13,17 +13,21 @@
     credentials {
         # system credentials as they appear in the auth DB, used for intra-service communications
         apiserver {
+            role: "system"
             user_key: "62T8CP7HGBC6647XF9314C2VY67RJO"
             user_secret: "FhS8VZv_I4%6Mo$8S1BWc$n$=o1dMYSivuiWU-Vguq7qGOKskG-d+b@tn_Iq"
         }
         webserver {
+            role: "system"
             user_key: "EYVQ385RW7Y2QQUH88CZ7DWIQ1WUHP"
             user_secret: "yfc8KQo*GMXb*9p((qcYC7ByFIpF7I&4VH3BfUYXH%o9vX1ZUZQEEw1Inc)S"
+            revoke_in_fixed_mode: true
         }
         tests {
+            role: "user"
+            display_name: "Default User"
             user_key: "EGRTCO8JMSIGI6S39GTP43NFWXDQOW"
             user_secret: "x!XTov_G-#vspE*Y(h$Anm&DIc5Ou-F)jsl$PdOyj5wG1&E!Z8"
-
         }
     }
 }

apiserver/config/default/services/auth.conf

@@ -0,0 +1,16 @@
+fixed_users {
+  guest {
+    enabled: false
+
+    default_company: "025315a9321f49f8be07f5ac48fbcf92"
+
+    name: "Guest"
+    username: "guest"
+    password: "guest"
+
+    # Allow access only to the following endpoints when using user/pass credentials
+    allow_endpoints: [
+      "auth.login"
+    ]
+  }
+}

apiserver/config/default/services/events.conf

@@ -0,0 +1,27 @@
+es_index_prefix: "events"
+
+ignore_iteration {
+    metrics: [":monitor:machine", ":monitor:gpu"]
+}
+
+
+events_retrieval {
+    state_expiration_sec: 3600
+
+    # max number of concurrent queries to ES when calculating events metrics
+    # should not exceed the amount of concurrent connections set in the ES driver
+    max_metrics_concurrency: 4
+
+    # the max amount of metrics to aggregate on
+    max_metrics_count: 100
+
+    # the max amount of variants to aggregate on
+    max_variants_count: 100
+}
+
+# if set then plot str will be checked for the valid json on plot add
+# and the result of the check is written to the db
+validate_plot_str: false
+
+# If not 0 then the plots equal or greater to the size will be stored compressed in the DB
+plot_compression_threshold: 100000

apiserver/config/default/services/organization.conf

@@ -0,0 +1,3 @@
+tags_cache {
+  expiration_seconds: 3600
+}

apiserver/config/default/services/projects.conf

@@ -0,0 +1,13 @@
+# Order of featured projects, by name or ID
+featured {
+  order: [
+    #  {id: "<project-id>"}
+    #  OR
+    #  {name: "<project-name>"}
+    #  OR
+    #  {name_regex: "<python-regex>"}
+  ]
+
+  # default featured index for public projects not specified in the order
+  public_default: 9999
+}

apiserver/config/default/services/tasks.conf

@@ -0,0 +1,11 @@
+non_responsive_tasks_watchdog {
+    enabled: true
+
+    # In-progress tasks older than this value in seconds will be stopped by the watchdog
+    threshold_sec: 7200
+
+    # Watchdog will sleep for this number of seconds after each cycle
+    watch_interval_sec: 900
+}
+
+multi_task_histogram_limit: 100

apiserver/config/info.py

@@ -0,0 +1,47 @@
+from functools import lru_cache
+from os import getenv
+from pathlib import Path
+
+from apiserver.config_repo import config
+from apiserver.version import __version__
+
+root = Path(__file__).parent.parent
+
+
+def _get(prop_name, env_suffix=None, default=""):
+    value = getenv(f"TRAINS_SERVER_{env_suffix or prop_name}")
+    if value:
+        return value
+
+    try:
+        return (root / prop_name).read_text().strip()
+    except FileNotFoundError:
+        return default
+
+
+@lru_cache()
+def get_build_number():
+    return _get("BUILD")
+
+
+@lru_cache()
+def get_version():
+    return _get("VERSION", default=__version__)
+
+
+@lru_cache()
+def get_commit_number():
+    return _get("COMMIT")
+
+
+@lru_cache()
+def get_deployment_type() -> str:
+    return _get("DEPLOY", env_suffix="DEPLOYMENT_TYPE", default="manual")
+
+
+def get_default_company():
+    return config.get("apiserver.default_company")
+
+
+missed_es_upgrade = False
+es_connection_error = False

apiserver/config_repo.py

@@ -0,0 +1,4 @@
+from apiserver.config import BasicConfig
+
+config = BasicConfig()
+config.initialize_logging()

apiserver/database/__init__.py

@@ -0,0 +1,98 @@
+from os import getenv
+
+from boltons.iterutils import first
+from furl import furl
+from jsonmodels import models
+from jsonmodels.errors import ValidationError
+from jsonmodels.fields import StringField
+from mongoengine import register_connection
+from mongoengine.connection import get_connection, disconnect
+
+from apiserver.config_repo import config
+from .defs import Database
+from .utils import get_items
+
+log = config.logger("database")
+
+strict = config.get("apiserver.mongo.strict", True)
+
+OVERRIDE_HOST_ENV_KEY = (
+    "TRAINS_MONGODB_SERVICE_HOST",
+    "MONGODB_SERVICE_HOST",
+    "MONGODB_SERVICE_SERVICE_HOST",
+)
+OVERRIDE_PORT_ENV_KEY = ("TRAINS_MONGODB_SERVICE_PORT", "MONGODB_SERVICE_PORT")
+
+
+class DatabaseEntry(models.Base):
+    host = StringField(required=True)
+    alias = StringField()
+
+
+class DatabaseFactory:
+    _entries = []
+
+    @classmethod
+    def initialize(cls):
+        db_entries = config.get("hosts.mongo", {})
+        missing = []
+        log.info("Initializing database connections")
+
+        override_hostname = first(map(getenv, OVERRIDE_HOST_ENV_KEY), None)
+        if override_hostname:
+            log.info(f"Using override mongodb host {override_hostname}")
+
+        override_port = first(map(getenv, OVERRIDE_PORT_ENV_KEY), None)
+        if override_port:
+            log.info(f"Using override mongodb port {override_port}")
+
+        for key, alias in get_items(Database).items():
+            if key not in db_entries:
+                missing.append(key)
+                continue
+
+            entry = DatabaseEntry(alias=alias, **db_entries.get(key))
+
+            if override_hostname:
+                entry.host = furl(entry.host).set(host=override_hostname).url
+
+            if override_port:
+                entry.host = furl(entry.host).set(port=override_port).url
+
+            try:
+                entry.validate()
+                log.info(
+                    "Registering connection to %(alias)s (%(host)s)" % entry.to_struct()
+                )
+                register_connection(alias=alias, host=entry.host)
+
+                cls._entries.append(entry)
+            except ValidationError as ex:
+                raise Exception("Invalid database entry `%s`: %s" % (key, ex.args[0]))
+        if missing:
+            raise ValueError("Missing database configuration for %s" % ", ".join(missing))
+
+    @classmethod
+    def get_entries(cls):
+        return cls._entries
+
+    @classmethod
+    def get_hosts(cls):
+        return [entry.host for entry in cls.get_entries()]
+
+    @classmethod
+    def get_aliases(cls):
+        return [entry.alias for entry in cls.get_entries()]
+
+    @classmethod
+    def reconnect(cls):
+        for entry in cls.get_entries():
+            # there is bug in the current implementation that prevents
+            # reconnection from work so workaround this
+            # get_connection(entry.alias, reconnect=True)
+            disconnect(entry.alias)
+            register_connection(alias=entry.alias, host=entry.host)
+            get_connection(entry.alias)
+
+
+db = DatabaseFactory()

apiserver/database/errors.py

@@ -1,6 +1,7 @@
 import re
 from contextlib import contextmanager
 from functools import wraps
+from textwrap import shorten
 
 import dpath
 from dpath.exceptions import InvalidKeyName
@@ -17,7 +18,7 @@ from mongoengine.errors import (
 )
 from pymongo.errors import PyMongoError, NotMasterError
 
-from apierrors import errors
+from apiserver.apierrors import errors
 
 
 class MakeGetAllQueryError(Exception):
@@ -33,7 +34,7 @@ class ParseCallError(Exception):
         self.params = kwargs
 
 
-def throws_default_error(err_cls):
+def throws_default_error(err_cls, shorten_width: int = None):
     """
     Used to make functions (Exception, str) -> Optional[str] searching for specialized error messages raise those
     messages in ``err_cls``. If the decorated function does not find a suitable error message,
@@ -45,25 +46,49 @@ def throws_default_error(err_cls):
         @wraps(func)
         def wrapper(self, e, message, **kwargs):
             extra_info = func(self, e, message, **kwargs)
-            raise err_cls(message, err=e, extra_info=extra_info)
+            err = str(e)
+            if shorten_width:
+                err = shorten(err, shorten_width, placeholder="...")
+            raise err_cls(message, err=err, extra_info=extra_info)
 
         return wrapper
 
     return decorator
 
 
+# noinspection RegExpRedundantEscape
 class ElasticErrorsHandler(object):
     @classmethod
-    @throws_default_error(errors.server_error.DataError)
+    def _bulk_meta_error(cls, error):
+        try:
+            _, err_type = next(dpath.search(error, "*/error/type", yielded=True))
+            _, reason = next(dpath.search(error, "*/error/reason", yielded=True))
+            if err_type == "cluster_block_exception":
+                raise errors.server_error.LowDiskSpace(
+                    "metrics, logs and all indexed data is in read-only mode!",
+                    reason=re.sub(r"^index\s\[.*?\]\s", "", reason) if reason else ""
+                )
+            return
+        except StopIteration:
+            pass
+
+    @classmethod
+    @throws_default_error(errors.server_error.DataError, shorten_width=200)
     def bulk_error(cls, e, _, **__):
         if not e.errors:
             return
 
+        # Currently we only handle the first error
+        error = e.errors[0]
+
+        cls._bulk_meta_error(error)
+
         # Else try returning a better error string
         for _, reason in dpath.search(e.errors[0], "*/error/reason", yielded=True):
             return reason
 
 
+# noinspection RegExpRedundantEscape
 class MongoEngineErrorsHandler(object):
     # NotUniqueError
     __not_unique_regex = re.compile(
@@ -81,6 +106,7 @@ class MongoEngineErrorsHandler(object):
     def validation_error(cls, e: ValidationError, message, **_):
         # Thrown when a document is validated. Documents are validated by default on save and on update
         err_dict = e.errors or {e.field_name: e.message}
+        err_dict = {key: str(value) for key, value in err_dict.items()}
         raise errors.bad_request.DataValidationError(message, **err_dict)
 
     @classmethod

apiserver/database/fields.py

@@ -1,5 +1,6 @@
-import re
+from operator import itemgetter
 from sys import maxsize
+from typing import Type, Tuple
 
 import six
 from mongoengine import (
@@ -11,7 +12,11 @@ from mongoengine import (
     SortedListField,
     MapField,
     DictField,
+    DynamicField,
 )
+from mongoengine.fields import key_not_string, key_starts_with_dollar, EmailField
+
+NoneType = type(None)
 
 
 class LengthRangeListField(ListField):
@@ -88,102 +93,22 @@ class CustomFloatField(FloatField):
             self.error("Float value must be greater than %s" % str(self.greater_than))
 
 
-# TODO: bucket name should be at most 63 characters....
-aws_s3_bucket_only_regex = (
-    r"^s3://"
-    r"(?:(?:\w[A-Z0-9\-]+\w)\.)*(?:\w[A-Z0-9\-]+\w)"  # bucket name
-)
+class CanonicEmailField(EmailField):
+    """email field that is always lower cased"""
+    def __set__(self, instance, value: str):
+        if value is not None:
+            try:
+                value = value.lower()
+            except AttributeError:
+                pass
+        super().__set__(instance, value)
 
-aws_s3_url_with_bucket_regex = (
-    r"^s3://"
-    r"(?:(?:\w[A-Z0-9\-]+\w)\.)*(?:\w[A-Z0-9\-]+\w)"  # bucket name
-    r"(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+(?:[A-Z]{2,6}\.?|[A-Z0-9-]{2,}(?<!-)\.?))"  # domain...
-)
-
-non_aws_s3_regex = (
-    r"^s3://"
-    r"(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+(?:[A-Z]{2,6}\.?|[A-Z0-9-]{2,}(?<!-)\.?)|"  # domain...
-    r"localhost|"  # localhost...
-    r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|"  # ...or ipv4
-    r"\[?[A-F0-9]*:[A-F0-9:]+\]?)"  # ...or ipv6
-    r"(?::\d+)?"  # optional port
-    r"(?:/(?:(?:\w[A-Z0-9\-]+\w)\.)*(?:\w[A-Z0-9\-]+\w))"  # bucket name
-)
-
-google_gs_bucket_only_regex = (
-    r"^gs://"
-    r"(?:(?:\w[A-Z0-9\-_]+\w)\.)*(?:\w[A-Z0-9\-_]+\w)"  # bucket name
-)
-
-file_regex = r"^file://"
-
-generic_url_regex = (
-    r"^%s://"  # scheme placeholder
-    r"(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+(?:[A-Z]{2,6}\.?|[A-Z0-9-]{2,}(?<!-)\.?)|"  # domain...
-    r"localhost|"  # localhost...
-    r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|"  # ...or ipv4
-    r"\[?[A-F0-9]*:[A-F0-9:]+\]?)"  # ...or ipv6
-    r"(?::\d+)?"  # optional port
-)
-
-path_suffix = r"(?:/?|[/?]\S+)$"
-file_path_suffix = r"(?:/\S*[^/]+)$"
-
-
-class _RegexURLField(StringField):
-    _regex = []
-
-    def __init__(self, regex, **kwargs):
-        super(_RegexURLField, self).__init__(**kwargs)
-        regex = regex if isinstance(regex, (tuple, list)) else [regex]
-        self._regex = [
-            re.compile(e, re.IGNORECASE) if isinstance(e, six.string_types) else e
-            for e in regex
-        ]
-
-    def validate(self, value):
-        # Check first if the scheme is valid
-        if not any(regex for regex in self._regex if regex.match(value)):
-            self.error("Invalid URL: {}".format(value))
-            return
-
-
-class OutputDestinationField(_RegexURLField):
-    """ A field representing task output URL """
-
-    schemes = ["s3", "gs", "file"]
-    _expressions = (
-        aws_s3_bucket_only_regex + path_suffix,
-        aws_s3_url_with_bucket_regex + path_suffix,
-        non_aws_s3_regex + path_suffix,
-        google_gs_bucket_only_regex + path_suffix,
-        file_regex + path_suffix,
-    )
-
-    def __init__(self, **kwargs):
-        super(OutputDestinationField, self).__init__(self._expressions, **kwargs)
-
-
-class SupportedURLField(_RegexURLField):
-    """ A field representing a model URL """
-
-    schemes = ["s3", "gs", "file", "http", "https"]
-
-    _expressions = tuple(
-        pattern + file_path_suffix
-        for pattern in (
-            aws_s3_bucket_only_regex,
-            aws_s3_url_with_bucket_regex,
-            non_aws_s3_regex,
-            google_gs_bucket_only_regex,
-            file_regex,
-            (generic_url_regex % "http"),
-            (generic_url_regex % "https"),
-        )
-    )
-
-    def __init__(self, **kwargs):
-        super(SupportedURLField, self).__init__(self._expressions, **kwargs)
+    def prepare_query_value(self, op, value):
+        if not isinstance(op, six.string_types):
+            return value
+        if value is not None:
+            value = value.lower()
+        return super().prepare_query_value(op, value)
 
 
 class StrippedStringField(StringField):
@@ -221,17 +146,82 @@ def contains_empty_key(d):
             return True
 
 
-class SafeMapField(MapField):
+class DictValidationMixin:
+    """
+    DictField validation in MongoEngine requires default alias and permissions to access DB version:
+    https://github.com/MongoEngine/mongoengine/issues/2239
+    This is a stripped down implementation that does not require any of the above and implies Mongo ver 3.6+
+    """
+
+    def _safe_validate(self: DictField, value):
+        if not isinstance(value, dict):
+            self.error("Only dictionaries may be used in a DictField")
+
+        if key_not_string(value):
+            msg = "Invalid dictionary key - documents must have only string keys"
+            self.error(msg)
+
+        if key_starts_with_dollar(value):
+            self.error(
+                'Invalid dictionary key name - keys may not startswith "$" characters'
+            )
+        super(DictField, self).validate(value)
+
+
+class SafeMapField(MapField, DictValidationMixin):
     def validate(self, value):
-        super(SafeMapField, self).validate(value)
+        self._safe_validate(value)
 
         if contains_empty_key(value):
             self.error("Empty keys are not allowed in a MapField")
 
 
-class SafeDictField(DictField):
+class SafeDictField(DictField, DictValidationMixin):
     def validate(self, value):
-        super(SafeDictField, self).validate(value)
+        self._safe_validate(value)
 
         if contains_empty_key(value):
             self.error("Empty keys are not allowed in a DictField")
+
+
+class SafeSortedListField(SortedListField):
+    """
+    SortedListField that does not raise an error in case items are not comparable
+    (in which case they will be sorted by their string representation)
+    """
+
+    def to_mongo(self, *args, **kwargs):
+        try:
+            return super(SafeSortedListField, self).to_mongo(*args, **kwargs)
+        except TypeError:
+            return self._safe_to_mongo(*args, **kwargs)
+
+    def _safe_to_mongo(self, value, use_db_field=True, fields=None):
+        value = super(SortedListField, self).to_mongo(value, use_db_field, fields)
+        if self._ordering is not None:
+
+            def key(v):
+                return str(itemgetter(self._ordering)(v))
+
+        else:
+            key = str
+        return sorted(value, key=key, reverse=self._order_reverse)
+
+
+class UnionField(DynamicField):
+    def __init__(self, types, *args, **kwargs):
+        super(UnionField, self).__init__(*args, **kwargs)
+        self.types: Tuple[Type] = tuple(types)
+
+    def validate(self, value, clean=True):
+        if not isinstance(value, self.types):
+            type_names = [t.__name__ for t in self.types]
+            expected = " or ".join(
+                filter(
+                    None,
+                    (", ".join(type_names[:-1]), type_names[-1]))
+            )
+            self.error(
+                f"Expected {expected}, got {type(value).__name__}: {value}"
+            )
+        super(UnionField, self).validate(value, clean)

apiserver/database/model/__init__.py

@@ -1,9 +1,11 @@
+from enum import Enum
+
 from mongoengine import Document, StringField
 
-from apierrors import errors
-from database.model.base import DbModelMixin, ABSTRACT_FLAG
-from database.model.company import Company
-from database.model.user import User
+from apiserver.apierrors import errors
+from apiserver.database.model.base import DbModelMixin, ABSTRACT_FLAG
+from apiserver.database.model.company import Company
+from apiserver.database.model.user import User
 
 
 class AttributedDocument(DbModelMixin, Document):
@@ -54,3 +56,7 @@ def validate_id(cls, company, **kwargs):
         **{name: obj_id for obj_id in missing for name in id_to_name[obj_id]}
     )
 
+
+class EntityVisibility(Enum):
+    active = "active"
+    archived = "archived"

apiserver/database/model/auth.py

@@ -6,10 +6,10 @@ from mongoengine import (
     DateTimeField,
 )
 
-from database import Database, strict
-from database.model import DbModelMixin
-from database.model.base import AuthDocument
-from database.utils import get_options
+from apiserver.database import Database, strict
+from apiserver.database.model import DbModelMixin
+from apiserver.database.model.base import AuthDocument
+from apiserver.database.utils import get_options
 
 
 class Entities(object):
@@ -32,6 +32,8 @@ class Role(object):
     """ Company user """
     annotator = "annotator"
     """ Annotator with limited access"""
+    guest = "guest"
+    """ Guest user. Read Only."""
 
     @classmethod
     def get_system_roles(cls) -> set:
@@ -43,15 +45,17 @@ class Role(object):
 
 
 class Credentials(EmbeddedDocument):
+    meta = {"strict": False}
     key = StringField(required=True)
     secret = StringField(required=True)
+    last_used = DateTimeField()
 
 
 class User(DbModelMixin, AuthDocument):
     meta = {"db_alias": Database.auth, "strict": strict}
 
     id = StringField(primary_key=True)
-    name = StringField(unique_with="company")
+    name = StringField()
 
     created = DateTimeField()
     """ User auth entry creation time """
@@ -68,5 +72,5 @@ class User(DbModelMixin, AuthDocument):
     credentials = EmbeddedDocumentListField(Credentials, default=list)
     """ Credentials generated for this user """
 
-    email = EmailField(unique=True, required=True)
+    email = EmailField(unique=True, sparse=True)
     """ Email uniquely identifying the user """

apiserver/database/model/base.py

@@ -1,19 +1,26 @@
 import re
 from collections import namedtuple
 from functools import reduce
-from typing import Collection
+from typing import Collection, Sequence, Union, Optional, Type, Tuple
 
+from boltons.iterutils import first, bucketize, partition
 from dateutil.parser import parse as parse_datetime
-from mongoengine import Q, Document
-from six import string_types
+from mongoengine import Q, Document, ListField, StringField
+from pymongo.command_cursor import CommandCursor
 
-from apierrors import errors
-from config import config
-from database.errors import MakeGetAllQueryError
-from database.projection import project_dict, ProjectionHelper
-from database.props import PropsMixin
-from database.query import RegexQ, RegexWrapper
-from database.utils import get_company_or_none_constraint, get_fields_with_attr
+from apiserver.apierrors import errors
+from apiserver.apierrors.base import BaseError
+from apiserver.config_repo import config
+from apiserver.database.errors import MakeGetAllQueryError
+from apiserver.database.projection import project_dict, ProjectionHelper
+from apiserver.database.props import PropsMixin
+from apiserver.database.query import RegexQ, RegexWrapper
+from apiserver.database.utils import (
+    get_company_or_none_constraint,
+    get_fields_choices,
+    field_does_not_exist,
+    field_exists,
+)
 
 log = config.logger("dbmodel")
 
@@ -28,7 +35,12 @@ class AuthDocument(Document):
 
 
 class ProperDictMixin(object):
-    def to_proper_dict(self, strip_private=True, only=None, extra_dict=None) -> dict:
+    def to_proper_dict(
+        self: Union["ProperDictMixin", Document],
+        strip_private=True,
+        only=None,
+        extra_dict=None,
+    ) -> dict:
         return self.properize_dict(
             self.to_mongo(use_db_field=False).to_dict(),
             strip_private=strip_private,
@@ -54,8 +66,9 @@ class ProperDictMixin(object):
 
 class GetMixin(PropsMixin):
     _text_score = "$text_score"
-
+    _projection_key = "projection"
     _ordering_key = "order_by"
+    _search_text_key = "search_text"
 
     _multi_field_param_sep = "__"
     _multi_field_param_prefix = {
@@ -64,11 +77,13 @@ class GetMixin(PropsMixin):
     }
     MultiFieldParameters = namedtuple("MultiFieldParameters", "pattern fields")
 
+    _field_collation_overrides = {}
+
     class QueryParameterOptions(object):
         def __init__(
             self,
             pattern_fields=("name",),
-            list_fields=("tags", "id"),
+            list_fields=("tags", "system_tags", "id"),
             datetime_fields=None,
             fields=None,
         ):
@@ -84,11 +99,56 @@ class GetMixin(PropsMixin):
             self.list_fields = list_fields
             self.pattern_fields = pattern_fields
 
+    class ListFieldBucketHelper:
+        op_prefix = "__$"
+        legacy_exclude_prefix = "-"
+
+        _default = "in"
+        _ops = {
+            "not": ("nin", False),
+            "all": ("all", True),
+            "and": ("all", True),
+        }
+        _next = _default
+        _sticky = False
+
+        def __init__(self, legacy=False):
+            self._legacy = legacy
+
+        def key(self, v):
+            if v is None:
+                self._next = self._default
+                return self._default
+            elif self._legacy and v.startswith(self.legacy_exclude_prefix):
+                self._next = self._default
+                return self._ops["not"][0]
+            elif v.startswith(self.op_prefix):
+                self._next, self._sticky = self._ops.get(
+                    v[len(self.op_prefix) :], (self._default, self._sticky)
+                )
+                return None
+
+            next_ = self._next
+            if not self._sticky:
+                self._next = self._default
+            return next_
+
+        def value_transform(self, v):
+            if self._legacy and v and v.startswith(self.legacy_exclude_prefix):
+                return v[len(self.legacy_exclude_prefix) :]
+            return v
+
     get_all_query_options = QueryParameterOptions()
 
     @classmethod
     def get(
-        cls, company, id, *, _only=None, include_public=False, **kwargs
+        cls: Union["GetMixin", Document],
+        company,
+        id,
+        *,
+        _only=None,
+        include_public=False,
+        **kwargs,
     ) -> "GetMixin":
         q = cls.objects(
             cls._prepare_perm_query(company, allow_public=include_public)
@@ -155,17 +215,7 @@ class GetMixin(PropsMixin):
             for field in tuple(opts.list_fields or ()):
                 data = parameters.pop(field, None)
                 if data:
-                    if not isinstance(data, (list, tuple)):
-                        raise MakeGetAllQueryError("expected list", field)
-                    exclude = [t for t in data if t.startswith("-")]
-                    include = list(set(data).difference(exclude))
-                    mongoengine_field = field.replace(".", "__")
-                    if include:
-                        dict_query[f"{mongoengine_field}__in"] = include
-                    if exclude:
-                        dict_query[f"{mongoengine_field}__nin"] = [
-                            t[1:] for t in exclude
-                        ]
+                    query &= cls.get_list_field_query(field, data)
 
             for field in opts.fields or []:
                 data = parameters.pop(field, None)
@@ -209,12 +259,72 @@ class GetMixin(PropsMixin):
 
         return query & RegexQ(**dict_query)
 
+    @classmethod
+    def get_list_field_query(cls, field: str, data: Sequence[Optional[str]]) -> Q:
+        """
+        Get a proper mongoengine Q object that represents an "or" query for the provided values
+        with respect to the given list field, with support for "none of empty" in case a None value
+        is included.
+
+        - Exclusion can be specified by a leading "-" for each value (API versions <2.8)
+            or by a preceding "__$not" value (operator)
+        - AND can be achieved using a preceding "__$all" or "__$and" value (operator)
+        """
+        if not isinstance(data, (list, tuple)):
+            raise MakeGetAllQueryError("expected list", field)
+
+        # TODO: backwards compatibility only for older API versions
+        helper = cls.ListFieldBucketHelper(legacy=True)
+        actions = bucketize(
+            data, key=helper.key, value_transform=helper.value_transform
+        )
+
+        allow_empty = None in actions.get("in", {})
+        mongoengine_field = field.replace(".", "__")
+
+        q = RegexQ()
+        for action in filter(None, actions):
+            q &= RegexQ(
+                **{
+                    f"{mongoengine_field}__{action}": list(
+                        set(filter(None, actions[action]))
+                    )
+                }
+            )
+
+        if not allow_empty:
+            return q
+
+        return (
+            q
+            | Q(**{f"{mongoengine_field}__exists": False})
+            | Q(**{mongoengine_field: []})
+        )
+
     @classmethod
     def _prepare_perm_query(cls, company, allow_public=False):
         if allow_public:
             return get_company_or_none_constraint(company)
         return Q(company=company)
 
+    @classmethod
+    def validate_order_by(cls, parameters, search_text) -> Sequence:
+        """
+        Validate and extract order_by params as a list
+        """
+        order_by = parameters.get(cls._ordering_key)
+        if not order_by:
+            return []
+
+        order_by = order_by if isinstance(order_by, list) else [order_by]
+        order_by = [cls._text_score if x == "@text_score" else x for x in order_by]
+        if not search_text and cls._text_score in order_by:
+            raise errors.bad_request.FieldsValueError(
+                "text score cannot be used in order_by when search text is not used"
+            )
+
+        return order_by
+
     @classmethod
     def validate_paging(
         cls, parameters=None, default_page=None, default_page_size=None
@@ -245,11 +355,40 @@ class GetMixin(PropsMixin):
             return override_projection
         if not parameters:
             return []
-        return parameters.get("projection") or parameters.get("only_fields", [])
+        return parameters.get(cls._projection_key) or parameters.get("only_fields", [])
 
     @classmethod
-    def set_default_ordering(cls, parameters, value):
-        parameters[cls._ordering_key] = parameters.get(cls._ordering_key) or value
+    def split_projection(
+        cls, projection: Sequence[str]
+    ) -> Tuple[Collection[str], Collection[str]]:
+        """Return include and exclude lists based on passed projection and class definition"""
+        if projection:
+            include, exclude = partition(
+                projection, key=lambda x: x[0] != ProjectionHelper.exclusion_prefix,
+            )
+        else:
+            include, exclude = [], []
+        exclude = {x.lstrip(ProjectionHelper.exclusion_prefix) for x in exclude}
+        return include, set(cls.get_exclude_fields()).union(exclude).difference(include)
+
+    @classmethod
+    def set_projection(cls, parameters: dict, value: Sequence[str]) -> Sequence[str]:
+        parameters.pop("only_fields", None)
+        parameters[cls._projection_key] = value
+        return value
+
+    @classmethod
+    def get_ordering(cls, parameters: dict) -> Optional[Sequence[str]]:
+        return parameters.get(cls._ordering_key)
+
+    @classmethod
+    def set_ordering(cls, parameters: dict, value: Sequence[str]) -> Sequence[str]:
+        parameters[cls._ordering_key] = value
+        return value
+
+    @classmethod
+    def set_default_ordering(cls, parameters: dict, value: Sequence[str]) -> None:
+        cls.set_ordering(parameters, cls.get_ordering(parameters) or value)
 
     @classmethod
     def get_many_with_join(
@@ -332,8 +471,9 @@ class GetMixin(PropsMixin):
                     `@text_score` keyword. A text index must be defined on the document type, otherwise an error will
                     be raised.
         :param return_dicts: Return a list of dictionaries. If True, a list of dicts is returned (if projection was
-            requested, each contains only the requested projection).
-            If False, a QuerySet object is returned (lazy evaluated)
+            requested, each contains only the requested projection). If False, a QuerySet object is returned
+            (lazy evaluated). If return_dicts is requested then the entities with the None value in order_by field
+            are returned last in the ordering.
         :param company: Company ID (required)
         :param parameters: Parameters dict from which paging ordering and searching parameters are extracted.
         :param query_dict: If provided, passed to prepare_query() along with all of the relevant arguments to produce
@@ -356,16 +496,38 @@ class GetMixin(PropsMixin):
             q = cls._prepare_perm_query(company, allow_public=allow_public)
         _query = (q & query) if query else q
 
+        if return_dicts:
+            return cls._get_many_override_none_ordering(
+                query=_query,
+                parameters=parameters,
+                override_projection=override_projection,
+            )
+
         return cls._get_many_no_company(
-            query=_query,
-            parameters=parameters,
-            override_projection=override_projection,
-            return_dicts=return_dicts,
+            query=_query, parameters=parameters, override_projection=override_projection
         )
 
+    @classmethod
+    def get_many_public(
+        cls, query: Q = None, projection: Collection[str] = None,
+    ):
+        """
+        Fetch all public documents matching a provided query.
+        :param query: Optional query object (mongoengine.Q).
+        :param projection: A list of projection fields.
+        :return: A list of documents matching the query.
+        """
+        q = get_company_or_none_constraint()
+        _query = (q & query) if query else q
+
+        return cls._get_many_no_company(query=_query, override_projection=projection)
+
     @classmethod
     def _get_many_no_company(
-        cls, query, parameters=None, override_projection=None, return_dicts=True
+        cls: Union["GetMixin", Document],
+        query: Q,
+        parameters=None,
+        override_projection=None,
     ):
         """
         Fetch all documents matching a provided query.
@@ -375,59 +537,138 @@ class GetMixin(PropsMixin):
         NOTE: BE VERY CAREFUL WITH THIS CALL, as it allows returning data across companies.
 
         :param query: Query object (mongoengine.Q)
-        :param return_dicts: Return a list of dictionaries. If True, a list of dicts is returned (if projection was
-            requested, each contains only the requested projection).
-            If False, a QuerySet object is returned (lazy evaluated)
         :param parameters: Parameters dict from which paging ordering and searching parameters are extracted.
         :param override_projection: A list of projection fields overriding any projection specified in the `param_dict`
             argument
         """
-        parameters = parameters or {}
-
         if not query:
             raise ValueError("query or call_data must be provided")
 
+        parameters = parameters or {}
+        search_text = parameters.get(cls._search_text_key)
+        order_by = cls.validate_order_by(parameters=parameters, search_text=search_text)
         page, page_size = cls.validate_paging(parameters=parameters)
-
-        order_by = parameters.get(cls._ordering_key)
-        if order_by:
-            order_by = order_by if isinstance(order_by, list) else [order_by]
-            order_by = [cls._text_score if x == "@text_score" else x for x in order_by]
-
-        search_text = parameters.get("search_text")
-
-        only = cls.get_projection(parameters, override_projection)
-
-        if not search_text and order_by and cls._text_score in order_by:
-            raise errors.bad_request.FieldsValueError(
-                "text score cannot be used in order_by when search text is not used"
-            )
+        include, exclude = cls.split_projection(
+            cls.get_projection(parameters, override_projection)
+        )
 
         qs = cls.objects(query)
         if search_text:
             qs = qs.search_text(search_text)
         if order_by:
             # add ordering
-            qs = (
-                qs.order_by(order_by)
-                if isinstance(order_by, string_types)
-                else qs.order_by(*order_by)
-            )
-        if only:
+            qs = qs.order_by(*order_by)
+
+        if include:
             # add projection
-            qs = qs.only(*only)
-        else:
-            exclude = set(cls.get_exclude_fields()).difference(only)
-            if exclude:
-                qs = qs.exclude(*exclude)
+            qs = qs.only(*include)
+
+        if exclude:
+            qs = qs.exclude(*exclude)
+
         if page is not None and page_size:
             # add paging
             qs = qs.skip(page * page_size).limit(page_size)
 
-        if return_dicts:
-            return [obj.to_proper_dict(only=only) for obj in qs]
         return qs
 
+    @classmethod
+    def _get_many_override_none_ordering(
+        cls: Union[Document, "GetMixin"],
+        query: Q = None,
+        parameters: dict = None,
+        override_projection: Collection[str] = None,
+    ) -> Sequence[dict]:
+        """
+        Fetch all documents matching a provided query. For the first order by field
+        the None values are sorted in the end regardless of the sorting order.
+        If the first order field is a user defined parameter (either from execution.parameters,
+        or from last_metrics) then the collation is set that sorts strings in numeric order where possible.
+        This is a company-less version for internal uses. We assume the caller has either added any necessary
+        constraints to the query or that no constraints are required.
+
+        NOTE: BE VERY CAREFUL WITH THIS CALL, as it allows returning data across companies.
+
+        :param query: Query object (mongoengine.Q)
+        :param parameters: Parameters dict from which paging ordering and searching parameters are extracted.
+        :param override_projection: A list of projection fields overriding any projection specified in the `param_dict`
+            argument
+        """
+        if not query:
+            raise ValueError("query or call_data must be provided")
+
+        parameters = parameters or {}
+        search_text = parameters.get(cls._search_text_key)
+        order_by = cls.validate_order_by(parameters=parameters, search_text=search_text)
+        page, page_size = cls.validate_paging(parameters=parameters)
+        include, exclude = cls.split_projection(
+            cls.get_projection(parameters, override_projection)
+        )
+
+        query_sets = [cls.objects(query)]
+        if order_by:
+            order_field = first(
+                field for field in order_by if not field.startswith("$")
+            )
+            if (
+                order_field
+                and not order_field.startswith("-")
+                and "[" not in order_field
+            ):
+                params = {}
+                mongo_field = order_field.replace(".", "__")
+                if mongo_field in cls.get_field_names_for_type(of_type=ListField):
+                    params["is_list"] = True
+                elif mongo_field in cls.get_field_names_for_type(of_type=StringField):
+                    params["empty_value"] = ""
+                non_empty = query & field_exists(mongo_field, **params)
+                empty = query & field_does_not_exist(mongo_field, **params)
+                query_sets = [cls.objects(non_empty), cls.objects(empty)]
+
+            query_sets = [qs.order_by(*order_by) for qs in query_sets]
+            if order_field:
+                collation_override = first(
+                    v
+                    for k, v in cls._field_collation_overrides.items()
+                    if order_field.startswith(k)
+                )
+                if collation_override:
+                    query_sets = [
+                        qs.collation(collation=collation_override) for qs in query_sets
+                    ]
+
+        if search_text:
+            query_sets = [qs.search_text(search_text) for qs in query_sets]
+
+        if include:
+            # add projection
+            query_sets = [qs.only(*include) for qs in query_sets]
+
+        if exclude:
+            query_sets = [qs.exclude(*exclude) for qs in query_sets]
+
+        if page is None or not page_size:
+            return [obj.to_proper_dict(only=include) for qs in query_sets for obj in qs]
+
+        # add paging
+        ret = []
+        start = page * page_size
+        for qs in query_sets:
+            qs_size = qs.count()
+            if qs_size < start:
+                start -= qs_size
+                continue
+            ret.extend(
+                obj.to_proper_dict(only=include)
+                for obj in qs.skip(start).limit(page_size)
+            )
+            if len(ret) >= page_size:
+                break
+            start = 0
+            page_size -= len(ret)
+
+        return ret
+
     @classmethod
     def get_for_writing(
         cls, *args, _only: Collection[str] = None, **kwargs
@@ -460,14 +701,24 @@ class GetMixin(PropsMixin):
 
 
 class UpdateMixin(object):
+    __user_set_allowed_fields = None
+    __locked_when_published_fields = None
+
     @classmethod
     def user_set_allowed(cls):
-        res = getattr(cls, "__user_set_allowed_fields", None)
-        if res is None:
-            res = cls.__user_set_allowed_fields = dict(
-                get_fields_with_attr(cls, "user_set_allowed")
+        if cls.__user_set_allowed_fields is None:
+            cls.__user_set_allowed_fields = dict(
+                get_fields_choices(cls, "user_set_allowed")
             )
-        return res
+        return cls.__user_set_allowed_fields
+
+    @classmethod
+    def locked_when_published(cls):
+        if cls.__locked_when_published_fields is None:
+            cls.__locked_when_published_fields = dict(
+                get_fields_choices(cls, "locked_when_published")
+            )
+        return cls.__locked_when_published_fields
 
     @classmethod
     def get_safe_update_dict(cls, fields):
@@ -488,7 +739,13 @@ class UpdateMixin(object):
         return update_dict
 
     @classmethod
-    def safe_update(cls, company_id, id, partial_update_dict, injected_update=None):
+    def safe_update(
+        cls: Union["UpdateMixin", Document],
+        company_id,
+        id,
+        partial_update_dict,
+        injected_update=None,
+    ):
         update_dict = cls.get_safe_update_dict(partial_update_dict)
         if not update_dict:
             return 0, {}
@@ -503,7 +760,52 @@ class UpdateMixin(object):
 class DbModelMixin(GetMixin, ProperDictMixin, UpdateMixin):
     """ Provide convenience methods for a subclass of mongoengine.Document """
 
-    pass
+    @classmethod
+    def aggregate(
+        cls: Union["DbModelMixin", Document],
+        pipeline: Sequence[dict],
+        allow_disk_use=None,
+        **kwargs,
+    ) -> CommandCursor:
+        """
+        Aggregate objects of this document class according to the provided pipeline.
+        :param pipeline: a list of dictionaries describing the pipeline stages
+        :param allow_disk_use: if True, allow the server to use disk space if aggregation query cannot fit in memory.
+        If None, default behavior will be used (see apiserver.conf/mongo/aggregate/allow_disk_use)
+        :param kwargs: additional keyword arguments passed to mongoengine
+        :return:
+        """
+        kwargs.update(
+            allowDiskUse=allow_disk_use
+            if allow_disk_use is not None
+            else config.get("apiserver.mongo.aggregate.allow_disk_use", True)
+        )
+        return cls.objects.aggregate(pipeline, **kwargs)
+
+    @classmethod
+    def set_public(
+        cls: Type[Document],
+        company_id: str,
+        ids: Sequence[str],
+        invalid_cls: Type[BaseError],
+        enabled: bool = True,
+    ):
+        if enabled:
+            items = list(cls.objects(id__in=ids, company=company_id).only("id"))
+            update = dict(set__company_origin=company_id, set__company="")
+        else:
+            items = list(
+                cls.objects(
+                    id__in=ids, company__in=(None, ""), company_origin=company_id
+                ).only("id")
+            )
+            update = dict(set__company=company_id, unset__company_origin=1)
+
+        if len(items) < len(ids):
+            missing = tuple(set(ids).difference(i.id for i in items))
+            raise invalid_cls(ids=missing)
+
+        return {"updated": cls.objects(id__in=ids).update(**update)}
 
 
 def validate_id(cls, company, **kwargs):
@@ -525,5 +827,5 @@ def validate_id(cls, company, **kwargs):
         id_to_name.setdefault(obj_id, []).append(name)
     raise errors.bad_request.ValidationError(
         "Invalid {} ids".format(cls.__name__.lower()),
-        **{name: obj_id for obj_id in missing for name in id_to_name[obj_id]}
+        **{name: obj_id for obj_id in missing for name in id_to_name[obj_id]},
     )

apiserver/database/model/company.py

@@ -0,0 +1,38 @@
+from mongoengine import (
+    Document,
+    EmbeddedDocument,
+    EmbeddedDocumentField,
+    StringField,
+    Q,
+    BooleanField,
+    DateTimeField,
+)
+
+from apiserver.database import Database, strict
+from apiserver.database.fields import StrippedStringField
+from apiserver.database.model import DbModelMixin
+
+
+class ReportStatsOption(EmbeddedDocument):
+    enabled = BooleanField(default=False)  # opt-in for statistics reporting
+    enabled_version = StringField()  # server version when enabled
+    enabled_time = DateTimeField()  # time when enabled
+    enabled_user = StringField()  # ID of user who enabled
+
+
+class CompanyDefaults(EmbeddedDocument):
+    cluster = StringField()
+    stats_option = EmbeddedDocumentField(ReportStatsOption, default=ReportStatsOption)
+
+
+class Company(DbModelMixin, Document):
+    meta = {"db_alias": Database.backend, "strict": strict}
+
+    id = StringField(primary_key=True)
+    name = StrippedStringField(min_length=3)
+    defaults = EmbeddedDocumentField(CompanyDefaults, default=CompanyDefaults)
+
+    @classmethod
+    def _prepare_perm_query(cls, company, allow_public=False):
+        """ Override default behavior since a 'company' constraint is not supported for this document... """
+        return Q()

apiserver/database/model/model.py

@@ -0,0 +1,75 @@
+from mongoengine import Document, StringField, DateTimeField, BooleanField
+
+from apiserver.database import Database, strict
+from apiserver.database.fields import StrippedStringField, SafeDictField, SafeSortedListField
+from apiserver.database.model import DbModelMixin
+from apiserver.database.model.base import GetMixin
+from apiserver.database.model.model_labels import ModelLabels
+from apiserver.database.model.company import Company
+from apiserver.database.model.project import Project
+from apiserver.database.model.task.task import Task
+from apiserver.database.model.user import User
+
+
+class Model(DbModelMixin, Document):
+    meta = {
+        "db_alias": Database.backend,
+        "strict": strict,
+        "indexes": [
+            "parent",
+            "project",
+            "task",
+            ("company", "framework"),
+            ("company", "name"),
+            ("company", "user"),
+            {
+                "name": "%s.model.main_text_index" % Database.backend,
+                "fields": ["$name", "$id", "$comment", "$parent", "$task", "$project"],
+                "default_language": "english",
+                "weights": {
+                    "name": 10,
+                    "id": 10,
+                    "comment": 10,
+                    "parent": 5,
+                    "task": 3,
+                    "project": 3,
+                },
+            },
+        ],
+    }
+    get_all_query_options = GetMixin.QueryParameterOptions(
+        pattern_fields=("name", "comment"),
+        fields=("ready",),
+        list_fields=(
+            "tags",
+            "system_tags",
+            "framework",
+            "uri",
+            "id",
+            "user",
+            "project",
+            "task",
+            "parent",
+        ),
+    )
+
+    id = StringField(primary_key=True)
+    name = StrippedStringField(user_set_allowed=True, min_length=3)
+    parent = StringField(reference_field="Model", required=False)
+    user = StringField(required=True, reference_field=User)
+    company = StringField(required=True, reference_field=Company)
+    project = StringField(reference_field=Project, user_set_allowed=True)
+    created = DateTimeField(required=True, user_set_allowed=True)
+    task = StringField(reference_field=Task)
+    comment = StringField(user_set_allowed=True)
+    tags = SafeSortedListField(StringField(required=True), user_set_allowed=True)
+    system_tags = SafeSortedListField(StringField(required=True), user_set_allowed=True)
+    uri = StrippedStringField(default="", user_set_allowed=True)
+    framework = StringField()
+    design = SafeDictField()
+    labels = ModelLabels()
+    ready = BooleanField(required=True)
+    ui_cache = SafeDictField(
+        default=dict, user_set_allowed=True, exclude_by_default=True
+    )
+    company_origin = StringField(exclude_by_default=True)

apiserver/database/model/model_labels.py

@@ -0,0 +1,14 @@
+from apiserver.database.fields import NoneType, UnionField, SafeMapField
+
+
+class ModelLabels(SafeMapField):
+    def __init__(self, *args, **kwargs):
+        super(ModelLabels, self).__init__(
+            field=UnionField(types=(int, NoneType)), *args, **kwargs
+        )
+
+    def validate(self, value):
+        super(ModelLabels, self).validate(value)
+        non_empty_values = list(filter(None, value.values()))
+        if non_empty_values and len(set(non_empty_values)) < len(non_empty_values):
+            self.error("Same label id appears more than once in model labels")

apiserver/database/model/project.py

@@ -1,27 +1,29 @@
-from mongoengine import StringField, DateTimeField, ListField
+from mongoengine import StringField, DateTimeField, IntField
 
-from database import Database, strict
-from database.fields import OutputDestinationField, StrippedStringField
-from database.model import AttributedDocument
-from database.model.base import GetMixin
+from apiserver.database import Database, strict
+from apiserver.database.fields import StrippedStringField, SafeSortedListField
+from apiserver.database.model import AttributedDocument
+from apiserver.database.model.base import GetMixin
 
 
 class Project(AttributedDocument):
 
     get_all_query_options = GetMixin.QueryParameterOptions(
-        pattern_fields=("name", "description"), list_fields=("tags", "id")
+        pattern_fields=("name", "description"),
+        list_fields=("tags", "system_tags", "id"),
     )
 
     meta = {
         "db_alias": Database.backend,
         "strict": strict,
         "indexes": [
+            ("company", "name"),
             {
                 "name": "%s.project.main_text_index" % Database.backend,
                 "fields": ["$name", "$id", "$description"],
                 "default_language": "english",
                 "weights": {"name": 10, "id": 10, "description": 10},
-            }
+            },
         ],
     }
 
@@ -34,6 +36,11 @@ class Project(AttributedDocument):
     )
     description = StringField(required=True)
     created = DateTimeField(required=True)
-    tags = ListField(StringField(required=True), default=list)
-    default_output_destination = OutputDestinationField()
+    tags = SafeSortedListField(StringField(required=True))
+    system_tags = SafeSortedListField(StringField(required=True))
+    default_output_destination = StrippedStringField()
     last_update = DateTimeField()
+    featured = IntField(default=9999)
+    logo_url = StringField()
+    logo_blob = StringField(exclude_by_default=True)
+    company_origin = StringField(exclude_by_default=True)

apiserver/database/model/queue.py

@@ -0,0 +1,46 @@
+from mongoengine import (
+    Document,
+    EmbeddedDocument,
+    StringField,
+    DateTimeField,
+    EmbeddedDocumentListField,
+)
+
+from apiserver.database import Database, strict
+from apiserver.database.fields import StrippedStringField, SafeSortedListField
+from apiserver.database.model import DbModelMixin
+from apiserver.database.model.base import ProperDictMixin, GetMixin
+from apiserver.database.model.company import Company
+from apiserver.database.model.task.task import Task
+
+
+class Entry(EmbeddedDocument, ProperDictMixin):
+    """ Entry representing a task waiting in the queue """
+    task = StringField(required=True, reference_field=Task)
+    ''' Task ID '''
+    added = DateTimeField(required=True)
+    ''' Added to the queue '''
+
+
+class Queue(DbModelMixin, Document):
+
+    get_all_query_options = GetMixin.QueryParameterOptions(
+        pattern_fields=("name",),
+        list_fields=("tags", "system_tags", "id"),
+    )
+
+    meta = {
+        'db_alias': Database.backend,
+        'strict': strict,
+    }
+
+    id = StringField(primary_key=True)
+    name = StrippedStringField(
+        required=True, unique_with="company", min_length=3, user_set_allowed=True
+    )
+    company = StringField(required=True, reference_field=Company)
+    created = DateTimeField(required=True)
+    tags = SafeSortedListField(StringField(required=True), default=list, user_set_allowed=True)
+    system_tags = SafeSortedListField(StringField(required=True), user_set_allowed=True)
+    entries = EmbeddedDocumentListField(Entry, default=list)
+    last_update = DateTimeField()

apiserver/database/model/settings.py

@@ -0,0 +1,57 @@
+from typing import Any, Optional, Sequence, Tuple
+
+from mongoengine import Document, StringField, DynamicField, Q
+from mongoengine.errors import NotUniqueError
+
+from apiserver.database import Database, strict
+from apiserver.database.model import DbModelMixin
+
+
+class SettingKeys:
+    server__uuid = "server.uuid"
+
+
+class Settings(DbModelMixin, Document):
+    meta = {
+        "db_alias": Database.backend,
+        "strict": strict,
+    }
+
+    key = StringField(primary_key=True)
+    value = DynamicField()
+
+    @classmethod
+    def get_by_key(cls, key: str, default: Optional[Any] = None, sep: str = ".") -> Any:
+        key = key.strip(sep)
+        res = Settings.objects(key=key).first()
+        if not res:
+            return default
+        return res.value
+
+    @classmethod
+    def get_by_prefix(
+        cls, key_prefix: str, default: Optional[Any] = None, sep: str = "."
+    ) -> Sequence[Tuple[str, Any]]:
+        key_prefix = key_prefix.strip(sep)
+        query = Q(key=key_prefix) | Q(key__startswith=key_prefix + sep)
+        res = Settings.objects(query)
+        if not res:
+            return default
+        return [(x.key, x.value) for x in res]
+
+    @classmethod
+    def set_or_add_value(cls, key: str, value: Any, sep: str = ".") -> bool:
+        """ Sets a new value or adds a new key/value setting (if key does not exist) """
+        key = key.strip(sep)
+        res = Settings.objects(key=key).update(key=key, value=value, upsert=True)
+        return bool(res)
+
+    @classmethod
+    def add_value(cls, key: str, value: Any, sep: str = ".") -> bool:
+        """ Adds a new key/value settings. Fails if key already exists. """
+        key = key.strip(sep)
+        try:
+            res = cls(key=key, value=value).save(force_insert=True)
+            return bool(res)
+        except NotUniqueError:
+            return False

apiserver/database/model/task/metrics.py

@@ -0,0 +1,39 @@
+from mongoengine import (
+    EmbeddedDocument,
+    StringField,
+    DynamicField,
+    LongField,
+    EmbeddedDocumentField,
+)
+
+from apiserver.database.fields import SafeMapField
+
+
+class MetricEvent(EmbeddedDocument):
+    meta = {
+        # For backwards compatibility reasons
+        "strict": False,
+    }
+
+    metric = StringField(required=True)
+    variant = StringField(required=True)
+    value = DynamicField(required=True)
+    min_value = DynamicField()  # for backwards compatibility reasons
+    max_value = DynamicField()  # for backwards compatibility reasons
+
+
+class EventStats(EmbeddedDocument):
+    meta = {
+        # For backwards compatibility reasons
+        "strict": False,
+    }
+    last_update = LongField()
+
+
+class MetricEventStats(EmbeddedDocument):
+    meta = {
+        # For backwards compatibility reasons
+        "strict": False,
+    }
+    metric = StringField(required=True)
+    event_stats_by_type = SafeMapField(field=EmbeddedDocumentField(EventStats))

apiserver/database/model/task/output.py

@@ -1,7 +1,7 @@
 from mongoengine import EmbeddedDocument, StringField
-from database.utils import get_options
 
-from database.fields import OutputDestinationField
+from apiserver.database.fields import StrippedStringField
+from apiserver.database.utils import get_options
 
 
 class Result(object):
@@ -10,7 +10,7 @@ class Result(object):
 
 
 class Output(EmbeddedDocument):
-    destination = OutputDestinationField()
+    destination = StrippedStringField()
     model = StringField(reference_field='Model')
     error = StringField(user_set_allowed=True)
     result = StringField(choices=get_options(Result))

apiserver/database/model/task/task.py

@@ -0,0 +1,238 @@
+from typing import Dict
+
+from mongoengine import (
+    StringField,
+    EmbeddedDocumentField,
+    EmbeddedDocument,
+    DateTimeField,
+    IntField,
+    ListField,
+    LongField,
+)
+
+from apiserver.database import Database, strict
+from apiserver.database.fields import (
+    StrippedStringField,
+    SafeMapField,
+    SafeDictField,
+    UnionField,
+    SafeSortedListField,
+)
+from apiserver.database.model import AttributedDocument
+from apiserver.database.model.base import ProperDictMixin, GetMixin
+from apiserver.database.model.model_labels import ModelLabels
+from apiserver.database.model.project import Project
+from apiserver.database.utils import get_options
+from .metrics import MetricEvent, MetricEventStats
+from .output import Output
+
+DEFAULT_LAST_ITERATION = 0
+
+
+class TaskStatus(object):
+    created = "created"
+    queued = "queued"
+    in_progress = "in_progress"
+    stopped = "stopped"
+    publishing = "publishing"
+    published = "published"
+    closed = "closed"
+    failed = "failed"
+    completed = "completed"
+    unknown = "unknown"
+
+
+class TaskStatusMessage(object):
+    stopping = "stopping"
+
+
+class TaskSystemTags(object):
+    development = "development"
+
+
+class Script(EmbeddedDocument, ProperDictMixin):
+    binary = StringField(default="python", strip=True)
+    repository = StringField(default="", strip=True)
+    tag = StringField(strip=True)
+    branch = StringField(strip=True)
+    version_num = StringField(strip=True)
+    entry_point = StringField(default="", strip=True)
+    working_dir = StringField(strip=True)
+    requirements = SafeDictField()
+    diff = StringField()
+
+
+class ArtifactTypeData(EmbeddedDocument):
+    preview = StringField()
+    content_type = StringField()
+    data_hash = StringField()
+
+
+class ArtifactModes:
+    input = "input"
+    output = "output"
+
+
+DEFAULT_ARTIFACT_MODE = ArtifactModes.output
+
+
+class Artifact(EmbeddedDocument):
+    key = StringField(required=True)
+    type = StringField(required=True)
+    mode = StringField(choices=get_options(ArtifactModes), default=DEFAULT_ARTIFACT_MODE)
+    uri = StringField()
+    hash = StringField()
+    content_size = LongField()
+    timestamp = LongField()
+    type_data = EmbeddedDocumentField(ArtifactTypeData)
+    display_data = SafeSortedListField(ListField(UnionField((int, float, str))))
+
+
+class ParamsItem(EmbeddedDocument, ProperDictMixin):
+    section = StringField(required=True)
+    name = StringField(required=True)
+    value = StringField(required=True)
+    type = StringField()
+    description = StringField()
+
+
+class ConfigurationItem(EmbeddedDocument, ProperDictMixin):
+    name = StringField(required=True)
+    value = StringField(required=True)
+    type = StringField()
+    description = StringField()
+
+
+class Execution(EmbeddedDocument, ProperDictMixin):
+    meta = {"strict": strict}
+    test_split = IntField(default=0)
+    parameters = SafeDictField(default=dict)
+    model = StringField(reference_field="Model")
+    model_desc = SafeMapField(StringField(default=""))
+    model_labels = ModelLabels()
+    framework = StringField()
+    artifacts: Dict[str, Artifact] = SafeMapField(field=EmbeddedDocumentField(Artifact))
+    docker_cmd = StringField()
+    queue = StringField()
+    """ Queue ID where task was queued """
+
+
+class TaskType(object):
+    training = "training"
+    testing = "testing"
+    inference = "inference"
+    data_processing = "data_processing"
+    application = "application"
+    monitor = "monitor"
+    controller = "controller"
+    optimizer = "optimizer"
+    service = "service"
+    qc = "qc"
+    custom = "custom"
+
+
+external_task_types = set(get_options(TaskType))
+
+
+class Task(AttributedDocument):
+    _numeric_locale = {"locale": "en_US", "numericOrdering": True}
+    _field_collation_overrides = {
+        "execution.parameters.": _numeric_locale,
+        "last_metrics.": _numeric_locale,
+        "hyperparams.": _numeric_locale,
+        "configuration.": _numeric_locale,
+    }
+
+    meta = {
+        "db_alias": Database.backend,
+        "strict": strict,
+        "indexes": [
+            "created",
+            "started",
+            "completed",
+            "active_duration",
+            "parent",
+            "project",
+            ("company", "name"),
+            ("company", "user"),
+            ("company", "status", "type"),
+            ("company", "system_tags", "last_update"),
+            ("company", "type", "system_tags", "status"),
+            ("company", "project", "type", "system_tags", "status"),
+            ("status", "last_update"),  # for maintenance tasks
+            {
+                "name": "%s.task.main_text_index" % Database.backend,
+                "fields": [
+                    "$name",
+                    "$id",
+                    "$comment",
+                    "$execution.model",
+                    "$output.model",
+                    "$script.repository",
+                    "$script.entry_point",
+                ],
+                "default_language": "english",
+                "weights": {
+                    "name": 10,
+                    "id": 10,
+                    "comment": 10,
+                    "execution.model": 2,
+                    "output.model": 2,
+                    "script.repository": 1,
+                    "script.entry_point": 1,
+                },
+            },
+        ],
+    }
+    get_all_query_options = GetMixin.QueryParameterOptions(
+        list_fields=("id", "user", "tags", "system_tags", "type", "status", "project", "parent"),
+        datetime_fields=("status_changed",),
+        pattern_fields=("name", "comment"),
+    )
+
+    id = StringField(primary_key=True)
+    name = StrippedStringField(
+        required=True, user_set_allowed=True, sparse=False, min_length=3
+    )
+
+    type = StringField(required=True, choices=get_options(TaskType))
+    status = StringField(default=TaskStatus.created, choices=get_options(TaskStatus))
+    status_reason = StringField()
+    status_message = StringField()
+    status_changed = DateTimeField()
+    comment = StringField(user_set_allowed=True)
+    created = DateTimeField(required=True, user_set_allowed=True)
+    started = DateTimeField()
+    completed = DateTimeField()
+    published = DateTimeField()
+    active_duration = IntField(default=None)
+    parent = StringField(reference_field="Task")
+    project = StringField(reference_field=Project, user_set_allowed=True)
+    output: Output = EmbeddedDocumentField(Output, default=Output)
+    execution: Execution = EmbeddedDocumentField(Execution, default=Execution)
+    tags = SafeSortedListField(StringField(required=True), user_set_allowed=True)
+    system_tags = SafeSortedListField(StringField(required=True), user_set_allowed=True)
+    script: Script = EmbeddedDocumentField(Script, default=Script)
+    last_worker = StringField()
+    last_worker_report = DateTimeField()
+    last_update = DateTimeField()
+    last_change = DateTimeField()
+    last_iteration = IntField(default=DEFAULT_LAST_ITERATION)
+    last_metrics = SafeMapField(field=SafeMapField(EmbeddedDocumentField(MetricEvent)))
+    metric_stats = SafeMapField(field=EmbeddedDocumentField(MetricEventStats))
+    company_origin = StringField(exclude_by_default=True)
+    duration = IntField()  # task duration in seconds
+    hyperparams = SafeMapField(field=SafeMapField(EmbeddedDocumentField(ParamsItem)))
+    configuration = SafeMapField(field=EmbeddedDocumentField(ConfigurationItem))
+    runtime = SafeDictField(default=dict)
+    docker_init_script = StringField()
+
+    def get_index_company(self) -> str:
+        """
+        Returns the company ID used for locating indices containing task data.
+        In case the task has a valid company, this is the company ID.
+        Otherwise, if the task has a company_origin, this is a task that has been made public and the
+         origin company should be used.
+        Otherwise, an empty company is used.
+        """
+        return self.company or self.company_origin or ""

apiserver/database/model/user.py

@@ -0,0 +1,22 @@
+from mongoengine import Document, StringField, DynamicField
+
+from apiserver.database import Database, strict
+from apiserver.database.model import DbModelMixin
+from apiserver.database.model.base import GetMixin
+from apiserver.database.model.company import Company
+
+
+class User(DbModelMixin, Document):
+    meta = {
+        "db_alias": Database.backend,
+        "strict": strict,
+    }
+    get_all_query_options = GetMixin.QueryParameterOptions(list_fields=("id",))
+
+    id = StringField(primary_key=True)
+    company = StringField(required=True, reference_field=Company)
+    name = StringField(required=True, user_set_allowed=True)
+    family_name = StringField(user_set_allowed=True)
+    given_name = StringField(user_set_allowed=True)
+    avatar = StringField()
+    preferences = DynamicField(default="", exclude_by_default=True)

apiserver/database/model/version.py

@@ -0,0 +1,18 @@
+from mongoengine import Document, DateTimeField, StringField
+
+from apiserver.database import Database, strict
+from apiserver.database.model import DbModelMixin
+
+
+class Version(DbModelMixin, Document):
+    meta = {
+        "collection": "versions",  # custom collection name ('version' is not a proper collection name...)
+        "db_alias": Database.backend,  # although we'll use this model for all databases, a default must be defined
+        "strict": strict,
+        "indexes": [("-created", "-num")],
+    }
+
+    id = StringField(primary_key=True)
+    num = StringField(required=True)
+    created = DateTimeField(required=True)
+    desc = StringField()

apiserver/database/projection.py

@@ -0,0 +1,377 @@
+import threading
+from concurrent.futures import ThreadPoolExecutor
+from itertools import groupby, chain
+from typing import Sequence, Dict, Callable, Tuple, Any, Type
+
+import dpath.path
+
+from apiserver.apierrors import errors
+from apiserver.database.props import PropsMixin
+
+SEP = "."
+
+
+def project_dict(data, projection, separator=SEP):
+    """
+    Project partial data from a dictionary into a new dictionary
+    :param data: Input dictionary
+    :param projection: List of dictionary paths (each a string with field names separated using a separator)
+    :param separator: Separator (default is '.')
+    :return: A new dictionary containing only the projected parts from the original dictionary
+    """
+    assert isinstance(data, dict)
+    result = {}
+
+    def copy_path(path_parts, source, destination):
+        src, dst = source, destination
+        try:
+            for depth, path_part in enumerate(path_parts[:-1]):
+                src_part = src[path_part]
+                if isinstance(src_part, dict):
+                    src = src_part
+                    dst = dst.setdefault(path_part, {})
+                elif isinstance(src_part, (list, tuple)):
+                    if path_part not in dst:
+                        dst[path_part] = [{} for _ in range(len(src_part))]
+                    elif not isinstance(dst[path_part], (list, tuple)):
+                        raise TypeError(
+                            "Incompatible destination type %s for %s (list expected)"
+                            % (type(dst), separator.join(path_parts[: depth + 1]))
+                        )
+                    elif not len(dst[path_part]) == len(src_part):
+                        raise ValueError(
+                            "Destination list length differs from source length for %s"
+                            % separator.join(path_parts[: depth + 1])
+                        )
+
+                    dst[path_part] = [
+                        copy_path(path_parts[depth + 1 :], s, d)
+                        for s, d in zip(src_part, dst[path_part])
+                    ]
+
+                    return destination
+                else:
+                    raise TypeError(
+                        "Unsupported projection type %s for %s"
+                        % (type(src), separator.join(path_parts[: depth + 1]))
+                    )
+
+            last_part = path_parts[-1]
+            dst[last_part] = src[last_part]
+        except KeyError:
+            # Projection field not in source, no biggie.
+            pass
+        return destination
+
+    for projection_path in sorted(projection):
+        copy_path(
+            path_parts=projection_path.split(separator), source=data, destination=result
+        )
+    return result
+
+
+class _ReferenceProxy(dict):
+    def __init__(self, id):
+        super(_ReferenceProxy, self).__init__(**({"id": id} if id else {}))
+
+
+class _ProxyManager:
+    lock = threading.Lock()
+
+    def __init__(self):
+        self._proxies: Dict[str, _ReferenceProxy] = {}
+
+    def add(self, id):
+        with self.lock:
+            proxy = self._proxies.get(id)
+            if proxy is None:
+                proxy = self._proxies[id] = _ReferenceProxy(id)
+            return proxy
+
+    def update(self, result):
+        proxy = self._proxies.get(result.get("id"))
+        if proxy is not None:
+            proxy.update(result)
+
+
+class ProjectionHelper(object):
+    pool = ThreadPoolExecutor()
+    exclusion_prefix = "-"
+
+    @property
+    def doc_projection(self):
+        return self._doc_projection
+
+    def __init__(self, doc_cls, projection, expand_reference_ids=False):
+        super(ProjectionHelper, self).__init__()
+        self._should_expand_reference_ids = expand_reference_ids
+        self._doc_cls = doc_cls
+        self._doc_projection = None
+        self._ref_projection = None
+        self._proxy_manager = _ProxyManager()
+
+        # Cached dpath paths for each of the result documents
+        self._cached_results_paths: Dict[int, Sequence[Tuple[Any, Type]]] = {}
+
+        self._parse_projection(projection)
+
+    def _collect_projection_fields(self, doc_cls, projection):
+        """
+        Collect projection for the given document into immediate document projection and reference documents projection
+        :param doc_cls: Document class
+        :param projection: List of projection fields
+        :return: A tuple of document projection and reference fields information
+        """
+        doc_projection = (
+            set()
+        )  # Projection fields for this class (used in the main query)
+        ref_projection_info = (
+            []
+        )  # Projection information for reference fields (used in join queries)
+        for field in projection:
+            field_ = field.lstrip(self.exclusion_prefix)
+            for ref_field, ref_field_cls in doc_cls.get_reference_fields().items():
+                if not field_.startswith(ref_field):
+                    # Doesn't start with a reference field
+                    continue
+                if field_ == ref_field:
+                    # Field is exactly a reference field. In this case we won't perform any inner projection (for that,
+                    # use '<reference field name>.*')
+                    continue
+                subfield = field_[len(ref_field) :]
+                if not subfield.startswith(SEP):
+                    # Starts with something that looks like a reference field, but isn't
+                    continue
+
+                ref_projection_info.append(
+                    (
+                        ref_field,
+                        ref_field_cls,
+                        ("" if field_[0] == field[0] else self.exclusion_prefix)
+                        + subfield[1:],
+                    )
+                )
+                break
+            else:
+                # Not a reference field, just add to the top-level projection
+                # We strip any trailing '*' since it means nothing for simple fields and for embedded documents
+                orig_field = field
+                if field.endswith(".*"):
+                    field = field[:-2]
+                if not field.lstrip(self.exclusion_prefix):
+                    raise errors.bad_request.InvalidFields(
+                        field=orig_field, object=doc_cls.__name__
+                    )
+                doc_projection.add(field)
+        return doc_projection, ref_projection_info
+
+    def _parse_projection(self, projection):
+        """
+        Prepare the projection data structures for get_many_with_join().
+        :param projection: A list of field names that should be returned by the query. Sub-fields can be specified
+            using '.' (i.e. "parent.name"). A field terminated by '.*' indicated that all of the field's sub-fields
+            should be returned (only relevant for fields that represent sub-documents or referenced documents)
+        :type projection: list of strings
+        :returns A tuple of (class fields projection, reference fields projection)
+        """
+        doc_cls = self._doc_cls
+        assert issubclass(doc_cls, PropsMixin)
+        if not projection:
+            return [], {}
+
+        doc_projection, ref_projection_info = self._collect_projection_fields(
+            doc_cls, projection
+        )
+
+        def normalize_cls_projection(cls_, fields):
+            """ Normalize projection for this class and group (expand *, for once) """
+            if "*" in fields:
+                return list(fields.difference("*").union(cls_.get_fields()))
+            return list(fields)
+
+        def compute_ref_cls_projection(cls_, group):
+            """ Compute inner projection for this class and group """
+            subfields = set([x[2] for x in group if x[2]])
+            return normalize_cls_projection(cls_, subfields)
+
+        def sort_key(proj_info):
+            return proj_info[:2]
+
+        # Aggregate by reference field. We'll leave out '*' from the projected items since
+        ref_projection = {
+            ref_field: dict(cls=ref_cls, only=compute_ref_cls_projection(ref_cls, g))
+            for (ref_field, ref_cls), g in groupby(
+                sorted(ref_projection_info, key=sort_key), sort_key
+            )
+        }
+
+        # Make sure this doesn't contain any reference field we'll join anyway
+        # (i.e. in case only_fields=[project, project.name])
+        doc_projection = normalize_cls_projection(
+            doc_cls, doc_projection.difference(ref_projection)
+        )
+
+        # Make sure that in case one or more field is a subfield of another field, we only use the the top-level field.
+        # This is done since in such a case, MongoDB will only use the most restrictive field (most nested field) and
+        # won't return some of the data we need.
+        # This way, we make sure to use the most inclusive field that contains all requested subfields.
+        projection_set = set(doc_projection)
+        doc_projection = [
+            field
+            for field in doc_projection
+            if not any(
+                field.startswith(f"{other_field}.")
+                for other_field in projection_set - {field}
+            )
+        ]
+
+        # Make sure we didn't get any invalid projection fields for this class
+        invalid_fields = [
+            f
+            for f in doc_projection
+            if f.partition(SEP)[0].lstrip(self.exclusion_prefix)
+            not in doc_cls.get_fields()
+        ]
+        if invalid_fields:
+            raise errors.bad_request.InvalidFields(
+                fields=invalid_fields, object=doc_cls.__name__
+            )
+
+        if ref_projection:
+            # Join mode - use both normal projection fields and top-level reference fields
+            doc_projection = set(doc_projection)
+            for field in set(ref_projection).difference(doc_projection):
+                if any(f for f in doc_projection if field.startswith(f)):
+                    continue
+                doc_projection.add(field)
+            doc_projection = list(doc_projection)
+
+        # If there are include fields (not only exclude) then add an id field
+        if (
+            not all(p.startswith(self.exclusion_prefix) for p in doc_projection)
+            and "id" not in doc_projection
+        ):
+            doc_projection.append("id")
+
+        self._doc_projection = doc_projection
+        self._ref_projection = ref_projection
+
+    def _search(
+        self,
+        doc_cls: PropsMixin,
+        obj: dict,
+        path: str,
+        factory: Callable[[str], dict] = None,
+    ) -> Sequence[str]:
+        """
+        Search for a path in the given object, return the list of values found for the
+        given path (multiple values may exist if the path is a glob expression)
+        :param doc_cls: The document class represented by the object
+        :param obj: Data object
+        :param path: Path to a leaf in the data object ("." separated, may contain "*")
+         (in case the path contains "*", there may be multiple values)
+        :param factory: If provided, replace each value found with an instance provided by the factory.
+        """
+        norm_path = doc_cls.get_dpath_translated_path(path)
+        globlist = norm_path.strip(SEP).split(SEP)
+
+        obj_paths = self._cached_results_paths.get(id(obj))
+        if obj_paths is None:
+            obj_paths = self._cached_results_paths[id(obj)] = list(
+                dpath.path.paths(obj, dirs=True, skip=True)
+            )
+
+        paths = [p for p in obj_paths if dpath.path.match(p, globlist)]
+
+        def search_and_replace(p: Sequence[Tuple[str, Type]]) -> Any:
+            parent = None
+            target = obj
+            for part in p:
+                parent = target
+                target = target[part[0]]
+            if parent and factory:
+                parent[p[-1][0]] = factory(target)
+            return target
+
+        return [search_and_replace(p) for p in paths]
+
+    def project(self, results, projection_func):
+        """
+        Perform projection on query results, using the provided projection func.
+        :param results: A list of results dictionaries on which projection should be performed
+        :param projection_func: A callable that receives a document type, list of ids and projection and returns query
+            results. This callable is used in order to perform sub-queries during projection
+        :return: Modified results (in-place)
+        """
+        cls = self._doc_cls
+        ref_projection = self._ref_projection
+
+        if ref_projection:
+            # Join mode - get results for each reference fields projection required (this is the join step)
+            # Note: this is a recursive step, so nested reference fields are supported
+
+            def collect_ids(ref_field_name):
+                """
+                Collect unique IDs for the given reference path from all result documents.
+                All collected IDs are replaced in the result dictionaries with a reference proxy generated by the
+                proxies manager to allow rapid update later on when projection results are obtained.
+                """
+                all_ids = (
+                    self._search(
+                        cls, res, ref_field_name, factory=self._proxy_manager.add
+                    )
+                    for res in results
+                )
+                return list(filter(None, set(chain.from_iterable(all_ids))))
+
+            items = [
+                tup
+                for tup in (
+                    (*item, collect_ids(item[0])) for item in ref_projection.items()
+                )
+                if tup[2]
+            ]
+
+            if items:
+
+                def do_projection(item):
+                    ref_field_name, data, ids = item
+
+                    doc_type = data["cls"]
+                    doc_only = list(filter(None, data["only"]))
+                    doc_only = list({"id"} | set(doc_only)) if doc_only else None
+
+                    for res in projection_func(
+                        doc_type=doc_type, projection=doc_only, ids=ids
+                    ):
+                        self._proxy_manager.update(res)
+
+                if len(ref_projection) == 1:
+                    do_projection(items[0])
+                else:
+                    for _ in self.pool.map(do_projection, items):
+                        # From ThreadPoolExecutor.map() documentation: If a call raises an exception then that exception
+                        #  will be raised when its value is retrieved from the map() iterator
+                        pass
+
+        def do_expand_reference_ids(result, skip_fields=None):
+            ref_fields = cls.get_reference_fields()
+            if skip_fields:
+                ref_fields = set(ref_fields) - set(skip_fields)
+            self._expand_reference_fields(cls, result, ref_fields)
+
+        # any reference field not projected should be expanded
+        if self._should_expand_reference_ids:
+            for result in results:
+                do_expand_reference_ids(
+                    result, skip_fields=list(ref_projection) if ref_projection else None
+                )
+
+        return results
+
+    def _expand_reference_fields(self, doc_cls, result, fields):
+        for ref_field_name in fields:
+            self._search(doc_cls, result, ref_field_name, factory=_ReferenceProxy)
+
+    def expand_reference_ids(self, doc_cls, result):
+        self._expand_reference_fields(doc_cls, result, doc_cls.get_reference_fields())

apiserver/database/props.py

@@ -1,17 +1,19 @@
-from collections import OrderedDict
+from collections import OrderedDict, defaultdict
+from itertools import chain
 from operator import attrgetter
 from threading import Lock
+from typing import Sequence
 
 import six
 from mongoengine import EmbeddedDocumentField, EmbeddedDocumentListField
-from mongoengine.base import get_document
+from mongoengine.base import get_document, BaseField
 
-from database.fields import (
+from apiserver.database.fields import (
     LengthRangeEmbeddedDocumentListField,
     UniqueEmbeddedDocumentListField,
     EmbeddedDocumentSortedListField,
 )
-from database.utils import get_fields, get_fields_and_attr
+from apiserver.database.utils import get_fields, get_fields_attr
 
 
 class PropsMixin(object):
@@ -19,6 +21,7 @@ class PropsMixin(object):
     __cached_reference_fields = None
     __cached_exclude_fields = None
     __cached_fields_with_instance = None
+    __cached_field_names_per_type = None
 
     __cached_dpath_computed_fields_lock = Lock()
     __cached_dpath_computed_fields = None
@@ -29,6 +32,39 @@ class PropsMixin(object):
             cls.__cached_fields = get_fields(cls)
         return cls.__cached_fields
 
+    @classmethod
+    def get_field_names_for_type(cls, of_type=BaseField):
+        """
+        Return field names per type including subfields
+        The fields of derived types are also returned
+        """
+        assert issubclass(of_type, BaseField)
+        if cls.__cached_field_names_per_type is None:
+            fields = defaultdict(list)
+            for name, field in get_fields(cls, return_instance=True, subfields=True):
+                fields[type(field)].append(name)
+            for type_ in fields:
+                fields[type_].extend(
+                    chain.from_iterable(
+                        fields[other_type]
+                        for other_type in fields
+                        if other_type != type_ and issubclass(other_type, type_)
+                    )
+                )
+            cls.__cached_field_names_per_type = fields
+
+        if of_type not in cls.__cached_field_names_per_type:
+            names = list(
+                chain.from_iterable(
+                    field_names
+                    for type_, field_names in cls.__cached_field_names_per_type.items()
+                    if issubclass(type_, of_type)
+                )
+            )
+            cls.__cached_field_names_per_type[of_type] = names
+
+        return cls.__cached_field_names_per_type[of_type]
+
     @classmethod
     def get_fields_with_instance(cls, doc_cls):
         if cls.__cached_fields_with_instance is None:
@@ -42,7 +78,7 @@ class PropsMixin(object):
     @staticmethod
     def _get_fields_with_attr(cls_, attr):
         """ Get all fields with the specified attribute (supports nested fields) """
-        res = get_fields_and_attr(cls_, attr=attr)
+        res = get_fields_attr(cls_, attr=attr)
 
         def resolve_doc(v):
             if not isinstance(v, six.string_types):
@@ -122,6 +158,14 @@ class PropsMixin(object):
             cls.__cached_reference_fields = OrderedDict(sorted(fields.items()))
         return cls.__cached_reference_fields
 
+    @classmethod
+    def get_extra_projection(cls, fields: Sequence) -> tuple:
+        if isinstance(fields, str):
+            fields = [fields]
+        return tuple(
+            set(fields).union(cls.get_fields()).difference(cls.get_exclude_fields())
+        )
+
     @classmethod
     def get_exclude_fields(cls):
         if cls.__cached_exclude_fields is None:
@@ -140,3 +184,18 @@ class PropsMixin(object):
                 result = separator.join(translated)
                 cls.__cached_dpath_computed_fields[path] = result
         return cls.__cached_dpath_computed_fields[path]
+
+    def get_field_value(self, field_path: str, default=None):
+        """
+        Return the document field_path value by the field_path name.
+        The path may contain '.'. If on any level the path is
+        not found then the default value is returned
+        """
+        path_elements = field_path.split(".")
+        current = self
+        for name in path_elements:
+            current = getattr(current, name, default)
+            if current == default:
+                break
+
+        return current

apiserver/database/query.py

@@ -1,8 +1,14 @@
 import copy
 import re
+from typing import Union
 
 from mongoengine import Q
-from mongoengine.queryset.visitor import QueryCompilerVisitor, SimplificationVisitor, QCombination
+from mongoengine.queryset.visitor import (
+    QueryCompilerVisitor,
+    SimplificationVisitor,
+    QCombination,
+    QNode,
+)
 
 
 class RegexWrapper(object):
@@ -17,17 +23,16 @@ class RegexWrapper(object):
 
 
 class RegexMixin(object):
-
-    def to_query(self, document):
+    def to_query(self: Union["RegexMixin", QNode], document):
         query = self.accept(SimplificationVisitor())
         query = query.accept(RegexQueryCompilerVisitor(document))
         return query
 
-    def _combine(self, other, operation):
+    def _combine(self: Union["RegexMixin", QNode], other, operation):
         """Combine this node with another node into a QCombination
         object.
         """
-        if getattr(other, 'empty', True):
+        if getattr(other, "empty", True):
             return self
 
         if self.empty:

apiserver/database/utils.py

@@ -0,0 +1,235 @@
+import hashlib
+from inspect import ismethod, getmembers
+from typing import Sequence, Tuple, Set, Optional, Callable, Any
+from uuid import uuid4
+
+from mongoengine import EmbeddedDocumentField, ListField, Document, Q
+from mongoengine.base import BaseField
+
+from .errors import translate_errors_context, ParseCallError
+
+
+def get_fields(cls, of_type=BaseField, return_instance=False, subfields=False):
+    return _get_fields(
+        cls,
+        of_type=of_type,
+        subfields=subfields,
+        selector=lambda k, v: (k, v) if return_instance else k,
+    )
+
+
+def get_fields_attr(cls, attr):
+    """ get field names from a class containing mongoengine fields """
+    return dict(
+        _get_fields(cls, with_attr=attr, selector=lambda k, v: (k, getattr(v, attr)))
+    )
+
+
+def get_fields_choices(cls, attr):
+    def get_choices(field_name: str, field: BaseField) -> Tuple:
+        if isinstance(field, ListField):
+            return field_name, field.field.choices
+        return field_name, field.choices
+
+    return dict(_get_fields(cls, with_attr=attr, subfields=True, selector=get_choices))
+
+
+def _get_fields(
+    cls,
+    with_attr=None,
+    of_type=BaseField,
+    subfields=False,
+    selector: Optional[Callable[[str, BaseField], Any]] = None,
+    path: Tuple[str, ...] = (),
+):
+    fields = []
+    for field_name, field in cls._fields.items():
+        field_path = path + (field_name,)
+        if isinstance(field, of_type) and (not with_attr or hasattr(field, with_attr)):
+            full_name = "__".join(field_path)
+            fields.append(selector(full_name, field) if selector else full_name)
+
+        if subfields and isinstance(field, EmbeddedDocumentField):
+            fields.extend(
+                _get_fields(
+                    field.document_type,
+                    with_attr=with_attr,
+                    of_type=of_type,
+                    subfields=subfields,
+                    selector=selector,
+                    path=field_path,
+                )
+            )
+
+    return fields
+
+
+def get_items(cls):
+    """ get key/value items from an enum-like class (members represent enumeration key/value) """
+
+    res = {k: v for k, v in getmembers(cls) if not (k.startswith("_") or ismethod(v))}
+    return res
+
+
+def get_options(cls):
+    """ get options from an enum-like class (members represent enumeration key/value) """
+    return list(get_items(cls).values())
+
+
+# return a dictionary of items which:
+# 1. are in the call_data
+# 2. are in the fields dictionary, and their value in the call_data matches the type in fields
+# 3. are in the cls_fields
+def parse_from_call(call_data, fields, cls_fields, discard_none_values=True):
+    if not isinstance(fields, dict):
+        # fields should be key=>type dict
+        fields = {k: None for k in fields}
+    fields = {k: v for k, v in fields.items() if k in cls_fields}
+    res = {}
+    with translate_errors_context("parsing call data"):
+        for field, desc in fields.items():
+            value = call_data.get(field)
+            if value is None:
+                if not discard_none_values and field in call_data:
+                    # we'll keep the None value in case the field actually exists in the call data
+                    res[field] = None
+                continue
+            if desc:
+                if issubclass(desc, Document):
+                    if not desc.objects(id=value).only("id"):
+                        raise ParseCallError(
+                            "expecting %s id" % desc.__name__, id=value, field=field
+                        )
+                elif callable(desc):
+                    try:
+                        desc(value)
+                    except TypeError:
+                        raise ParseCallError(f"expecting {desc.__name__}", field=field)
+                    except Exception as ex:
+                        raise ParseCallError(str(ex), field=field)
+            res[field] = value
+        return res
+
+
+def init_cls_from_base(cls, instance):
+    return cls(
+        **{
+            k: v
+            for k, v in instance.to_mongo(use_db_field=False).to_dict().items()
+            if k[0] != "_"
+        }
+    )
+
+
+def get_company_or_none_constraint(company=None):
+    return Q(company__in=(company, None, "")) | Q(company__exists=False)
+
+
+def field_does_not_exist(field: str, empty_value=None, is_list=False) -> Q:
+    """
+    Creates a query object used for finding a field that doesn't exist, or has None or an empty value.
+    :param field: Field name
+    :param empty_value: The empty value to test for (None means no specific empty value will be used)
+    :param is_list: Is this a list (array) field. In this case, instead of testing for an empty value,
+                    the length of the array will be used (len==0 means empty)
+    :return:
+    """
+    query = Q(**{f"{field}__exists": False}) | Q(
+        **{f"{field}__in": {empty_value, None}}
+    )
+    if is_list:
+        query |= Q(**{f"{field}__size": 0})
+    return query
+
+
+def field_exists(field: str, empty_value=None, is_list=False) -> Q:
+    """
+    Creates a query object used for finding a field that exists and is not None or empty.
+    :param field: Field name
+    :param empty_value: The empty value to test for (None means no specific empty value will be used)
+    :param is_list: Is this a list (array) field. In this case, instead of testing for an empty value,
+                    the length of the array will be used (len==0 means empty)
+    :return:
+    """
+    query = Q(**{f"{field}__exists": True}) & Q(
+        **{f"{field}__nin": {empty_value, None}}
+    )
+    if is_list:
+        query &= Q(**{f"{field}__not__size": 0})
+    return query
+
+
+def get_subkey(d, key_path, default=None):
+    """ Get a key from a nested dictionary. kay_path is a '.' separated string of keys used to traverse
+        the nested dictionary.
+    """
+    keys = key_path.split(".")
+    for i, key in enumerate(keys):
+        if not isinstance(d, dict):
+            raise KeyError(
+                "Expecting a dict (%s)" % (".".join(keys[:i]) if i else "bad input")
+            )
+        d = d.get(key)
+        if d is None:
+            return default
+    return d
+
+
+def id():
+    return str(uuid4()).replace("-", "")
+
+
+def hash_field_name(s):
+    """ Hash field name into a unique safe string """
+    return hashlib.md5(s.encode()).hexdigest()
+
+
+def merge_dicts(*dicts):
+    base = {}
+    for dct in dicts:
+        base.update(dct)
+    return base
+
+
+def filter_fields(cls, fields):
+    """From the fields dictionary return only the fields that match cls fields"""
+    return {key: fields[key] for key in fields if key in get_fields(cls)}
+
+
+def _names_set(*names: str) -> Set[str]:
+    """
+    Given a list of names return set with names and '-names'
+    """
+    return set(names) | set(f"-{name}" for name in names)
+
+
+system_tag_names = {
+    "model": _names_set("active", "archived"),
+    "project": _names_set("archived", "public", "default"),
+    "task": _names_set("active", "archived", "development"),
+    "queue": _names_set("default"),
+}
+
+system_tag_prefixes = {"task": _names_set("annotat")}
+
+
+def partition_tags(
+    entity: str, tags: Sequence[str], system_tags: Optional[Sequence[str]] = ()
+) -> Tuple[Sequence[str], Sequence[str]]:
+    """
+    Partition the given tags sequence into system and user-defined tags
+    :param entity: The name of the entity that defines the list of the system tags
+    :param tags: The tags to partition
+    :param system_tags: Optional. If passed then these tags are considered system together
+    with those defined for the entity.
+    :return: a tuple where the first element is the sequence of user-defined tags and
+    the second element is the sequence of system tags
+    """
+    tags = set(tags)
+    system_tags = set(system_tags)
+    system_tags |= tags & system_tag_names[entity]
+
+    prefixes = system_tag_prefixes.get(entity, [])
+    system_tags |= {t for t in tags for p in prefixes if t.lower().startswith(p)}
+
+    return list(tags - system_tags), list(system_tags)

apiserver/elastic/apply_mappings.py

@@ -0,0 +1,58 @@
+#!/usr/bin/env python3
+"""
+Apply elasticsearch mappings to given hosts.
+"""
+import argparse
+import json
+from pathlib import Path
+from typing import Optional, Sequence
+
+from elasticsearch import Elasticsearch
+
+HERE = Path(__file__).resolve().parent
+
+
+def apply_mappings_to_cluster(
+    hosts: Sequence, key: Optional[str] = None, es_args: dict = None
+):
+    """Hosts maybe a sequence of strings or dicts in the form {"host": <host>, "port": <port>}"""
+
+    def _send_template(f):
+        with f.open() as json_data:
+            data = json.load(json_data)
+            template_name = f.stem
+            res = es.indices.put_template(template_name, body=data)
+            return {"mapping": template_name, "result": res}
+
+    p = HERE / "mappings"
+    if key:
+        files = (p / key).glob("*.json")
+    else:
+        files = p.glob("**/*.json")
+
+    es = Elasticsearch(hosts=hosts, **(es_args or {}))
+    return [_send_template(f) for f in files]
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(
+        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
+    )
+    parser.add_argument("--key", help="host key, e.g. events, datasets etc.")
+    parser.add_argument(
+        "--hosts",
+        nargs="+",
+        help="list of es hosts from the same cluster, where each host is http[s]://[user:password@]host:port",
+    )
+    return parser.parse_args()
+
+
+def main():
+    args = parse_args()
+    print(">>>>> Applying mapping to " + str(args.hosts))
+    res = apply_mappings_to_cluster(args.hosts, args.key)
+    print(res)
+
+
+if __name__ == "__main__":
+    main()