Version bump to v1.6

Add queues.peek_task and queues.get_num_entries

.gitignore

@@ -12,7 +12,6 @@ test-reports
 .pytest_cache
 venv
 *.noseids
-build
 *.egg-info
 .cache
 .mypy_cache

README.md

@@ -1,45 +1,68 @@
-# Trains Server
+<div align="center">
 
-##  Auto-Magical Experiment Manager & Version Control for AI - ε Devops Included!
+<img src="docs/clearml_server_logo.png" width="250px">
+
+**ClearML - Auto-Magical Suite of tools to streamline your ML workflow 
+</br>Experiment Manager, ML-Ops and Data-Management**
 
 [![GitHub license](https://img.shields.io/badge/license-SSPL-green.svg)](https://img.shields.io/badge/license-SSPL-green.svg)
 [![Python versions](https://img.shields.io/badge/python-3.6%20%7C%203.7-blue.svg)](https://img.shields.io/badge/python-3.6%20%7C%203.7-blue.svg)
 [![GitHub version](https://img.shields.io/github/release-pre/allegroai/trains-server.svg)](https://img.shields.io/github/release-pre/allegroai/trains-server.svg)
-[![PyPI status](https://img.shields.io/badge/status-beta-yellow.svg)](https://img.shields.io/badge/status-beta-yellow.svg)
+[![Artifact Hub](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/allegroai)](https://artifacthub.io/packages/search?repo=allegroai)
 
-### Help improve Trains by filling our 2-min [user survey](https://allegro.ai/lp/trains-user-survey/)
+</div>
 
-## :rocket: Trains-Agent Services is now included, for more information see [services](https://github.com/allegroai/trains-server#services)
+---
+<div align="center">
 
-## v0.16 Upgrade Notice
+**Note regarding Apache Log4j2 Remote Code Execution (RCE) Vulnerability - CVE-2021-44228 - ESA-2021-31**
 
-In v0.16, the Elasticsearch subsystem of Trains Server has been upgraded from version 5.6 to version 7.6. This change necessitates the migration of the database contents to accommodate the change in index structure across the different versions.  
+</div>
 
-Follow [this procedure](https://allegro.ai/docs/deploying_trains/trains_server_es7_migration/) to migrate existing data.
+According to [ElasticSearch's latest report](https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476), 
+supported versions of Elasticsearch (6.8.9+, 7.8+) used with recent versions of the JDK (JDK9+) **are not susceptible to either remote code execution or information leakage**
+due to Elasticsearch’s usage of the Java Security Manager.
 
-## Introduction
+**As the latest version of ClearML Server uses Elasticsearch 7.10+ with JDK15, it is not affected by these vulnerabilities.**
 
-The **trains-server** is the backend service infrastructure for [Trains](https://github.com/allegroai/trains).
+As a precaution, we've upgraded the ES version to 7.16.2 and added the mitigation recommended by ElasticSearch to our latest [docker-compose.yml](https://github.com/allegroai/clearml-server/blob/cfccbe05c158b75e520581f86e9668291da5c70a/docker/docker-compose.yml#L42) file.
+
+While previous Elasticsearch versions (5.6.11+, 6.4.0+ and 7.0.0+) used by older ClearML Server versions are only susceptible to the information leakage vulnerability
+(which in any case **does not permit access to data within the Elasticsearch cluster**), 
+we still recommend upgrading to the latest version of ClearML Server. Alternatively, you can apply the mitigation as implemented in our latest 
+[docker-compose.yml](https://github.com/allegroai/clearml-server/blob/cfccbe05c158b75e520581f86e9668291da5c70a/docker/docker-compose.yml#L42) file.    
+
+**Update 15 December**: A further vulnerability (CVE-2021-45046) was disclosed on December 14th.
+ElasticSearch's guidance for Elasticsearch remains unchanged by this new vulnerability, thus **not affecting ClearML Server**.
+
+**Update 22 December**: To keep with ElasticSearch's recommendations, we've upgraded the ES version to the newly released 7.16.2
+
+---
+
+## ClearML Server
+#### *Formerly known as Trains Server*
+
+The **ClearML Server** is the backend service infrastructure for [ClearML](https://github.com/allegroai/clearml).
 It allows multiple users to collaborate and manage their experiments.
-By default, **Trains** is set up to work with the **Trains** demo server, which is open to anyone and resets periodically.
-In order to host your own server, you will need to launch **trains-server** and point **Trains** to it.
+**ClearML** offers a [free hosted service](https://app.clear.ml/), which is maintained by **ClearML** and open to anyone.
+In order to host your own server, you will need to launch the **ClearML Server** and point **ClearML** to it.
 
-**trains-server** contains the following components:
+The **ClearML Server** contains the following components:
 
-* The **Trains** Web-App, a single-page UI for experiment management and browsing
+* The **ClearML** Web-App, a single-page UI for experiment management and browsing
 * RESTful API for:
     * Documenting and logging experiment information, statistics and results
     * Querying experiments history, logs and results
 * Locally-hosted file server for storing images and models making them easily accessible using the Web-App
 
-You can quickly [deploy](#launching-trains-server)  your **trains-server** using Docker, AWS EC2 AMI, or Kubernetes. 
+You can quickly [deploy](#launching-the-clearml-server)  your **ClearML Server** using Docker, AWS EC2 AMI, or Kubernetes. 
 
 ## System design
 
 
-![Alt Text](https://github.com/allegroai/trains/blob/master/docs/system_diagram.png?raw=true)
+![Alt Text](docs/ClearML_Server_Diagram.png)
 
-**trains-server** has two supported configurations:
+The **ClearML Server** has two supported configurations:
 - Single IP (domain) with the following open ports
     - Web application on port 8080
     - API service on port 8008
@@ -50,11 +73,11 @@ You can quickly [deploy](#launching-trains-server)  your **trains-server** using
     - API service on sub-domain: api.\*.\*
     - File storage service on sub-domain: files.\*.\*
     
-## Launching trains-server
+## Launching The ClearML Server
 
 ### Prerequisites
 
-The ports 8080/8081/8008 must be available for the **trains-server** services.
+The ports 8080/8081/8008 must be available for the **ClearML Server** services.
    
 For example, to see if port `8080` is in use:
 
@@ -68,24 +91,23 @@ For example, to see if port `8080` is in use:
    
 ### Launching   
     
-Launch **trains-server** in any of the following formats:
+Launch The **ClearML Server** in any of the following formats:
 
-- Pre-built [AWS EC2 AMI](https://allegro.ai/docs/deploying_trains/trains_server_aws_ec2_ami/)
-- Pre-built [GCP Custom Image](https://allegro.ai/docs/deploying_trains/trains_server_gcp/)
+- Pre-built [AWS EC2 AMI](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_aws_ec2_ami)
+- Pre-built [GCP Custom Image](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_gcp)
 - Pre-built Docker Image
-    - [Linux](https://allegro.ai/docs/deploying_trains/trains_server_linux_mac/)
-    - [macOS](https://allegro.ai/docs/deploying_trains/trains_server_linux_mac/)
-    - [Windows 10](https://allegro.ai/docs/deploying_trains/trains_server_win/)
+    - [Linux](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_linux_mac)
+    - [macOS](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_linux_mac)
+    - [Windows 10](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_win)
 - Kubernetes    
-    - [Kubernetes Helm](https://allegro.ai/docs/deploying_trains/trains_server_kubernetes_helm/)
-    - Manual [Kubernetes installation](https://allegro.ai/docs/deploying_trains/trains_server_kubernetes/)
+    - [Kubernetes Helm](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_kubernetes_helm)
+    - Manual [Kubernetes installation](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_kubernetes)
 
-## Connecting Trains to your trains-server
+## Connecting ClearML to your ClearML Server
 
-By default, the **Trains** client is set up to work with the [**Trains** demo server](https://demoapp.trains.allegro.ai/).  
-To have the **Trains** client use your **trains-server** instead:
-- Run the `trains-init` command for an interactive setup.
-- Or manually edit `~/trains.conf` file, making sure the server settings (`api_server`, `web_server`, `file_server`) are configured correctly, for example:
+In order to set up the **ClearML** client to work with your **ClearML Server**:
+- Run the `clearml-init` command for an interactive setup.
+- Or manually edit `~/clearml.conf` file, making sure the server settings (`api_server`, `web_server`, `file_server`) are configured correctly, for example:
 
         api {
             # API server on port 8008
@@ -98,44 +120,44 @@ To have the **Trains** client use your **trains-server** instead:
             files_server: "http://localhost:8081"
         }
 
-**Note**: If you have set up **trains-server** in a sub-domain configuration, then there is no need to specify a port number,
+**Note**: If you have set up your **ClearML Server** in a sub-domain configuration, then there is no need to specify a port number,
 it will be inferred from the http/s scheme.
 
-After launching the **trains-server** and configuring the **Trains** client to use the **trains-server**,
-you can [use](https://github.com/allegroai/trains#using-trains) **Trains** in your experiments and view them in your **trains-server** web server,
+After launching the **ClearML Server** and configuring the **ClearML** client to use the **ClearML Server**,
+you can [use](https://github.com/allegroai/clearml) **ClearML** in your experiments and view them in your **ClearML Server** web server,
 for example http://localhost:8080.  
-For more information about the Trains client, see [**Trains**](https://github.com/allegroai/trains).
+For more information about the ClearML client, see [**ClearML**](https://github.com/allegroai/clearml).
 
-## Trains-Agent Services  <a name="services"></a> 
+## ClearML-Agent Services  <a name="services"></a> 
 
-As of version 0.15 of **trains-server**, dockerized deployment includes a **Trains-Agent Services** container running as 
+As of version 0.15 of **ClearML Server**, dockerized deployment includes a **ClearML-Agent Services** container running as 
 part of the docker container collection.
 
-Trains-Agent Services is an extension of Trains-Agent that provides the ability to launch long-lasting jobs 
+ClearML-Agent Services is an extension of ClearML-Agent that provides the ability to launch long-lasting jobs 
 that previously had to be executed on local / dedicated machines. It allows a single agent to 
 launch multiple dockers (Tasks) for different use cases. To name a few use cases, auto-scaler service (spinning instances 
 when the need arises and the budget allows), Controllers (Implementing pipelines and more sophisticated DevOps logic),
 Optimizer (such as Hyper-parameter Optimization or sweeping), and Application (such as interactive Bokeh apps for 
 increased data transparency)
 
-Trains-Agent Services container will spin **any** task enqueued into the dedicated `services` queue. 
-Every task launched by Trains-Agent Services  will be registered as a new node in the system, 
+ClearML-Agent Services container will spin **any** task enqueued into the dedicated `services` queue. 
+Every task launched by ClearML-Agent Services  will be registered as a new node in the system, 
 providing tracking and transparency capabilities.  
-You can also run the Trains-Agent Services manually, see details in [trains-agent services mode](https://github.com/allegroai/trains-agent#trains-agent-services-mode-)
+You can also run the ClearML-Agent Services manually, see details in [ClearML-agent services mode](https://github.com/allegroai/clearml-agent#clearml-agent-services-mode-)
 
 **Note**: It is the user's responsibility to make sure the proper tasks are pushed into the `services` queue. 
 Do not enqueue training / inference tasks into the `services` queue, as it will put unnecessary load on the server.
 
 ## Advanced Functionality
 
-**trains-server** provides a few additional useful features, which can be manually enabled:
+The **ClearML Server** provides a few additional useful features, which can be manually enabled:
  
-* [Web login authentication](https://allegro.ai/docs/faq/faq/#web-auth)
-* [Non-responsive experiments watchdog](https://allegro.ai/docs/faq/faq/#watchdog)  
+* [Web login authentication](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_config#web-login-authentication)
+* [Non-responsive experiments watchdog](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_config#non-responsive-task-watchdog)  
 
-## Restarting trains-server
+## Restarting ClearML Server
 
-To restart the **trains-server**, you must first stop the containers, and then restart them.
+To restart the **ClearML Server**, you must first stop the containers, and then restart them.
 
    ```bash
    docker-compose down
@@ -144,12 +166,12 @@ To restart the **trains-server**, you must first stop the containers, and then r
 
 ## Upgrading <a name="upgrade"></a>
 
-**trains-server** releases are also reflected in the [docker compose configuration file](https://github.com/allegroai/trains-server/blob/master/docker-compose.yml).  
-We strongly encourage you to keep your **trains-server** up to date, by keeping up with the current release.
+**ClearML Server** releases are also reflected in the [docker compose configuration file](https://github.com/allegroai/trains-server/blob/master/docker/docker-compose.yml).  
+We strongly encourage you to keep your **ClearML Server** up to date, by keeping up with the current release.
 
 **Note**: The following upgrade instructions use the Linux OS as an example.
 
-To upgrade your existing **trains-server** deployment:
+To upgrade your existing **ClearML Server** deployment:
 
 1. Shut down the docker containers
    ```bash
@@ -158,10 +180,10 @@ To upgrade your existing **trains-server** deployment:
 
 1. We highly recommend backing up your data directory before upgrading.
 
-   Assuming your data directory is `/opt/trains`, to archive all data into `~/trains_backup.tgz` execute:
+   Assuming your data directory is `/opt/clearml`, to archive all data into `~/clearml_backup.tgz` execute:
 
    ```bash
-   sudo tar czvf ~/trains_backup.tgz /opt/trains/data
+   sudo tar czvf ~/clearml_backup.tgz /opt/clearml/data
    ```    
 
    <details>
@@ -169,51 +191,51 @@ To upgrade your existing **trains-server** deployment:
 
    To restore this example backup, execute:
    ```bash
-   sudo rm -R /opt/trains/data
-   sudo tar -xzf ~/trains_backup.tgz -C /opt/trains/data
+   sudo rm -R /opt/clearml/data
+   sudo tar -xzf ~/clearml_backup.tgz -C /opt/clearml/data
    ```
    </details>
 
 1. Download the latest `docker-compose.yml` file.
 
    ```bash
-   curl https://raw.githubusercontent.com/allegroai/trains-server/master/docker-compose.yml -o docker-compose.yml 
+   curl https://raw.githubusercontent.com/allegroai/trains-server/master/docker/docker-compose.yml -o docker-compose.yml 
    ```
 
-1. Configure the Trains-Agent Services (not supported on Windows installation). 
-   If `TRAINS_HOST_IP` is not provided, Trains-Agent Services will use the external 
-   public address of the **trains-server**. If `TRAINS_AGENT_GIT_USER` / `TRAINS_AGENT_GIT_PASS` are not provided, 
-   the Trains-Agent Services will not be able to access any private repositories for running service tasks.
+1. Configure the ClearML-Agent Services (not supported on Windows installation). 
+   If `CLEARML_HOST_IP` is not provided, ClearML-Agent Services will use the external 
+   public address of the **ClearML Server**. If `CLEARML_AGENT_GIT_USER` / `CLEARML_AGENT_GIT_PASS` are not provided, 
+   the ClearML-Agent Services will not be able to access any private repositories for running service tasks.
    
    ```bash
-   export TRAINS_HOST_IP=server_host_ip_here
-   export TRAINS_AGENT_GIT_USER=git_username_here
-   export TRAINS_AGENT_GIT_PASS=git_password_here
+   export CLEARML_HOST_IP=server_host_ip_here
+   export CLEARML_AGENT_GIT_USER=git_username_here
+   export CLEARML_AGENT_GIT_PASS=git_password_here
    ```
 
-1. Spin up the docker containers, it will automatically pull the latest **trains-server** build    
+1. Spin up the docker containers, it will automatically pull the latest **ClearML Server** build    
    ```bash
    docker-compose -f docker-compose.yml pull
    docker-compose -f docker-compose.yml up
    ```
 
-**\* If something went wrong along the way, check our FAQ: [Common Docker Upgrade Errors](https://allegro.ai/docs/faq/faq/#common-docker-upgrade-errors).**
+**\* If something went wrong along the way, check our FAQ: [Common Docker Upgrade Errors](https://clear.ml/docs/latest/docs/faq/).**
 
 
 ## Community & Support
 
-If you have any questions, look to the Trains [FAQ](https://allegro.ai/docs/faq/faq/), or
-tag your questions on [stackoverflow](https://stackoverflow.com/questions/tagged/trains) with '**trains**' tag.
+If you have any questions, look to the ClearML [FAQ](https://clear.ml/docs/latest/docs/faq), or
+tag your questions on [stackoverflow](https://stackoverflow.com/questions/tagged/clearml) with '**clearml**' tag.
 
-For feature requests or bug reports, please use [GitHub issues](https://github.com/allegroai/trains-server/issues).
+For feature requests or bug reports, please use [GitHub issues](https://github.com/allegroai/clearml-server/issues).
 
-Additionally, you can always find us at *trains@allegro.ai*
+Additionally, you can always find us at *clearml@allegro.ai*
 
 ## License
 
 [Server Side Public License v1.0](https://github.com/mongodb/mongo/blob/master/LICENSE-Community.txt)
 
-**trains-server** relies on both [MongoDB](https://github.com/mongodb/mongo) and [ElasticSearch](https://github.com/elastic/elasticsearch).
+The **ClearML Server** relies on both [MongoDB](https://github.com/mongodb/mongo) and [ElasticSearch](https://github.com/elastic/elasticsearch).
 With the recent changes in both MongoDB's and ElasticSearch's OSS license, we feel it is our responsibility as a
 member of the community to support the projects we love and cherish.
 We believe the cause for the license change in both cases is more than just,

apiserver/apierrors/__init__.py

@@ -0,0 +1,6 @@
+from .apierror import APIError
+from .base import BaseError
+
+from apiserver.apierrors_generator import ErrorsGenerator
+
+ErrorsGenerator.generate_python_files()

apiserver/apierrors/apierror.py

@@ -1,9 +1,10 @@
 class APIError(Exception):
-    def __init__(self, msg, code=500, subcode=0, **_):
+    def __init__(self, msg, code=500, subcode=0, error_data=None, **_):
         super(APIError, self).__init__()
         self._msg = msg
         self._code = code
         self._subcode = subcode
+        self._error_data = error_data or {}
 
     @property
     def msg(self):
@@ -17,5 +18,9 @@ class APIError(Exception):
     def subcode(self):
         return self._subcode
 
+    @property
+    def error_data(self):
+        return self._error_data
+
     def __str__(self):
         return self.msg

apiserver/apierrors/base.py

@@ -1,9 +1,13 @@
-import six
-from boltons.typeutils import classproperty
 from typing import Tuple
 
+import six
+from boltons.iterutils import is_collection, remap
+from boltons.typeutils import classproperty
+
 from .apierror import APIError
 
+jsonable_types = (dict, list, tuple, str, int, float, bool, type(None))
+
 
 class BaseError(APIError):
     _default_code = 500
@@ -19,15 +23,26 @@ class BaseError(APIError):
                 f"{k}={self._format_kwarg(v)}" for k, v in kwargs.items()
             )
             message += f": {kwargs_msg}"
-        params = kwargs.copy()
-        params.update(
-            code=self._default_code, subcode=self._default_subcode, msg=message
+
+        super(BaseError, self).__init__(
+            code=self._default_code,
+            subcode=self._default_subcode,
+            msg=message,
+            error_data=self._to_safe_json_types(kwargs),
         )
-        super(BaseError, self).__init__(**params)
+
+    @staticmethod
+    def _to_safe_json_types(data):
+        def visit(_, k, v):
+            if not isinstance(v, jsonable_types):
+                v = str(v)
+            return k, v
+
+        return remap(data, visit=visit)
 
     @staticmethod
     def _format_kwarg(value):
-        if isinstance(value, (tuple, list)):
+        if is_collection(value):
             return f'({", ".join(str(v) for v in value)})'
         elif isinstance(value, six.string_types):
             return value

apiserver/apierrors/errors.conf

@@ -0,0 +1,148 @@
+301 {
+    _: "moved_permanently"
+    1: ["not_supported", "this endpoint is no longer supported for the requested API version"]
+}
+
+400 {
+    _: "bad_request"
+    1: ["not_supported", "endpoint is not supported"]
+    2: ["request_path_has_invalid_version", "request path has invalid version"]
+    5: ["invalid_headers", "invalid headers"]
+    6: ["impersonation_error", "impersonation error"]
+
+    10: ["invalid_id", "invalid object id"]
+    11: ["missing_required_fields", "missing required fields"]
+    12: ["validation_error", "validation error"]
+    13: ["fields_not_allowed_for_role", "fields not allowed for role"]
+    14: ["invalid fields", "fields not defined for object"]
+    15: ["fields_conflict", "conflicting fields"]
+    16: ["fields_value_error", "invalid value for fields"]
+    17: ["batch_contains_no_items", "batch request contains no items"]
+    18: ["batch_validation_error", "batch request validation error"]
+    19: ["invalid_lucene_syntax", "malformed lucene query"]
+    20: ["fields_type_error", "invalid type for fields"]
+    21: ["invalid_regex_error", "malformed regular expression"]
+    22: ["invalid_email_address", "malformed email address"]
+    23: ["invalid_domain_name", "malformed domain name"]
+    24: ["not_public_object", "object is not public"]
+
+    # Auth / Login
+    75: ["invalid_access_key", "access key not found for user"]
+
+    # Tasks
+    100: ["task_error", "general task error"]
+    101: ["invalid_task_id", "invalid task id"]
+    102: ["task_validation_error", "task validation error"]
+    110: ["invalid_task_status", "invalid task status"]
+    111: ["task_not_started", "task not started (invalid task status)"]
+    112: ["task_in_progress", "task in progress (invalid task status)"]
+    113: ["task_published", "task published (invalid task status)"]
+    114: ["task_status_unknown", "task unknown (invalid task status)"]
+    120: ["invalid_task_execution_progress", "invalid task execution progress"]
+    121: ["failed_changing_task_status", "failed changing task status. probably someone changed it before you"]
+    122: ["missing_task_fields", "task is missing expected fields"]
+    123: ["task_cannot_be_deleted", "task cannot be deleted"]
+    125: ["task_has_jobs_running", "task has jobs that haven't completed yet"]
+    126: ["invalid_task_type", "invalid task type for this operations"]
+    127: ["invalid_task_input", "invalid task output"]
+    128: ["invalid_task_output", "invalid task output"]
+    129: ["task_publish_in_progress", "Task publish in progress"]
+    130: ["task_not_found", "task not found"]
+    131: ["events_not_added", "events not added"]
+
+    # Models
+    200: ["model_error", "general task error"]
+    201: ["invalid_model_id", "invalid model id"]
+    202: ["model_not_ready", "model is not ready"]
+    203: ["model_is_ready", "model is ready"]
+    204: ["invalid_model_uri", "invalid model URI"]
+    205: ["model_in_use", "model is used by tasks"]
+    206: ["model_creating_task_exists", "task that created this model exists"]
+
+    # Users
+    300: ["invalid_user", "invalid user"]
+    301: ["invalid_user_id", "invalid user id"]
+    302: ["user_id_exists", "user id already exists"]
+    305: ["invalid_preferences_update", "Malformed key and/or value"]
+
+    # Projects
+    401: ["invalid_project_id", "invalid project id"]
+    402: ["project_has_tasks", "project has associated tasks"]
+    403: ["project_not_found", "project not found"]
+    405: ["project_has_models", "project has associated models"]
+    407: ["invalid_project_name", "invalid project name"]
+    408: ["cannot_update_project_location", "Cannot update project location. Use projects.move instead"]
+    409: ["project_path_exceeds_max", "Project path exceed the maximum allowed depth"]
+    410: ["project_source_and_destination_are_the_same", "Project has the same source and destination paths"]
+    411: ["project_cannot_be_moved_under_itself", "Project can not be moved under itself in the projects hierarchy"]
+    412: ["project_cannot_be_merged_into_its_child", "Project can not be merged into its own child"]
+
+    # Queues
+    701: ["invalid_queue_id", "invalid queue id"]
+    702: ["queue_not_empty", "queue is not empty"]
+    703: ["invalid_queue_or_task_not_queued", "invalid queue id or task not in queue"]
+    704: ["removed_during_reposition", "task was removed by another party during reposition"]
+    705: ["failed_adding_during_reposition", "failed adding task back to queue during reposition"]
+    706: ["task_already_queued", "failed adding task to queue since task is already queued"]
+    707: ["no_default_queue", "no queue is tagged as the default queue for this company"]
+    708: ["multiple_default_queues", "more than one queue is tagged as the default queue for this company"]
+
+    # Database
+    800: ["data_validation_error", "data validation error"]
+    801: ["expected_unique_data", "value combination already exists (unique field already contains this value)"]
+
+    # Workers
+    1001: ["invalid_worker_id", "invalid worker id"]
+    1002: ["worker_registration_failed", "worker registration failed"]
+    1003: ["worker_registered", "worker is already registered"]
+    1004: ["worker_not_registered", "worker is not registered"]
+    1005: ["worker_stats_not_found", "worker stats not found"]
+
+    1104: ["invalid_scroll_id", "Invalid scroll id"]
+}
+
+401 {
+    _:  "unauthorized"
+    1:  ["not_authorized", "unauthorized (not authorized for endpoint)"]
+    2:  ["entity_not_allowed", "unauthorized (entity not allowed)"]
+    10: ["bad_auth_type", "unauthorized (bad authentication header type)"]
+    20: ["no_credentials", "unauthorized (missing credentials)"]
+    21: ["bad_credentials", "unauthorized (malformed credentials)"]
+    22: ["invalid_credentials", "unauthorized (invalid credentials)"]
+    30: ["invalid_token", "invalid token"]
+    31: ["blocked_token", "token is blocked"]
+    40: ["invalid_fixed_user", "fixed user ID was not found"]
+}
+
+403: {
+    _:  "forbidden"
+    10: ["routing_error", "forbidden (routing error)"]
+    12: ["blocked_internal_endpoint", "forbidden (blocked internal endpoint)"]
+    20: ["role_not_allowed", "forbidden (not allowed for role)"]
+    21: ["no_write_permission", "forbidden (modification not allowed)"]
+}
+
+410: {
+    _:  "gone"
+    1: ["not_supported", "thus endpoint is not supported any more"]
+}
+
+500 {
+    _:   "server_error"
+    0:   ["general_error", "general server error"]
+    1:   ["internal_error", "internal server error"]
+    2:   ["config_error", "configuration error"]
+    3:   ["build_info_error", "build info unavailable or corrupted"]
+    4:   ["low_disk_space", "Critical server error! Server reports low or insufficient disk space. Please resolve immediately by allocating additional disk space or freeing up storage space."]
+    10:  ["transaction_error", "a transaction call has returned with an error"]
+    # Database-related issues
+    100: ["data_error", "general data error"]
+    101: ["inconsistent_data", "inconsistent data encountered in document"]
+    102: ["database_unavailable", "database is temporarily unavailable"]
+    110: ["update_failed", "update failed"]
+
+    # Index-related issues
+    201: ["missing_index", "missing internal index"]
+
+    9999: ["not_implemented", "action is not yet implemented"]
+}

apiserver/apierrors_generator/__init__.py

@@ -0,0 +1 @@
+from .errors_generator import ErrorsGenerator

apiserver/apierrors_generator/__main__.py

@@ -0,0 +1,4 @@
+from .errors_generator import ErrorsGenerator
+
+if __name__ == '__main__':
+    ErrorsGenerator.generate_python_files()

apiserver/apierrors_generator/errors_generator.py

@@ -0,0 +1,31 @@
+from functools import reduce
+from pathlib import Path
+from typing import Union
+
+from pyhocon import ConfigFactory, ConfigTree
+
+from .generator import Generator
+
+
+class ErrorsGenerator:
+    _apierrors_path = Path(__file__).parents[1] / "apierrors"
+    _files = [_apierrors_path / "errors.conf"]
+
+    @classmethod
+    def _get_codes(cls):
+        return {
+            (k, v.pop("_")): v
+            for k, v in reduce(
+                ConfigTree.merge_configs, map(ConfigFactory.parse_file, cls._files),
+            ).items()
+        }
+
+    @classmethod
+    def add_errors_file(cls, path: Union[Path, str]):
+        cls._files.append(path)
+
+    @classmethod
+    def generate_python_files(cls):
+        Generator(cls._apierrors_path / "errors", format_pep8=False).make_errors(
+            cls._get_codes()
+        )

apiserver/apierrors_generator/generator.py

@@ -8,9 +8,12 @@ from pathlib import Path
 
 env = jinja2.Environment(
     loader=jinja2.FileSystemLoader(str(Path(__file__).parent)),
-    autoescape=jinja2.select_autoescape(disabled_extensions=('py',), default_for_string=False),
+    autoescape=jinja2.select_autoescape(
+        disabled_extensions=("py",), default_for_string=False
+    ),
     trim_blocks=True,
-    lstrip_blocks=True)
+    lstrip_blocks=True,
+)
 
 
 def env_filter(name=None):
@@ -19,14 +22,14 @@ def env_filter(name=None):
 
 @env_filter()
 def cls_name(name):
-    delims = list(map(re.escape, (' ', '_')))
-    parts = re.split('|'.join(delims), name)
-    return ''.join(x.capitalize() for x in parts)
+    delims = list(map(re.escape, (" ", "_")))
+    parts = re.split("|".join(delims), name)
+    return "".join(x.capitalize() for x in parts)
 
 
 class Generator(object):
-    _base_class_name = 'BaseError'
-    _base_class_module = 'apierrors.base'
+    _base_class_name = "BaseError"
+    _base_class_module = "apiserver.apierrors.base"
 
     def __init__(self, path, format_pep8=True, use_md5=True):
         self._use_md5 = use_md5
@@ -35,29 +38,37 @@ class Generator(object):
         self._path.mkdir(parents=True, exist_ok=True)
 
     def _make_init_file(self, path):
-        (self._path / path / '__init__.py').write_bytes('')
+        (self._path / path / "__init__.py").write_bytes(b"")
 
     def _do_render(self, file, template, context):
-        with file.open('w') as f:
+        with file.open("w") as f:
             result = template.render(
                 base_class_name=self._base_class_name,
                 base_class_module=self._base_class_module,
-                **context)
+                **context
+            )
             if self._format_pep8:
-                result = autopep8.fix_code(result, options={'aggressive': 1, 'verbose': 0, 'max_line_length': 120})
+                import autopep8
+
+                result = autopep8.fix_code(
+                    result,
+                    options={"aggressive": 1, "verbose": 0, "max_line_length": 120},
+                )
             f.write(result)
 
     def _make_section(self, name, code, subcodes):
         self._do_render(
-            file=(self._path / name).with_suffix('.py'),
-            template=env.get_template('templates/section.jinja2'),
-            context=dict(code=code, subcodes=list(subcodes.items()),))
+            file=(self._path / name).with_suffix(".py"),
+            template=env.get_template("templates/section.jinja2"),
+            context=dict(code=code, subcodes=list(subcodes.items()),),
+        )
 
     def _make_init(self, sections):
         self._do_render(
-            file=(self._path / '__init__.py'),
-            template=env.get_template('templates/init.jinja2'),
-            context=dict(sections=sections,))
+            file=(self._path / "__init__.py"),
+            template=env.get_template("templates/init.jinja2"),
+            context=dict(sections=sections,),
+        )
 
     def _key_to_str(self, data):
         if isinstance(data, dict):
@@ -66,11 +77,11 @@ class Generator(object):
 
     def _calc_digest(self, data):
         data = json.dumps(self._key_to_str(data), sort_keys=True)
-        return hashlib.md5(data.encode('utf8')).hexdigest()
+        return hashlib.md5(data.encode("utf8")).hexdigest()
 
     def make_errors(self, errors):
         digest = None
-        digest_file = self._path / 'digest.md5'
+        digest_file = self._path / "digest.md5"
         if self._use_md5:
             digest = self._calc_digest(errors)
             if digest_file.is_file():
@@ -79,7 +90,7 @@ class Generator(object):
 
         self._make_init(errors)
         for (code, section_name), subcodes in errors.items():
-            self._make_section(section_name, code, subcodes)
+            self._make_section(section_name, int(code), subcodes)
 
         if self._use_md5:
             digest_file.write_text(digest)

apiserver/apierrors_generator/templates/section.jinja2

@@ -5,5 +5,5 @@ from {{ base_class_module }} import {{ base_class_name }}
 {% for subcode, (name, msg) in subcodes %}
 
 
-{{ error_class(name|cls_name, msg, code, subcode) -}}
+{{ error_class(name|cls_name, msg, code, subcode|int) -}}
 {% endfor %}

apiserver/apimodels/__init__.py

@@ -1,5 +1,3 @@
-from __future__ import absolute_import
-
 from enum import Enum
 from typing import Union, Type, Iterable
 
@@ -9,11 +7,29 @@ from jsonmodels import fields
 from jsonmodels.fields import _LazyType, NotSet
 from jsonmodels.models import Base as ModelBase
 from jsonmodels.validators import Enum as EnumValidator
-from luqum.parser import parser, ParseError
+from mongoengine.base import BaseDocument
 from validators import email as email_validator, domain as domain_validator
 
-from apierrors import errors
-from utilities.json import loads, dumps
+from apiserver.apierrors import errors
+from apiserver.utilities.json import loads, dumps
+
+
+class EmailField(fields.StringField):
+    def validate(self, value):
+        super().validate(value)
+        if value is None:
+            return
+        if email_validator(value) is not True:
+            raise errors.bad_request.InvalidEmailAddress()
+
+
+class DomainField(fields.StringField):
+    def validate(self, value):
+        super().validate(value)
+        if value is None:
+            return
+        if domain_validator(value) is not True:
+            raise errors.bad_request.InvalidDomainName()
 
 
 def make_default(field_cls, default_value):
@@ -35,6 +51,8 @@ class ListField(fields.ListField):
         try:
             return super(ListField, self)._cast_value(value)
         except TypeError:
+            if len(self.items_types) == 1 and issubclass(self.items_types[0], Enum):
+                return self.items_types[0](value)
             return value
 
     def validate_single_value(self, item):
@@ -43,6 +61,12 @@ class ListField(fields.ListField):
             item.validate()
 
 
+# since there is no distinction between None and empty DictField
+# this value can be used as sentinel in order to distinguish
+# between not set and empty DictField
+DictFieldNotSet = {}
+
+
 class DictField(fields.BaseField):
     types = (dict,)
 
@@ -71,6 +95,31 @@ class DictField(fields.BaseField):
             for type_ in value_types
         )
 
+    def parse_value(self, values):
+        """Cast value to proper collection."""
+        result = self.get_default_value()
+
+        if values is None:
+            return result
+
+        if not self.value_types or not isinstance(values, dict):
+            return values
+
+        return {key: self._cast_value(value) for key, value in values.items()}
+
+    def _cast_value(self, value):
+        if isinstance(value, self.value_types):
+            return value
+        else:
+            if len(self.value_types) != 1:
+                tpl = 'Cannot decide which type to choose from "{types}".'
+                raise jsonmodels.errors.ValidationError(
+                    tpl.format(
+                        types=', '.join([t.__name__ for t in self.value_types])
+                    )
+                )
+            return self.value_types[0](**value)
+
     def validate(self, value):
         super(DictField, self).validate(value)
 
@@ -96,6 +145,15 @@ class DictField(fields.BaseField):
                 )
             )
 
+    def _elem_to_struct(self, value):
+        try:
+            return value.to_struct()
+        except AttributeError:
+            return value
+
+    def to_struct(self, values):
+        return {k: self._elem_to_struct(v) for k, v in values.items()}
+
 
 class IntField(fields.IntField):
     def parse_value(self, value):
@@ -105,23 +163,6 @@ class IntField(fields.IntField):
             return value
 
 
-def validate_lucene_query(value):
-    if value == "":
-        return
-    try:
-        parser.parse(value)
-    except ParseError as e:
-        raise errors.bad_request.InvalidLuceneSyntax(error=e)
-
-
-class LuceneQueryField(fields.StringField):
-    def validate(self, value):
-        super(LuceneQueryField, self).validate(value)
-        if value is None:
-            return
-        validate_lucene_query(value)
-
-
 class NullableEnumValidator(EnumValidator):
     """Validator for enums that allows a None value."""
 
@@ -177,7 +218,7 @@ class ActualEnumField(fields.StringField):
         )
 
     def parse_value(self, value):
-        if value is None and not self.required:
+        if value is NotSet and not self.required:
             return self.get_default_value()
         try:
             # noinspection PyArgumentList
@@ -189,24 +230,6 @@ class ActualEnumField(fields.StringField):
         return super().to_struct(value.value)
 
 
-class EmailField(fields.StringField):
-    def validate(self, value):
-        super().validate(value)
-        if value is None:
-            return
-        if email_validator(value) is not True:
-            raise errors.bad_request.InvalidEmailAddress()
-
-
-class DomainField(fields.StringField):
-    def validate(self, value):
-        super().validate(value)
-        if value is None:
-            return
-        if domain_validator(value) is not True:
-            raise errors.bad_request.InvalidDomainName()
-
-
 class JsonSerializableMixin:
     def to_json(self: ModelBase):
         return dumps(self.to_struct())
@@ -214,3 +237,67 @@ class JsonSerializableMixin:
     @classmethod
     def from_json(cls: Type[ModelBase], s):
         return cls(**loads(s))
+
+
+def callable_default(cls: Type[fields.BaseField]) -> Type[fields.BaseField]:
+    class _Wrapped(cls):
+        _callable_default = None
+
+        def get_default_value(self):
+            if self._callable_default:
+                return self._callable_default()
+            return super(_Wrapped, self).get_default_value()
+
+        def __init__(self, *args, default=None, **kwargs):
+            if default and callable(default):
+                self._callable_default = default
+                default = default()
+            super(_Wrapped, self).__init__(*args, default=default, **kwargs)
+
+    return _Wrapped
+
+
+class MongoengineFieldsDict(DictField):
+    """
+    DictField representing mongoengine field names/value mapping.
+    Used to convert mongoengine-style field/subfield notation to user-presentable syntax, including handling update
+        operators.
+    """
+
+    mongoengine_update_operators = (
+        "inc",
+        "dec",
+        "push",
+        "push_all",
+        "pop",
+        "pull",
+        "pull_all",
+        "add_to_set",
+    )
+
+    @staticmethod
+    def _normalize_mongo_value(value):
+        if isinstance(value, BaseDocument):
+            return value.to_mongo()
+        return value
+
+    @classmethod
+    def _normalize_mongo_field_path(cls, path, value):
+        parts = path.split("__")
+        if len(parts) > 1:
+            if parts[0] == "set":
+                parts = parts[1:]
+            elif parts[0] == "unset":
+                parts = parts[1:]
+                value = None
+            elif parts[0] in cls.mongoengine_update_operators:
+                return None, None
+        return ".".join(parts), cls._normalize_mongo_value(value)
+
+    def parse_value(self, value):
+        value = super(MongoengineFieldsDict, self).parse_value(value)
+        return {
+            k: v
+            for k, v in (self._normalize_mongo_field_path(*p) for p in value.items())
+            if k is not None
+        }

apiserver/apimodels/auth.py

@@ -2,10 +2,10 @@ from jsonmodels.fields import IntField, StringField, BoolField, EmbeddedField, D
 from jsonmodels.models import Base
 from jsonmodels.validators import Max, Enum
 
-from apimodels import ListField, EnumField
-from config import config
-from database.model.auth import Role
-from database.utils import get_options
+from apiserver.apimodels import ListField, EnumField
+from apiserver.config_repo import config
+from apiserver.database.model.auth import Role
+from apiserver.database.utils import get_options
 
 
 class GetTokenRequest(Base):
@@ -75,11 +75,17 @@ class CreateUserResponse(Base):
 class Credentials(Base):
     access_key = StringField(required=True)
     secret_key = StringField(required=True)
+    label = StringField()
 
 
 class CredentialsResponse(Credentials):
     secret_key = StringField()
     last_used = DateTimeField(default=None)
+    last_used_from = StringField()
+
+
+class CreateCredentialsRequest(Base):
+    label = StringField()
 
 
 class CreateCredentialsResponse(Base):
@@ -90,6 +96,11 @@ class GetCredentialsResponse(Base):
     credentials = ListField(CredentialsResponse)
 
 
+class EditCredentialsRequest(Base):
+    access_key = StringField(required=True)
+    label = StringField()
+
+
 class RevokeCredentialsRequest(Base):
     access_key = StringField(required=True)
 

apiserver/apimodels/base.py

@@ -0,0 +1,28 @@
+from jsonmodels import models, fields
+from jsonmodels.validators import Length
+
+from apiserver.apimodels import MongoengineFieldsDict, ListField
+
+
+class UpdateResponse(models.Base):
+    updated = fields.IntField(required=True)
+    fields = MongoengineFieldsDict()
+
+
+class PagedRequest(models.Base):
+    page = fields.IntField()
+    page_size = fields.IntField()
+
+
+class IdResponse(models.Base):
+    id = fields.StringField(required=True)
+
+
+class MakePublicRequest(models.Base):
+    ids = ListField(items_types=str, validators=[Length(minimum_value=1)])
+
+
+class MoveRequest(models.Base):
+    ids = ListField([str], validators=Length(minimum_value=1))
+    project = fields.StringField()
+    project_name = fields.StringField()

apiserver/apimodels/batch.py

@@ -0,0 +1,25 @@
+from typing import Sequence
+
+from jsonmodels.fields import StringField
+from jsonmodels.models import Base
+from jsonmodels.validators import Length
+
+from apiserver.apimodels import ListField
+from apiserver.apimodels.base import UpdateResponse
+
+
+class BatchRequest(Base):
+    ids: Sequence[str] = ListField([str], validators=Length(minimum_value=1))
+
+
+class BatchResponse(Base):
+    succeeded: Sequence[dict] = ListField([dict])
+    failed: Sequence[dict] = ListField([dict])
+
+
+class UpdateBatchItem(UpdateResponse):
+    id: str = StringField()
+
+
+class UpdateBatchResponse(BatchResponse):
+    succeeded: Sequence[UpdateBatchItem] = ListField(UpdateBatchItem)

apiserver/apimodels/custom_validators/__init__.py

@@ -0,0 +1,34 @@
+import validators
+from jsonmodels.errors import ValidationError
+
+
+class ForEach(object):
+    def __init__(self, validator):
+        self.validator = validator
+
+    def validate(self, values):
+        for value in values:
+            self.validator.validate(value)
+
+    def modify_schema(self, field_schema):
+        return self.validator.modify_schema(field_schema)
+
+
+class Hostname(object):
+
+    def validate(self, value):
+        if validators.domain(value) is not True:
+            raise ValidationError(f"Value '{value}' is not a valid hostname")
+
+    def modify_schema(self, field_schema):
+        field_schema["format"] = "hostname"
+
+
+class Email(object):
+
+    def validate(self, value):
+        if validators.email(value) is not True:
+            raise ValidationError(f"Value '{value}' is not a valid email address")
+
+    def modify_schema(self, field_schema):
+        field_schema["format"] = "email"

apiserver/apimodels/events.py

@@ -0,0 +1,157 @@
+from enum import auto
+from typing import Sequence, Optional
+
+from jsonmodels import validators
+from jsonmodels.fields import StringField, BoolField, EmbeddedField
+from jsonmodels.models import Base
+from jsonmodels.validators import Length, Min, Max
+
+from apiserver.apimodels import ListField, IntField, ActualEnumField
+from apiserver.bll.event.event_common import EventType
+from apiserver.bll.event.scalar_key import ScalarKeyEnum
+from apiserver.config_repo import config
+from apiserver.utilities.stringenum import StringEnum
+
+
+class HistogramRequestBase(Base):
+    samples: int = IntField(default=2000, validators=[Min(1), Max(6000)])
+    key: ScalarKeyEnum = ActualEnumField(ScalarKeyEnum, default=ScalarKeyEnum.iter)
+
+
+class MetricVariants(Base):
+    metric: str = StringField(required=True)
+    variants: Sequence[str] = ListField(items_types=str)
+
+
+class ScalarMetricsIterHistogramRequest(HistogramRequestBase):
+    task: str = StringField(required=True)
+    metrics: Sequence[MetricVariants] = ListField(items_types=MetricVariants)
+
+
+class MultiTaskScalarMetricsIterHistogramRequest(HistogramRequestBase):
+    tasks: Sequence[str] = ListField(
+        items_types=str,
+        validators=[
+            Length(
+                minimum_value=1,
+                maximum_value=config.get(
+                    "services.tasks.multi_task_histogram_limit", 10
+                ),
+            )
+        ],
+    )
+
+
+class TaskMetric(Base):
+    task: str = StringField(required=True)
+    metric: str = StringField(default=None)
+    variants: Sequence[str] = ListField(items_types=str)
+
+
+class MetricEventsRequest(Base):
+    metrics: Sequence[TaskMetric] = ListField(
+        items_types=TaskMetric, validators=[Length(minimum_value=1)]
+    )
+    iters: int = IntField(default=1, validators=validators.Min(1))
+    navigate_earlier: bool = BoolField(default=True)
+    refresh: bool = BoolField(default=False)
+    scroll_id: str = StringField()
+
+
+class TaskMetricVariant(Base):
+    task: str = StringField(required=True)
+    metric: str = StringField(required=True)
+    variant: str = StringField(required=True)
+
+
+class GetHistorySampleRequest(TaskMetricVariant):
+    iteration: Optional[int] = IntField()
+    refresh: bool = BoolField(default=False)
+    scroll_id: Optional[str] = StringField()
+    navigate_current_metric: bool = BoolField(default=True)
+
+
+class NextHistorySampleRequest(Base):
+    task: str = StringField(required=True)
+    scroll_id: Optional[str] = StringField()
+    navigate_earlier: bool = BoolField(default=True)
+
+
+class LogOrderEnum(StringEnum):
+    asc = auto()
+    desc = auto()
+
+
+class TaskEventsRequestBase(Base):
+    task: str = StringField(required=True)
+    batch_size: int = IntField(default=500)
+
+
+class TaskEventsRequest(TaskEventsRequestBase):
+    metrics: Sequence[MetricVariants] = ListField(items_types=MetricVariants)
+    event_type: EventType = ActualEnumField(EventType, default=EventType.all)
+    order: Optional[str] = ActualEnumField(LogOrderEnum, default=LogOrderEnum.asc)
+    scroll_id: str = StringField()
+    count_total: bool = BoolField(default=True)
+
+
+class LogEventsRequest(TaskEventsRequestBase):
+    batch_size: int = IntField(default=5000)
+    navigate_earlier: bool = BoolField(default=True)
+    from_timestamp: Optional[int] = IntField()
+    order: Optional[str] = ActualEnumField(LogOrderEnum)
+
+
+class ScalarMetricsIterRawRequest(TaskEventsRequestBase):
+    batch_size: int = IntField()
+    key: ScalarKeyEnum = ActualEnumField(ScalarKeyEnum, default=ScalarKeyEnum.iter)
+    metric: MetricVariants = EmbeddedField(MetricVariants, required=True)
+    count_total: bool = BoolField(default=False)
+    scroll_id: str = StringField()
+
+
+class IterationEvents(Base):
+    iter: int = IntField()
+    events: Sequence[dict] = ListField(items_types=dict)
+
+
+class MetricEvents(Base):
+    task: str = StringField()
+    iterations: Sequence[IterationEvents] = ListField(items_types=IterationEvents)
+
+
+class MetricEventsResponse(Base):
+    metrics: Sequence[MetricEvents] = ListField(items_types=MetricEvents)
+    scroll_id: str = StringField()
+
+
+class MultiTasksRequestBase(Base):
+    tasks: Sequence[str] = ListField(
+        items_types=str, validators=[Length(minimum_value=1)]
+    )
+
+
+class SingleValueMetricsRequest(MultiTasksRequestBase):
+    pass
+
+
+class TaskMetricsRequest(MultiTasksRequestBase):
+    event_type: EventType = ActualEnumField(EventType, required=True)
+
+
+class TaskPlotsRequest(Base):
+    task: str = StringField(required=True)
+    iters: int = IntField(default=1)
+    scroll_id: str = StringField()
+    no_scroll: bool = BoolField(default=False)
+    metrics: Sequence[MetricVariants] = ListField(items_types=MetricVariants)
+
+
+class ClearScrollRequest(Base):
+    scroll_id: str = StringField()
+
+
+class ClearTaskLogRequest(Base):
+    task: str = StringField(required=True)
+    threshold_sec = IntField()
+    allow_locked = BoolField(default=False)

apiserver/apimodels/login.py

@@ -0,0 +1,34 @@
+from jsonmodels.fields import StringField, BoolField, EmbeddedField, ListField
+from jsonmodels.models import Base
+
+from apiserver.apimodels import DictField, callable_default
+
+
+class GetSupportedModesRequest(Base):
+    state = StringField(help_text="ASCII base64 encoded application state")
+    callback_url_prefix = StringField()
+
+
+class BasicGuestMode(Base):
+    enabled = BoolField(default=False)
+    name = StringField()
+    username = StringField()
+    password = StringField()
+
+
+class BasicMode(Base):
+    enabled = BoolField(default=False)
+    guest = callable_default(EmbeddedField)(BasicGuestMode, default=BasicGuestMode)
+
+
+class ServerErrors(Base):
+    missed_es_upgrade = BoolField(default=False)
+    es_connection_error = BoolField(default=False)
+
+
+class GetSupportedModesResponse(Base):
+    basic = EmbeddedField(BasicMode)
+    server_errors = EmbeddedField(ServerErrors)
+    sso = DictField([str, type(None)])
+    sso_providers = ListField([dict])
+    authenticated = BoolField(default=False)

apiserver/apimodels/metadata.py

@@ -0,0 +1,24 @@
+from typing import Sequence
+
+from jsonmodels import validators
+from jsonmodels.fields import StringField, BoolField
+from jsonmodels.models import Base
+
+from apiserver.apimodels import ListField
+
+
+class MetadataItem(Base):
+    key = StringField(required=True)
+    type = StringField(required=True)
+    value = StringField(required=True)
+
+
+class DeleteMetadata(Base):
+    keys: Sequence[str] = ListField(str, validators=validators.Length(minimum_value=1))
+
+
+class AddOrUpdateMetadata(Base):
+    metadata: Sequence[MetadataItem] = ListField(
+        [MetadataItem], validators=validators.Length(minimum_value=1)
+    )
+    replace_metadata = BoolField(default=False)

apiserver/apimodels/models.py

@@ -1,9 +1,14 @@
 from jsonmodels import models, fields
 from six import string_types
 
-from apimodels import ListField, DictField
-from apimodels.base import UpdateResponse
-from apimodels.tasks import PublishResponse as TaskPublishResponse
+from apiserver.apimodels import ListField, DictField
+from apiserver.apimodels.base import UpdateResponse
+from apiserver.apimodels.batch import BatchRequest
+from apiserver.apimodels.metadata import (
+    MetadataItem,
+    DeleteMetadata,
+    AddOrUpdateMetadata,
+)
 
 
 class GetFrameworksRequest(models.Base):
@@ -13,7 +18,7 @@ class GetFrameworksRequest(models.Base):
 class CreateModelRequest(models.Base):
     name = fields.StringField(required=True)
     uri = fields.StringField(required=True)
-    labels = DictField(value_types=string_types+(int,))
+    labels = DictField(value_types=string_types + (int,))
     tags = ListField(items_types=string_types)
     system_tags = ListField(items_types=string_types)
     comment = fields.StringField()
@@ -25,6 +30,7 @@ class CreateModelRequest(models.Base):
     ready = fields.BoolField(default=True)
     ui_cache = DictField()
     task = fields.StringField()
+    metadata = DictField(value_types=[MetadataItem])
 
 
 class CreateModelResponse(models.Base):
@@ -32,17 +38,44 @@ class CreateModelResponse(models.Base):
     created = fields.BoolField(required=True)
 
 
-class PublishModelRequest(models.Base):
+class ModelRequest(models.Base):
     model = fields.StringField(required=True)
+
+
+class DeleteModelRequest(ModelRequest):
+    force = fields.BoolField(default=False)
+
+
+class ModelsDeleteManyRequest(BatchRequest):
+    force = fields.BoolField(default=False)
+
+
+class PublishModelRequest(ModelRequest):
     force_publish_task = fields.BoolField(default=False)
     publish_task = fields.BoolField(default=True)
 
 
 class ModelTaskPublishResponse(models.Base):
     id = fields.StringField(required=True)
-    data = fields.EmbeddedField(TaskPublishResponse)
+    data = fields.EmbeddedField(UpdateResponse)
 
 
 class PublishModelResponse(UpdateResponse):
     published_task = fields.EmbeddedField(ModelTaskPublishResponse)
-    updated = fields.IntField()
+
+
+class ModelsPublishManyRequest(BatchRequest):
+    force_publish_task = fields.BoolField(default=False)
+    publish_task = fields.BoolField(default=True)
+
+
+class DeleteMetadataRequest(DeleteMetadata):
+    model = fields.StringField(required=True)
+
+
+class AddOrUpdateMetadataRequest(AddOrUpdateMetadata):
+    model = fields.StringField(required=True)
+
+
+class ModelsGetRequest(models.Base):
+    include_stats = fields.BoolField(default=False)

apiserver/apimodels/organization.py

@@ -1,5 +1,7 @@
 from jsonmodels import fields, models
 
+from apiserver.apimodels import DictField
+
 
 class Filter(models.Base):
     tags = fields.ListField([str])
@@ -9,3 +11,11 @@ class Filter(models.Base):
 class TagsRequest(models.Base):
     include_system = fields.BoolField(default=False)
     filter = fields.EmbeddedField(Filter)
+
+
+class EntitiesCountRequest(models.Base):
+    projects = DictField()
+    tasks = DictField()
+    models = DictField()
+    pipelines = DictField()
+    datasets = DictField()

apiserver/apimodels/pipelines.py

@@ -0,0 +1,19 @@
+from jsonmodels import models, fields
+
+from apiserver.apimodels import ListField
+
+
+class Arg(models.Base):
+    name = fields.StringField(required=True)
+    value = fields.StringField(required=True)
+
+
+class StartPipelineRequest(models.Base):
+    task = fields.StringField(required=True)
+    queue = fields.StringField(required=True)
+    args = ListField(Arg)
+
+
+class StartPipelineResponse(models.Base):
+    pipeline = fields.StringField(required=True)
+    enqueued = fields.BoolField(required=True)

apiserver/apimodels/projects.py

@@ -0,0 +1,69 @@
+from jsonmodels import models, fields
+
+from apiserver.apimodels import ListField, ActualEnumField, DictField
+from apiserver.apimodels.organization import TagsRequest
+from apiserver.database.model import EntityVisibility
+
+
+class ProjectRequest(models.Base):
+    project = fields.StringField(required=True)
+
+
+class MergeRequest(ProjectRequest):
+    destination_project = fields.StringField()
+
+
+class MoveRequest(ProjectRequest):
+    new_location = fields.StringField()
+
+
+class DeleteRequest(ProjectRequest):
+    force = fields.BoolField(default=False)
+    delete_contents = fields.BoolField(default=False)
+
+
+class ProjectOrNoneRequest(models.Base):
+    project = fields.StringField()
+    include_subprojects = fields.BoolField(default=True)
+
+
+class GetParamsRequest(ProjectOrNoneRequest):
+    page = fields.IntField(default=0)
+    page_size = fields.IntField(default=500)
+
+
+class ProjectTagsRequest(TagsRequest):
+    projects = ListField(str)
+
+
+class MultiProjectRequest(models.Base):
+    projects = fields.ListField(str)
+    include_subprojects = fields.BoolField(default=True)
+
+
+class ProjectTaskParentsRequest(MultiProjectRequest):
+    tasks_state = ActualEnumField(EntityVisibility)
+
+
+class ProjectHyperparamValuesRequest(MultiProjectRequest):
+    section = fields.StringField(required=True)
+    name = fields.StringField(required=True)
+    allow_public = fields.BoolField(default=True)
+
+
+class ProjectModelMetadataValuesRequest(MultiProjectRequest):
+    key = fields.StringField(required=True)
+    allow_public = fields.BoolField(default=True)
+
+
+class ProjectsGetRequest(models.Base):
+    include_dataset_stats = fields.BoolField(default=False)
+    include_stats = fields.BoolField(default=False)
+    include_stats_filter = DictField()
+    stats_with_children = fields.BoolField(default=True)
+    stats_for_state = ActualEnumField(EntityVisibility, default=EntityVisibility.active)
+    non_public = fields.BoolField(default=False)
+    active_users = fields.ListField(str)
+    check_own_contents = fields.BoolField(default=False)
+    shallow_search = fields.BoolField(default=False)
+    search_hidden = fields.BoolField(default=False)

apiserver/apimodels/queues.py

@@ -2,7 +2,12 @@ from jsonmodels import validators
 from jsonmodels.fields import StringField, IntField, BoolField, FloatField
 from jsonmodels.models import Base
 
-from apimodels import ListField
+from apiserver.apimodels import ListField, DictField
+from apiserver.apimodels.metadata import (
+    MetadataItem,
+    DeleteMetadata,
+    AddOrUpdateMetadata,
+)
 
 
 class GetDefaultResp(Base):
@@ -14,12 +19,22 @@ class CreateRequest(Base):
     name = StringField(required=True)
     tags = ListField(items_types=[str])
     system_tags = ListField(items_types=[str])
+    metadata = DictField(value_types=[MetadataItem])
 
 
 class QueueRequest(Base):
     queue = StringField(required=True)
 
 
+class GetByIdRequest(QueueRequest):
+    max_task_entries = IntField()
+
+
+class GetNextTaskRequest(QueueRequest):
+    queue = StringField(required=True)
+    get_task_info = BoolField(default=False)
+
+
 class DeleteRequest(QueueRequest):
     force = BoolField(default=False)
 
@@ -28,6 +43,7 @@ class UpdateRequest(QueueRequest):
     name = StringField()
     tags = ListField(items_types=[str])
     system_tags = ListField(items_types=[str])
+    metadata = DictField(value_types=[MetadataItem])
 
 
 class TaskRequest(QueueRequest):
@@ -47,6 +63,7 @@ class GetMetricsRequest(Base):
     from_date = FloatField(required=True, validators=validators.Min(0))
     to_date = FloatField(required=True, validators=validators.Min(0))
     interval = IntField(required=True, validators=validators.Min(1))
+    refresh = BoolField(default=False)
 
 
 class QueueMetrics(Base):
@@ -58,3 +75,11 @@ class QueueMetrics(Base):
 
 class GetMetricsResponse(Base):
     queues = ListField(QueueMetrics)
+
+
+class DeleteMetadataRequest(DeleteMetadata):
+    queue = StringField(required=True)
+
+
+class AddOrUpdateMetadataRequest(AddOrUpdateMetadata):
+    queue = StringField(required=True)

apiserver/apimodels/tasks.py

@@ -0,0 +1,312 @@
+from typing import Sequence
+
+from jsonmodels import models
+from jsonmodels.fields import StringField, BoolField, IntField, EmbeddedField
+from jsonmodels.validators import Enum, Length
+
+from apiserver.apimodels import DictField, ListField
+from apiserver.apimodels.base import UpdateResponse
+from apiserver.apimodels.batch import BatchRequest, UpdateBatchItem, BatchResponse
+from apiserver.database.model.task.task import (
+    TaskType,
+    ArtifactModes,
+    DEFAULT_ARTIFACT_MODE,
+    TaskModelTypes,
+)
+from apiserver.database.utils import get_options
+
+
+class ArtifactTypeData(models.Base):
+    preview = StringField()
+    content_type = StringField()
+    data_hash = StringField()
+
+
+class Artifact(models.Base):
+    key = StringField(required=True)
+    type = StringField(required=True)
+    mode = StringField(
+        validators=Enum(*get_options(ArtifactModes)), default=DEFAULT_ARTIFACT_MODE
+    )
+    uri = StringField()
+    hash = StringField()
+    content_size = IntField()
+    timestamp = IntField()
+    type_data = EmbeddedField(ArtifactTypeData)
+    display_data = ListField([list])
+
+
+class StartedResponse(UpdateResponse):
+    started = IntField()
+
+
+class EnqueueResponse(UpdateResponse):
+    queued = IntField()
+
+
+class EnqueueBatchItem(UpdateBatchItem):
+    queued: bool = BoolField()
+
+
+class EnqueueManyResponse(BatchResponse):
+    succeeded: Sequence[EnqueueBatchItem] = ListField(EnqueueBatchItem)
+
+
+class DequeueResponse(UpdateResponse):
+    dequeued = IntField()
+
+
+class DequeueBatchItem(UpdateBatchItem):
+    dequeued: bool = BoolField()
+
+
+class DequeueManyResponse(BatchResponse):
+    succeeded: Sequence[DequeueBatchItem] = ListField(DequeueBatchItem)
+
+
+class ResetResponse(UpdateResponse):
+    dequeued = DictField()
+    events = DictField()
+    deleted_models = IntField()
+    urls = DictField()
+
+
+class ResetBatchItem(UpdateBatchItem):
+    dequeued: bool = BoolField()
+    deleted_models = IntField()
+    urls = DictField()
+
+
+class ResetManyResponse(BatchResponse):
+    succeeded: Sequence[ResetBatchItem] = ListField(ResetBatchItem)
+
+
+class TaskRequest(models.Base):
+    task = StringField(required=True)
+
+
+class TaskUpdateRequest(TaskRequest):
+    force = BoolField(default=False)
+
+
+class UpdateRequest(TaskUpdateRequest):
+    status_reason = StringField(default="")
+    status_message = StringField(default="")
+
+
+class EnqueueRequest(UpdateRequest):
+    queue = StringField()
+    queue_name = StringField()
+
+
+class DeleteRequest(UpdateRequest):
+    move_to_trash = BoolField(default=True)
+    return_file_urls = BoolField(default=False)
+    delete_output_models = BoolField(default=True)
+
+
+class SetRequirementsRequest(TaskRequest):
+    requirements = DictField(required=True)
+
+
+class CompletedRequest(UpdateRequest):
+    publish = BoolField(default=False)
+
+
+class CompletedResponse(UpdateResponse):
+    published = IntField(default=0)
+
+
+class PublishRequest(UpdateRequest):
+    publish_model = BoolField(default=True)
+
+
+class TaskData(models.Base):
+    """
+    This is a partial description of task can be updated incrementally
+    """
+
+
+class CreateRequest(TaskData):
+    name = StringField(required=True)
+    type = StringField(required=True, validators=Enum(*get_options(TaskType)))
+
+
+class PingRequest(TaskRequest):
+    pass
+
+
+class GetTypesRequest(models.Base):
+    projects = ListField(items_types=[str])
+
+
+class TaskInputModel(models.Base):
+    name = StringField()
+    model = StringField()
+
+
+class CloneRequest(TaskRequest):
+    new_task_name = StringField()
+    new_task_comment = StringField()
+    new_task_tags = ListField([str])
+    new_task_system_tags = ListField([str])
+    new_task_parent = StringField()
+    new_task_project = StringField()
+    new_task_hyperparams = DictField()
+    new_task_configuration = DictField()
+    new_task_container = DictField()
+    new_task_input_models = ListField([TaskInputModel])
+    execution_overrides = DictField()
+    validate_references = BoolField(default=False)
+    new_project_name = StringField()
+
+
+class AddOrUpdateArtifactsRequest(TaskUpdateRequest):
+    artifacts = ListField([Artifact], validators=Length(minimum_value=1))
+
+
+class ArtifactId(models.Base):
+    key = StringField(required=True)
+    mode = StringField(
+        validators=Enum(*get_options(ArtifactModes)), default=DEFAULT_ARTIFACT_MODE
+    )
+
+
+class DeleteArtifactsRequest(TaskUpdateRequest):
+    artifacts = ListField([ArtifactId], validators=Length(minimum_value=1))
+
+
+class ResetRequest(UpdateRequest):
+    clear_all = BoolField(default=False)
+    return_file_urls = BoolField(default=False)
+    delete_output_models = BoolField(default=True)
+
+
+class MultiTaskRequest(models.Base):
+    tasks = ListField([str], validators=Length(minimum_value=1))
+
+
+class GetHyperParamsRequest(MultiTaskRequest):
+    pass
+
+
+class HyperParamItem(models.Base):
+    section = StringField(required=True, validators=Length(minimum_value=1))
+    name = StringField(required=True, validators=Length(minimum_value=1))
+    value = StringField(required=True)
+    type = StringField()
+    description = StringField()
+
+
+class ReplaceHyperparams(object):
+    none = "none"
+    section = "section"
+    all = "all"
+
+
+class EditHyperParamsRequest(TaskUpdateRequest):
+    hyperparams: Sequence[HyperParamItem] = ListField(
+        [HyperParamItem], validators=Length(minimum_value=1)
+    )
+    replace_hyperparams = StringField(
+        validators=Enum(*get_options(ReplaceHyperparams)),
+        default=ReplaceHyperparams.none,
+    )
+
+
+class HyperParamKey(models.Base):
+    section = StringField(required=True, validators=Length(minimum_value=1))
+    name = StringField(nullable=True)
+
+
+class DeleteHyperParamsRequest(TaskUpdateRequest):
+    hyperparams: Sequence[HyperParamKey] = ListField(
+        [HyperParamKey], validators=Length(minimum_value=1)
+    )
+
+
+class GetConfigurationsRequest(MultiTaskRequest):
+    names = ListField([str])
+
+
+class GetConfigurationNamesRequest(MultiTaskRequest):
+    skip_empty = BoolField(default=True)
+
+
+class Configuration(models.Base):
+    name = StringField(required=True, validators=Length(minimum_value=1))
+    value = StringField(required=True)
+    type = StringField()
+    description = StringField()
+
+
+class EditConfigurationRequest(TaskUpdateRequest):
+    configuration: Sequence[Configuration] = ListField(
+        [Configuration], validators=Length(minimum_value=1)
+    )
+    replace_configuration = BoolField(default=False)
+
+
+class DeleteConfigurationRequest(TaskUpdateRequest):
+    configuration: Sequence[str] = ListField([str], validators=Length(minimum_value=1))
+
+
+class ArchiveRequest(MultiTaskRequest):
+    status_reason = StringField(default="")
+    status_message = StringField(default="")
+
+
+class ArchiveResponse(models.Base):
+    archived = IntField()
+
+
+class TaskBatchRequest(BatchRequest):
+    status_reason = StringField(default="")
+    status_message = StringField(default="")
+
+
+class StopManyRequest(TaskBatchRequest):
+    force = BoolField(default=False)
+
+
+class EnqueueManyRequest(TaskBatchRequest):
+    queue = StringField()
+    queue_name = StringField()
+    validate_tasks = BoolField(default=False)
+
+
+class DeleteManyRequest(TaskBatchRequest):
+    move_to_trash = BoolField(default=True)
+    return_file_urls = BoolField(default=False)
+    delete_output_models = BoolField(default=True)
+    force = BoolField(default=False)
+
+
+class ResetManyRequest(TaskBatchRequest):
+    clear_all = BoolField(default=False)
+    return_file_urls = BoolField(default=False)
+    delete_output_models = BoolField(default=True)
+    force = BoolField(default=False)
+
+
+class PublishManyRequest(TaskBatchRequest):
+    publish_model = BoolField(default=True)
+    force = BoolField(default=False)
+
+
+class AddUpdateModelRequest(TaskRequest):
+    name = StringField(required=True)
+    model = StringField(required=True)
+    type = StringField(required=True, validators=Enum(*get_options(TaskModelTypes)))
+    iteration = IntField()
+
+
+class ModelItemKey(models.Base):
+    name = StringField(required=True)
+    type = StringField(required=True, validators=Enum(*get_options(TaskModelTypes)))
+
+
+class DeleteModelsRequest(TaskRequest):
+    models: Sequence[ModelItemKey] = ListField(
+        [ModelItemKey], validators=Length(minimum_value=1)
+    )

apiserver/apimodels/users.py

@@ -1,7 +1,7 @@
 from jsonmodels.fields import StringField
 from jsonmodels.models import Base
 
-from apimodels import DictField
+from apiserver.apimodels import DictField
 
 
 class CreateRequest(Base):

apiserver/apimodels/workers.py

@@ -12,7 +12,7 @@ from jsonmodels.fields import (
 )
 from jsonmodels.models import Base
 
-from apimodels import make_default, ListField, EnumField, JsonSerializableMixin
+from apiserver.apimodels import make_default, ListField, EnumField, JsonSerializableMixin
 
 DEFAULT_TIMEOUT = 10 * 60
 
@@ -96,6 +96,7 @@ class WorkerResponseEntry(WorkerEntry):
 
 class GetAllRequest(Base):
     last_seen = IntField(default=3600)
+    tags = ListField(str)
 
 
 class GetAllResponse(Base):

apiserver/bll/auth/__init__.py

@@ -1,17 +1,21 @@
 from datetime import datetime
 
-import database
-from apierrors import errors
-from apimodels.auth import GetTokenResponse, CreateUserRequest, Credentials as CredModel
-from apimodels.users import CreateRequest as Users_CreateRequest
-from bll.user import UserBLL
-from config import config
-from config.info import get_version, get_build_number
-from database.errors import translate_errors_context
-from database.model.auth import User, Role, Credentials
-from database.model.company import Company
-from service_repo import APICall, ServiceRepo
-from service_repo.auth import Identity, Token, get_client_id, get_secret_key
+from apiserver import database
+from apiserver.apierrors import errors
+from apiserver.apimodels.auth import (
+    GetTokenResponse,
+    CreateUserRequest,
+    Credentials as CredModel,
+)
+from apiserver.apimodels.users import CreateRequest as Users_CreateRequest
+from apiserver.bll.user import UserBLL
+from apiserver.config_repo import config
+from apiserver.config.info import get_version, get_build_number
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.auth import User, Role, Credentials
+from apiserver.database.model.company import Company
+from apiserver.service_repo import APICall, ServiceRepo
+from apiserver.service_repo.auth import Identity, Token, get_client_id, get_secret_key
 
 log = config.logger("AuthBLL")
 
@@ -57,9 +61,10 @@ class AuthBLL:
                 api_version=str(ServiceRepo.max_endpoint_version()),
                 server_version=str(get_version()),
                 server_build=str(get_build_number()),
+                feature_set="basic",
             )
 
-            return GetTokenResponse(token=token.decode("ascii"))
+            return GetTokenResponse(token=token)
 
     @staticmethod
     def create_user(request: CreateUserRequest, call: APICall = None) -> str:
@@ -144,7 +149,7 @@ class AuthBLL:
 
     @classmethod
     def create_credentials(
-        cls, user_id: str, company_id: str, role: str = None
+        cls, user_id: str, company_id: str, role: str = None, label: str = None,
     ) -> CredModel:
 
         with translate_errors_context():
@@ -153,9 +158,11 @@ class AuthBLL:
             if not user:
                 raise errors.bad_request.InvalidUserId(**query)
 
-            cred = CredModel(access_key=get_client_id(), secret_key=get_secret_key())
+            cred = CredModel(
+                access_key=get_client_id(), secret_key=get_secret_key(), label=label
+            )
             user.credentials.append(
-                Credentials(key=cred.access_key, secret=cred.secret_key)
+                Credentials(key=cred.access_key, secret=cred.secret_key, label=label)
             )
             user.save()
 

apiserver/bll/event/event_bll.py

@@ -1,48 +1,91 @@
+import base64
 import hashlib
+import re
+import zlib
 from collections import defaultdict
 from contextlib import closing
 from datetime import datetime
 from operator import attrgetter
-from typing import Sequence, Set, Tuple, Optional
+from typing import Sequence, Set, Tuple, Optional, List, Mapping, Union
 
-import six
-from elasticsearch import helpers
+import elasticsearch
+from elasticsearch.helpers import BulkIndexError
 from mongoengine import Q
 from nested_dict import nested_dict
 
-import database.utils as dbutils
-import es_factory
-from apierrors import errors
-from bll.event.debug_images_iterator import DebugImagesIterator
-from bll.event.event_metrics import EventMetrics, EventType
-from bll.event.log_events_iterator import LogEventsIterator, TaskEventsResult
-from bll.task import TaskBLL
-from config import config
-from database.errors import translate_errors_context
-from database.model.task.task import Task, TaskStatus
-from redis_manager import redman
-from timing_context import TimingContext
-from tools import safe_get
-from utilities.dicts import flatten_nested_items
+from apiserver.bll.event.event_common import (
+    EventType,
+    get_index_name,
+    check_empty_data,
+    search_company_events,
+    delete_company_events,
+    MetricVariants,
+    get_metric_variants_condition,
+    uncompress_plot,
+    get_max_metric_and_variant_counts,
+)
+from apiserver.bll.event.events_iterator import EventsIterator, TaskEventsResult
+from apiserver.bll.event.history_debug_image_iterator import HistoryDebugImageIterator
+from apiserver.bll.event.history_plot_iterator import HistoryPlotIterator
+from apiserver.bll.event.metric_debug_images_iterator import MetricDebugImagesIterator
+from apiserver.bll.event.metric_plots_iterator import MetricPlotsIterator
+from apiserver.bll.util import parallel_chunked_decorator
+from apiserver.database import utils as dbutils
+from apiserver.es_factory import es_factory
+from apiserver.apierrors import errors
+from apiserver.bll.event.event_metrics import EventMetrics
+from apiserver.bll.task import TaskBLL
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.task.task import Task, TaskStatus
+from apiserver.redis_manager import redman
+from apiserver.timing_context import TimingContext
+from apiserver.tools import safe_get
+from apiserver.utilities.dicts import flatten_nested_items, nested_get
+from apiserver.utilities.json import loads
 
 # noinspection PyTypeChecker
-EVENT_TYPES = set(map(attrgetter("value"), EventType))
+EVENT_TYPES: Set[str] = set(map(attrgetter("value"), EventType))
 LOCKED_TASK_STATUSES = (TaskStatus.publishing, TaskStatus.published)
+MAX_LONG = 2 ** 63 - 1
+MIN_LONG = -(2 ** 63)
+
+
+log = config.logger(__file__)
+
+
+class PlotFields:
+    valid_plot = "valid_plot"
+    plot_len = "plot_len"
+    plot_str = "plot_str"
+    plot_data = "plot_data"
+    source_urls = "source_urls"
 
 
 class EventBLL(object):
     id_fields = ("task", "iter", "metric", "variant", "key")
     empty_scroll = "FFFF"
+    img_source_regex = re.compile(
+        r"['\"]source['\"]:\s?['\"]([a-z][a-z0-9+\-.]*://.*?)['\"]",
+        flags=re.IGNORECASE,
+    )
 
     def __init__(self, events_es=None, redis=None):
         self.es = events_es or es_factory.connect("events")
+        self.redis = redis or redman.connection("apiserver")
         self._metrics = EventMetrics(self.es)
         self._skip_iteration_for_metric = set(
             config.get("services.events.ignore_iteration.metrics", [])
         )
-        self.redis = redis or redman.connection("apiserver")
-        self.debug_images_iterator = DebugImagesIterator(es=self.es, redis=self.redis)
-        self.log_events_iterator = LogEventsIterator(es=self.es)
+        self.debug_images_iterator = MetricDebugImagesIterator(
+            es=self.es, redis=self.redis
+        )
+        self.debug_image_sample_history = HistoryDebugImageIterator(
+            es=self.es, redis=self.redis
+        )
+        self.plots_iterator = MetricPlotsIterator(es=self.es, redis=self.redis)
+        self.plot_sample_history = HistoryPlotIterator(es=self.es, redis=self.redis)
+        self.events_iterator = EventsIterator(es=self.es)
 
     @property
     def metrics(self) -> EventMetrics:
@@ -64,7 +107,7 @@ class EventBLL(object):
     def add_events(
         self, company_id, events, worker, allow_locked_tasks=False
     ) -> Tuple[int, int, dict]:
-        actions = []
+        actions: List[dict] = []
         task_ids = set()
         task_iteration = defaultdict(lambda: 0)
         task_last_scalar_events = nested_dict(
@@ -74,6 +117,7 @@ class EventBLL(object):
             3, dict
         )  # task_id -> metric_hash -> event_type -> MetricEvent
         errors_per_type = defaultdict(int)
+        invalid_iteration_error = f"Iteration number should not exceed {MAX_LONG}"
         valid_tasks = self._get_valid_tasks(
             company_id,
             task_ids={
@@ -81,6 +125,7 @@ class EventBLL(object):
             },
             allow_locked_tasks=allow_locked_tasks,
         )
+
         for event in events:
             # remove spaces from event type
             event_type = event.get("type")
@@ -122,6 +167,9 @@ class EventBLL(object):
             iter = event.get("iter")
             if iter is not None:
                 iter = int(iter)
+                if iter > MAX_LONG or iter < MIN_LONG:
+                    errors_per_type[invalid_iteration_error] += 1
+                    continue
                 event["iter"] = iter
 
             # used to have "values" to indicate array. no need anymore
@@ -132,7 +180,7 @@ class EventBLL(object):
             event["metric"] = event.get("metric") or ""
             event["variant"] = event.get("variant") or ""
 
-            index_name = EventMetrics.get_index_name(company_id, event_type)
+            index_name = get_index_name(company_id, event_type)
             es_action = {
                 "_op_type": "index",  # overwrite if exists with same ID
                 "_index": index_name,
@@ -140,7 +188,7 @@ class EventBLL(object):
             }
 
             # for "log" events, don't assing custom _id - whatever is sent, is written (not overwritten)
-            if event_type != "log":
+            if event_type != EventType.task_log.value:
                 es_action["_id"] = self._get_event_id(event)
             else:
                 es_action["_id"] = dbutils.id()
@@ -162,51 +210,71 @@ class EventBLL(object):
 
             actions.append(es_action)
 
+        plot_actions = [
+            action["_source"]
+            for action in actions
+            if action["_source"]["type"] == EventType.metrics_plot.value
+        ]
+        if plot_actions:
+            self.validate_and_compress_plots(
+                plot_actions,
+                validate_json=config.get("services.events.validate_plot_str", False),
+                compression_threshold=config.get(
+                    "services.events.plot_compression_threshold", 100_000
+                ),
+            )
+
         added = 0
-        if actions:
-            chunk_size = 500
-            with translate_errors_context(), TimingContext("es", "events_add_batch"):
-                # TODO: replace it with helpers.parallel_bulk in the future once the parallel pool leak is fixed
-                with closing(
-                    helpers.streaming_bulk(
-                        self.es,
-                        actions,
-                        chunk_size=chunk_size,
-                        # thread_count=8,
-                        refresh=True,
-                    )
-                ) as it:
-                    for success, info in it:
-                        if success:
-                            added += chunk_size
-                        else:
-                            errors_per_type["Error when indexing events batch"] += 1
+        with translate_errors_context():
+            if actions:
+                chunk_size = 500
+                with TimingContext("es", "events_add_batch"):
+                    # TODO: replace it with helpers.parallel_bulk in the future once the parallel pool leak is fixed
+                    with closing(
+                        elasticsearch.helpers.streaming_bulk(
+                            self.es,
+                            actions,
+                            chunk_size=chunk_size,
+                            # thread_count=8,
+                            refresh=True,
+                        )
+                    ) as it:
+                        for success, info in it:
+                            if success:
+                                added += 1
+                            else:
+                                errors_per_type["Error when indexing events batch"] += 1
 
-                remaining_tasks = set()
-                now = datetime.utcnow()
-                for task_id in task_ids:
-                    # Update related tasks. For reasons of performance, we prefer to update
-                    # all of them and not only those who's events were successful
-                    updated = self._update_task(
-                        company_id=company_id,
-                        task_id=task_id,
-                        now=now,
-                        iter_max=task_iteration.get(task_id),
-                        last_scalar_events=task_last_scalar_events.get(task_id),
-                        last_events=task_last_events.get(task_id),
-                    )
+                    remaining_tasks = set()
+                    now = datetime.utcnow()
+                    for task_id in task_ids:
+                        # Update related tasks. For reasons of performance, we prefer to update
+                        # all of them and not only those who's events were successful
+                        updated = self._update_task(
+                            company_id=company_id,
+                            task_id=task_id,
+                            now=now,
+                            iter_max=task_iteration.get(task_id),
+                            last_scalar_events=task_last_scalar_events.get(task_id),
+                            last_events=task_last_events.get(task_id),
+                        )
 
-                    if not updated:
-                        remaining_tasks.add(task_id)
-                        continue
+                        if not updated:
+                            remaining_tasks.add(task_id)
+                            continue
 
-                if remaining_tasks:
-                    TaskBLL.set_last_update(
-                        remaining_tasks, company_id, last_update=now
-                    )
+                    if remaining_tasks:
+                        TaskBLL.set_last_update(
+                            remaining_tasks, company_id, last_update=now
+                        )
 
-        # Compensate for always adding chunk_size on success (last chunk is probably smaller)
-        added = min(added, len(actions))
+            # this is for backwards compatibility with streaming bulk throwing exception on those
+            invalid_iterations_count = errors_per_type.get(invalid_iteration_error)
+            if invalid_iterations_count:
+                raise BulkIndexError(
+                    f"{invalid_iterations_count} document(s) failed to index.",
+                    [invalid_iteration_error],
+                )
 
         if not added:
             raise errors.bad_request.EventsNotAdded(**errors_per_type)
@@ -214,6 +282,53 @@ class EventBLL(object):
         errors_count = sum(errors_per_type.values())
         return added, errors_count, errors_per_type
 
+    @parallel_chunked_decorator(chunk_size=10)
+    def validate_and_compress_plots(
+        self,
+        plot_events: Sequence[dict],
+        validate_json: bool,
+        compression_threshold: int,
+    ):
+        for event in plot_events:
+            validate = validate_json and not event.pop("skip_validation", False)
+            plot_str = event.get(PlotFields.plot_str)
+            if not plot_str:
+                event[PlotFields.plot_len] = 0
+                if validate:
+                    event[PlotFields.valid_plot] = False
+                continue
+
+            plot_len = len(plot_str)
+            event[PlotFields.plot_len] = plot_len
+            if validate:
+                event[PlotFields.valid_plot] = self._is_valid_json(plot_str)
+
+            urls = {match for match in self.img_source_regex.findall(plot_str)}
+            if urls:
+                event[PlotFields.source_urls] = list(urls)
+
+            if compression_threshold and plot_len >= compression_threshold:
+                event[PlotFields.plot_data] = base64.encodebytes(
+                    zlib.compress(plot_str.encode(), level=1)
+                ).decode("ascii")
+                event.pop(PlotFields.plot_str, None)
+
+    @parallel_chunked_decorator(chunk_size=10)
+    def uncompress_plots(self, plot_events: Sequence[dict]):
+        for event in plot_events:
+            uncompress_plot(event)
+
+    @staticmethod
+    def _is_valid_json(text: str) -> bool:
+        """Check str for valid json"""
+        if not text:
+            return False
+        try:
+            loads(text)
+        except Exception:
+            return False
+        return True
+
     def _update_last_scalar_events_for_task(self, last_events, event):
         """
         Update last_events structure with the provided event details if this event is more
@@ -315,10 +430,10 @@ class EventBLL(object):
 
     def scroll_task_events(
         self,
-        company_id,
-        task_id,
-        order,
-        event_type=None,
+        company_id: str,
+        task_id: str,
+        order: str,
+        event_type: EventType,
         batch_size=10000,
         scroll_id=None,
     ):
@@ -330,12 +445,7 @@ class EventBLL(object):
                 es_res = self.es.scroll(scroll_id=scroll_id, scroll="1h")
         else:
             size = min(batch_size, 10000)
-            if event_type is None:
-                event_type = "*"
-
-            es_index = EventMetrics.get_index_name(company_id, event_type)
-
-            if not self.es.indices.exists(es_index):
+            if check_empty_data(self.es, company_id=company_id, event_type=event_type):
                 return [], None, 0
 
             es_req = {
@@ -345,30 +455,58 @@ class EventBLL(object):
             }
 
             with translate_errors_context(), TimingContext("es", "scroll_task_events"):
-                es_res = self.es.search(index=es_index, body=es_req, scroll="1h")
+                es_res = search_company_events(
+                    self.es,
+                    company_id=company_id,
+                    event_type=event_type,
+                    body=es_req,
+                    scroll="1h",
+                )
 
         events, total_events, next_scroll_id = self._get_events_from_es_res(es_res)
+        if event_type in (EventType.metrics_plot, EventType.all):
+            self.uncompress_plots(events)
+
         return events, next_scroll_id, total_events
 
     def get_last_iterations_per_event_metric_variant(
-        self, es_index: str, task_id: str, num_last_iterations: int, event_type: str
+        self,
+        company_id: str,
+        task_id: str,
+        num_last_iterations: int,
+        event_type: EventType,
+        metric_variants: MetricVariants = None,
     ):
-        if not self.es.indices.exists(es_index):
+        if check_empty_data(self.es, company_id=company_id, event_type=event_type):
             return []
 
+        must = [{"term": {"task": task_id}}]
+        if metric_variants:
+            must.append(get_metric_variants_condition(metric_variants))
+        query = {"bool": {"must": must}}
+        search_args = dict(
+            es=self.es, company_id=company_id, event_type=event_type
+        )
+        max_metrics, max_variants = get_max_metric_and_variant_counts(
+            query=query, **search_args,
+        )
+        max_variants = int(max_variants // num_last_iterations)
+
         es_req: dict = {
             "size": 0,
             "aggs": {
                 "metrics": {
                     "terms": {
                         "field": "metric",
-                        "size": EventMetrics.MAX_METRICS_COUNT,
+                        "size": max_metrics,
+                        "order": {"_key": "asc"},
                     },
                     "aggs": {
                         "variants": {
                             "terms": {
                                 "field": "variant",
-                                "size": EventMetrics.MAX_VARIANTS_COUNT,
+                                "size": max_variants,
+                                "order": {"_key": "asc"},
                             },
                             "aggs": {
                                 "iters": {
@@ -383,15 +521,14 @@ class EventBLL(object):
                     },
                 }
             },
-            "query": {"bool": {"must": [{"term": {"task": task_id}}]}},
+            "query": query,
         }
-        if event_type:
-            es_req["query"]["bool"]["must"].append({"term": {"type": event_type}})
 
         with translate_errors_context(), TimingContext(
             "es", "task_last_iter_metric_variant"
         ):
-            es_res = self.es.search(index=es_index, body=es_req)
+            es_res = search_company_events(body=es_req, **search_args)
+
         if "aggregations" not in es_res:
             return []
 
@@ -410,27 +547,47 @@ class EventBLL(object):
         sort=None,
         size: int = 500,
         scroll_id: str = None,
+        no_scroll: bool = False,
+        metric_variants: MetricVariants = None,
     ):
         if scroll_id == self.empty_scroll:
-            return [], scroll_id, 0
+            return TaskEventsResult()
 
         if scroll_id:
             with translate_errors_context(), TimingContext("es", "get_task_events"):
                 es_res = self.es.scroll(scroll_id=scroll_id, scroll="1h")
         else:
-            event_type = "plot"
-            es_index = EventMetrics.get_index_name(company_id, event_type)
-            if not self.es.indices.exists(es_index):
+            event_type = EventType.metrics_plot
+            if check_empty_data(self.es, company_id=company_id, event_type=event_type):
                 return TaskEventsResult()
 
-            must = []
+            plot_valid_condition = {
+                "bool": {
+                    "should": [
+                        {"term": {PlotFields.valid_plot: True}},
+                        {
+                            "bool": {
+                                "must_not": {"exists": {"field": PlotFields.valid_plot}}
+                            }
+                        },
+                    ]
+                }
+            }
+            must = [plot_valid_condition]
+
             if last_iterations_per_plot is None:
                 must.append({"terms": {"task": tasks}})
+                if metric_variants:
+                    must.append(get_metric_variants_condition(metric_variants))
             else:
                 should = []
                 for i, task_id in enumerate(tasks):
                     last_iters = self.get_last_iterations_per_event_metric_variant(
-                        es_index, task_id, last_iterations_per_plot, event_type
+                        company_id=company_id,
+                        task_id=task_id,
+                        num_last_iterations=last_iterations_per_plot,
+                        event_type=event_type,
+                        metric_variants=metric_variants,
                     )
                     if not last_iters:
                         continue
@@ -462,11 +619,17 @@ class EventBLL(object):
             }
 
             with translate_errors_context(), TimingContext("es", "get_task_plots"):
-                es_res = self.es.search(
-                    index=es_index, body=es_req, ignore=404, scroll="1h",
+                es_res = search_company_events(
+                    self.es,
+                    company_id=company_id,
+                    event_type=event_type,
+                    body=es_req,
+                    ignore=404,
+                    **({} if no_scroll else {"scroll": "1h"}),
                 )
 
         events, total_events, next_scroll_id = self._get_events_from_es_res(es_res)
+        self.uncompress_plots(events)
         return TaskEventsResult(
             events=events, next_scroll_id=next_scroll_id, total_events=total_events
         )
@@ -485,33 +648,102 @@ class EventBLL(object):
 
         return events, total_events, next_scroll_id
 
+    def get_debug_image_urls(
+        self, company_id: str, task_id: str, after_key: dict = None
+    ) -> Tuple[Sequence[str], Optional[dict]]:
+        if check_empty_data(self.es, company_id, EventType.metrics_image):
+            return [], None
+
+        es_req = {
+            "size": 0,
+            "aggs": {
+                "debug_images": {
+                    "composite": {
+                        "size": 1000,
+                        **({"after": after_key} if after_key else {}),
+                        "sources": [{"url": {"terms": {"field": "url"}}}],
+                    }
+                }
+            },
+            "query": {
+                "bool": {
+                    "must": [{"term": {"task": task_id}}, {"exists": {"field": "url"}}]
+                }
+            },
+        }
+
+        es_response = search_company_events(
+            self.es,
+            company_id=company_id,
+            event_type=EventType.metrics_image,
+            body=es_req,
+        )
+        res = nested_get(es_response, ("aggregations", "debug_images"))
+        if not res:
+            return [], None
+
+        return [bucket["key"]["url"] for bucket in res["buckets"]], res.get("after_key")
+
+    def get_plot_image_urls(
+        self, company_id: str, task_id: str, scroll_id: Optional[str]
+    ) -> Tuple[Sequence[dict], Optional[str]]:
+        if scroll_id == self.empty_scroll:
+            return [], None
+
+        if scroll_id:
+            es_res = self.es.scroll(scroll_id=scroll_id, scroll="10m")
+        else:
+            if check_empty_data(self.es, company_id, EventType.metrics_plot):
+                return [], None
+
+            es_req = {
+                "size": 1000,
+                "_source": [PlotFields.source_urls],
+                "query": {
+                    "bool": {
+                        "must": [
+                            {"term": {"task": task_id}},
+                            {"exists": {"field": PlotFields.source_urls}},
+                        ]
+                    }
+                },
+            }
+            es_res = search_company_events(
+                self.es,
+                company_id=company_id,
+                event_type=EventType.metrics_plot,
+                body=es_req,
+                scroll="10m",
+            )
+
+        events, _, next_scroll_id = self._get_events_from_es_res(es_res)
+        return events, next_scroll_id
+
     def get_task_events(
         self,
-        company_id,
-        task_id,
-        event_type=None,
+        company_id: str,
+        task_id: str,
+        event_type: EventType,
         metric=None,
         variant=None,
         last_iter_count=None,
         sort=None,
         size=500,
         scroll_id=None,
-    ):
+        no_scroll=False,
+    ) -> TaskEventsResult:
         if scroll_id == self.empty_scroll:
-            return [], scroll_id, 0
+            return TaskEventsResult()
 
         if scroll_id:
             with translate_errors_context(), TimingContext("es", "get_task_events"):
                 es_res = self.es.scroll(scroll_id=scroll_id, scroll="1h")
         else:
-            task_ids = [task_id] if isinstance(task_id, six.string_types) else task_id
-            if event_type is None:
-                event_type = "*"
-
-            es_index = EventMetrics.get_index_name(company_id, event_type)
-            if not self.es.indices.exists(es_index):
+            if check_empty_data(self.es, company_id=company_id, event_type=event_type):
                 return TaskEventsResult()
 
+            task_ids = [task_id] if isinstance(task_id, str) else task_id
+
             must = []
             if metric:
                 must.append({"term": {"metric": metric}})
@@ -521,23 +753,24 @@ class EventBLL(object):
             if last_iter_count is None:
                 must.append({"terms": {"task": task_ids}})
             else:
-                should = []
-                for i, task_id in enumerate(task_ids):
-                    last_iters = self.get_last_iters(
-                        es_index, task_id, event_type, last_iter_count
-                    )
-                    if not last_iters:
-                        continue
-                    should.append(
-                        {
-                            "bool": {
-                                "must": [
-                                    {"term": {"task": task_id}},
-                                    {"terms": {"iter": last_iters}},
-                                ]
-                            }
+                tasks_iters = self.get_last_iters(
+                    company_id=company_id,
+                    event_type=event_type,
+                    task_id=task_ids,
+                    iters=last_iter_count,
+                )
+                should = [
+                    {
+                        "bool": {
+                            "must": [
+                                {"term": {"task": task}},
+                                {"terms": {"iter": last_iters}},
+                            ]
                         }
-                    )
+                    }
+                    for task, last_iters in tasks_iters.items()
+                    if last_iters
+                ]
                 if not should:
                     return TaskEventsResult()
                 must.append({"bool": {"should": should}})
@@ -552,47 +785,63 @@ class EventBLL(object):
             }
 
             with translate_errors_context(), TimingContext("es", "get_task_events"):
-                es_res = self.es.search(
-                    index=es_index, body=es_req, ignore=404, scroll="1h",
+                es_res = search_company_events(
+                    self.es,
+                    company_id=company_id,
+                    event_type=event_type,
+                    body=es_req,
+                    ignore=404,
+                    **({} if no_scroll else {"scroll": "1h"}),
                 )
 
         events, total_events, next_scroll_id = self._get_events_from_es_res(es_res)
+        if event_type in (EventType.metrics_plot, EventType.all):
+            self.uncompress_plots(events)
+
         return TaskEventsResult(
             events=events, next_scroll_id=next_scroll_id, total_events=total_events
         )
 
-    def get_metrics_and_variants(self, company_id, task_id, event_type):
-
-        es_index = EventMetrics.get_index_name(company_id, event_type)
-
-        if not self.es.indices.exists(es_index):
+    def get_metrics_and_variants(
+        self, company_id: str, task_id: str, event_type: EventType
+    ):
+        if check_empty_data(self.es, company_id=company_id, event_type=event_type):
             return {}
 
+        query = {"bool": {"must": [{"term": {"task": task_id}}]}}
+        search_args = dict(
+            es=self.es, company_id=company_id, event_type=event_type
+        )
+        max_metrics, max_variants = get_max_metric_and_variant_counts(
+            query=query, **search_args,
+        )
         es_req = {
             "size": 0,
             "aggs": {
                 "metrics": {
                     "terms": {
                         "field": "metric",
-                        "size": EventMetrics.MAX_METRICS_COUNT,
+                        "size": max_metrics,
+                        "order": {"_key": "asc"},
                     },
                     "aggs": {
                         "variants": {
                             "terms": {
                                 "field": "variant",
-                                "size": EventMetrics.MAX_VARIANTS_COUNT,
+                                "size": max_variants,
+                                "order": {"_key": "asc"},
                             }
                         }
                     },
                 }
             },
-            "query": {"bool": {"must": [{"term": {"task": task_id}}]}},
+            "query": query,
         }
 
         with translate_errors_context(), TimingContext(
             "es", "events_get_metrics_and_variants"
         ):
-            es_res = self.es.search(index=es_index, body=es_req)
+            es_res = search_company_events(body=es_req, **search_args)
 
         metrics = {}
         for metric_bucket in es_res["aggregations"]["metrics"].get("buckets"):
@@ -603,34 +852,42 @@ class EventBLL(object):
 
         return metrics
 
-    def get_task_latest_scalar_values(self, company_id, task_id):
-        es_index = EventMetrics.get_index_name(company_id, "training_stats_scalar")
-
-        if not self.es.indices.exists(es_index):
-            return {}
+    def get_task_latest_scalar_values(
+        self, company_id, task_id
+    ) -> Tuple[Sequence[dict], int]:
+        event_type = EventType.metrics_scalar
+        if check_empty_data(self.es, company_id=company_id, event_type=event_type):
+            return [], 0
 
+        query = {
+            "bool": {
+                "must": [
+                    {"query_string": {"query": "value:>0"}},
+                    {"term": {"task": task_id}},
+                ]
+            }
+        }
+        search_args = dict(
+            es=self.es, company_id=company_id, event_type=event_type
+        )
+        max_metrics, max_variants = get_max_metric_and_variant_counts(
+            query=query, **search_args,
+        )
         es_req = {
             "size": 0,
-            "query": {
-                "bool": {
-                    "must": [
-                        {"query_string": {"query": "value:>0"}},
-                        {"term": {"task": task_id}},
-                    ]
-                }
-            },
+            "query": query,
             "aggs": {
                 "metrics": {
                     "terms": {
                         "field": "metric",
-                        "size": EventMetrics.MAX_METRICS_COUNT,
+                        "size": max_metrics,
                         "order": {"_key": "asc"},
                     },
                     "aggs": {
                         "variants": {
                             "terms": {
                                 "field": "variant",
-                                "size": EventMetrics.MAX_VARIANTS_COUNT,
+                                "size": max_variants,
                                 "order": {"_key": "asc"},
                             },
                             "aggs": {
@@ -661,7 +918,7 @@ class EventBLL(object):
         with translate_errors_context(), TimingContext(
             "es", "events_get_metrics_and_variants"
         ):
-            es_res = self.es.search(index=es_index, body=es_req)
+            es_res = search_company_events(body=es_req, **search_args)
 
         metrics = []
         max_timestamp = 0
@@ -688,9 +945,8 @@ class EventBLL(object):
         return metrics, max_timestamp
 
     def get_vector_metrics_per_iter(self, company_id, task_id, metric, variant):
-
-        es_index = EventMetrics.get_index_name(company_id, "training_stats_vector")
-        if not self.es.indices.exists(es_index):
+        event_type = EventType.metrics_vector
+        if check_empty_data(self.es, company_id=company_id, event_type=event_type):
             return [], []
 
         es_req = {
@@ -708,7 +964,9 @@ class EventBLL(object):
             "sort": ["iter"],
         }
         with translate_errors_context(), TimingContext("es", "task_stats_vector"):
-            es_res = self.es.search(index=es_index, body=es_req)
+            es_res = search_company_events(
+                self.es, company_id=company_id, event_type=event_type, body=es_req
+            )
 
         vectors = []
         iterations = []
@@ -718,49 +976,150 @@ class EventBLL(object):
 
         return iterations, vectors
 
-    def get_last_iters(self, es_index, task_id, event_type, iters):
-        if not self.es.indices.exists(es_index):
-            return []
+    def get_last_iters(
+        self,
+        company_id: str,
+        event_type: EventType,
+        task_id: Union[str, Sequence[str]],
+        iters: int,
+    ) -> Mapping[str, Sequence]:
+        if check_empty_data(self.es, company_id=company_id, event_type=event_type):
+            return {}
 
+        task_ids = [task_id] if isinstance(task_id, str) else task_id
         es_req: dict = {
             "size": 0,
             "aggs": {
-                "iters": {
-                    "terms": {
-                        "field": "iter",
-                        "size": iters,
-                        "order": {"_key": "desc"},
-                    }
+                "tasks": {
+                    "terms": {"field": "task"},
+                    "aggs": {
+                        "iters": {
+                            "terms": {
+                                "field": "iter",
+                                "size": iters,
+                                "order": {"_key": "desc"},
+                            }
+                        }
+                    },
                 }
             },
-            "query": {"bool": {"must": [{"term": {"task": task_id}}]}},
+            "query": {"bool": {"must": [{"terms": {"task": task_ids}}]}},
         }
-        if event_type:
-            es_req["query"]["bool"]["must"].append({"term": {"type": event_type}})
 
         with translate_errors_context(), TimingContext("es", "task_last_iter"):
-            es_res = self.es.search(index=es_index, body=es_req)
-        if "aggregations" not in es_res:
-            return []
+            es_res = search_company_events(
+                self.es, company_id=company_id, event_type=event_type, body=es_req,
+            )
 
-        return [b["key"] for b in es_res["aggregations"]["iters"]["buckets"]]
+        if "aggregations" not in es_res:
+            return {}
+
+        return {
+            tb["key"]: [ib["key"] for ib in tb["iters"]["buckets"]]
+            for tb in es_res["aggregations"]["tasks"]["buckets"]
+        }
+
+    @staticmethod
+    def _validate_task_state(company_id: str, task_id: str, allow_locked: bool = False):
+        extra_msg = None
+        query = Q(id=task_id, company=company_id)
+        if not allow_locked:
+            query &= Q(status__nin=LOCKED_TASK_STATUSES)
+            extra_msg = "or task published"
+        res = Task.objects(query).only("id").first()
+        if not res:
+            raise errors.bad_request.InvalidTaskId(
+                extra_msg, company=company_id, id=task_id
+            )
 
     def delete_task_events(self, company_id, task_id, allow_locked=False):
-        with translate_errors_context():
-            extra_msg = None
-            query = Q(id=task_id, company=company_id)
-            if not allow_locked:
-                query &= Q(status__nin=LOCKED_TASK_STATUSES)
-                extra_msg = "or task published"
-            res = Task.objects(query).only("id").first()
-            if not res:
-                raise errors.bad_request.InvalidTaskId(
-                    extra_msg, company=company_id, id=task_id
-                )
+        self._validate_task_state(
+            company_id=company_id, task_id=task_id, allow_locked=allow_locked
+        )
 
-        es_index = EventMetrics.get_index_name(company_id, "*")
         es_req = {"query": {"term": {"task": task_id}}}
         with translate_errors_context(), TimingContext("es", "delete_task_events"):
-            es_res = self.es.delete_by_query(index=es_index, body=es_req, refresh=True)
+            es_res = delete_company_events(
+                es=self.es,
+                company_id=company_id,
+                event_type=EventType.all,
+                body=es_req,
+                refresh=True,
+            )
 
         return es_res.get("deleted", 0)
+
+    def clear_task_log(
+        self,
+        company_id: str,
+        task_id: str,
+        allow_locked: bool = False,
+        threshold_sec: int = None,
+    ):
+        self._validate_task_state(
+            company_id=company_id, task_id=task_id, allow_locked=allow_locked
+        )
+        if check_empty_data(
+            self.es, company_id=company_id, event_type=EventType.task_log
+        ):
+            return 0
+
+        with translate_errors_context(), TimingContext("es", "clear_task_log"):
+            must = [{"term": {"task": task_id}}]
+            sort = None
+            if threshold_sec:
+                timestamp_ms = int(threshold_sec * 1000)
+                must.append(
+                    {
+                        "range": {
+                            "timestamp": {
+                                "lt": (es_factory.get_timestamp_millis() - timestamp_ms)
+                            }
+                        }
+                    }
+                )
+                sort = {"timestamp": {"order": "desc"}}
+            es_req = {
+                "query": {"bool": {"must": must}},
+                **({"sort": sort} if sort else {}),
+            }
+            es_res = delete_company_events(
+                es=self.es,
+                company_id=company_id,
+                event_type=EventType.task_log,
+                body=es_req,
+                refresh=True,
+            )
+            return es_res.get("deleted", 0)
+
+    def delete_multi_task_events(self, company_id: str, task_ids: Sequence[str]):
+        """
+        Delete mutliple task events. No check is done for tasks write access
+        so it should be checked by the calling code
+        """
+        es_req = {"query": {"terms": {"task": task_ids}}}
+        with translate_errors_context(), TimingContext(
+            "es", "delete_multi_tasks_events"
+        ):
+            es_res = delete_company_events(
+                es=self.es,
+                company_id=company_id,
+                event_type=EventType.all,
+                body=es_req,
+                refresh=True,
+            )
+
+        return es_res.get("deleted", 0)
+
+    def clear_scroll(self, scroll_id: str):
+        if scroll_id == self.empty_scroll:
+            return
+        # noinspection PyBroadException
+        try:
+            self.es.clear_scroll(scroll_id=scroll_id)
+        except elasticsearch.exceptions.NotFoundError:
+            pass
+        except elasticsearch.exceptions.RequestError:
+            pass
+        except Exception as ex:
+            log.exception("Failed clearing scroll %s", scroll_id)

apiserver/bll/event/event_common.py

@@ -0,0 +1,166 @@
+import base64
+import zlib
+from enum import Enum
+from typing import Union, Sequence, Mapping, Tuple
+
+from boltons.typeutils import classproperty
+from elasticsearch import Elasticsearch
+
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.timing_context import TimingContext
+from apiserver.tools import safe_get
+
+
+class EventType(Enum):
+    metrics_scalar = "training_stats_scalar"
+    metrics_vector = "training_stats_vector"
+    metrics_image = "training_debug_image"
+    metrics_plot = "plot"
+    task_log = "log"
+    all = "*"
+
+
+SINGLE_SCALAR_ITERATION = -2**31
+MetricVariants = Mapping[str, Sequence[str]]
+
+
+class EventSettings:
+    _max_es_allowed_aggregation_buckets = 10000
+
+    @classproperty
+    def max_workers(self):
+        return config.get("services.events.events_retrieval.max_metrics_concurrency", 4)
+
+    @classproperty
+    def state_expiration_sec(self):
+        return config.get(
+            f"services.events.events_retrieval.state_expiration_sec", 3600
+        )
+
+    @classproperty
+    def max_es_buckets(self):
+        percentage = (
+            min(
+                100,
+                config.get(
+                    "services.events.events_retrieval.dynamic_metrics_count_threshold",
+                    80,
+                ),
+            )
+            / 100
+        )
+        return int(self._max_es_allowed_aggregation_buckets * percentage)
+
+
+def get_index_name(company_id: str, event_type: str):
+    event_type = event_type.lower().replace(" ", "_")
+    return f"events-{event_type}-{company_id.lower()}"
+
+
+def check_empty_data(es: Elasticsearch, company_id: str, event_type: EventType) -> bool:
+    es_index = get_index_name(company_id, event_type.value)
+    if not es.indices.exists(es_index):
+        return True
+    return False
+
+
+def search_company_events(
+    es: Elasticsearch,
+    company_id: Union[str, Sequence[str]],
+    event_type: EventType,
+    body: dict,
+    **kwargs,
+) -> dict:
+    es_index = get_index_name(company_id, event_type.value)
+    return es.search(index=es_index, body=body, **kwargs)
+
+
+def delete_company_events(
+    es: Elasticsearch, company_id: str, event_type: EventType, body: dict, **kwargs
+) -> dict:
+    es_index = get_index_name(company_id, event_type.value)
+    return es.delete_by_query(
+        index=es_index, body=body, conflicts="proceed", **kwargs
+    )
+
+
+def count_company_events(
+    es: Elasticsearch, company_id: str, event_type: EventType, body: dict, **kwargs
+) -> dict:
+    es_index = get_index_name(company_id, event_type.value)
+    return es.count(index=es_index, body=body, **kwargs)
+
+
+def get_max_metric_and_variant_counts(
+    es: Elasticsearch,
+    company_id: Union[str, Sequence[str]],
+    event_type: EventType,
+    query: dict,
+    **kwargs,
+) -> Tuple[int, int]:
+    dynamic = config.get(
+        "services.events.events_retrieval.dynamic_metrics_count", False
+    )
+    max_metrics_count = config.get(
+        "services.events.events_retrieval.max_metrics_count", 100
+    )
+    max_variants_count = config.get(
+        "services.events.events_retrieval.max_variants_count", 100
+    )
+    if not dynamic:
+        return max_metrics_count, max_variants_count
+
+    es_req: dict = {
+        "size": 0,
+        "query": query,
+        "aggs": {"metrics_count": {"cardinality": {"field": "metric"}}},
+    }
+    with translate_errors_context(), TimingContext(
+        "es", "get_max_metric_and_variant_counts"
+    ):
+        es_res = search_company_events(
+            es, company_id=company_id, event_type=event_type, body=es_req, **kwargs,
+        )
+
+    metrics_count = safe_get(
+        es_res, "aggregations/metrics_count/value", max_metrics_count
+    )
+    if not metrics_count:
+        return max_metrics_count, max_variants_count
+
+    return metrics_count, int(EventSettings.max_es_buckets / metrics_count)
+
+
+def get_metric_variants_condition(metric_variants: MetricVariants,) -> Sequence:
+    conditions = [
+        {
+            "bool": {
+                "must": [
+                    {"term": {"metric": metric}},
+                    {"terms": {"variant": variants}},
+                ]
+            }
+        }
+        if variants
+        else {"term": {"metric": metric}}
+        for metric, variants in metric_variants.items()
+    ]
+
+    return {"bool": {"should": conditions}}
+
+
+class PlotFields:
+    valid_plot = "valid_plot"
+    plot_len = "plot_len"
+    plot_str = "plot_str"
+    plot_data = "plot_data"
+    source_urls = "source_urls"
+
+
+def uncompress_plot(event: dict):
+    plot_data = event.pop(PlotFields.plot_data, None)
+    if plot_data and event.get(PlotFields.plot_str) is None:
+        event[PlotFields.plot_str] = zlib.decompress(
+            base64.b64decode(plot_data)
+        ).decode()

apiserver/bll/event/event_metrics.py

@@ -2,87 +2,99 @@ import itertools
 import math
 from collections import defaultdict
 from concurrent.futures.thread import ThreadPoolExecutor
-from enum import Enum
 from functools import partial
 from operator import itemgetter
-from typing import Sequence, Tuple
+from typing import Sequence, Tuple, Mapping
 
+from boltons.iterutils import bucketize
 from elasticsearch import Elasticsearch
 from mongoengine import Q
 
-from apierrors import errors
-from bll.event.scalar_key import ScalarKey, ScalarKeyEnum
-from config import config
-from database.errors import translate_errors_context
-from database.model.task.task import Task
-from timing_context import TimingContext
-from tools import safe_get
+from apiserver.apierrors import errors
+from apiserver.bll.event.event_common import (
+    EventType,
+    EventSettings,
+    search_company_events,
+    check_empty_data,
+    MetricVariants,
+    get_metric_variants_condition,
+    get_max_metric_and_variant_counts,
+    SINGLE_SCALAR_ITERATION,
+)
+from apiserver.bll.event.scalar_key import ScalarKey, ScalarKeyEnum
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.task.task import Task
+from apiserver.timing_context import TimingContext
+from apiserver.tools import safe_get
 
 log = config.logger(__file__)
 
 
-class EventType(Enum):
-    metrics_scalar = "training_stats_scalar"
-    metrics_vector = "training_stats_vector"
-    metrics_image = "training_debug_image"
-    metrics_plot = "plot"
-    task_log = "log"
-
-
 class EventMetrics:
-    MAX_METRICS_COUNT = 100
-    MAX_VARIANTS_COUNT = 100
     MAX_AGGS_ELEMENTS_COUNT = 50
     MAX_SAMPLE_BUCKETS = 6000
 
     def __init__(self, es: Elasticsearch):
         self.es = es
 
-    @property
-    def _max_concurrency(self):
-        return config.get("services.events.max_metrics_concurrency", 4)
-
-    @staticmethod
-    def get_index_name(company_id, event_type):
-        event_type = event_type.lower().replace(" ", "_")
-        return f"events-{event_type}-{company_id}"
-
     def get_scalar_metrics_average_per_iter(
-        self, company_id: str, task_id: str, samples: int, key: ScalarKeyEnum
+        self,
+        company_id: str,
+        task_id: str,
+        samples: int,
+        key: ScalarKeyEnum,
+        metric_variants: MetricVariants = None,
     ) -> dict:
         """
         Get scalar metric histogram per metric and variant
         The amount of points in each histogram should not exceed
         the requested samples
         """
-        es_index = self.get_index_name(company_id, "training_stats_scalar")
-        if not self.es.indices.exists(es_index):
+        event_type = EventType.metrics_scalar
+        if check_empty_data(self.es, company_id=company_id, event_type=event_type):
             return {}
 
         return self._get_scalar_average_per_iter_core(
-            task_id, es_index, samples, ScalarKey.resolve(key)
+            task_id=task_id,
+            company_id=company_id,
+            event_type=event_type,
+            samples=samples,
+            key=ScalarKey.resolve(key),
+            metric_variants=metric_variants,
         )
 
     def _get_scalar_average_per_iter_core(
         self,
         task_id: str,
-        es_index: str,
+        company_id: str,
+        event_type: EventType,
         samples: int,
         key: ScalarKey,
         run_parallel: bool = True,
+        metric_variants: MetricVariants = None,
     ) -> dict:
         intervals = self._get_task_metric_intervals(
-            es_index=es_index, task_id=task_id, samples=samples, field=key.field
+            company_id=company_id,
+            event_type=event_type,
+            task_id=task_id,
+            samples=samples,
+            field=key.field,
+            metric_variants=metric_variants,
         )
         if not intervals:
             return {}
         interval_groups = self._group_task_metric_intervals(intervals)
 
         get_scalar_average = partial(
-            self._get_scalar_average, task_id=task_id, es_index=es_index, key=key
+            self._get_scalar_average,
+            task_id=task_id,
+            company_id=company_id,
+            event_type=event_type,
+            key=key,
         )
         if run_parallel:
-            with ThreadPoolExecutor(max_workers=self._max_concurrency) as pool:
+            with ThreadPoolExecutor(max_workers=EventSettings.max_workers) as pool:
                 metrics = itertools.chain.from_iterable(
                     pool.map(get_scalar_average, interval_groups)
                 )
@@ -115,7 +127,7 @@ class EventMetrics:
                 company=company_id,
                 query=Q(id__in=task_ids),
                 allow_public=allow_public,
-                override_projection=("id", "name", "company"),
+                override_projection=("id", "name", "company", "company_origin"),
                 return_dicts=False,
             )
             if len(task_objs) < len(task_ids):
@@ -123,24 +135,26 @@ class EventMetrics:
                 raise errors.bad_request.InvalidTaskId(company=company_id, ids=invalid)
             task_name_by_id = {t.id: t.name for t in task_objs}
 
-        companies = {t.company for t in task_objs}
+        companies = {t.get_index_company() for t in task_objs}
         if len(companies) > 1:
             raise errors.bad_request.InvalidTaskId(
                 "only tasks from the same company are supported"
             )
 
-        es_index = self.get_index_name(next(iter(companies)), "training_stats_scalar")
-        if not self.es.indices.exists(es_index):
+        event_type = EventType.metrics_scalar
+        company_id = next(iter(companies))
+        if check_empty_data(self.es, company_id=company_id, event_type=event_type):
             return {}
 
         get_scalar_average_per_iter = partial(
             self._get_scalar_average_per_iter_core,
-            es_index=es_index,
+            company_id=company_id,
+            event_type=event_type,
             samples=samples,
             key=ScalarKey.resolve(key),
             run_parallel=False,
         )
-        with ThreadPoolExecutor(max_workers=self._max_concurrency) as pool:
+        with ThreadPoolExecutor(max_workers=EventSettings.max_workers) as pool:
             task_metrics = zip(
                 task_ids, pool.map(get_scalar_average_per_iter, task_ids)
             )
@@ -155,6 +169,57 @@ class EventMetrics:
 
         return res
 
+    def get_task_single_value_metrics(
+        self, company_id: str, task_ids: Sequence[str]
+    ) -> Mapping[str, dict]:
+        """
+        For the requested tasks return all the events delivered for the single iteration (-2**31)
+        """
+        if check_empty_data(
+            self.es, company_id=company_id, event_type=EventType.metrics_scalar
+        ):
+            return {}
+
+        with TimingContext("es", "get_task_single_value_metrics"):
+            task_events = self._get_task_single_value_metrics(company_id, task_ids)
+
+        def _get_value(event: dict):
+            return {
+                field: event.get(field)
+                for field in ("metric", "variant", "value", "timestamp")
+            }
+
+        return {
+            task: [_get_value(e) for e in events]
+            for task, events in bucketize(task_events, itemgetter("task")).items()
+        }
+
+    def _get_task_single_value_metrics(
+        self, company_id: str, task_ids: Sequence[str]
+    ) -> Sequence[dict]:
+        es_req = {
+            "size": 10000,
+            "query": {
+                "bool": {
+                    "must": [
+                        {"terms": {"task": task_ids}},
+                        {"term": {"iter": SINGLE_SCALAR_ITERATION}},
+                    ]
+                }
+            },
+        }
+        with translate_errors_context():
+            es_res = search_company_events(
+                body=es_req,
+                es=self.es,
+                company_id=company_id,
+                event_type=EventType.metrics_scalar,
+            )
+            if not es_res["hits"]["total"]["value"]:
+                return []
+
+        return [hit["_source"] for hit in es_res["hits"]["hits"]]
+
     MetricInterval = Tuple[str, str, int, int]
     MetricIntervalGroup = Tuple[int, Sequence[Tuple[str, str]]]
 
@@ -194,7 +259,13 @@ class EventMetrics:
         return metric_interval_groups
 
     def _get_task_metric_intervals(
-        self, es_index, task_id: str, samples: int, field: str = "iter"
+        self,
+        company_id: str,
+        event_type: EventType,
+        task_id: str,
+        samples: int,
+        field: str = "iter",
+        metric_variants: MetricVariants = None,
     ) -> Sequence[MetricInterval]:
         """
         Calculate interval per task metric variant so that the resulting
@@ -202,17 +273,33 @@ class EventMetrics:
         Return the list og metric variant intervals as the following tuple:
         (metric, variant, interval, samples)
         """
+        must = self._task_conditions(task_id)
+        if metric_variants:
+            must.append(get_metric_variants_condition(metric_variants))
+        query = {"bool": {"must": must}}
+        search_args = dict(
+            es=self.es, company_id=company_id, event_type=event_type
+        )
+        max_metrics, max_variants = get_max_metric_and_variant_counts(
+            query=query, **search_args,
+        )
+        max_variants = int(max_variants // 2)
         es_req = {
             "size": 0,
-            "query": {"term": {"task": task_id}},
+            "query": query,
             "aggs": {
                 "metrics": {
-                    "terms": {"field": "metric", "size": self.MAX_METRICS_COUNT},
+                    "terms": {
+                        "field": "metric",
+                        "size": max_metrics,
+                        "order": {"_key": "asc"},
+                    },
                     "aggs": {
                         "variants": {
                             "terms": {
                                 "field": "variant",
-                                "size": self.MAX_VARIANTS_COUNT,
+                                "size": max_variants,
+                                "order": {"_key": "asc"},
                             },
                             "aggs": {
                                 "count": {"value_count": {"field": field}},
@@ -226,7 +313,7 @@ class EventMetrics:
         }
 
         with translate_errors_context(), TimingContext("es", "task_stats_get_interval"):
-            es_res = self.es.search(index=es_index, body=es_req)
+            es_res = search_company_events(body=es_req, **search_args)
 
         aggs_result = es_res.get("aggregations")
         if not aggs_result:
@@ -269,7 +356,8 @@ class EventMetrics:
         self,
         metrics_interval: MetricIntervalGroup,
         task_id: str,
-        es_index: str,
+        company_id: str,
+        event_type: EventType,
         key: ScalarKey,
     ) -> Sequence[MetricData]:
         """
@@ -277,29 +365,42 @@ class EventMetrics:
         """
         interval, metrics = metrics_interval
         aggregation = self._add_aggregation_average(key.get_aggregation(interval))
-        aggs = {
-            "metrics": {
-                "terms": {
-                    "field": "metric",
-                    "size": self.MAX_METRICS_COUNT,
-                    "order": {"_key": "desc"},
-                },
-                "aggs": {
-                    "variants": {
-                        "terms": {
-                            "field": "variant",
-                            "size": self.MAX_VARIANTS_COUNT,
-                            "order": {"_key": "desc"},
-                        },
-                        "aggs": aggregation,
-                    }
-                },
-            }
-        }
-        aggs_result = self._query_aggregation_for_task_metrics(
-            es_index, aggs=aggs, task_id=task_id, metrics=metrics
+        query = self._get_task_metrics_query(task_id=task_id, metrics=metrics)
+        search_args = dict(
+            es=self.es, company_id=company_id, event_type=event_type
         )
+        max_metrics, max_variants = get_max_metric_and_variant_counts(
+            query=query, **search_args,
+        )
+        max_variants = int(max_variants // 2)
+        es_req = {
+            "size": 0,
+            "query": query,
+            "aggs": {
+                "metrics": {
+                    "terms": {
+                        "field": "metric",
+                        "size": max_metrics,
+                        "order": {"_key": "asc"},
+                    },
+                    "aggs": {
+                        "variants": {
+                            "terms": {
+                                "field": "variant",
+                                "size": max_variants,
+                                "order": {"_key": "asc"},
+                            },
+                            "aggs": aggregation,
+                        }
+                    },
+                }
+            },
+        }
 
+        with translate_errors_context():
+            es_res = search_company_events(body=es_req, **search_args)
+
+        aggs_result = es_res.get("aggregations")
         if not aggs_result:
             return {}
 
@@ -326,18 +427,18 @@ class EventMetrics:
             for key, value in aggregation.items()
         }
 
-    def _query_aggregation_for_task_metrics(
-        self,
-        es_index: str,
-        aggs: dict,
-        task_id: str,
-        metrics: Sequence[Tuple[str, str]],
-    ) -> dict:
-        """
-        Return the result of elastic search query for the given aggregation filtered
-        by the given task_ids and metrics
-        """
-        must = [{"term": {"task": task_id}}]
+    @staticmethod
+    def _task_conditions(task_id: str) -> list:
+        return [
+            {"term": {"task": task_id}},
+            {"range": {"iter": {"gt": SINGLE_SCALAR_ITERATION}}},
+        ]
+
+    @classmethod
+    def _get_task_metrics_query(
+        cls, task_id: str, metrics: Sequence[Tuple[str, str]],
+    ):
+        must = cls._task_conditions(task_id)
         if metrics:
             should = [
                 {
@@ -352,57 +453,50 @@ class EventMetrics:
             ]
             must.append({"bool": {"should": should}})
 
-        es_req = {
-            "size": 0,
-            "query": {"bool": {"must": must}},
-            "aggs": aggs,
-        }
+        return {"bool": {"must": must}}
 
-        with translate_errors_context(), TimingContext("es", "task_stats_scalar"):
-            es_res = self.es.search(index=es_index, body=es_req)
-
-        return es_res.get("aggregations")
-
-    def get_tasks_metrics(
+    def get_task_metrics(
         self, company_id, task_ids: Sequence, event_type: EventType
     ) -> Sequence:
         """
         For the requested tasks return all the metrics that
         reported events of the requested types
         """
-        es_index = EventMetrics.get_index_name(company_id, event_type.value)
-        if not self.es.indices.exists(es_index):
+        if check_empty_data(self.es, company_id, event_type):
             return {}
 
-        with ThreadPoolExecutor(self._max_concurrency) as pool:
+        with ThreadPoolExecutor(EventSettings.max_workers) as pool:
             res = pool.map(
                 partial(
-                    self._get_task_metrics, es_index=es_index, event_type=event_type,
+                    self._get_task_metrics,
+                    company_id=company_id,
+                    event_type=event_type,
                 ),
                 task_ids,
             )
         return list(zip(task_ids, res))
 
-    def _get_task_metrics(self, task_id, es_index, event_type: EventType) -> Sequence:
+    def _get_task_metrics(
+        self, task_id: str, company_id: str, event_type: EventType
+    ) -> Sequence:
         es_req = {
             "size": 0,
-            "query": {
-                "bool": {
-                    "must": [
-                        {"term": {"task": task_id}},
-                        {"term": {"type": event_type.value}},
-                    ]
-                }
-            },
+            "query": {"bool": {"must": self._task_conditions(task_id)}},
             "aggs": {
                 "metrics": {
-                    "terms": {"field": "metric", "size": self.MAX_METRICS_COUNT}
+                    "terms": {
+                        "field": "metric",
+                        "size": EventSettings.max_es_buckets,
+                        "order": {"_key": "asc"},
+                    }
                 }
             },
         }
 
         with translate_errors_context(), TimingContext("es", "_get_task_metrics"):
-            es_res = self.es.search(index=es_index, body=es_req)
+            es_res = search_company_events(
+                self.es, company_id=company_id, event_type=event_type, body=es_req
+            )
 
         return [
             metric["key"]

apiserver/bll/event/events_iterator.py

@@ -0,0 +1,198 @@
+from typing import Optional, Tuple, Sequence, Any
+
+import attr
+import jsonmodels.models
+import jwt
+from elasticsearch import Elasticsearch
+from jwt.algorithms import get_default_algorithms
+
+from apiserver.bll.event.event_common import (
+    check_empty_data,
+    search_company_events,
+    EventType,
+    MetricVariants,
+    get_metric_variants_condition,
+    count_company_events,
+)
+from apiserver.bll.event.scalar_key import ScalarKeyEnum, ScalarKey
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.timing_context import TimingContext
+
+
+@attr.s(auto_attribs=True)
+class TaskEventsResult:
+    total_events: int = 0
+    next_scroll_id: str = None
+    events: list = attr.Factory(list)
+
+
+class EventsIterator:
+    def __init__(self, es: Elasticsearch):
+        self.es = es
+
+    def get_task_events(
+        self,
+        event_type: EventType,
+        company_id: str,
+        task_id: str,
+        batch_size: int,
+        navigate_earlier: bool = True,
+        from_key_value: Optional[Any] = None,
+        metric_variants: MetricVariants = None,
+        key: ScalarKeyEnum = ScalarKeyEnum.timestamp,
+        **kwargs,
+    ) -> TaskEventsResult:
+        if check_empty_data(self.es, company_id, event_type):
+            return TaskEventsResult()
+
+        from_key_value = kwargs.pop("from_timestamp", from_key_value)
+
+        res = TaskEventsResult()
+        res.events, res.total_events = self._get_events(
+            event_type=event_type,
+            company_id=company_id,
+            task_id=task_id,
+            batch_size=batch_size,
+            navigate_earlier=navigate_earlier,
+            from_key_value=from_key_value,
+            metric_variants=metric_variants,
+            key=ScalarKey.resolve(key),
+        )
+        return res
+
+    def count_task_events(
+        self,
+        event_type: EventType,
+        company_id: str,
+        task_id: str,
+        metric_variants: MetricVariants = None,
+    ) -> int:
+        if check_empty_data(self.es, company_id, event_type):
+            return 0
+
+        query, _ = self._get_initial_query_and_must(task_id, metric_variants)
+        es_req = {
+            "query": query,
+        }
+
+        with translate_errors_context(), TimingContext("es", "count_task_events"):
+            es_result = count_company_events(
+                self.es, company_id=company_id, event_type=event_type, body=es_req,
+            )
+
+            return es_result["count"]
+
+    def _get_events(
+        self,
+        event_type: EventType,
+        company_id: str,
+        task_id: str,
+        batch_size: int,
+        navigate_earlier: bool,
+        key: ScalarKey,
+        from_key_value: Optional[Any],
+        metric_variants: MetricVariants = None,
+    ) -> Tuple[Sequence[dict], int]:
+        """
+        Return up to 'batch size' events starting from the previous key-field value (timestamp or iter) either in the
+        direction of earlier events (navigate_earlier=True) or in the direction of later events.
+        If from_key_field is not set then start either from latest or earliest.
+        For the last key-field value all the events are brought (even if the resulting size exceeds batch_size)
+        so that events with this value will not be lost between the calls.
+        """
+        query, must = self._get_initial_query_and_must(task_id, metric_variants)
+
+        # retrieve the next batch of events
+        es_req = {
+            "size": batch_size,
+            "query": query,
+            "sort": {key.field: "desc" if navigate_earlier else "asc"},
+        }
+
+        if from_key_value:
+            es_req["search_after"] = [from_key_value]
+
+        with translate_errors_context(), TimingContext("es", "get_task_events"):
+            es_result = search_company_events(
+                self.es, company_id=company_id, event_type=event_type, body=es_req,
+            )
+            hits = es_result["hits"]["hits"]
+            hits_total = es_result["hits"]["total"]["value"]
+            if not hits:
+                return [], hits_total
+
+            events = [hit["_source"] for hit in hits]
+
+            # retrieve the events that match the last event timestamp
+            # but did not make it into the previous call due to batch_size limitation
+            es_req = {
+                "size": 10000,
+                "query": {
+                    "bool": {
+                        "must": must + [{"term": {key.field: events[-1][key.field]}}]
+                    }
+                },
+            }
+            es_result = search_company_events(
+                self.es, company_id=company_id, event_type=event_type, body=es_req,
+            )
+            last_second_hits = es_result["hits"]["hits"]
+            if not last_second_hits or len(last_second_hits) < 2:
+                # if only one element is returned for the last timestamp
+                # then it is already present in the events
+                return events, hits_total
+
+            already_present_ids = set(hit["_id"] for hit in hits)
+            last_second_events = [
+                hit["_source"]
+                for hit in last_second_hits
+                if hit["_id"] not in already_present_ids
+            ]
+
+            # return the list merged from original query results +
+            # leftovers from the last timestamp
+            return (
+                [*events, *last_second_events],
+                hits_total,
+            )
+
+    @staticmethod
+    def _get_initial_query_and_must(
+        task_id: str, metric_variants: MetricVariants = None
+    ) -> Tuple[dict, list]:
+        if not metric_variants:
+            must = [{"term": {"task": task_id}}]
+            query = {"term": {"task": task_id}}
+        else:
+            must = [
+                {"term": {"task": task_id}},
+                get_metric_variants_condition(metric_variants),
+            ]
+            query = {"bool": {"must": must}}
+        return query, must
+
+
+class Scroll(jsonmodels.models.Base):
+    def get_scroll_id(self) -> str:
+        return jwt.encode(
+            self.to_struct(),
+            key=config.get(
+                "services.events.events_retrieval.scroll_id_key", "1234567890"
+            ),
+        )
+
+    @classmethod
+    def from_scroll_id(cls, scroll_id: str):
+        try:
+            return cls(
+                **jwt.decode(
+                    scroll_id,
+                    key=config.get(
+                        "services.events.events_retrieval.scroll_id_key", "1234567890"
+                    ),
+                    algorithms=get_default_algorithms(),
+                )
+            )
+        except jwt.PyJWTError:
+            raise ValueError("Invalid Scroll ID")

apiserver/bll/event/history_debug_image_iterator.py

@@ -0,0 +1,56 @@
+from typing import Sequence, Tuple, Callable
+
+from elasticsearch import Elasticsearch
+from redis.client import StrictRedis
+
+from apiserver.utilities.dicts import nested_get
+from .event_common import EventType
+from .history_sample_iterator import HistorySampleIterator, VariantState
+
+
+class HistoryDebugImageIterator(HistorySampleIterator):
+    def __init__(self, redis: StrictRedis, es: Elasticsearch):
+        super().__init__(redis, es, EventType.metrics_image)
+
+    def _get_extra_conditions(self) -> Sequence[dict]:
+        return [{"exists": {"field": "url"}}]
+
+    def _get_variants_conditions(self, variants: Sequence[VariantState]) -> dict:
+        variants_conditions = [
+            {
+                "bool": {
+                    "must": [
+                        {"term": {"variant": v.name}},
+                        {"range": {"iter": {"gte": v.min_iteration}}},
+                    ]
+                }
+            }
+            for v in variants
+        ]
+        return {"bool": {"should": variants_conditions}}
+
+    def _process_event(self, event: dict) -> dict:
+        return event
+
+    def _get_min_max_aggs(self) -> Tuple[dict, Callable[[dict], Tuple[int, int]]]:
+        # The min iteration is the lowest iteration that contains non-recycled image url
+        aggs = {
+            "last_iter": {"max": {"field": "iter"}},
+            "urls": {
+                # group by urls and choose the minimal iteration
+                # from all the maximal iterations per url
+                "terms": {"field": "url", "order": {"max_iter": "asc"}, "size": 1},
+                "aggs": {
+                    # find max iteration for each url
+                    "max_iter": {"max": {"field": "iter"}}
+                },
+            },
+        }
+
+        def get_min_max_data(variant_bucket: dict) -> Tuple[int, int]:
+            urls = nested_get(variant_bucket, ("urls", "buckets"))
+            min_iter = int(urls[0]["max_iter"]["value"])
+            max_iter = int(variant_bucket["last_iter"]["value"])
+            return min_iter, max_iter
+
+        return aggs, get_min_max_data

apiserver/bll/event/history_plot_iterator.py

@@ -0,0 +1,36 @@
+from typing import Sequence, Tuple, Callable
+
+from elasticsearch import Elasticsearch
+from redis.client import StrictRedis
+
+from .event_common import EventType, uncompress_plot
+from .history_sample_iterator import HistorySampleIterator, VariantState
+
+
+class HistoryPlotIterator(HistorySampleIterator):
+    def __init__(self, redis: StrictRedis, es: Elasticsearch):
+        super().__init__(redis, es, EventType.metrics_plot)
+
+    def _get_extra_conditions(self) -> Sequence[dict]:
+        return []
+
+    def _get_variants_conditions(self, variants: Sequence[VariantState]) -> dict:
+        return {"terms": {"variant": [v.name for v in variants]}}
+
+    def _process_event(self, event: dict) -> dict:
+        uncompress_plot(event)
+        return event
+
+    def _get_min_max_aggs(self) -> Tuple[dict, Callable[[dict], Tuple[int, int]]]:
+        # The min iteration is the lowest iteration that contains non-recycled image url
+        aggs = {
+            "last_iter": {"max": {"field": "iter"}},
+            "first_iter": {"min": {"field": "iter"}},
+        }
+
+        def get_min_max_data(variant_bucket: dict) -> Tuple[int, int]:
+            min_iter = int(variant_bucket["first_iter"]["value"])
+            max_iter = int(variant_bucket["last_iter"]["value"])
+            return min_iter, max_iter
+
+        return aggs, get_min_max_data

apiserver/bll/event/history_sample_iterator.py

@@ -0,0 +1,442 @@
+import abc
+import operator
+from operator import attrgetter
+from typing import Sequence, Tuple, Optional, Callable, Mapping
+
+import attr
+from boltons.iterutils import first, bucketize
+from elasticsearch import Elasticsearch
+from jsonmodels.fields import StringField, ListField, IntField, BoolField
+from jsonmodels.models import Base
+from redis import StrictRedis
+
+from apiserver.apierrors import errors
+from apiserver.apimodels import JsonSerializableMixin
+from apiserver.bll.event.event_common import (
+    EventSettings,
+    EventType,
+    check_empty_data,
+    search_company_events,
+    get_max_metric_and_variant_counts,
+)
+from apiserver.bll.redis_cache_manager import RedisCacheManager
+from apiserver.database.errors import translate_errors_context
+from apiserver.timing_context import TimingContext
+from apiserver.utilities.dicts import nested_get
+
+
+class VariantState(Base):
+    name: str = StringField(required=True)
+    metric: str = StringField(default=None)
+    min_iteration: int = IntField()
+    max_iteration: int = IntField()
+
+
+class HistorySampleState(Base, JsonSerializableMixin):
+    id: str = StringField(required=True)
+    iteration: int = IntField()
+    variant: str = StringField()
+    task: str = StringField()
+    metric: str = StringField()
+    reached_first: bool = BoolField()
+    reached_last: bool = BoolField()
+    variant_states: Sequence[VariantState] = ListField([VariantState])
+    warning: str = StringField()
+    navigate_current_metric = BoolField(default=True)
+
+
+@attr.s(auto_attribs=True)
+class HistorySampleResult(object):
+    scroll_id: str = None
+    event: dict = None
+    min_iteration: int = None
+    max_iteration: int = None
+
+
+class HistorySampleIterator(abc.ABC):
+    def __init__(self, redis: StrictRedis, es: Elasticsearch, event_type: EventType):
+        self.es = es
+        self.event_type = event_type
+        self.cache_manager = RedisCacheManager(
+            state_class=HistorySampleState,
+            redis=redis,
+            expiration_interval=EventSettings.state_expiration_sec,
+        )
+
+    def get_next_sample(
+        self, company_id: str, task: str, state_id: str, navigate_earlier: bool
+    ) -> HistorySampleResult:
+        """
+        Get the sample for next/prev variant on the current iteration
+        If does not exist then try getting sample for the first/last variant from next/prev iteration
+        """
+        res = HistorySampleResult(scroll_id=state_id)
+        state = self.cache_manager.get_state(state_id)
+        if not state or state.task != task:
+            raise errors.bad_request.InvalidScrollId(scroll_id=state_id)
+
+        if check_empty_data(self.es, company_id=company_id, event_type=self.event_type):
+            return res
+
+        event = self._get_next_for_current_iteration(
+            company_id=company_id, navigate_earlier=navigate_earlier, state=state
+        ) or self._get_next_for_another_iteration(
+            company_id=company_id, navigate_earlier=navigate_earlier, state=state
+        )
+        if not event:
+            return res
+
+        self._fill_res_and_update_state(event=event, res=res, state=state)
+        self.cache_manager.set_state(state=state)
+        return res
+
+    def _fill_res_and_update_state(
+        self, event: dict, res: HistorySampleResult, state: HistorySampleState
+    ):
+        self._process_event(event)
+        state.variant = event["variant"]
+        state.metric = event["metric"]
+        state.iteration = event["iter"]
+        res.event = event
+        var_state = first(
+            vs
+            for vs in state.variant_states
+            if vs.name == state.variant and vs.metric == state.metric
+        )
+        if var_state:
+            res.min_iteration = var_state.min_iteration
+            res.max_iteration = var_state.max_iteration
+
+    @abc.abstractmethod
+    def _get_extra_conditions(self) -> Sequence[dict]:
+        pass
+
+    @abc.abstractmethod
+    def _process_event(self, event: dict) -> dict:
+        pass
+
+    @abc.abstractmethod
+    def _get_variants_conditions(self, variants: Sequence[VariantState]) -> dict:
+        pass
+
+    def _get_metric_variants_condition(self, variants: Sequence[VariantState]) -> dict:
+        metrics = bucketize(variants, key=attrgetter("metric"))
+        metrics_conditions = [
+            {
+                "bool": {
+                    "must": [
+                        {"term": {"metric": metric}},
+                        self._get_variants_conditions(vs),
+                    ]
+                }
+            }
+            for metric, vs in metrics.items()
+        ]
+        return {"bool": {"should": metrics_conditions}}
+
+    def _get_next_for_current_iteration(
+        self, company_id: str, navigate_earlier: bool, state: HistorySampleState
+    ) -> Optional[dict]:
+        """
+        Get the sample for next (if navigate_earlier is False) or previous variant sorted by name for the same iteration
+        Only variants for which the iteration falls into their valid range are considered
+        Return None if no such variant or sample is found
+        """
+        if state.navigate_current_metric:
+            variants = [
+                var_state
+                for var_state in state.variant_states
+                if var_state.metric == state.metric
+            ]
+        else:
+            variants = state.variant_states
+
+        cmp = operator.lt if navigate_earlier else operator.gt
+        variants = [
+            var_state
+            for var_state in variants
+            if cmp((var_state.metric, var_state.name), (state.metric, state.variant))
+            and var_state.min_iteration <= state.iteration
+        ]
+        if not variants:
+            return
+
+        must_conditions = [
+            {"term": {"task": state.task}},
+            {"term": {"iter": state.iteration}},
+            self._get_metric_variants_condition(variants),
+            *self._get_extra_conditions(),
+        ]
+        order = "desc" if navigate_earlier else "asc"
+        es_req = {
+            "size": 1,
+            "sort": [{"metric": order}, {"variant": order}],
+            "query": {"bool": {"must": must_conditions}},
+        }
+
+        with translate_errors_context(), TimingContext(
+            "es", "get_next_for_current_iteration"
+        ):
+            es_res = search_company_events(
+                self.es,
+                company_id=company_id,
+                event_type=self.event_type,
+                body=es_req,
+            )
+
+        hits = nested_get(es_res, ("hits", "hits"))
+        if not hits:
+            return
+
+        return hits[0]["_source"]
+
+    def _get_next_for_another_iteration(
+        self, company_id: str, navigate_earlier: bool, state: HistorySampleState
+    ) -> Optional[dict]:
+        """
+        Get the sample for the first variant for the next iteration (if navigate_earlier is set to False)
+        or from the last variant for the previous iteration (otherwise)
+        The variants for which the sample falls in invalid range are discarded
+        If no suitable sample is found then None is returned
+        """
+        if state.navigate_current_metric:
+            variants = [
+                var_state
+                for var_state in state.variant_states
+                if var_state.metric == state.metric
+            ]
+        else:
+            variants = state.variant_states
+
+        if navigate_earlier:
+            range_operator = "lt"
+            order = "desc"
+            variants = [
+                var_state
+                for var_state in variants
+                if var_state.min_iteration < state.iteration
+            ]
+        else:
+            range_operator = "gt"
+            order = "asc"
+            variants = variants
+
+        if not variants:
+            return
+
+        must_conditions = [
+            {"term": {"task": state.task}},
+            self._get_metric_variants_condition(variants),
+            {"range": {"iter": {range_operator: state.iteration}}},
+            *self._get_extra_conditions(),
+        ]
+        es_req = {
+            "size": 1,
+            "sort": [{"iter": order}, {"metric": order}, {"variant": order}],
+            "query": {"bool": {"must": must_conditions}},
+        }
+        with translate_errors_context(), TimingContext(
+            "es", "get_next_for_another_iteration"
+        ):
+            es_res = search_company_events(
+                self.es,
+                company_id=company_id,
+                event_type=self.event_type,
+                body=es_req,
+            )
+
+        hits = nested_get(es_res, ("hits", "hits"))
+        if not hits:
+            return
+
+        return hits[0]["_source"]
+
+    def get_sample_for_variant(
+        self,
+        company_id: str,
+        task: str,
+        metric: str,
+        variant: str,
+        iteration: Optional[int] = None,
+        refresh: bool = False,
+        state_id: str = None,
+        navigate_current_metric: bool = True,
+    ) -> HistorySampleResult:
+        """
+        Get the sample for the requested iteration or the latest before it
+        If the iteration is not passed then get the latest event
+        """
+        res = HistorySampleResult()
+        if check_empty_data(self.es, company_id=company_id, event_type=self.event_type):
+            return res
+
+        def init_state(state_: HistorySampleState):
+            state_.task = task
+            state_.metric = metric
+            state_.navigate_current_metric = navigate_current_metric
+            self._reset_variant_states(company_id=company_id, state=state_)
+
+        def validate_state(state_: HistorySampleState):
+            if (
+                state_.task != task
+                or state_.navigate_current_metric != navigate_current_metric
+                or (state_.navigate_current_metric and state_.metric != metric)
+            ):
+                raise errors.bad_request.InvalidScrollId(
+                    "Task and metric stored in the state do not match the passed ones",
+                    scroll_id=state_.id,
+                )
+            # fix old variant states:
+            for vs in state_.variant_states:
+                if vs.metric is None:
+                    vs.metric = metric
+            if refresh:
+                self._reset_variant_states(company_id=company_id, state=state_)
+
+        state: HistorySampleState
+        with self.cache_manager.get_or_create_state(
+            state_id=state_id, init_state=init_state, validate_state=validate_state,
+        ) as state:
+            res.scroll_id = state.id
+
+            var_state = first(
+                vs
+                for vs in state.variant_states
+                if vs.name == variant and vs.metric == metric
+            )
+            if not var_state:
+                return res
+
+            res.min_iteration = var_state.min_iteration
+            res.max_iteration = var_state.max_iteration
+
+            must_conditions = [
+                {"term": {"task": task}},
+                {"term": {"metric": metric}},
+                {"term": {"variant": variant}},
+                *self._get_extra_conditions(),
+            ]
+            if iteration is not None:
+                must_conditions.append(
+                    {
+                        "range": {
+                            "iter": {"lte": iteration, "gte": var_state.min_iteration}
+                        }
+                    }
+                )
+            else:
+                must_conditions.append(
+                    {"range": {"iter": {"gte": var_state.min_iteration}}}
+                )
+
+            es_req = {
+                "size": 1,
+                "sort": {"iter": "desc"},
+                "query": {"bool": {"must": must_conditions}},
+            }
+
+            with translate_errors_context(), TimingContext(
+                "es", "get_history_sample_for_variant"
+            ):
+                es_res = search_company_events(
+                    self.es,
+                    company_id=company_id,
+                    event_type=self.event_type,
+                    body=es_req,
+                )
+
+            hits = nested_get(es_res, ("hits", "hits"))
+            if not hits:
+                return res
+
+            self._fill_res_and_update_state(
+                event=hits[0]["_source"], res=res, state=state
+            )
+            return res
+
+    def _reset_variant_states(self, company_id: str, state: HistorySampleState):
+        metrics = self._get_metric_variant_iterations(
+            company_id=company_id,
+            task=state.task,
+            metric=state.metric if state.navigate_current_metric else None,
+        )
+        state.variant_states = [
+            VariantState(
+                metric=metric,
+                name=var_name,
+                min_iteration=min_iter,
+                max_iteration=max_iter,
+            )
+            for metric, variants in metrics.items()
+            for var_name, min_iter, max_iter in variants
+        ]
+
+    @abc.abstractmethod
+    def _get_min_max_aggs(self) -> Tuple[dict, Callable[[dict], Tuple[int, int]]]:
+        pass
+
+    def _get_metric_variant_iterations(
+        self, company_id: str, task: str, metric: str,
+    ) -> Mapping[str, Tuple[str, str, int, int]]:
+        """
+        Return valid min and max iterations that the task reported events of the required type
+        """
+        must = [
+            {"term": {"task": task}},
+            *self._get_extra_conditions(),
+        ]
+        if metric is not None:
+            must.append({"term": {"metric": metric}})
+        query = {"bool": {"must": must}}
+
+        search_args = dict(
+            es=self.es, company_id=company_id, event_type=self.event_type
+        )
+        max_metrics, max_variants = get_max_metric_and_variant_counts(
+            query=query, **search_args
+        )
+        max_variants = int(max_variants // 2)
+        min_max_aggs, get_min_max_data = self._get_min_max_aggs()
+        es_req: dict = {
+            "size": 0,
+            "query": query,
+            "aggs": {
+                "metrics": {
+                    "terms": {
+                        "field": "metric",
+                        "size": max_metrics,
+                        "order": {"_key": "asc"},
+                    },
+                    "aggs": {
+                        "variants": {
+                            "terms": {
+                                "field": "variant",
+                                "size": max_variants,
+                                "order": {"_key": "asc"},
+                            },
+                            "aggs": min_max_aggs,
+                        }
+                    },
+                }
+            },
+        }
+
+        with translate_errors_context(), TimingContext(
+            "es", "get_history_sample_iterations"
+        ):
+            es_res = search_company_events(body=es_req, **search_args,)
+
+        def get_variant_data(variant_bucket: dict) -> Tuple[str, int, int]:
+            variant = variant_bucket["key"]
+            min_iter, max_iter = get_min_max_data(variant_bucket)
+            return variant, min_iter, max_iter
+
+        return {
+            metric_bucket["key"]: [
+                get_variant_data(variant_bucket)
+                for variant_bucket in nested_get(metric_bucket, ("variants", "buckets"))
+            ]
+            for metric_bucket in nested_get(
+                es_res, ("aggregations", "metrics", "buckets")
+            )
+        }

apiserver/bll/event/metric_debug_images_iterator.py

@@ -0,0 +1,53 @@
+from typing import Sequence, Tuple, Callable
+
+from elasticsearch import Elasticsearch
+from redis.client import StrictRedis
+
+from apiserver.utilities.dicts import nested_get
+from .event_common import EventType
+from .metric_events_iterator import MetricEventsIterator, VariantState
+
+
+class MetricDebugImagesIterator(MetricEventsIterator):
+    def __init__(self, redis: StrictRedis, es: Elasticsearch):
+        super().__init__(redis, es, EventType.metrics_image)
+
+    def _get_extra_conditions(self) -> Sequence[dict]:
+        return [{"exists": {"field": "url"}}]
+
+    def _get_variant_state_aggs(self) -> Tuple[dict, Callable[[dict, VariantState], None]]:
+        aggs = {
+            "urls": {
+                "terms": {
+                    "field": "url",
+                    "order": {"max_iter": "desc"},
+                    "size": 1,  # we need only one url from the most recent iteration
+                },
+                "aggs": {
+                    "max_iter": {"max": {"field": "iter"}},
+                    "iters": {
+                        "top_hits": {
+                            "sort": {"iter": {"order": "desc"}},
+                            "size": 2,  # need two last iterations so that we can take
+                            # the second one as invalid
+                            "_source": "iter",
+                        }
+                    },
+                },
+            }
+        }
+
+        def fill_variant_state_data(variant_bucket: dict,  state: VariantState):
+            """If the image urls get recycled then fill the last_invalid_iteration field"""
+            top_iter_url = nested_get(variant_bucket, ("urls", "buckets"))[0]
+            iters = nested_get(top_iter_url, ("iters", "hits", "hits"))
+            if len(iters) > 1:
+                state.last_invalid_iteration = nested_get(iters[1], ("_source", "iter"))
+
+        return aggs, fill_variant_state_data
+
+    def _process_event(self, event: dict) -> dict:
+        return event
+
+    def _get_same_variant_events_order(self) -> dict:
+        return {"url": {"order": "desc"}}

apiserver/bll/event/metric_events_iterator.py

@@ -0,0 +1,439 @@
+import abc
+from concurrent.futures.thread import ThreadPoolExecutor
+from datetime import datetime
+from functools import partial
+from operator import itemgetter
+from typing import Sequence, Tuple, Optional, Mapping, Callable
+
+import attr
+import dpath
+from boltons.iterutils import first
+from elasticsearch import Elasticsearch
+from jsonmodels.fields import StringField, ListField, IntField
+from jsonmodels.models import Base
+from redis import StrictRedis
+
+from apiserver.apimodels import JsonSerializableMixin
+from apiserver.bll.event.event_common import (
+    EventSettings,
+    check_empty_data,
+    search_company_events,
+    EventType,
+    get_metric_variants_condition, get_max_metric_and_variant_counts,
+)
+from apiserver.bll.redis_cache_manager import RedisCacheManager
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.task.metrics import MetricEventStats
+from apiserver.database.model.task.task import Task
+from apiserver.timing_context import TimingContext
+
+
+class VariantState(Base):
+    variant: str = StringField(required=True)
+    last_invalid_iteration: int = IntField()
+
+
+class MetricState(Base):
+    metric: str = StringField(required=True)
+    variants: Sequence[VariantState] = ListField([VariantState], required=True)
+    timestamp: int = IntField(default=0)
+
+
+class TaskScrollState(Base):
+    task: str = StringField(required=True)
+    metrics: Sequence[MetricState] = ListField([MetricState], required=True)
+    last_min_iter: Optional[int] = IntField()
+    last_max_iter: Optional[int] = IntField()
+
+    def reset(self):
+        """Reset the scrolling state for the metric"""
+        self.last_min_iter = self.last_max_iter = None
+
+
+class MetricEventsScrollState(Base, JsonSerializableMixin):
+    id: str = StringField(required=True)
+    tasks: Sequence[TaskScrollState] = ListField([TaskScrollState])
+    warning: str = StringField()
+
+
+@attr.s(auto_attribs=True)
+class MetricEventsResult(object):
+    metric_events: Sequence[tuple] = []
+    next_scroll_id: str = None
+
+
+class MetricEventsIterator:
+    def __init__(self, redis: StrictRedis, es: Elasticsearch, event_type: EventType):
+        self.es = es
+        self.event_type = event_type
+        self.cache_manager = RedisCacheManager(
+            state_class=MetricEventsScrollState,
+            redis=redis,
+            expiration_interval=EventSettings.state_expiration_sec,
+        )
+
+    def get_task_events(
+        self,
+        company_id: str,
+        task_metrics: Mapping[str, dict],
+        iter_count: int,
+        navigate_earlier: bool = True,
+        refresh: bool = False,
+        state_id: str = None,
+    ) -> MetricEventsResult:
+        if check_empty_data(self.es, company_id, self.event_type):
+            return MetricEventsResult()
+
+        def init_state(state_: MetricEventsScrollState):
+            state_.tasks = self._init_task_states(company_id, task_metrics)
+
+        def validate_state(state_: MetricEventsScrollState):
+            """
+            Validate that the metrics stored in the state are the same
+            as requested in the current call.
+            Refresh the state if requested
+            """
+            if refresh:
+                self._reinit_outdated_task_states(company_id, state_, task_metrics)
+
+        with self.cache_manager.get_or_create_state(
+            state_id=state_id, init_state=init_state, validate_state=validate_state
+        ) as state:
+            res = MetricEventsResult(next_scroll_id=state.id)
+            specific_variants_requested = any(
+                variants
+                for t, metrics in task_metrics.items()
+                if metrics
+                for m, variants in metrics.items()
+            )
+            with ThreadPoolExecutor(EventSettings.max_workers) as pool:
+                res.metric_events = list(
+                    pool.map(
+                        partial(
+                            self._get_task_metric_events,
+                            company_id=company_id,
+                            iter_count=iter_count,
+                            navigate_earlier=navigate_earlier,
+                            specific_variants_requested=specific_variants_requested,
+                        ),
+                        state.tasks,
+                    )
+                )
+
+            return res
+
+    def _reinit_outdated_task_states(
+        self,
+        company_id,
+        state: MetricEventsScrollState,
+        task_metrics: Mapping[str, dict],
+    ):
+        """
+        Determine the metrics for which new event_type events were added
+        since their states were initialized and re-init these states
+        """
+        tasks = Task.objects(id__in=list(task_metrics), company=company_id).only(
+            "id", "metric_stats"
+        )
+
+        def get_last_update_times_for_task_metrics(
+            task: Task,
+        ) -> Mapping[str, datetime]:
+            """For metrics that reported event_type events get mapping of the metric name to the last update times"""
+            metric_stats: Mapping[str, MetricEventStats] = task.metric_stats
+            if not metric_stats:
+                return {}
+
+            requested_metrics = task_metrics[task.id]
+            return {
+                stats.metric: stats.event_stats_by_type[
+                    self.event_type.value
+                ].last_update
+                for stats in metric_stats.values()
+                if self.event_type.value in stats.event_stats_by_type
+                and (not requested_metrics or stats.metric in requested_metrics)
+            }
+
+        update_times = {
+            task.id: get_last_update_times_for_task_metrics(task) for task in tasks
+        }
+        task_metric_states = {
+            task_state.task: {
+                metric_state.metric: metric_state for metric_state in task_state.metrics
+            }
+            for task_state in state.tasks
+        }
+        task_metrics_to_recalc = {}
+        for task, metrics_times in update_times.items():
+            old_metric_states = task_metric_states[task]
+            metrics_to_recalc = {
+                m: task_metrics[task].get(m)
+                for m, t in metrics_times.items()
+                if m not in old_metric_states or old_metric_states[m].timestamp < t
+            }
+            if metrics_to_recalc:
+                task_metrics_to_recalc[task] = metrics_to_recalc
+
+        updated_task_states = self._init_task_states(company_id, task_metrics_to_recalc)
+
+        def merge_with_updated_task_states(
+            old_state: TaskScrollState, updates: Sequence[TaskScrollState]
+        ) -> TaskScrollState:
+            task = old_state.task
+            updated_state = first(uts for uts in updates if uts.task == task)
+            if not updated_state:
+                old_state.reset()
+                return old_state
+
+            updated_metrics = [m.metric for m in updated_state.metrics]
+            return TaskScrollState(
+                task=task,
+                metrics=[
+                    *updated_state.metrics,
+                    *(
+                        old_metric
+                        for old_metric in old_state.metrics
+                        if old_metric.metric not in updated_metrics
+                    ),
+                ],
+            )
+
+        state.tasks = [
+            merge_with_updated_task_states(task_state, updated_task_states)
+            for task_state in state.tasks
+        ]
+
+    def _init_task_states(
+        self, company_id: str, task_metrics: Mapping[str, dict]
+    ) -> Sequence[TaskScrollState]:
+        """
+        Returned initialized metric scroll stated for the requested task metrics
+        """
+        with ThreadPoolExecutor(EventSettings.max_workers) as pool:
+            task_metric_states = pool.map(
+                partial(self._init_metric_states_for_task, company_id=company_id),
+                task_metrics.items(),
+            )
+
+        return [
+            TaskScrollState(task=task, metrics=metric_states,)
+            for task, metric_states in zip(task_metrics, task_metric_states)
+        ]
+
+    @abc.abstractmethod
+    def _get_extra_conditions(self) -> Sequence[dict]:
+        pass
+
+    @abc.abstractmethod
+    def _get_variant_state_aggs(self) -> Tuple[dict, Callable[[dict, VariantState], None]]:
+        pass
+
+    def _init_metric_states_for_task(
+        self, task_metrics: Tuple[str, dict], company_id: str
+    ) -> Sequence[MetricState]:
+        """
+        Return metric scroll states for the task filled with the variant states
+        for the variants that reported any event_type events
+        """
+        task, metrics = task_metrics
+        must = [{"term": {"task": task}}, *self._get_extra_conditions()]
+        if metrics:
+            must.append(get_metric_variants_condition(metrics))
+        query = {"bool": {"must": must}}
+
+        search_args = dict(
+            es=self.es, company_id=company_id, event_type=self.event_type
+        )
+        max_metrics, max_variants = get_max_metric_and_variant_counts(
+            query=query, **search_args
+        )
+        max_variants = int(max_variants // 2)
+        variant_state_aggs, fill_variant_state_data = self._get_variant_state_aggs()
+        es_req: dict = {
+            "size": 0,
+            "query": query,
+            "aggs": {
+                "metrics": {
+                    "terms": {
+                        "field": "metric",
+                        "size": max_metrics,
+                        "order": {"_key": "asc"},
+                    },
+                    "aggs": {
+                        "last_event_timestamp": {"max": {"field": "timestamp"}},
+                        "variants": {
+                            "terms": {
+                                "field": "variant",
+                                "size": max_variants,
+                                "order": {"_key": "asc"},
+                            },
+                            **({"aggs": variant_state_aggs} if variant_state_aggs else {}),
+                        },
+                    },
+                }
+            },
+        }
+
+        with translate_errors_context(), TimingContext("es", "_init_metric_states"):
+            es_res = search_company_events(body=es_req, **search_args)
+        if "aggregations" not in es_res:
+            return []
+
+        def init_variant_state(variant: dict):
+            """
+            Return new variant state for the passed variant bucket
+            """
+            state = VariantState(variant=variant["key"])
+            if fill_variant_state_data:
+                fill_variant_state_data(variant, state)
+
+            return state
+
+        return [
+            MetricState(
+                metric=metric["key"],
+                timestamp=dpath.get(metric, "last_event_timestamp/value"),
+                variants=[
+                    init_variant_state(variant)
+                    for variant in dpath.get(metric, "variants/buckets")
+                ],
+            )
+            for metric in dpath.get(es_res, "aggregations/metrics/buckets")
+        ]
+
+    @abc.abstractmethod
+    def _process_event(self, event: dict) -> dict:
+        pass
+
+    @abc.abstractmethod
+    def _get_same_variant_events_order(self) -> dict:
+        pass
+
+    def _get_task_metric_events(
+        self,
+        task_state: TaskScrollState,
+        company_id: str,
+        iter_count: int,
+        navigate_earlier: bool,
+        specific_variants_requested: bool,
+    ) -> Tuple:
+        """
+        Return task metric events grouped by iterations
+        Update task scroll state
+        """
+        if not task_state.metrics:
+            return task_state.task, []
+
+        if task_state.last_max_iter is None:
+            # the first fetch is always from the latest iteration to the earlier ones
+            navigate_earlier = True
+
+        must_conditions = [
+            {"term": {"task": task_state.task}},
+            {"terms": {"metric": [m.metric for m in task_state.metrics]}},
+            *self._get_extra_conditions(),
+        ]
+
+        range_condition = None
+        if navigate_earlier and task_state.last_min_iter is not None:
+            range_condition = {"lt": task_state.last_min_iter}
+        elif not navigate_earlier and task_state.last_max_iter is not None:
+            range_condition = {"gt": task_state.last_max_iter}
+        if range_condition:
+            must_conditions.append({"range": {"iter": range_condition}})
+
+        metrics_count = len(task_state.metrics)
+        max_variants = int(EventSettings.max_es_buckets / (metrics_count * iter_count))
+        es_req = {
+            "size": 0,
+            "query": {"bool": {"must": must_conditions}},
+            "aggs": {
+                "iters": {
+                    "terms": {
+                        "field": "iter",
+                        "size": iter_count,
+                        "order": {"_key": "desc" if navigate_earlier else "asc"},
+                    },
+                    "aggs": {
+                        "metrics": {
+                            "terms": {
+                                "field": "metric",
+                                "size": metrics_count,
+                                "order": {"_key": "asc"},
+                            },
+                            "aggs": {
+                                "variants": {
+                                    "terms": {
+                                        "field": "variant",
+                                        "size": max_variants,
+                                        "order": {"_key": "asc"},
+                                    },
+                                    "aggs": {
+                                        "events": {
+                                            "top_hits": {
+                                                "sort": self._get_same_variant_events_order()
+                                            }
+                                        }
+                                    },
+                                }
+                            },
+                        }
+                    },
+                }
+            },
+        }
+        with translate_errors_context(), TimingContext("es", "_get_task_metric_events"):
+            es_res = search_company_events(
+                self.es,
+                company_id=company_id,
+                event_type=self.event_type,
+                body=es_req,
+            )
+        if "aggregations" not in es_res:
+            return task_state.task, []
+
+        invalid_iterations = {
+            (m.metric, v.variant): v.last_invalid_iteration
+            for m in task_state.metrics
+            for v in m.variants
+        }
+        allow_uninitialized = (
+            False
+            if specific_variants_requested
+            else config.get(
+                "services.events.events_retrieval.debug_images.allow_uninitialized_variants",
+                False,
+            )
+        )
+
+        def is_valid_event(event: dict) -> bool:
+            key = event.get("metric"), event.get("variant")
+            if key not in invalid_iterations:
+                return allow_uninitialized
+
+            max_invalid = invalid_iterations[key]
+            return max_invalid is None or event.get("iter") > max_invalid
+
+        def get_iteration_events(it_: dict) -> Sequence:
+            return [
+                self._process_event(ev["_source"])
+                for m in dpath.get(it_, "metrics/buckets")
+                for v in dpath.get(m, "variants/buckets")
+                for ev in dpath.get(v, "events/hits/hits")
+                if is_valid_event(ev["_source"])
+            ]
+
+        iterations = []
+        for it in dpath.get(es_res, "aggregations/iters/buckets"):
+            events = get_iteration_events(it)
+            if events:
+                iterations.append({"iter": it["key"], "events": events})
+
+        if not navigate_earlier:
+            iterations.sort(key=itemgetter("iter"), reverse=True)
+        if iterations:
+            task_state.last_max_iter = iterations[0]["iter"]
+            task_state.last_min_iter = iterations[-1]["iter"]
+
+        return task_state.task, iterations

apiserver/bll/event/metric_plots_iterator.py

@@ -0,0 +1,25 @@
+from typing import Sequence
+
+from elasticsearch import Elasticsearch
+from redis.client import StrictRedis
+
+from .event_common import EventType, uncompress_plot
+from .metric_events_iterator import MetricEventsIterator
+
+
+class MetricPlotsIterator(MetricEventsIterator):
+    def __init__(self, redis: StrictRedis, es: Elasticsearch):
+        super().__init__(redis, es, EventType.metrics_plot)
+
+    def _get_extra_conditions(self) -> Sequence[dict]:
+        return []
+
+    def _get_variant_state_aggs(self):
+        return None, None
+
+    def _process_event(self, event: dict) -> dict:
+        uncompress_plot(event)
+        return event
+
+    def _get_same_variant_events_order(self) -> dict:
+        return {"timestamp": {"order": "desc"}}

apiserver/bll/event/scalar_key.py

@@ -4,9 +4,11 @@ Module for polymorphism over different types of X axes in scalar aggregations
 from abc import ABC, abstractmethod
 from enum import auto
 
-from utilities.stringenum import StringEnum
-from bll.util import extract_properties_to_lists
-from config import config
+from typing import Any
+
+from apiserver.utilities import extract_properties_to_lists
+from apiserver.utilities.stringenum import StringEnum
+from apiserver.config_repo import config
 
 log = config.logger(__file__)
 
@@ -96,6 +98,10 @@ class ScalarKey(ABC):
         """
         return int(iter_data[self.bucket_key_key]), iter_data["avg_val"]["value"]
 
+    def cast_value(self, value: Any) -> Any:
+        """Cast value to appropriate type"""
+        return value
+
 
 class TimestampKey(ScalarKey):
     """
@@ -117,6 +123,9 @@ class TimestampKey(ScalarKey):
             }
         }
 
+    def cast_value(self, value: Any) -> int:
+        return int(value)
+
 
 class IterKey(ScalarKey):
     """
@@ -134,6 +143,9 @@ class IterKey(ScalarKey):
             }
         }
 
+    def cast_value(self, value: Any) -> int:
+        return int(value)
+
 
 class ISOTimeKey(ScalarKey):
     """

apiserver/bll/model/__init__.py

@@ -0,0 +1,160 @@
+from datetime import datetime
+from typing import Callable, Tuple, Sequence, Dict
+
+from apiserver.apierrors import errors
+from apiserver.apimodels.models import ModelTaskPublishResponse
+from apiserver.bll.task.utils import deleted_prefix
+from apiserver.database.model import EntityVisibility
+from apiserver.database.model.model import Model
+from apiserver.database.model.task.task import Task, TaskStatus
+from .metadata import Metadata
+
+
+class ModelBLL:
+    @classmethod
+    def get_company_model_by_id(
+        cls, company_id: str, model_id: str, only_fields=None
+    ) -> Model:
+        query = dict(company=company_id, id=model_id)
+        qs = Model.objects(**query)
+        if only_fields:
+            qs = qs.only(*only_fields)
+        model = qs.first()
+        if not model:
+            raise errors.bad_request.InvalidModelId(**query)
+        return model
+
+    @classmethod
+    def publish_model(
+        cls,
+        model_id: str,
+        company_id: str,
+        force_publish_task: bool = False,
+        publish_task_func: Callable[[str, str, bool], dict] = None,
+    ) -> Tuple[int, ModelTaskPublishResponse]:
+        model = cls.get_company_model_by_id(company_id=company_id, model_id=model_id)
+        if model.ready:
+            raise errors.bad_request.ModelIsReady(company=company_id, model=model_id)
+
+        published_task = None
+        if model.task and publish_task_func:
+            task = (
+                Task.objects(id=model.task, company=company_id)
+                .only("id", "status")
+                .first()
+            )
+            if task and task.status != TaskStatus.published:
+                task_publish_res = publish_task_func(
+                    model.task, company_id, force_publish_task
+                )
+                published_task = ModelTaskPublishResponse(
+                    id=model.task, data=task_publish_res
+                )
+
+        updated = model.update(upsert=False, ready=True, last_update=datetime.utcnow())
+        return updated, published_task
+
+    @classmethod
+    def delete_model(
+        cls, model_id: str, company_id: str, force: bool
+    ) -> Tuple[int, Model]:
+        model = cls.get_company_model_by_id(
+            company_id=company_id,
+            model_id=model_id,
+            only_fields=("id", "task", "project", "uri"),
+        )
+        deleted_model_id = f"{deleted_prefix}{model_id}"
+
+        using_tasks = Task.objects(models__input__model=model_id).only("id")
+        if using_tasks:
+            if not force:
+                raise errors.bad_request.ModelInUse(
+                    "as execution model, use force=True to delete",
+                    num_tasks=len(using_tasks),
+                )
+            # update deleted model id in using tasks
+            Task._get_collection().update_many(
+                filter={"_id": {"$in": [t.id for t in using_tasks]}},
+                update={"$set": {"models.input.$[elem].model": deleted_model_id}},
+                array_filters=[{"elem.model": model_id}],
+                upsert=False,
+            )
+
+        if model.task:
+            task = Task.objects(id=model.task).first()
+            if task and task.status == TaskStatus.published:
+                if not force:
+                    raise errors.bad_request.ModelCreatingTaskExists(
+                        "and published, use force=True to delete", task=model.task
+                    )
+                if task.models.output and model_id in task.models.output:
+                    now = datetime.utcnow()
+                    Task._get_collection().update_one(
+                        filter={"_id": model.task, "models.output.model": model_id},
+                        update={
+                            "$set": {
+                                "models.output.$[elem].model": deleted_model_id,
+                                "output.error": f"model deleted on {now.isoformat()}",
+                            },
+                            "last_change": now,
+                        },
+                        array_filters=[{"elem.model": model_id}],
+                        upsert=False,
+                    )
+
+        del_count = Model.objects(id=model_id, company=company_id).delete()
+        return del_count, model
+
+    @classmethod
+    def archive_model(cls, model_id: str, company_id: str):
+        cls.get_company_model_by_id(
+            company_id=company_id, model_id=model_id, only_fields=("id",)
+        )
+        archived = Model.objects(company=company_id, id=model_id).update(
+            add_to_set__system_tags=EntityVisibility.archived.value,
+            last_update=datetime.utcnow(),
+        )
+
+        return archived
+
+    @classmethod
+    def unarchive_model(cls, model_id: str, company_id: str):
+        cls.get_company_model_by_id(
+            company_id=company_id, model_id=model_id, only_fields=("id",)
+        )
+        unarchived = Model.objects(company=company_id, id=model_id).update(
+            pull__system_tags=EntityVisibility.archived.value,
+            last_update=datetime.utcnow(),
+        )
+
+        return unarchived
+
+    @classmethod
+    def get_model_stats(
+        cls, company: str, model_ids: Sequence[str],
+    ) -> Dict[str, dict]:
+        if not model_ids:
+            return {}
+
+        result = Model.aggregate(
+            [
+                {
+                    "$match": {
+                        "company": {"$in": [None, "", company]},
+                        "_id": {"$in": model_ids},
+                    }
+                },
+                {
+                    "$addFields": {
+                        "labels_count": {"$size": {"$objectToArray": "$labels"}}
+                    }
+                },
+                {
+                    "$project": {"labels_count": 1},
+                },
+            ]
+        )
+        return {
+            r.pop("_id"): r
+            for r in result
+        }

apiserver/bll/model/metadata.py

@@ -0,0 +1,111 @@
+from typing import Sequence, Union, Mapping
+
+from mongoengine import Document
+
+from apiserver.apierrors import errors
+from apiserver.apimodels.metadata import MetadataItem
+from apiserver.database.model.base import GetMixin
+from apiserver.service_repo import APICall
+from apiserver.utilities.parameter_key_escaper import (
+    ParameterKeyEscaper,
+    mongoengine_safe,
+)
+from apiserver.config_repo import config
+from apiserver.timing_context import TimingContext
+
+log = config.logger(__file__)
+
+
+class Metadata:
+    @staticmethod
+    def metadata_from_api(
+        api_data: Union[Mapping[str, MetadataItem], Sequence[MetadataItem]]
+    ) -> dict:
+        if not api_data:
+            return {}
+
+        if isinstance(api_data, dict):
+            return {
+                ParameterKeyEscaper.escape(k): v.to_struct()
+                for k, v in api_data.items()
+            }
+
+        return {
+            ParameterKeyEscaper.escape(item.key): item.to_struct() for item in api_data
+        }
+
+    @classmethod
+    def edit_metadata(
+        cls,
+        obj: Document,
+        items: Sequence[MetadataItem],
+        replace_metadata: bool,
+        **more_updates,
+    ) -> int:
+        with TimingContext("mongo", "edit_metadata"):
+            update_cmds = dict()
+            metadata = cls.metadata_from_api(items)
+            if replace_metadata:
+                update_cmds["set__metadata"] = metadata
+            else:
+                for key, value in metadata.items():
+                    update_cmds[f"set__metadata__{mongoengine_safe(key)}"] = value
+
+            return obj.update(**update_cmds, **more_updates)
+
+    @classmethod
+    def delete_metadata(cls, obj: Document, keys: Sequence[str], **more_updates) -> int:
+        with TimingContext("mongo", "delete_metadata"):
+            return obj.update(
+                **{
+                    f"unset__metadata__{ParameterKeyEscaper.escape(key)}": 1
+                    for key in set(keys)
+                },
+                **more_updates,
+            )
+
+    @staticmethod
+    def _process_path(path: str):
+        """
+        Frontend does a partial escaping on the path so the all '.' in key names are escaped
+        Need to unescape and apply a full mongo escaping
+        """
+        parts = path.split(".")
+        if len(parts) < 2 or len(parts) > 3:
+            raise errors.bad_request.ValidationError("invalid field", path=path)
+        return ".".join(
+            ParameterKeyEscaper.escape(ParameterKeyEscaper.unescape(p)) for p in parts
+        )
+
+    @classmethod
+    def escape_paths(cls, paths: Sequence[str]) -> Sequence[str]:
+        for prefix in (
+            "metadata.",
+            "-metadata.",
+        ):
+            paths = [
+                cls._process_path(path) if path.startswith(prefix) else path
+                for path in paths
+            ]
+        return paths
+
+    @classmethod
+    def escape_query_parameters(cls, call: APICall) -> dict:
+        if not call.data:
+            return call.data
+
+        keys = list(call.data)
+        call_data = {
+            safe_key: call.data[key]
+            for key, safe_key in zip(keys, Metadata.escape_paths(keys))
+        }
+
+        projection = GetMixin.get_projection(call_data)
+        if projection:
+            GetMixin.set_projection(call_data, Metadata.escape_paths(projection))
+
+        ordering = GetMixin.get_ordering(call_data)
+        if ordering:
+            GetMixin.set_ordering(call_data, Metadata.escape_paths(ordering))
+
+        return call_data

apiserver/bll/organization/__init__.py

@@ -0,0 +1,63 @@
+from collections import defaultdict
+from enum import Enum
+from typing import Sequence, Dict
+
+from apiserver.config_repo import config
+from apiserver.database.model.model import Model
+from apiserver.database.model.task.task import Task
+from apiserver.redis_manager import redman
+from .tags_cache import _TagsCache
+
+log = config.logger(__file__)
+
+
+class Tags(Enum):
+    Task = "task"
+    Model = "model"
+
+
+class OrgBLL:
+    def __init__(self, redis=None):
+        self.redis = redis or redman.connection("apiserver")
+        self._task_tags = _TagsCache(Task, self.redis)
+        self._model_tags = _TagsCache(Model, self.redis)
+
+    def get_tags(
+        self,
+        company_id: str,
+        entity: Tags,
+        include_system: bool = False,
+        filter_: Dict[str, Sequence[str]] = None,
+        projects: Sequence[str] = None,
+    ) -> dict:
+        tags_cache = self._get_tags_cache_for_entity(entity)
+        if not projects:
+            return tags_cache.get_tags(
+                company_id, include_system=include_system, filter_=filter_
+            )
+
+        ret = defaultdict(set)
+        for project in projects:
+            project_tags = tags_cache.get_tags(
+                company_id,
+                include_system=include_system,
+                filter_=filter_,
+                project=project,
+            )
+            for field, tags in project_tags.items():
+                ret[field] |= tags
+
+        return ret
+
+    def update_tags(
+        self, company_id: str, entity: Tags, project: str, tags=None, system_tags=None,
+    ):
+        tags_cache = self._get_tags_cache_for_entity(entity)
+        tags_cache.update_tags(company_id, project, tags, system_tags)
+
+    def reset_tags(self, company_id: str, entity: Tags, projects: Sequence[str]):
+        tags_cache = self._get_tags_cache_for_entity(entity)
+        tags_cache.reset_tags(company_id, projects=projects)
+
+    def _get_tags_cache_for_entity(self, entity: Tags) -> _TagsCache:
+        return self._task_tags if entity == Tags.Task else self._model_tags

apiserver/bll/organization/tags_cache.py

@@ -1,17 +1,15 @@
-from collections import defaultdict
-from enum import Enum
 from itertools import chain
 from typing import Sequence, Union, Type, Dict
 
 from mongoengine import Q
 from redis import Redis
 
-from config import config
-from database.model.base import GetMixin
-from database.model.model import Model
-from database.model.task.task import Task
-from redis_manager import redman
-from utilities import json
+from apiserver.config_repo import config
+from apiserver.bll.project import project_ids_with_children
+from apiserver.database.model import EntityVisibility
+from apiserver.database.model.base import GetMixin
+from apiserver.database.model.model import Model
+from apiserver.database.model.task.task import Task
 
 log = config.logger(__file__)
 _settings_prefix = "services.organization"
@@ -20,6 +18,8 @@ _settings_prefix = "services.organization"
 class _TagsCache:
     _tags_field = "tags"
     _system_tags_field = "system_tags"
+    _dummy_tag = "__dummy__"
+    # prepend our list in redis with this tag since empty lists are auto deleted
 
     def __init__(self, db_cls: Union[Type[Model], Type[Task]], redis: Redis):
         self.db_cls = db_cls
@@ -31,24 +31,26 @@ class _TagsCache:
 
     def _get_tags_from_db(
         self,
-        company: str,
+        company_id: str,
         field: str,
         project: str = None,
         filter_: Dict[str, Sequence[str]] = None,
     ) -> set:
-        query = Q(company=company)
+        query = Q(company=company_id)
         if filter_:
             for name, vals in filter_.items():
                 if vals:
                     query &= GetMixin.get_list_field_query(name, vals)
         if project:
-            query &= Q(project=project)
+            query &= Q(project__in=project_ids_with_children([project]))
+        else:
+            query &= Q(system_tags__nin=[EntityVisibility.hidden.value])
 
         return self.db_cls.objects(query).distinct(field)
 
     def _get_tags_cache_key(
         self,
-        company: str,
+        company_id: str,
         field: str,
         project: str = None,
         filter_: Dict[str, Sequence[str]] = None,
@@ -64,12 +66,12 @@ class _TagsCache:
             filter_str = "_".join(
                 ["filter", *chain.from_iterable([f, *v] for f, v in filter_.items())]
             )
-        key_parts = [company, project, self.db_cls.__name__, field, filter_str]
+        key_parts = [field, company_id, project, self.db_cls.__name__, filter_str]
         return "_".join(filter(None, key_parts))
 
     def get_tags(
         self,
-        company: str,
+        company_id: str,
         include_system: bool = False,
         filter_: Dict[str, Sequence[str]] = None,
         project: str = None,
@@ -83,27 +85,29 @@ class _TagsCache:
         fields = [self._tags_field]
         if include_system:
             fields.append(self._system_tags_field)
-        redis_keys = [
-            self._get_tags_cache_key(company, field=f, project=project, filter_=filter_)
-            for f in fields
-        ]
-        cached = self.redis.mget(redis_keys)
+
         ret = {}
-        for field, tag_data, key in zip(fields, cached, redis_keys):
-            if tag_data is not None:
-                tags = json.loads(tag_data)
+        for field in fields:
+            redis_key = self._get_tags_cache_key(
+                company_id, field=field, project=project, filter_=filter_
+            )
+            cached_tags = self.redis.lrange(redis_key, 0, -1)
+            if cached_tags:
+                tags = [c.decode() for c in cached_tags[1:]]
             else:
-                tags = list(self._get_tags_from_db(company, field, project, filter_))
-                self.redis.setex(
-                    key,
-                    time=self._tags_cache_expiration_seconds,
-                    value=json.dumps(tags),
+                tags = list(
+                    self._get_tags_from_db(
+                        company_id, field=field, project=project, filter_=filter_
+                    )
                 )
+                self.redis.rpush(redis_key, self._dummy_tag, *tags)
+                self.redis.expire(redis_key, self._tags_cache_expiration_seconds)
+
             ret[field] = set(tags)
 
         return ret
 
-    def update_tags(self, company: str, project: str, tags=None, system_tags=None):
+    def update_tags(self, company_id: str, project: str, tags=None, system_tags=None):
         """
         Updates tags. If reset is set then both tags and system_tags
         are recalculated. Otherwise only those that are not 'None'
@@ -119,22 +123,22 @@ class _TagsCache:
         if not fields:
             return
 
-        self._delete_redis_keys(company, projects=[project], fields=fields)
+        self._delete_redis_keys(company_id, projects=[project], fields=fields)
 
-    def reset_tags(self, company: str, projects: Sequence[str]):
+    def reset_tags(self, company_id: str, projects: Sequence[str]):
         self._delete_redis_keys(
-            company,
+            company_id,
             projects=projects,
             fields=(self._tags_field, self._system_tags_field),
         )
 
     def _delete_redis_keys(
-        self, company: str, projects: [Sequence[str]], fields: Sequence[str]
+        self, company_id: str, projects: [Sequence[str]], fields: Sequence[str]
     ):
         redis_keys = list(
             chain.from_iterable(
                 self.redis.keys(
-                    self._get_tags_cache_key(company, field=f, project=p) + "*"
+                    self._get_tags_cache_key(company_id, field=f, project=p) + "*"
                 )
                 for f in fields
                 for p in set(projects) | {None}
@@ -142,52 +146,3 @@ class _TagsCache:
         )
         if redis_keys:
             self.redis.delete(*redis_keys)
-
-
-class Tags(Enum):
-    Task = "task"
-    Model = "model"
-
-
-class OrgBLL:
-    def __init__(self, redis=None):
-        self.redis = redis or redman.connection("apiserver")
-        self._task_tags = _TagsCache(Task, self.redis)
-        self._model_tags = _TagsCache(Model, self.redis)
-
-    def get_tags(
-        self,
-        company: str,
-        entity: Tags,
-        include_system: bool = False,
-        filter_: Dict[str, Sequence[str]] = None,
-        projects: Sequence[str] = None,
-    ) -> dict:
-        tags_cache = self._get_tags_cache_for_entity(entity)
-        if not projects:
-            return tags_cache.get_tags(
-                company, include_system=include_system, filter_=filter_
-            )
-
-        ret = defaultdict(set)
-        for project in projects:
-            project_tags = tags_cache.get_tags(
-                company, include_system=include_system, filter_=filter_, project=project
-            )
-            for field, tags in project_tags.items():
-                ret[field] |= tags
-
-        return ret
-
-    def update_tags(
-        self, company: str, entity: Tags, project: str, tags=None, system_tags=None,
-    ):
-        tags_cache = self._get_tags_cache_for_entity(entity)
-        tags_cache.update_tags(company, project, tags, system_tags)
-
-    def reset_tags(self, company: str, entity: Tags, projects: Sequence[str]):
-        tags_cache = self._get_tags_cache_for_entity(entity)
-        tags_cache.reset_tags(company, projects=projects)
-
-    def _get_tags_cache_for_entity(self, entity: Tags) -> _TagsCache:
-        return self._task_tags if entity == Tags.Task else self._model_tags

apiserver/bll/project/__init__.py

@@ -0,0 +1,3 @@
+from .project_bll import ProjectBLL
+from .project_queries import ProjectQueries
+from .sub_projects import _ids_with_children as project_ids_with_children

apiserver/bll/project/project_bll.py

@@ -0,0 +1,939 @@
+import itertools
+from collections import defaultdict
+from datetime import datetime, timedelta
+from functools import reduce
+from itertools import groupby
+from operator import itemgetter
+from typing import (
+    Sequence,
+    Optional,
+    Type,
+    Tuple,
+    Dict,
+    Set,
+    TypeVar,
+    Callable,
+    Mapping,
+    Any,
+)
+
+from boltons.iterutils import partition
+from mongoengine import Q, Document
+
+from apiserver import database
+from apiserver.apierrors import errors
+from apiserver.config_repo import config
+from apiserver.database.model import EntityVisibility, AttributedDocument
+from apiserver.database.model.base import GetMixin
+from apiserver.database.model.model import Model
+from apiserver.database.model.project import Project
+from apiserver.database.model.task.task import Task, TaskStatus, external_task_types
+from apiserver.database.utils import get_options, get_company_or_none_constraint
+from apiserver.timing_context import TimingContext
+from apiserver.utilities.dicts import nested_get
+from .sub_projects import (
+    _reposition_project_with_children,
+    _ensure_project,
+    _validate_project_name,
+    _update_subproject_names,
+    _save_under_parent,
+    _get_sub_projects,
+    _ids_with_children,
+    _ids_with_parents,
+    _get_project_depth,
+)
+
+log = config.logger(__file__)
+max_depth = config.get("services.projects.sub_projects.max_depth", 10)
+
+
+class ProjectBLL:
+    @classmethod
+    def merge_project(
+        cls, company, source_id: str, destination_id: str
+    ) -> Tuple[int, int, Set[str]]:
+        """
+        Move all the tasks and sub projects from the source project to the destination
+        Remove the source project
+        Return the amounts of moved entities and subprojects + set of all the affected project ids
+        """
+        with TimingContext("mongo", "move_project"):
+            if source_id == destination_id:
+                raise errors.bad_request.ProjectSourceAndDestinationAreTheSame(
+                    source=source_id
+                )
+            source = Project.get(company, source_id)
+            if destination_id:
+                destination = Project.get(company, destination_id)
+                if source_id in destination.path:
+                    raise errors.bad_request.ProjectCannotBeMergedIntoItsChild(
+                        source=source_id, destination=destination_id
+                    )
+            else:
+                destination = None
+
+            children = _get_sub_projects(
+                [source.id], _only=("id", "name", "parent", "path")
+            )[source.id]
+            if destination:
+                cls.validate_projects_depth(
+                    projects=children,
+                    old_parent_depth=len(source.path) + 1,
+                    new_parent_depth=len(destination.path) + 1,
+                )
+
+            moved_entities = 0
+            for entity_type in (Task, Model):
+                moved_entities += entity_type.objects(
+                    company=company,
+                    project=source_id,
+                    system_tags__nin=[EntityVisibility.archived.value],
+                ).update(upsert=False, project=destination_id)
+
+            moved_sub_projects = 0
+            for child in Project.objects(company=company, parent=source_id):
+                _reposition_project_with_children(
+                    project=child,
+                    children=[c for c in children if c.parent == child.id],
+                    parent=destination,
+                )
+                moved_sub_projects += 1
+
+            affected = {source.id, *(source.path or [])}
+            source.delete()
+
+            if destination:
+                destination.update(last_update=datetime.utcnow())
+                affected.update({destination.id, *(destination.path or [])})
+
+        return moved_entities, moved_sub_projects, affected
+
+    @staticmethod
+    def validate_projects_depth(
+        projects: Sequence[Project], old_parent_depth: int, new_parent_depth: int
+    ):
+        for current in projects:
+            current_depth = len(current.path) + 1
+            if current_depth - old_parent_depth + new_parent_depth > max_depth:
+                raise errors.bad_request.ProjectPathExceedsMax(max_depth=max_depth)
+
+    @classmethod
+    def move_project(
+        cls, company: str, user: str, project_id: str, new_location: str
+    ) -> Tuple[int, Set[str]]:
+        """
+        Move project with its sub projects from its current location to the target one.
+        If the target location does not exist then it will be created. If it exists then
+        it should be writable. The source location should be writable too.
+        Return the number of moved projects + set of all the affected project ids
+        """
+        with TimingContext("mongo", "move_project"):
+            project = Project.get(company, project_id)
+            old_parent_id = project.parent
+            old_parent = (
+                Project.get_for_writing(company=project.company, id=old_parent_id)
+                if old_parent_id
+                else None
+            )
+
+            children = _get_sub_projects([project.id], _only=("id", "name", "path"))[
+                project.id
+            ]
+            cls.validate_projects_depth(
+                projects=[project, *children],
+                old_parent_depth=len(project.path),
+                new_parent_depth=_get_project_depth(new_location),
+            )
+
+            new_parent = _ensure_project(company=company, user=user, name=new_location)
+            new_parent_id = new_parent.id if new_parent else None
+            if old_parent_id == new_parent_id:
+                raise errors.bad_request.ProjectSourceAndDestinationAreTheSame(
+                    location=new_parent.name if new_parent else ""
+                )
+            if new_parent and (
+                project_id == new_parent.id or project_id in new_parent.path
+            ):
+                raise errors.bad_request.ProjectCannotBeMovedUnderItself(
+                    project=project_id, parent=new_parent.id
+                )
+            moved = _reposition_project_with_children(
+                project, children=children, parent=new_parent
+            )
+
+            now = datetime.utcnow()
+            affected = set()
+            for p in filter(None, (old_parent, new_parent)):
+                p.update(last_update=now)
+                affected.update({p.id, *(p.path or [])})
+
+            return moved, affected
+
+    @classmethod
+    def update(cls, company: str, project_id: str, **fields):
+        with TimingContext("mongo", "projects_update"):
+            project = Project.get_for_writing(company=company, id=project_id)
+            if not project:
+                raise errors.bad_request.InvalidProjectId(id=project_id)
+
+            new_name = fields.pop("name", None)
+            if new_name:
+                new_name, new_location = _validate_project_name(new_name)
+                old_name, old_location = _validate_project_name(project.name)
+                if new_location != old_location:
+                    raise errors.bad_request.CannotUpdateProjectLocation(name=new_name)
+                fields["name"] = new_name
+                fields["basename"] = new_name.split("/")[-1]
+
+            fields["last_update"] = datetime.utcnow()
+            updated = project.update(upsert=False, **fields)
+
+            if new_name:
+                old_name = project.name
+                project.name = new_name
+                children = _get_sub_projects(
+                    [project.id], _only=("id", "name", "path")
+                )[project.id]
+                _update_subproject_names(
+                    project=project, children=children, old_name=old_name
+                )
+
+            return updated
+
+    @classmethod
+    def create(
+        cls,
+        user: str,
+        company: str,
+        name: str,
+        description: str = "",
+        tags: Sequence[str] = None,
+        system_tags: Sequence[str] = None,
+        default_output_destination: str = None,
+        parent_creation_params: dict = None,
+    ) -> str:
+        """
+        Create a new project.
+        Returns project ID
+        """
+        if _get_project_depth(name) > max_depth:
+            raise errors.bad_request.ProjectPathExceedsMax(max_depth=max_depth)
+
+        name, location = _validate_project_name(name)
+        now = datetime.utcnow()
+        project = Project(
+            id=database.utils.id(),
+            user=user,
+            company=company,
+            name=name,
+            basename=name.split("/")[-1],
+            description=description,
+            tags=tags,
+            system_tags=system_tags,
+            default_output_destination=default_output_destination,
+            created=now,
+            last_update=now,
+        )
+        parent = _ensure_project(
+            company=company,
+            user=user,
+            name=location,
+            creation_params=parent_creation_params,
+        )
+        _save_under_parent(project=project, parent=parent)
+        if parent:
+            parent.update(last_update=now)
+
+        return project.id
+
+    @classmethod
+    def find_or_create(
+        cls,
+        user: str,
+        company: str,
+        project_name: str,
+        description: str,
+        project_id: str = None,
+        tags: Sequence[str] = None,
+        system_tags: Sequence[str] = None,
+        default_output_destination: str = None,
+        parent_creation_params: dict = None,
+    ) -> str:
+        """
+        Find a project named `project_name` or create a new one.
+        Returns project ID
+        """
+        if not project_id and not project_name:
+            raise errors.bad_request.ValidationError("project id or name required")
+
+        if project_id:
+            project = Project.objects(company=company, id=project_id).only("id").first()
+            if not project:
+                raise errors.bad_request.InvalidProjectId(id=project_id)
+            return project_id
+
+        project_name, _ = _validate_project_name(project_name)
+        project = Project.objects(company=company, name=project_name).only("id").first()
+        if project:
+            return project.id
+
+        return cls.create(
+            user=user,
+            company=company,
+            name=project_name,
+            description=description,
+            tags=tags,
+            system_tags=system_tags,
+            default_output_destination=default_output_destination,
+            parent_creation_params=parent_creation_params,
+        )
+
+    @classmethod
+    def move_under_project(
+        cls,
+        entity_cls: Type[Document],
+        user: str,
+        company: str,
+        ids: Sequence[str],
+        project: str = None,
+        project_name: str = None,
+    ):
+        """
+        Move a batch of entities to `project` or a project named `project_name` (create if does not exist)
+        """
+        with TimingContext("mongo", "move_under_project"):
+            project = cls.find_or_create(
+                user=user,
+                company=company,
+                project_id=project,
+                project_name=project_name,
+                description="",
+            )
+            extra = (
+                {"set__last_change": datetime.utcnow()}
+                if hasattr(entity_cls, "last_change")
+                else {}
+            )
+            entity_cls.objects(company=company, id__in=ids).update(
+                set__project=project, **extra
+            )
+
+            return project
+
+    archived_tasks_cond = {"$in": [EntityVisibility.archived.value, "$system_tags"]}
+    visibility_states = [EntityVisibility.archived, EntityVisibility.active]
+
+    @classmethod
+    def make_projects_get_all_pipelines(
+        cls,
+        company_id: str,
+        project_ids: Sequence[str],
+        specific_state: Optional[EntityVisibility] = None,
+        filter_: Mapping[str, Any] = None,
+        users: Sequence[str] = None,
+    ) -> Tuple[Sequence, Sequence]:
+        archived = EntityVisibility.archived.value
+
+        def ensure_valid_fields():
+            """
+            Make sure system tags is always an array (required by subsequent $in in archived_tasks_cond
+            """
+            return {
+                "$addFields": {
+                    "system_tags": {
+                        "$cond": {
+                            "if": {"$ne": [{"$type": "$system_tags"}, "array"]},
+                            "then": [],
+                            "else": "$system_tags",
+                        }
+                    },
+                    "status": {"$ifNull": ["$status", "unknown"]},
+                }
+            }
+
+        status_count_pipeline = [
+            # count tasks per project per status
+            {
+                "$match": cls.get_match_conditions(
+                    company=company_id,
+                    project_ids=project_ids,
+                    filter_=filter_,
+                    users=users,
+                )
+            },
+            ensure_valid_fields(),
+            {
+                "$group": {
+                    "_id": {
+                        "project": "$project",
+                        "status": "$status",
+                        archived: cls.archived_tasks_cond,
+                    },
+                    "count": {"$sum": 1},
+                }
+            },
+            # for each project, create a list of (status, count, archived)
+            {
+                "$group": {
+                    "_id": "$_id.project",
+                    "counts": {
+                        "$push": {
+                            "status": "$_id.status",
+                            "count": "$count",
+                            archived: "$_id.%s" % archived,
+                        }
+                    },
+                }
+            },
+        ]
+
+        def completed_after_subquery(additional_cond, time_thresh: datetime):
+            return {
+                # the sum of
+                "$sum": {
+                    # for each task
+                    "$cond": {
+                        # if completed after the time_thresh
+                        "if": {
+                            "$and": [
+                                "$completed",
+                                {"$gt": ["$completed", time_thresh]},
+                                additional_cond,
+                            ]
+                        },
+                        "then": 1,
+                        "else": 0,
+                    }
+                }
+            }
+
+        def max_started_subquery(condition):
+            return {
+                "$max": {
+                    "$cond": {
+                        "if": condition,
+                        "then": "$started",
+                        "else": datetime.min,
+                    }
+                }
+            }
+
+        def runtime_subquery(additional_cond):
+            return {
+                # the sum of
+                "$sum": {
+                    # for each task
+                    "$cond": {
+                        # if completed and started and completed > started
+                        "if": {
+                            "$and": [
+                                "$started",
+                                "$completed",
+                                {"$gt": ["$completed", "$started"]},
+                                additional_cond,
+                            ]
+                        },
+                        # then: floor((completed - started) / 1000)
+                        "then": {
+                            "$floor": {
+                                "$divide": [
+                                    {"$subtract": ["$completed", "$started"]},
+                                    1000.0,
+                                ]
+                            }
+                        },
+                        "else": 0,
+                    }
+                }
+            }
+
+        group_step = {"_id": "$project"}
+        time_thresh = datetime.utcnow() - timedelta(hours=24)
+        for state in cls.visibility_states:
+            if specific_state and state != specific_state:
+                continue
+            cond = (
+                cls.archived_tasks_cond
+                if state == EntityVisibility.archived
+                else {"$not": cls.archived_tasks_cond}
+            )
+            group_step[state.value] = runtime_subquery(cond)
+            group_step[f"{state.value}_recently_completed"] = completed_after_subquery(
+                cond, time_thresh=time_thresh
+            )
+            group_step[f"{state.value}_max_task_started"] = max_started_subquery(cond)
+
+        def add_state_to_filter(f: Mapping[str, Any]) -> Mapping[str, Any]:
+            if not specific_state:
+                return f
+
+            f = f or {}
+            new_f = {k: v for k, v in f.items() if k != "system_tags"}
+            system_tags = [
+                tag
+                for tag in f.get("system_tags", [])
+                if tag
+                not in (
+                    EntityVisibility.archived.value,
+                    f"-{EntityVisibility.archived.value}",
+                )
+            ]
+
+            if specific_state == EntityVisibility.archived:
+                system_tags.append(EntityVisibility.archived.value)
+            else:
+                system_tags.append(f"-{EntityVisibility.archived.value}")
+            new_f["system_tags"] = system_tags
+
+            return new_f
+
+        runtime_pipeline = [
+            # only count run time for these types of tasks
+            {
+                "$match": cls.get_match_conditions(
+                    company=company_id,
+                    project_ids=project_ids,
+                    filter_=add_state_to_filter(filter_),
+                    users=users,
+                )
+            },
+            ensure_valid_fields(),
+            {
+                # for each project
+                "$group": group_step
+            },
+        ]
+
+        return status_count_pipeline, runtime_pipeline
+
+    T = TypeVar("T")
+
+    @staticmethod
+    def aggregate_project_data(
+        func: Callable[[T, T], T],
+        project_ids: Sequence[str],
+        child_projects: Mapping[str, Sequence[Project]],
+        data: Mapping[str, T],
+    ) -> Dict[str, T]:
+        """
+        Given a list of project ids and data collected over these projects and their subprojects
+        For each project aggregates the data from all of its subprojects
+        """
+        aggregated = {}
+        if not data:
+            return aggregated
+        for pid in project_ids:
+            relevant_projects = {p.id for p in child_projects.get(pid, [])} | {pid}
+            relevant_data = [data for p, data in data.items() if p in relevant_projects]
+            if not relevant_data:
+                continue
+            aggregated[pid] = reduce(func, relevant_data)
+        return aggregated
+
+    @classmethod
+    def get_dataset_stats(
+        cls, company: str, project_ids: Sequence[str], users: Sequence[str] = None,
+    ) -> Dict[str, dict]:
+        if not project_ids:
+            return {}
+
+        task_runtime_pipeline = [
+            {
+                "$match": {
+                    **cls.get_match_conditions(
+                        company=company,
+                        project_ids=project_ids,
+                        users=users,
+                        filter_={
+                            "system_tags": [f"-{EntityVisibility.archived.value}"]
+                        },
+                    ),
+                    "runtime": {"$exists": True, "$gt": {}},
+                }
+            },
+            {"$project": {"project": 1, "runtime": 1, "last_update": 1}},
+            {"$sort": {"project": 1, "last_update": 1}},
+            {"$group": {"_id": "$project", "runtime": {"$last": "$runtime"}}},
+        ]
+
+        return {
+            r["_id"]: {
+                "file_count": r["runtime"].get("ds_file_count", 0),
+                "total_size": r["runtime"].get("ds_total_size", 0),
+            }
+            for r in Task.aggregate(task_runtime_pipeline)
+        }
+
+    @classmethod
+    def get_project_stats(
+        cls,
+        company: str,
+        project_ids: Sequence[str],
+        specific_state: Optional[EntityVisibility] = None,
+        include_children: bool = True,
+        search_hidden: bool = False,
+        filter_: Mapping[str, Any] = None,
+        users: Sequence[str] = None,
+        user_active_project_ids: Sequence[str] = None,
+    ) -> Tuple[Dict[str, dict], Dict[str, dict]]:
+        if not project_ids:
+            return {}, {}
+
+        child_projects = (
+            _get_sub_projects(
+                project_ids,
+                _only=("id", "name"),
+                search_hidden=search_hidden,
+                allowed_ids=user_active_project_ids,
+            )
+            if include_children
+            else {}
+        )
+        project_ids_with_children = set(project_ids) | {
+            c.id for c in itertools.chain.from_iterable(child_projects.values())
+        }
+        status_count_pipeline, runtime_pipeline = cls.make_projects_get_all_pipelines(
+            company,
+            project_ids=list(project_ids_with_children),
+            specific_state=specific_state,
+            filter_=filter_,
+            users=users,
+        )
+
+        default_counts = dict.fromkeys(get_options(TaskStatus), 0)
+
+        def set_default_count(entry):
+            return dict(default_counts, **entry)
+
+        status_count = defaultdict(lambda: {})
+        key = itemgetter(EntityVisibility.archived.value)
+        for result in Task.aggregate(status_count_pipeline):
+            for k, group in groupby(sorted(result["counts"], key=key), key):
+                section = (
+                    EntityVisibility.archived if k else EntityVisibility.active
+                ).value
+                status_count[result["_id"]][section] = set_default_count(
+                    {
+                        count_entry["status"]: count_entry["count"]
+                        for count_entry in group
+                    }
+                )
+
+        def sum_status_count(
+            a: Mapping[str, Mapping], b: Mapping[str, Mapping]
+        ) -> Dict[str, dict]:
+            return {
+                section: {
+                    status: nested_get(a, (section, status), default=0)
+                    + nested_get(b, (section, status), default=0)
+                    for status in set(a.get(section, {})) | set(b.get(section, {}))
+                }
+                for section in set(a) | set(b)
+            }
+
+        status_count = cls.aggregate_project_data(
+            func=sum_status_count,
+            project_ids=project_ids,
+            child_projects=child_projects,
+            data=status_count,
+        )
+
+        runtime = {
+            result["_id"]: {k: v for k, v in result.items() if k != "_id"}
+            for result in Task.aggregate(runtime_pipeline)
+        }
+
+        def sum_runtime(
+            a: Mapping[str, Mapping], b: Mapping[str, Mapping]
+        ) -> Dict[str, dict]:
+            return {
+                section: a.get(section, 0) + b.get(section, 0)
+                if not section.endswith("max_task_started")
+                else max(a.get(section) or datetime.min, b.get(section) or datetime.min)
+                for section in set(a) | set(b)
+            }
+
+        runtime = cls.aggregate_project_data(
+            func=sum_runtime,
+            project_ids=project_ids,
+            child_projects=child_projects,
+            data=runtime,
+        )
+
+        def get_status_counts(project_id, section):
+            project_runtime = runtime.get(project_id, {})
+            project_section_statuses = nested_get(
+                status_count, (project_id, section), default=default_counts
+            )
+
+            def get_time_or_none(value):
+                return value if value != datetime.min else None
+
+            return {
+                "status_count": project_section_statuses,
+                "total_tasks": sum(project_section_statuses.values()),
+                "total_runtime": project_runtime.get(section, 0),
+                "completed_tasks_24h": project_runtime.get(
+                    f"{section}_recently_completed", 0
+                ),
+                "last_task_run": get_time_or_none(
+                    project_runtime.get(f"{section}_max_task_started", datetime.min)
+                ),
+            }
+
+        report_for_states = [
+            s
+            for s in cls.visibility_states
+            if not specific_state or specific_state == s
+        ]
+
+        stats = {
+            project: {
+                task_state.value: get_status_counts(project, task_state.value)
+                for task_state in report_for_states
+            }
+            for project in project_ids
+        }
+
+        children = {
+            project: sorted(
+                [{"id": c.id, "name": c.name} for c in child_projects.get(project, [])],
+                key=itemgetter("name"),
+            )
+            for project in project_ids
+        }
+        return stats, children
+
+    @classmethod
+    def get_active_users(
+        cls,
+        company,
+        project_ids: Sequence[str],
+        user_ids: Optional[Sequence[str]] = None,
+    ) -> Set[str]:
+        """
+        Get the set of user ids that created tasks/models in the given projects
+        If project_ids is empty then all projects are examined
+        If user_ids are passed then only subset of these users is returned
+        """
+        with TimingContext("mongo", "active_users_in_projects"):
+            query = Q(company=company)
+            if user_ids:
+                query &= Q(user__in=user_ids)
+
+            projects_query = query
+            if project_ids:
+                project_ids = _ids_with_children(project_ids)
+                query &= Q(project__in=project_ids)
+                projects_query &= Q(id__in=project_ids)
+
+            res = set(Project.objects(projects_query).distinct(field="user"))
+            for cls_ in (Task, Model):
+                res |= set(cls_.objects(query).distinct(field="user"))
+
+            return res
+
+    @classmethod
+    def get_project_tags(
+        cls,
+        company_id: str,
+        include_system: bool,
+        projects: Sequence[str] = None,
+        filter_: Dict[str, Sequence[str]] = None,
+    ) -> Tuple[Sequence[str], Sequence[str]]:
+        with TimingContext("mongo", "get_tags_from_db"):
+            query = Q(company=company_id)
+            if filter_:
+                for name, vals in filter_.items():
+                    if vals:
+                        query &= GetMixin.get_list_field_query(name, vals)
+
+            if projects:
+                query &= Q(id__in=_ids_with_children(projects))
+
+            tags = Project.objects(query).distinct("tags")
+            system_tags = (
+                Project.objects(query).distinct("system_tags") if include_system else []
+            )
+            return tags, system_tags
+
+    @classmethod
+    def get_projects_with_active_user(
+        cls,
+        company: str,
+        users: Sequence[str],
+        project_ids: Optional[Sequence[str]] = None,
+        allow_public: bool = True,
+    ) -> Tuple[Sequence[str], Sequence[str]]:
+        """
+        Get the projects ids where user created any tasks including all the parents of these projects
+        If project ids are specified then filter the results by these project ids
+        """
+        query = Q(user__in=users)
+
+        if allow_public:
+            query &= get_company_or_none_constraint(company)
+        else:
+            query &= Q(company=company)
+
+        user_projects_query = query
+        if project_ids:
+            ids_with_children = _ids_with_children(project_ids)
+            query &= Q(project__in=ids_with_children)
+            user_projects_query &= Q(id__in=ids_with_children)
+
+        res = {p.id for p in Project.objects(user_projects_query).only("id")}
+        for cls_ in (Task, Model):
+            res |= set(cls_.objects(query).distinct(field="project"))
+
+        res = list(res)
+        if not res:
+            return res, res
+
+        user_active_project_ids = _ids_with_parents(res)
+        filtered_ids = (
+            list(set(user_active_project_ids) & set(project_ids))
+            if project_ids
+            else list(user_active_project_ids)
+        )
+
+        return filtered_ids, user_active_project_ids
+
+    @classmethod
+    def get_task_parents(
+        cls,
+        company_id: str,
+        projects: Sequence[str],
+        include_subprojects: bool,
+        state: Optional[EntityVisibility] = None,
+    ) -> Sequence[dict]:
+        """
+        Get list of unique parent tasks sorted by task name for the passed company projects
+        If projects is None or empty then get parents for all the company tasks
+        """
+        query = Q(company=company_id)
+
+        if projects:
+            if include_subprojects:
+                projects = _ids_with_children(projects)
+            query &= Q(project__in=projects)
+        else:
+            query &= Q(system_tags__nin=[EntityVisibility.hidden.value])
+
+        if state == EntityVisibility.archived:
+            query &= Q(system_tags__in=[EntityVisibility.archived.value])
+        elif state == EntityVisibility.active:
+            query &= Q(system_tags__nin=[EntityVisibility.archived.value])
+
+        parent_ids = set(Task.objects(query).distinct("parent"))
+        if not parent_ids:
+            return []
+
+        parents = Task.get_many_with_join(
+            company_id,
+            query=Q(id__in=parent_ids),
+            allow_public=True,
+            override_projection=("id", "name", "project.name"),
+        )
+        return sorted(parents, key=itemgetter("name"))
+
+    @classmethod
+    def get_task_types(cls, company, project_ids: Optional[Sequence]) -> set:
+        """
+        Return the list of unique task types used by company and public tasks
+        If project ids passed then only tasks from these projects are considered
+        """
+        query = get_company_or_none_constraint(company)
+        if project_ids:
+            project_ids = _ids_with_children(project_ids)
+            query &= Q(project__in=project_ids)
+        else:
+            query &= Q(system_tags__nin=[EntityVisibility.hidden.value])
+        res = Task.objects(query).distinct(field="type")
+        return set(res).intersection(external_task_types)
+
+    @classmethod
+    def get_model_frameworks(cls, company, project_ids: Optional[Sequence]) -> Sequence:
+        """
+        Return the list of unique frameworks used by company and public models
+        If project ids passed then only models from these projects are considered
+        """
+        query = get_company_or_none_constraint(company)
+        if project_ids:
+            project_ids = _ids_with_children(project_ids)
+            query &= Q(project__in=project_ids)
+        return Model.objects(query).distinct(field="framework")
+
+    @staticmethod
+    def get_match_conditions(
+        company: str,
+        project_ids: Sequence[str],
+        filter_: Mapping[str, Any],
+        users: Sequence[str],
+    ):
+        conditions = {
+            "company": {"$in": [None, "", company]},
+            "project": {"$in": project_ids},
+        }
+        if users:
+            conditions["user"] = {"$in": users}
+
+        if not filter_:
+            return conditions
+
+        for field, field_filter in filter_.items():
+            if not (
+                field_filter
+                and isinstance(field_filter, list)
+                and all(isinstance(t, str) for t in field_filter)
+            ):
+                raise errors.bad_request.ValidationError(
+                    f"List of strings expected for the field: {field}"
+                )
+            exclude, include = partition(field_filter, lambda x: x.startswith("-"))
+            conditions[field] = {
+                **({"$in": include} if include else {}),
+                **({"$nin": [e[1:] for e in exclude]} if exclude else {}),
+            }
+
+        return conditions
+
+    @classmethod
+    def calc_own_contents(
+        cls,
+        company: str,
+        project_ids: Sequence[str],
+        filter_: Mapping[str, Any] = None,
+        users: Sequence[str] = None,
+    ) -> Dict[str, dict]:
+        """
+        Returns the amount of task/models per requested project
+        Use separate aggregation calls on Task/Model instead of lookup
+        aggregation on projects in order not to hit memory limits on large tasks
+        """
+        if not project_ids:
+            return {}
+
+        pipeline = [
+            {
+                "$match": cls.get_match_conditions(
+                    company=company,
+                    project_ids=project_ids,
+                    filter_=filter_,
+                    users=users,
+                )
+            },
+            {"$project": {"project": 1}},
+            {"$group": {"_id": "$project", "count": {"$sum": 1}}},
+        ]
+
+        def get_agrregate_res(cls_: Type[AttributedDocument]) -> dict:
+            return {data["_id"]: data["count"] for data in cls_.aggregate(pipeline)}
+
+        with TimingContext("mongo", "get_security_groups"):
+            tasks = get_agrregate_res(Task)
+            models = get_agrregate_res(Model)
+            return {
+                pid: {"own_tasks": tasks.get(pid, 0), "own_models": models.get(pid, 0)}
+                for pid in project_ids
+            }

apiserver/bll/project/project_cleanup.py

@@ -0,0 +1,189 @@
+from typing import Tuple, Set, Sequence
+
+import attr
+
+from apiserver.apierrors import errors
+from apiserver.bll.event import EventBLL
+from apiserver.bll.task.task_cleanup import (
+    collect_debug_image_urls,
+    collect_plot_image_urls,
+    TaskUrls,
+)
+from apiserver.config_repo import config
+from apiserver.database.model import EntityVisibility
+from apiserver.database.model.model import Model
+from apiserver.database.model.project import Project
+from apiserver.database.model.task.task import Task, ArtifactModes, TaskType
+from apiserver.timing_context import TimingContext
+from .sub_projects import _ids_with_children
+
+log = config.logger(__file__)
+event_bll = EventBLL()
+
+
+@attr.s(auto_attribs=True)
+class DeleteProjectResult:
+    deleted: int = 0
+    disassociated_tasks: int = 0
+    deleted_models: int = 0
+    deleted_tasks: int = 0
+    urls: TaskUrls = None
+
+
+def validate_project_delete(company: str, project_id: str):
+    project = Project.get_for_writing(
+        company=company, id=project_id, _only=("id", "path", "system_tags")
+    )
+    if not project:
+        raise errors.bad_request.InvalidProjectId(id=project_id)
+    is_pipeline = "pipeline" in (project.system_tags or [])
+    project_ids = _ids_with_children([project_id])
+    ret = {}
+    for cls in (Task, Model):
+        ret[f"{cls.__name__.lower()}s"] = cls.objects(project__in=project_ids).count()
+    for cls in (Task, Model):
+        query = dict(
+            project__in=project_ids, system_tags__nin=[EntityVisibility.archived.value]
+        )
+        name = f"non_archived_{cls.__name__.lower()}s"
+        if not is_pipeline:
+            ret[name] = cls.objects(**query).count()
+        else:
+            ret[name] = (
+                cls.objects(**query, type=TaskType.controller).count()
+                if cls == Task
+                else 0
+            )
+
+    return ret
+
+
+def delete_project(
+    company: str, project_id: str, force: bool, delete_contents: bool
+) -> Tuple[DeleteProjectResult, Set[str]]:
+    project = Project.get_for_writing(
+        company=company, id=project_id, _only=("id", "path", "system_tags")
+    )
+    if not project:
+        raise errors.bad_request.InvalidProjectId(id=project_id)
+    is_pipeline = "pipeline" in (project.system_tags or [])
+    project_ids = _ids_with_children([project_id])
+    if not force:
+        query = dict(
+            project__in=project_ids, system_tags__nin=[EntityVisibility.archived.value]
+        )
+        if not is_pipeline:
+            for cls, error in (
+                (Task, errors.bad_request.ProjectHasTasks),
+                (Model, errors.bad_request.ProjectHasModels),
+            ):
+                non_archived = cls.objects(**query).only("id")
+                if non_archived:
+                    raise error("use force=true to delete", id=project_id)
+        else:
+            non_archived = Task.objects(**query, type=TaskType.controller).only("id")
+            if non_archived:
+                raise errors.bad_request.ProjectHasTasks(
+                    "please archive all the runs inside the project", id=project_id
+                )
+
+    if not delete_contents:
+        with TimingContext("mongo", "update_children"):
+            for cls in (Model, Task):
+                updated_count = cls.objects(project__in=project_ids).update(
+                    project=None
+                )
+        res = DeleteProjectResult(disassociated_tasks=updated_count)
+    else:
+        deleted_models, model_urls = _delete_models(projects=project_ids)
+        deleted_tasks, event_urls, artifact_urls = _delete_tasks(
+            company=company, projects=project_ids
+        )
+        res = DeleteProjectResult(
+            deleted_tasks=deleted_tasks,
+            deleted_models=deleted_models,
+            urls=TaskUrls(
+                model_urls=list(model_urls),
+                event_urls=list(event_urls),
+                artifact_urls=list(artifact_urls),
+            ),
+        )
+
+    affected = {*project_ids, *(project.path or [])}
+    res.deleted = Project.objects(id__in=project_ids).delete()
+
+    return res, affected
+
+
+def _delete_tasks(company: str, projects: Sequence[str]) -> Tuple[int, Set, Set]:
+    """
+    Delete only the task themselves and their non published version.
+    Child models under the same project are deleted separately.
+    Children tasks should be deleted in the same api call.
+    If any child entities are left in another projects then updated their parent task to None
+    """
+    tasks = Task.objects(project__in=projects).only("id", "execution__artifacts")
+    if not tasks:
+        return 0, set(), set()
+
+    task_ids = {t.id for t in tasks}
+    with TimingContext("mongo", "delete_tasks_update_children"):
+        Task.objects(parent__in=task_ids, project__nin=projects).update(parent=None)
+        Model.objects(task__in=task_ids, project__nin=projects).update(task=None)
+
+    event_urls, artifact_urls = set(), set()
+    for task in tasks:
+        event_urls.update(collect_debug_image_urls(company, task.id))
+        event_urls.update(collect_plot_image_urls(company, task.id))
+        if task.execution and task.execution.artifacts:
+            artifact_urls.update(
+                {
+                    a.uri
+                    for a in task.execution.artifacts.values()
+                    if a.mode == ArtifactModes.output and a.uri
+                }
+            )
+
+    event_bll.delete_multi_task_events(company, list(task_ids))
+    deleted = tasks.delete()
+    return deleted, event_urls, artifact_urls
+
+
+def _delete_models(projects: Sequence[str]) -> Tuple[int, Set[str]]:
+    """
+    Delete project models and update the tasks from other projects
+    that reference them to reference None.
+    """
+    with TimingContext("mongo", "delete_models"):
+        models = Model.objects(project__in=projects).only("task", "id", "uri")
+        if not models:
+            return 0, set()
+
+        model_ids = list({m.id for m in models})
+
+        Task._get_collection().update_many(
+            filter={
+                "project": {"$nin": projects},
+                "models.input.model": {"$in": model_ids},
+            },
+            update={"$set": {"models.input.$[elem].model": None}},
+            array_filters=[{"elem.model": {"$in": model_ids}}],
+            upsert=False,
+        )
+
+        model_tasks = list({m.task for m in models if m.task})
+        if model_tasks:
+            Task._get_collection().update_many(
+                filter={
+                    "_id": {"$in": model_tasks},
+                    "project": {"$nin": projects},
+                    "models.output.model": {"$in": model_ids},
+                },
+                update={"$set": {"models.output.$[elem].model": None}},
+                array_filters=[{"elem.model": {"$in": model_ids}}],
+                upsert=False,
+            )
+
+        urls = {m.uri for m in models if m.uri}
+        deleted = models.delete()
+        return deleted, urls

apiserver/bll/project/project_queries.py

@@ -0,0 +1,370 @@
+import json
+from collections import OrderedDict
+from datetime import datetime
+from typing import (
+    Sequence,
+    Optional,
+    Tuple,
+)
+
+from redis import StrictRedis
+
+from apiserver.config_repo import config
+from apiserver.database.model.model import Model
+from apiserver.database.model.task.task import Task
+from apiserver.redis_manager import redman
+from apiserver.utilities.dicts import nested_get
+from apiserver.utilities.parameter_key_escaper import ParameterKeyEscaper
+from .sub_projects import _ids_with_children
+
+log = config.logger(__file__)
+
+
+class ProjectQueries:
+    def __init__(self, redis=None):
+        self.redis: StrictRedis = redis or redman.connection("apiserver")
+
+    @staticmethod
+    def _get_project_constraint(
+        project_ids: Sequence[str], include_subprojects: bool
+    ) -> dict:
+        """
+        If passed projects is None means top level projects
+        If passed projects is empty means no project filtering
+        """
+        if include_subprojects:
+            if not project_ids:
+                return {}
+            project_ids = _ids_with_children(project_ids)
+
+        if project_ids is None:
+            project_ids = [None]
+        if not project_ids:
+            return {}
+
+        return {"project": {"$in": project_ids}}
+
+    @staticmethod
+    def _get_company_constraint(company_id: str, allow_public: bool = True) -> dict:
+        if allow_public:
+            return {"company": {"$in": [None, "", company_id]}}
+
+        return {"company": company_id}
+
+    @classmethod
+    def get_aggregated_project_parameters(
+        cls,
+        company_id,
+        project_ids: Sequence[str],
+        include_subprojects: bool,
+        page: int = 0,
+        page_size: int = 500,
+    ) -> Tuple[int, int, Sequence[dict]]:
+        page = max(0, page)
+        page_size = max(1, page_size)
+        pipeline = [
+            {
+                "$match": {
+                    **cls._get_company_constraint(company_id),
+                    **cls._get_project_constraint(project_ids, include_subprojects),
+                    "hyperparams": {"$exists": True, "$gt": {}},
+                }
+            },
+            {"$project": {"sections": {"$objectToArray": "$hyperparams"}}},
+            {"$unwind": "$sections"},
+            {
+                "$project": {
+                    "section": "$sections.k",
+                    "names": {"$objectToArray": "$sections.v"},
+                }
+            },
+            {"$unwind": "$names"},
+            {"$group": {"_id": {"section": "$section", "name": "$names.k"}}},
+            {"$sort": OrderedDict({"_id.section": 1, "_id.name": 1})},
+            {"$skip": page * page_size},
+            {"$limit": page_size},
+            {
+                "$group": {
+                    "_id": 1,
+                    "total": {"$sum": 1},
+                    "results": {"$push": "$$ROOT"},
+                }
+            },
+        ]
+
+        result = next(Task.aggregate(pipeline), None)
+
+        total = 0
+        remaining = 0
+        results = []
+
+        if result:
+            total = int(result.get("total", -1))
+            results = [
+                {
+                    "section": ParameterKeyEscaper.unescape(
+                        nested_get(r, ("_id", "section"))
+                    ),
+                    "name": ParameterKeyEscaper.unescape(
+                        nested_get(r, ("_id", "name"))
+                    ),
+                }
+                for r in result.get("results", [])
+            ]
+            remaining = max(0, total - (len(results) + page * page_size))
+
+        return total, remaining, results
+
+    ParamValues = Tuple[int, Sequence[str]]
+
+    def _get_cached_param_values(
+        self, key: str, last_update: datetime, allowed_delta_sec=0
+    ) -> Optional[ParamValues]:
+        try:
+            cached = self.redis.get(key)
+            if not cached:
+                return
+
+            data = json.loads(cached)
+            cached_last_update = datetime.fromtimestamp(data["last_update"])
+            if (last_update - cached_last_update).total_seconds() <= allowed_delta_sec:
+                return data["total"], data["values"]
+        except Exception as ex:
+            log.error(f"Error retrieving params cached values: {str(ex)}")
+
+    def get_task_hyperparam_distinct_values(
+        self,
+        company_id: str,
+        project_ids: Sequence[str],
+        section: str,
+        name: str,
+        include_subprojects: bool,
+        allow_public: bool = True,
+    ) -> ParamValues:
+        company_constraint = self._get_company_constraint(company_id, allow_public)
+        project_constraint = self._get_project_constraint(
+            project_ids, include_subprojects
+        )
+        key_path = f"hyperparams.{ParameterKeyEscaper.escape(section)}.{ParameterKeyEscaper.escape(name)}"
+        last_updated_task = (
+            Task.objects(
+                **company_constraint,
+                **project_constraint,
+                **{f"{key_path.replace('.', '__')}__exists": True},
+            )
+            .only("last_update")
+            .order_by("-last_update")
+            .limit(1)
+            .first()
+        )
+        if not last_updated_task:
+            return 0, []
+
+        redis_key = f"hyperparam_values_{company_id}_{'_'.join(project_ids)}_{section}_{name}_{allow_public}"
+        last_update = last_updated_task.last_update or datetime.utcnow()
+        cached_res = self._get_cached_param_values(
+            key=redis_key,
+            last_update=last_update,
+            allowed_delta_sec=config.get(
+                "services.tasks.hyperparam_values.cache_allowed_outdate_sec", 60
+            ),
+        )
+        if cached_res:
+            return cached_res
+
+        max_values = config.get("services.tasks.hyperparam_values.max_count", 100)
+        pipeline = [
+            {
+                "$match": {
+                    **company_constraint,
+                    **project_constraint,
+                    key_path: {"$exists": True},
+                }
+            },
+            {"$project": {"value": f"${key_path}.value"}},
+            {"$group": {"_id": "$value"}},
+            {"$sort": {"_id": 1}},
+            {"$limit": max_values},
+            {
+                "$group": {
+                    "_id": 1,
+                    "total": {"$sum": 1},
+                    "results": {"$push": "$$ROOT._id"},
+                }
+            },
+        ]
+
+        result = next(Task.aggregate(pipeline, collation=Task._numeric_locale), None)
+        if not result:
+            return 0, []
+
+        total = int(result.get("total", 0))
+        values = result.get("results", [])
+
+        ttl = config.get("services.tasks.hyperparam_values.cache_ttl_sec", 86400)
+        cached = dict(last_update=last_update.timestamp(), total=total, values=values)
+        self.redis.setex(redis_key, ttl, json.dumps(cached))
+
+        return total, values
+
+    @classmethod
+    def get_unique_metric_variants(
+        cls, company_id, project_ids: Sequence[str], include_subprojects: bool
+    ):
+        pipeline = [
+            {
+                "$match": {
+                    **cls._get_company_constraint(company_id),
+                    **cls._get_project_constraint(project_ids, include_subprojects),
+                }
+            },
+            {"$project": {"metrics": {"$objectToArray": "$last_metrics"}}},
+            {"$unwind": "$metrics"},
+            {
+                "$project": {
+                    "metric": "$metrics.k",
+                    "variants": {"$objectToArray": "$metrics.v"},
+                }
+            },
+            {"$unwind": "$variants"},
+            {
+                "$group": {
+                    "_id": {
+                        "metric": "$variants.v.metric",
+                        "variant": "$variants.v.variant",
+                    },
+                    "metrics": {
+                        "$addToSet": {
+                            "metric": "$variants.v.metric",
+                            "metric_hash": "$metric",
+                            "variant": "$variants.v.variant",
+                            "variant_hash": "$variants.k",
+                        }
+                    },
+                }
+            },
+            {"$sort": OrderedDict({"_id.metric": 1, "_id.variant": 1})},
+        ]
+
+        result = Task.aggregate(pipeline)
+        return [r["metrics"][0] for r in result]
+
+    @classmethod
+    def get_model_metadata_keys(
+        cls,
+        company_id,
+        project_ids: Sequence[str],
+        include_subprojects: bool,
+        page: int = 0,
+        page_size: int = 500,
+    ) -> Tuple[int, int, Sequence[dict]]:
+        page = max(0, page)
+        page_size = max(1, page_size)
+        pipeline = [
+            {
+                "$match": {
+                    **cls._get_company_constraint(company_id),
+                    **cls._get_project_constraint(project_ids, include_subprojects),
+                    "metadata": {"$exists": True, "$gt": {}},
+                }
+            },
+            {"$project": {"metadata": {"$objectToArray": "$metadata"}}},
+            {"$unwind": "$metadata"},
+            {"$group": {"_id": "$metadata.k"}},
+            {"$sort": {"_id": 1}},
+            {"$skip": page * page_size},
+            {"$limit": page_size},
+            {
+                "$group": {
+                    "_id": 1,
+                    "total": {"$sum": 1},
+                    "results": {"$push": "$$ROOT"},
+                }
+            },
+        ]
+
+        result = next(Model.aggregate(pipeline), None)
+
+        total = 0
+        remaining = 0
+        results = []
+
+        if result:
+            total = int(result.get("total", -1))
+            results = [
+                ParameterKeyEscaper.unescape(r.get("_id"))
+                for r in result.get("results", [])
+            ]
+            remaining = max(0, total - (len(results) + page * page_size))
+
+        return total, remaining, results
+
+    def get_model_metadata_distinct_values(
+        self,
+        company_id: str,
+        project_ids: Sequence[str],
+        key: str,
+        include_subprojects: bool,
+        allow_public: bool = True,
+    ) -> ParamValues:
+        company_constraint = self._get_company_constraint(company_id, allow_public)
+        project_constraint = self._get_project_constraint(
+            project_ids, include_subprojects
+        )
+        key_path = f"metadata.{ParameterKeyEscaper.escape(key)}"
+        last_updated_model = (
+            Model.objects(
+                **company_constraint,
+                **project_constraint,
+                **{f"{key_path.replace('.', '__')}__exists": True},
+            )
+            .only("last_update")
+            .order_by("-last_update")
+            .limit(1)
+            .first()
+        )
+        if not last_updated_model:
+            return 0, []
+
+        redis_key = f"modelmetadata_values_{company_id}_{'_'.join(project_ids)}_{key}_{allow_public}"
+        last_update = last_updated_model.last_update or datetime.utcnow()
+        cached_res = self._get_cached_param_values(
+            key=redis_key, last_update=last_update
+        )
+        if cached_res:
+            return cached_res
+
+        max_values = config.get("services.models.metadata_values.max_count", 100)
+        pipeline = [
+            {
+                "$match": {
+                    **company_constraint,
+                    **project_constraint,
+                    key_path: {"$exists": True},
+                }
+            },
+            {"$project": {"value": f"${key_path}.value"}},
+            {"$group": {"_id": "$value"}},
+            {"$sort": {"_id": 1}},
+            {"$limit": max_values},
+            {
+                "$group": {
+                    "_id": 1,
+                    "total": {"$sum": 1},
+                    "results": {"$push": "$$ROOT._id"},
+                }
+            },
+        ]
+
+        result = next(Model.aggregate(pipeline, collation=Model._numeric_locale), None)
+        if not result:
+            return 0, []
+
+        total = int(result.get("total", 0))
+        values = result.get("results", [])
+
+        ttl = config.get("services.models.metadata_values.cache_ttl_sec", 86400)
+        cached = dict(last_update=last_update.timestamp(), total=total, values=values)
+        self.redis.setex(redis_key, ttl, json.dumps(cached))
+
+        return total, values

apiserver/bll/project/sub_projects.py

@@ -0,0 +1,189 @@
+import itertools
+from datetime import datetime
+from typing import Tuple, Optional, Sequence, Mapping
+
+from apiserver import database
+from apiserver.apierrors import errors
+from apiserver.database.model import EntityVisibility
+from apiserver.database.model.project import Project
+
+name_separator = "/"
+
+
+def _get_project_depth(project_name: str) -> int:
+    return len(list(filter(None, project_name.split(name_separator))))
+
+
+def _validate_project_name(project_name: str) -> Tuple[str, str]:
+    """
+    Remove redundant '/' characters. Ensure that the project name is not empty
+    Return the cleaned up project name and location
+    """
+    name_parts = list(filter(None, project_name.split(name_separator)))
+    if not name_parts:
+        raise errors.bad_request.InvalidProjectName(name=project_name)
+
+    return name_separator.join(name_parts), name_separator.join(name_parts[:-1])
+
+
+def _ensure_project(
+    company: str, user: str, name: str, creation_params: dict = None
+) -> Optional[Project]:
+    """
+    Makes sure that the project with the given name exists
+    If needed auto-create the project and all the missing projects in the path to it
+    Return the project
+    """
+    name = name.strip(name_separator)
+    if not name:
+        return None
+
+    project = _get_writable_project_from_name(company, name)
+    if project:
+        return project
+
+    now = datetime.utcnow()
+    name, location = _validate_project_name(name)
+    project = Project(
+        id=database.utils.id(),
+        user=user,
+        company=company,
+        created=now,
+        last_update=now,
+        name=name,
+        basename=name.split("/")[-1],
+        **(creation_params or dict(description="")),
+    )
+    parent = _ensure_project(company, user, location, creation_params=creation_params)
+    _save_under_parent(project=project, parent=parent)
+    if parent:
+        parent.update(last_update=now)
+
+    return project
+
+
+def _save_under_parent(project: Project, parent: Optional[Project]):
+    """
+    Save the project under the given parent project or top level (parent=None)
+    Check that the project location matches the parent name
+    """
+    location, _, _ = project.name.rpartition(name_separator)
+    if not parent:
+        if location:
+            raise ValueError(
+                f"Project location {location} does not match empty parent name"
+            )
+        project.parent = None
+        project.path = []
+        project.save()
+        return
+
+    if location != parent.name:
+        raise ValueError(
+            f"Project location {location} does not match parent name {parent.name}"
+        )
+    project.parent = parent.id
+    project.path = [*(parent.path or []), parent.id]
+    project.save()
+
+
+def _get_writable_project_from_name(
+    company,
+    name,
+    _only: Optional[Sequence[str]] = ("id", "name", "path", "company", "parent"),
+) -> Optional[Project]:
+    """
+    Return a project from name. If the project not found then return None
+    """
+    qs = Project.objects(company=company, name=name)
+    if _only:
+        qs = qs.only(*_only)
+    return qs.first()
+
+
+def _get_sub_projects(
+    project_ids: Sequence[str],
+    _only: Sequence[str] = ("id", "path"),
+    search_hidden=True,
+    allowed_ids: Sequence[str] = None,
+) -> Mapping[str, Sequence[Project]]:
+    """
+    Return the list of child projects of all the levels for the parent project ids
+    """
+    query = dict(path__in=project_ids)
+    if not search_hidden:
+        query["system_tags__nin"] = [EntityVisibility.hidden.value]
+    if allowed_ids:
+        query["id__in"] = allowed_ids
+
+    qs = Project.objects(**query)
+    if _only:
+        _only = set(_only) | {"path"}
+        qs = qs.only(*_only)
+    subprojects = list(qs)
+
+    return {
+        pid: [s for s in subprojects if pid in (s.path or [])] for pid in project_ids
+    }
+
+
+def _ids_with_parents(project_ids: Sequence[str]) -> Sequence[str]:
+    """
+    Return project ids with all the parent projects
+    """
+    projects = Project.objects(id__in=project_ids).only("id", "path")
+    parent_ids = set(itertools.chain.from_iterable(p.path for p in projects if p.path))
+    return list({*(p.id for p in projects), *parent_ids})
+
+
+def _ids_with_children(project_ids: Sequence[str]) -> Sequence[str]:
+    """
+    Return project ids with the ids of all the subprojects
+    """
+    subprojects = Project.objects(path__in=project_ids).only("id")
+    return list({*project_ids, *(child.id for child in subprojects)})
+
+
+def _update_subproject_names(
+    project: Project,
+    children: Sequence[Project],
+    old_name: str,
+    update_path: bool = False,
+    old_path: Sequence[str] = None,
+) -> int:
+    """
+    Update sub project names when the base project name changes
+    Optionally update the paths
+    """
+    updated = 0
+    for child in children:
+        child_suffix = name_separator.join(
+            child.name.split(name_separator)[len(old_name.split(name_separator)) :]
+        )
+        updates = {"name": name_separator.join((project.name, child_suffix))}
+        if update_path:
+            updates["path"] = project.path + child.path[len(old_path) :]
+        updated += child.update(upsert=False, **updates)
+
+    return updated
+
+
+def _reposition_project_with_children(
+    project: Project, children: Sequence[Project], parent: Project
+) -> int:
+    new_location = parent.name if parent else None
+    old_name = project.name
+    old_path = project.path
+    project.name = name_separator.join(
+        filter(None, (new_location, project.name.split(name_separator)[-1]))
+    )
+    _save_under_parent(project, parent=parent)
+
+    moved = 1 + _update_subproject_names(
+        project=project,
+        children=children,
+        old_name=old_name,
+        update_path=True,
+        old_path=old_path,
+    )
+    return moved

apiserver/bll/query/builder.py

@@ -1,6 +1,6 @@
 from typing import Optional, Sequence, Iterable, Union
 
-from config import config
+from apiserver.config_repo import config
 
 log = config.logger(__file__)
 

apiserver/bll/queue/queue_bll.py

@@ -4,14 +4,14 @@ from typing import Callable, Sequence, Optional, Tuple
 
 from elasticsearch import Elasticsearch
 
-import database
-import es_factory
-from apierrors import errors
-from bll.queue.queue_metrics import QueueMetrics
-from bll.workers import WorkerBLL
-from config import config
-from database.errors import translate_errors_context
-from database.model.queue import Queue, Entry
+from apiserver import database
+from apiserver.es_factory import es_factory
+from apiserver.apierrors import errors
+from apiserver.bll.queue.queue_metrics import QueueMetrics, MetricsRefresher
+from apiserver.bll.workers import WorkerBLL
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.queue import Queue, Entry
 
 log = config.logger(__file__)
 
@@ -32,6 +32,7 @@ class QueueBLL(object):
         name: str,
         tags: Optional[Sequence[str]] = None,
         system_tags: Optional[Sequence[str]] = None,
+        metadata: Optional[dict] = None,
     ) -> Queue:
         """Creates a queue"""
         with translate_errors_context():
@@ -43,13 +44,31 @@ class QueueBLL(object):
                 name=name,
                 tags=tags or [],
                 system_tags=system_tags or [],
+                metadata=metadata,
                 last_update=now,
             )
             queue.save()
             return queue
 
+    def get_by_name(
+        self, company_id: str, queue_name: str, only: Optional[Sequence[str]] = None,
+    ) -> Queue:
+        qs = Queue.objects(name=queue_name, company=company_id)
+        if only:
+            qs = qs.only(*only)
+
+        return qs.first()
+
+    @staticmethod
+    def _get_task_entries_projection(max_task_entries: int) -> dict:
+        return dict(slice__entries=max_task_entries)
+
     def get_by_id(
-        self, company_id: str, queue_id: str, only: Optional[Sequence[str]] = None
+        self,
+        company_id: str,
+        queue_id: str,
+        only: Optional[Sequence[str]] = None,
+        max_task_entries: int = None,
     ) -> Queue:
         """
         Get queue by id
@@ -60,6 +79,8 @@ class QueueBLL(object):
             qs = Queue.objects(**query)
             if only:
                 qs = qs.only(*only)
+            if max_task_entries:
+                qs = qs.fields(**self._get_task_entries_projection(max_task_entries))
             queue = qs.first()
             if not queue:
                 raise errors.bad_request.InvalidQueueId(**query)
@@ -124,14 +145,32 @@ class QueueBLL(object):
                 )
             queue.delete()
 
-    def get_all(self, company_id: str, query_dict: dict) -> Sequence[dict]:
+    def get_all(
+        self,
+        company_id: str,
+        query_dict: dict,
+        max_task_entries: int = None,
+        ret_params: dict = None,
+    ) -> Sequence[dict]:
         """Get all the queues according to the query"""
         with translate_errors_context():
             return Queue.get_many(
-                company=company_id, parameters=query_dict, query_dict=query_dict
+                company=company_id,
+                parameters=query_dict,
+                query_dict=query_dict,
+                projection_fields=self._get_task_entries_projection(max_task_entries)
+                if max_task_entries
+                else None,
+                ret_params=ret_params,
             )
 
-    def get_queue_infos(self, company_id: str, query_dict: dict) -> Sequence[dict]:
+    def get_queue_infos(
+        self,
+        company_id: str,
+        query_dict: dict,
+        max_task_entries: int = None,
+        ret_params: dict = None,
+    ) -> Sequence[dict]:
         """
         Get infos on all the company queues, including queue tasks and workers
         """
@@ -141,6 +180,10 @@ class QueueBLL(object):
                 company=company_id,
                 query_dict=query_dict,
                 override_projection=projection,
+                projection_fields=self._get_task_entries_projection(max_task_entries)
+                if max_task_entries
+                else None,
+                ret_params=ret_params,
             )
 
             queue_workers = defaultdict(list)
@@ -171,13 +214,15 @@ class QueueBLL(object):
             if any(e.task == task_id for e in queue.entries):
                 raise errors.bad_request.TaskAlreadyQueued(task=task_id)
 
-            self.metrics.log_queue_metrics_to_es(company_id=company_id, queues=[queue])
-
             entry = Entry(added=datetime.utcnow(), task=task_id)
             query = dict(id=queue_id, company=company_id)
             res = Queue.objects(entries__task__ne=task_id, **query).update_one(
                 push__entries=entry, last_update=datetime.utcnow(), upsert=False
             )
+
+            queue.reload()
+            self.metrics.log_queue_metrics_to_es(company_id=company_id, queues=[queue])
+
             if not res:
                 raise errors.bad_request.InvalidQueueOrTaskNotQueued(
                     task=task_id, **query
@@ -217,7 +262,6 @@ class QueueBLL(object):
             queue = self.get_queue_with_task(
                 company_id=company_id, queue_id=queue_id, task_id=task_id
             )
-            self.metrics.log_queue_metrics_to_es(company_id, queues=[queue])
 
             entries_to_remove = [e for e in queue.entries if e.task == task_id]
             query = dict(id=queue_id, company=company_id)
@@ -225,6 +269,9 @@ class QueueBLL(object):
                 pull_all__entries=entries_to_remove, last_update=datetime.utcnow()
             )
 
+            queue.reload()
+            self.metrics.log_queue_metrics_to_es(company_id=company_id, queues=[queue])
+
             return len(entries_to_remove) if res else 0
 
     def reposition_task(
@@ -268,3 +315,25 @@ class QueueBLL(object):
                     )
 
             return new_position
+
+    def count_entries(self, company: str, queue_id: str) -> Optional[int]:
+        res = next(
+            Queue.aggregate(
+                [
+                    {
+                        "$match": {
+                            "company": {"$in": [None, "", company]},
+                            "_id": queue_id,
+                        }
+                    },
+                    {"$project": {"count": {"$size": "$entries"}}},
+                ]
+            ),
+            None,
+        )
+        if res is None:
+            raise errors.bad_request.InvalidQueueId(queue_id=queue_id)
+        return int(res.get("count"))
+
+
+MetricsRefresher.start(queue_metrics=QueueBLL().metrics)

apiserver/bll/queue/queue_metrics.py

@@ -1,35 +1,43 @@
+import json
 from collections import defaultdict
 from datetime import datetime
+from time import sleep
 from typing import Sequence
 
-import elasticsearch.helpers
+from boltons.typeutils import classproperty
 from elasticsearch import Elasticsearch
 
-import es_factory
-from apierrors.errors import bad_request
-from bll.query import Builder as QueryBuilder
-from config import config
-from database.errors import translate_errors_context
-from database.model.queue import Queue, Entry
-from timing_context import TimingContext
+from apiserver.es_factory import es_factory
+from apiserver.apierrors.errors import bad_request
+from apiserver.bll.query import Builder as QueryBuilder
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.queue import Queue, Entry
+from apiserver.redis_manager import redman
+from apiserver.timing_context import TimingContext
+from apiserver.utilities.threads_manager import ThreadsManager
 
 log = config.logger(__file__)
+_conf = config.get("services.queues")
+_queue_metrics_key_pattern = "queue_metrics_{queue}"
+redis = redman.connection("apiserver")
+
+
+class EsKeys:
+    WAITING_TIME_FIELD = "average_waiting_time"
+    QUEUE_LENGTH_FIELD = "queue_length"
+    TIMESTAMP_FIELD = "timestamp"
+    QUEUE_FIELD = "queue"
 
 
 class QueueMetrics:
-    class EsKeys:
-        WAITING_TIME_FIELD = "average_waiting_time"
-        QUEUE_LENGTH_FIELD = "queue_length"
-        TIMESTAMP_FIELD = "timestamp"
-        QUEUE_FIELD = "queue"
-
     def __init__(self, es: Elasticsearch):
         self.es = es
 
     @staticmethod
     def _queue_metrics_prefix_for_company(company_id: str) -> str:
         """Returns the es index prefix for the company"""
-        return f"queue_metrics_{company_id}_"
+        return f"queue_metrics_{company_id.lower()}_"
 
     @staticmethod
     def _get_es_index_suffix():
@@ -49,7 +57,7 @@ class QueueMetrics:
         total_waiting_in_secs = sum((now - e.added).total_seconds() for e in entries)
         return total_waiting_in_secs / len(entries)
 
-    def log_queue_metrics_to_es(self, company_id: str, queues: Sequence[Queue]) -> bool:
+    def log_queue_metrics_to_es(self, company_id: str, queues: Sequence[Queue]) -> int:
         """
         Calculate and write queue statistics (avg waiting time and queue length) to Elastic
         :return: True if the write to es was successful, false otherwise
@@ -63,23 +71,22 @@ class QueueMetrics:
 
         def make_doc(queue: Queue) -> dict:
             entries = [e for e in queue.entries if e.added]
-            return dict(
-                _index=es_index,
-                _source={
-                    self.EsKeys.TIMESTAMP_FIELD: timestamp,
-                    self.EsKeys.QUEUE_FIELD: queue.id,
-                    self.EsKeys.WAITING_TIME_FIELD: self._calc_avg_waiting_time(
-                        entries
-                    ),
-                    self.EsKeys.QUEUE_LENGTH_FIELD: len(entries),
-                },
-            )
+            return {
+                EsKeys.TIMESTAMP_FIELD: timestamp,
+                EsKeys.QUEUE_FIELD: queue.id,
+                EsKeys.WAITING_TIME_FIELD: self._calc_avg_waiting_time(entries),
+                EsKeys.QUEUE_LENGTH_FIELD: len(entries),
+            }
 
-        actions = list(map(make_doc, queues))
+        logged = 0
+        for q in queues:
+            queue_doc = make_doc(q)
+            self.es.index(index=es_index, body=queue_doc)
+            redis_key = _queue_metrics_key_pattern.format(queue=q.id)
+            redis.set(redis_key, json.dumps(queue_doc))
+            logged += 1
 
-        es_res = elasticsearch.helpers.bulk(self.es, actions)
-        added, errors = es_res[:2]
-        return (added == len(actions)) and not errors
+        return logged
 
     def _log_current_metrics(self, company_id: str, queue_ids=Sequence[str]):
         query = dict(company=company_id)
@@ -90,8 +97,7 @@ class QueueMetrics:
 
     def _search_company_metrics(self, company_id: str, es_req: dict) -> dict:
         return self.es.search(
-            index=f"{self._queue_metrics_prefix_for_company(company_id)}*",
-            body=es_req,
+            index=f"{self._queue_metrics_prefix_for_company(company_id)}*", body=es_req,
         )
 
     @classmethod
@@ -105,13 +111,13 @@ class QueueMetrics:
         return {
             "dates": {
                 "date_histogram": {
-                    "field": cls.EsKeys.TIMESTAMP_FIELD,
+                    "field": EsKeys.TIMESTAMP_FIELD,
                     "fixed_interval": f"{interval}s",
                     "min_doc_count": 1,
                 },
                 "aggs": {
                     "queues": {
-                        "terms": {"field": cls.EsKeys.QUEUE_FIELD},
+                        "terms": {"field": EsKeys.QUEUE_FIELD},
                         "aggs": cls._get_top_waiting_agg(),
                     }
                 },
@@ -128,13 +134,13 @@ class QueueMetrics:
             "top_avg_waiting": {
                 "top_hits": {
                     "sort": [
-                        {cls.EsKeys.WAITING_TIME_FIELD: {"order": "desc"}},
-                        {cls.EsKeys.QUEUE_LENGTH_FIELD: {"order": "desc"}},
+                        {EsKeys.WAITING_TIME_FIELD: {"order": "desc"}},
+                        {EsKeys.QUEUE_LENGTH_FIELD: {"order": "desc"}},
                     ],
                     "_source": {
                         "includes": [
-                            cls.EsKeys.WAITING_TIME_FIELD,
-                            cls.EsKeys.QUEUE_LENGTH_FIELD,
+                            EsKeys.WAITING_TIME_FIELD,
+                            EsKeys.QUEUE_LENGTH_FIELD,
                         ]
                     },
                     "size": 1,
@@ -149,6 +155,7 @@ class QueueMetrics:
         to_date: float,
         interval: int,
         queue_ids: Sequence[str],
+        refresh: bool = False,
     ) -> dict:
         """
         Get the company queue metrics in the specified time range.
@@ -158,7 +165,8 @@ class QueueMetrics:
         In case no queue ids are specified the avg across all the
         company queues is calculated for each metric
         """
-        # self._log_current_metrics(company, queue_ids=queue_ids)
+        if refresh:
+            self._log_current_metrics(company_id, queue_ids=queue_ids)
 
         if from_date >= to_date:
             raise bad_request.FieldsValueError("from_date must be less than to_date")
@@ -256,7 +264,47 @@ class QueueMetrics:
                 continue
             res = queue_data["top_avg_waiting"]["hits"]["hits"][0]["_source"]
             queue_metrics[queue_data["key"]] = {
-                "queue_length": res[cls.EsKeys.QUEUE_LENGTH_FIELD],
-                "avg_waiting_time": res[cls.EsKeys.WAITING_TIME_FIELD],
+                "queue_length": res[EsKeys.QUEUE_LENGTH_FIELD],
+                "avg_waiting_time": res[EsKeys.WAITING_TIME_FIELD],
             }
         return queue_metrics
+
+
+class MetricsRefresher:
+    threads = ThreadsManager()
+
+    @classproperty
+    def watch_interval_sec(self):
+        return _conf.get("metrics_refresh_interval_sec", 300)
+
+    @classmethod
+    @threads.register("queue_metrics_refresh_watchdog", daemon=True)
+    def start(cls, queue_metrics: QueueMetrics):
+        if not cls.watch_interval_sec:
+            return
+
+        sleep(10)
+        while not ThreadsManager.terminating:
+            try:
+                for queue in Queue.objects():
+                    timestamp = es_factory.get_timestamp_millis()
+                    doc_time = 0
+                    try:
+                        redis_key = _queue_metrics_key_pattern.format(queue=queue.id)
+                        data = redis.get(redis_key)
+                        if data:
+                            queue_doc = json.loads(data)
+                            doc_time = int(queue_doc.get(EsKeys.TIMESTAMP_FIELD))
+                    except Exception as ex:
+                        log.exception(
+                            f"Error reading queue metrics data for queue {queue.id}: {str(ex)}"
+                        )
+
+                    if (
+                        not doc_time
+                        or (timestamp - doc_time) > cls.watch_interval_sec * 1000
+                    ):
+                        queue_metrics.log_queue_metrics_to_es(queue.company, [queue])
+            except Exception as ex:
+                log.exception(f"Failed collecting queue metrics: {str(ex)}")
+            sleep(60)

apiserver/bll/redis_cache_manager.py

@@ -3,8 +3,8 @@ from typing import Optional, TypeVar, Generic, Type, Callable
 
 from redis import StrictRedis
 
-import database
-from timing_context import TimingContext
+from apiserver import database
+from apiserver.timing_context import TimingContext
 
 T = TypeVar("T")
 
@@ -49,6 +49,21 @@ class RedisCacheManager(Generic[T]):
     def _get_redis_key(self, state_id):
         return f"{self.state_class}/{state_id}"
 
+    def get_or_create_state_core(
+        self,
+        state_id=None,
+        init_state: Callable[[T], None] = _do_nothing,
+        validate_state: Callable[[T], None] = _do_nothing,
+    ) -> T:
+        state = self.get_state(state_id) if state_id else None
+        if state:
+            validate_state(state)
+        else:
+            state = self.state_class(id=database.utils.id())
+            init_state(state)
+
+        return state
+
     @contextmanager
     def get_or_create_state(
         self,
@@ -66,12 +81,9 @@ class RedisCacheManager(Generic[T]):
         :param validate_state: user callback to validate the state if retrieved from cache
         Should throw an exception if the state is not valid. If not passed then no validation is done
         """
-        state = self.get_state(state_id) if state_id else None
-        if state:
-            validate_state(state)
-        else:
-            state = self.state_class(id=database.utils.id())
-            init_state(state)
+        state = self.get_or_create_state_core(
+            state_id=state_id, init_state=init_state, validate_state=validate_state
+        )
 
         try:
             yield state

apiserver/bll/statistics/resource_monitor.py

@@ -6,7 +6,7 @@ from time import sleep
 import attr
 import psutil
 
-from utilities.threads_manager import ThreadsManager
+from apiserver.utilities.threads_manager import ThreadsManager
 
 
 class ResourceMonitor(Thread):

apiserver/bll/statistics/stats_reporter.py

@@ -11,18 +11,18 @@ import requests
 from requests.adapters import HTTPAdapter
 from requests.packages.urllib3.util.retry import Retry
 
-from bll.query import Builder as QueryBuilder
-from bll.util import get_server_uuid
-from bll.workers import WorkerStats, WorkerBLL
-from config import config
-from config.info import get_deployment_type
-from database.model import Company, User
-from database.model.queue import Queue
-from database.model.task.task import Task
-from tools import safe_get
-from utilities.json import dumps
-from utilities.threads_manager import ThreadsManager
-from version import __version__ as current_version
+from apiserver.bll.query import Builder as QueryBuilder
+from apiserver.bll.util import get_server_uuid
+from apiserver.bll.workers import WorkerStats, WorkerBLL
+from apiserver.config_repo import config
+from apiserver.config.info import get_deployment_type
+from apiserver.database.model import Company, User
+from apiserver.database.model.queue import Queue
+from apiserver.database.model.task.task import Task
+from apiserver.tools import safe_get
+from apiserver.utilities.json import dumps
+from apiserver.utilities.threads_manager import ThreadsManager
+from apiserver.version import __version__ as current_version
 from .resource_monitor import ResourceMonitor
 
 log = config.logger(__file__)
@@ -45,7 +45,7 @@ class StatisticsReporter:
     def start_reporter(cls):
         """
         Periodically send statistics reports for companies who have opted in.
-        Note: in trains we usually have only a single company
+        Note: in clearml we usually have only a single company
         """
         if not cls.supported:
             return

apiserver/bll/task/__init__.py

@@ -3,5 +3,4 @@ from .utils import (
     ChangeStatusRequest,
     update_project_time,
     validate_status_change,
-    split_by,
 )

apiserver/bll/task/artifacts.py

@@ -0,0 +1,97 @@
+from operator import itemgetter
+from typing import Sequence
+
+from apiserver.apimodels.tasks import Artifact as ApiArtifact, ArtifactId
+from apiserver.bll.task.utils import get_task_for_update, update_task
+from apiserver.database.model.task.task import DEFAULT_ARTIFACT_MODE, Artifact
+from apiserver.database.utils import hash_field_name
+from apiserver.timing_context import TimingContext
+from apiserver.utilities.dicts import nested_get, nested_set
+from apiserver.utilities.parameter_key_escaper import mongoengine_safe
+
+
+def get_artifact_id(artifact: dict):
+    """
+    Calculate id from 'key' and 'mode' fields
+    Return hash on on the id so that it will not contain mongo illegal characters
+    """
+    key_hash: str = hash_field_name(artifact["key"])
+    mode: str = artifact.get("mode", DEFAULT_ARTIFACT_MODE)
+    return f"{key_hash}_{mode}"
+
+
+def artifacts_prepare_for_save(fields: dict):
+    artifacts_field = ("execution", "artifacts")
+    artifacts = nested_get(fields, artifacts_field)
+    if artifacts is None:
+        return
+
+    nested_set(
+        fields, artifacts_field, value={get_artifact_id(a): a for a in artifacts}
+    )
+
+
+def artifacts_unprepare_from_saved(fields):
+    artifacts_field = ("execution", "artifacts")
+    artifacts = nested_get(fields, artifacts_field)
+    if artifacts is None:
+        return
+
+    nested_set(
+        fields,
+        artifacts_field,
+        value=sorted(artifacts.values(), key=itemgetter("key")),
+    )
+
+
+class Artifacts:
+    @classmethod
+    def add_or_update_artifacts(
+        cls,
+        company_id: str,
+        task_id: str,
+        artifacts: Sequence[ApiArtifact],
+        force: bool,
+    ) -> int:
+        with TimingContext("mongo", "update_artifacts"):
+            task = get_task_for_update(
+                company_id=company_id,
+                task_id=task_id,
+                force=force,
+            )
+
+            artifacts = {
+                get_artifact_id(a): Artifact(**a)
+                for a in (api_artifact.to_struct() for api_artifact in artifacts)
+            }
+
+            update_cmds = {
+                f"set__execution__artifacts__{mongoengine_safe(name)}": value
+                for name, value in artifacts.items()
+            }
+            return update_task(task, update_cmds=update_cmds)
+
+    @classmethod
+    def delete_artifacts(
+        cls,
+        company_id: str,
+        task_id: str,
+        artifact_ids: Sequence[ArtifactId],
+        force: bool,
+    ) -> int:
+        with TimingContext("mongo", "delete_artifacts"):
+            task = get_task_for_update(
+                company_id=company_id,
+                task_id=task_id,
+                force=force,
+            )
+
+            artifact_ids = [
+                get_artifact_id(a)
+                for a in (artifact_id.to_struct() for artifact_id in artifact_ids)
+            ]
+            delete_cmds = {
+                f"unset__execution__artifacts__{id_}": 1 for id_ in set(artifact_ids)
+            }
+
+            return update_task(task, update_cmds=delete_cmds)

apiserver/bll/task/hyperparams.py

@@ -0,0 +1,247 @@
+from itertools import chain
+from operator import attrgetter
+from typing import Sequence, Dict
+
+from boltons import iterutils
+
+from apiserver.apierrors import errors
+from apiserver.apimodels.tasks import (
+    HyperParamKey,
+    HyperParamItem,
+    ReplaceHyperparams,
+    Configuration,
+)
+from apiserver.bll.task import TaskBLL
+from apiserver.bll.task.utils import get_task_for_update, update_task
+from apiserver.config_repo import config
+from apiserver.database.model.task.task import ParamsItem, Task, ConfigurationItem
+from apiserver.timing_context import TimingContext
+from apiserver.utilities.parameter_key_escaper import (
+    ParameterKeyEscaper,
+    mongoengine_safe,
+)
+
+log = config.logger(__file__)
+task_bll = TaskBLL()
+
+
+class HyperParams:
+    _properties_section = "properties"
+
+    @classmethod
+    def get_params(cls, company_id: str, task_ids: Sequence[str]) -> Dict[str, dict]:
+        only = ("id", "hyperparams")
+        tasks = task_bll.assert_exists(
+            company_id=company_id, task_ids=task_ids, only=only, allow_public=True,
+        )
+
+        return {
+            task.id: {"hyperparams": cls._get_params_list(items=task.hyperparams)}
+            for task in tasks
+        }
+
+    @classmethod
+    def _get_params_list(
+        cls, items: Dict[str, Dict[str, ParamsItem]]
+    ) -> Sequence[dict]:
+        ret = list(chain.from_iterable(v.values() for v in items.values()))
+        return [
+            p.to_proper_dict() for p in sorted(ret, key=attrgetter("section", "name"))
+        ]
+
+    @classmethod
+    def _normalize_params(cls, params: Sequence) -> bool:
+        """
+        Lower case properties section and return True if it is the only section
+        """
+        for p in params:
+            if p.section.lower() == cls._properties_section:
+                p.section = cls._properties_section
+
+        return all(p.section == cls._properties_section for p in params)
+
+    @classmethod
+    def delete_params(
+        cls,
+        company_id: str,
+        task_id: str,
+        hyperparams: Sequence[HyperParamKey],
+        force: bool,
+    ) -> int:
+        with TimingContext("mongo", "delete_hyperparams"):
+            properties_only = cls._normalize_params(hyperparams)
+            task = get_task_for_update(
+                company_id=company_id,
+                task_id=task_id,
+                allow_all_statuses=properties_only,
+                force=force,
+            )
+
+            with_param, without_param = iterutils.partition(
+                hyperparams, key=lambda p: bool(p.name)
+            )
+            sections_to_delete = {p.section for p in without_param}
+            delete_cmds = {
+                f"unset__hyperparams__{ParameterKeyEscaper.escape(section)}": 1
+                for section in sections_to_delete
+            }
+
+            for item in with_param:
+                section = ParameterKeyEscaper.escape(item.section)
+                if item.section in sections_to_delete:
+                    raise errors.bad_request.FieldsConflict(
+                        "Cannot delete section field if the whole section was scheduled for deletion"
+                    )
+                name = ParameterKeyEscaper.escape(item.name)
+                delete_cmds[f"unset__hyperparams__{section}__{name}"] = 1
+
+            return update_task(
+                task, update_cmds=delete_cmds, set_last_update=not properties_only
+            )
+
+    @classmethod
+    def edit_params(
+        cls,
+        company_id: str,
+        task_id: str,
+        hyperparams: Sequence[HyperParamItem],
+        replace_hyperparams: str,
+        force: bool,
+    ) -> int:
+        with TimingContext("mongo", "edit_hyperparams"):
+            properties_only = cls._normalize_params(hyperparams)
+            task = get_task_for_update(
+                company_id=company_id,
+                task_id=task_id,
+                allow_all_statuses=properties_only,
+                force=force,
+            )
+
+            update_cmds = dict()
+            hyperparams = cls._db_dicts_from_list(hyperparams)
+            if replace_hyperparams == ReplaceHyperparams.all:
+                update_cmds["set__hyperparams"] = hyperparams
+            elif replace_hyperparams == ReplaceHyperparams.section:
+                for section, value in hyperparams.items():
+                    update_cmds[
+                        f"set__hyperparams__{mongoengine_safe(section)}"
+                    ] = value
+            else:
+                for section, section_params in hyperparams.items():
+                    for name, value in section_params.items():
+                        update_cmds[
+                            f"set__hyperparams__{section}__{mongoengine_safe(name)}"
+                        ] = value
+
+            return update_task(
+                task, update_cmds=update_cmds, set_last_update=not properties_only
+            )
+
+    @classmethod
+    def _db_dicts_from_list(cls, items: Sequence[HyperParamItem]) -> Dict[str, dict]:
+        sections = iterutils.bucketize(items, key=attrgetter("section"))
+        return {
+            ParameterKeyEscaper.escape(section): {
+                ParameterKeyEscaper.escape(param.name): ParamsItem(**param.to_struct())
+                for param in params
+            }
+            for section, params in sections.items()
+        }
+
+    @classmethod
+    def get_configurations(
+        cls, company_id: str, task_ids: Sequence[str], names: Sequence[str]
+    ) -> Dict[str, dict]:
+        only = ["id"]
+        if names:
+            only.extend(
+                f"configuration.{ParameterKeyEscaper.escape(name)}" for name in names
+            )
+        else:
+            only.append("configuration")
+        tasks = task_bll.assert_exists(
+            company_id=company_id, task_ids=task_ids, only=only, allow_public=True,
+        )
+
+        return {
+            task.id: {
+                "configuration": [
+                    c.to_proper_dict()
+                    for c in sorted(task.configuration.values(), key=attrgetter("name"))
+                ]
+            }
+            for task in tasks
+        }
+
+    @classmethod
+    def get_configuration_names(
+        cls, company_id: str, task_ids: Sequence[str], skip_empty: bool
+    ) -> Dict[str, list]:
+        skip_empty_condition = {"$match": {"items.v.value": {"$nin": [None, ""]}}}
+        pipeline = [
+            {
+                "$match": {
+                    "company": {"$in": [None, "", company_id]},
+                    "_id": {"$in": task_ids},
+                }
+            },
+            {"$project": {"items": {"$objectToArray": "$configuration"}}},
+            {"$unwind": "$items"},
+            *([skip_empty_condition] if skip_empty else []),
+            {"$group": {"_id": "$_id", "names": {"$addToSet": "$items.k"}}},
+        ]
+
+        with TimingContext("mongo", "get_configuration_names"):
+            tasks = Task.aggregate(pipeline)
+
+            return {
+                task["_id"]: {
+                    "names": sorted(
+                        ParameterKeyEscaper.unescape(name) for name in task["names"]
+                    )
+                }
+                for task in tasks
+            }
+
+    @classmethod
+    def edit_configuration(
+        cls,
+        company_id: str,
+        task_id: str,
+        configuration: Sequence[Configuration],
+        replace_configuration: bool,
+        force: bool,
+    ) -> int:
+        with TimingContext("mongo", "edit_configuration"):
+            task = get_task_for_update(
+                company_id=company_id, task_id=task_id, force=force
+            )
+
+            update_cmds = dict()
+            configuration = {
+                ParameterKeyEscaper.escape(c.name): ConfigurationItem(**c.to_struct())
+                for c in configuration
+            }
+            if replace_configuration:
+                update_cmds["set__configuration"] = configuration
+            else:
+                for name, value in configuration.items():
+                    update_cmds[f"set__configuration__{mongoengine_safe(name)}"] = value
+
+            return update_task(task, update_cmds=update_cmds)
+
+    @classmethod
+    def delete_configuration(
+        cls, company_id: str, task_id: str, configuration: Sequence[str], force: bool
+    ) -> int:
+        with TimingContext("mongo", "delete_configuration"):
+            task = get_task_for_update(
+                company_id=company_id, task_id=task_id, force=force
+            )
+
+            delete_cmds = {
+                f"unset__configuration__{ParameterKeyEscaper.escape(name)}": 1
+                for name in set(configuration)
+            }
+
+            return update_task(task, update_cmds=delete_cmds)

apiserver/bll/task/non_responsive_tasks_watchdog.py

@@ -1,11 +1,10 @@
 from datetime import timedelta, datetime
 from time import sleep
 
-from apierrors import errors
-from bll.task import ChangeStatusRequest
-from config import config
-from database.model.task.task import TaskStatus, Task
-from utilities.threads_manager import ThreadsManager
+from apiserver.bll.task import update_project_time
+from apiserver.config_repo import config
+from apiserver.database.model.task.task import TaskStatus, Task
+from apiserver.utilities.threads_manager import ThreadsManager
 
 log = config.logger(__file__)
 
@@ -71,19 +70,29 @@ class NonResponsiveTasksWatchdog:
             return 0
 
         err_count = 0
+        project_ids = set()
+        now = datetime.utcnow()
         for task in tasks:
             log.info(
                 f"Stopping {task.id} ({task.name}), last updated at {task.last_update}"
             )
+            # noinspection PyBroadException
             try:
-                ChangeStatusRequest(
-                    task=task,
-                    new_status=TaskStatus.stopped,
+                updated = Task.objects(id=task.id, status=task.status).update(
+                    status=TaskStatus.stopped,
                     status_reason="Forced stop (non-responsive)",
                     status_message="Forced stop (non-responsive)",
-                    force=True,
-                ).execute()
-            except errors.bad_request.FailedChangingTaskStatus:
-                err_count += 1
+                    status_changed=now,
+                    last_update=now,
+                    last_change=now,
+                )
+                if updated:
+                    project_ids.add(task.project)
+                else:
+                    err_count += 1
+            except Exception as ex:
+                log.error("Failed setting status: %s", str(ex))
+
+        update_project_time(list(project_ids))
 
         return len(tasks) - err_count

apiserver/bll/task/param_utils.py

@@ -1,12 +1,11 @@
 import itertools
-from typing import Sequence, Tuple
+from typing import Sequence, Tuple, Optional
 
-import dpath
 
-from apierrors import errors
-from database.model.task.task import Task
-from tools import safe_get
-from utilities.parameter_key_escaper import ParameterKeyEscaper
+from apiserver.apierrors import errors
+from apiserver.database.model.task.task import Task
+from apiserver.utilities.dicts import nested_get, nested_delete, nested_set
+from apiserver.utilities.parameter_key_escaper import ParameterKeyEscaper
 
 
 hyperparams_default_section = "Args"
@@ -14,7 +13,7 @@ hyperparams_legacy_type = "legacy"
 tf_define_section = "TF_DEFINE"
 
 
-def split_param_name(full_name: str, default_section: str) -> Tuple[str, str]:
+def split_param_name(full_name: str, default_section: str) -> Tuple[Optional[str], str]:
     """
     Return parameter section and name. The section is either TF_DEFINE or the default one
     """
@@ -62,7 +61,7 @@ def _remove_legacy_params(data: dict, with_sections: bool = False) -> int:
     return removed
 
 
-def _get_legacy_params(data: dict, with_sections: bool = False) -> Sequence[str]:
+def _get_legacy_params(data: dict, with_sections: bool = False) -> Sequence[dict]:
     """
     Remove the legacy params from the data dict and return the number of removed params
     If the path not found then return 0
@@ -71,8 +70,10 @@ def _get_legacy_params(data: dict, with_sections: bool = False) -> Sequence[str]
         return []
 
     if with_sections:
-        return itertools.chain.from_iterable(
-            _get_legacy_params(section_data) for section_data in data.values()
+        return list(
+            itertools.chain.from_iterable(
+                _get_legacy_params(section_data) for section_data in data.values()
+            )
         )
 
     return [
@@ -86,15 +87,15 @@ def params_prepare_for_save(fields: dict, previous_task: Task = None):
     Escape all the section and param names for hyper params and configuration to make it mongo sage
     """
     for old_params_field, new_params_field, default_section in (
-        ("execution/parameters", "hyperparams", hyperparams_default_section),
-        ("execution/model_desc", "configuration", None),
+        (("execution", "parameters"), "hyperparams", hyperparams_default_section),
+        (("execution", "model_desc"), "configuration", None),
     ):
-        legacy_params = safe_get(fields, old_params_field)
+        legacy_params = nested_get(fields, old_params_field)
         if legacy_params is None:
             continue
 
         if (
-            not safe_get(fields, new_params_field)
+            not fields.get(new_params_field)
             and previous_task
             and previous_task[new_params_field]
         ):
@@ -117,21 +118,34 @@ def params_prepare_for_save(fields: dict, previous_task: Task = None):
             new_param = dict(name=name, type=hyperparams_legacy_type, value=str(value))
             if section is not None:
                 new_param["section"] = section
-            dpath.new(fields, new_path, new_param)
-        dpath.delete(fields, old_params_field)
+            nested_set(fields, new_path, new_param)
+        nested_delete(fields, old_params_field)
 
-    for param_field in ("hyperparams", "configuration"):
-        params = safe_get(fields, param_field)
-        if params:
-            escaped_params = {
-                ParameterKeyEscaper.escape(key): {
-                    ParameterKeyEscaper.escape(k): v for k, v in value.items()
-                }
-                if isinstance(value, dict)
-                else value
-                for key, value in params.items()
+    def ensure_non_empty(k: str, desc: str) -> str:
+        if not k:
+            raise errors.bad_request.ValidationError(
+                f"Empty {desc} name is not allowed"
+            )
+        return k
+
+    params = fields.get("hyperparams")
+    if params:
+        escaped_params = {
+            ParameterKeyEscaper.escape(ensure_non_empty(key, "section")): {
+                ParameterKeyEscaper.escape(ensure_non_empty(k, "parameter")): v
+                for k, v in value.items()
             }
-            dpath.set(fields, param_field, escaped_params)
+            for key, value in params.items()
+        }
+        fields["hyperparams"] = escaped_params
+
+    params = fields.get("configuration")
+    if params:
+        escaped_params = {
+            ParameterKeyEscaper.escape(ensure_non_empty(key, "configuration")): value
+            for key, value in params.items()
+        }
+        fields["configuration"] = escaped_params
 
 
 def params_unprepare_from_saved(fields, copy_to_legacy=False):
@@ -140,7 +154,7 @@ def params_unprepare_from_saved(fields, copy_to_legacy=False):
     If copy_to_legacy is set then copy hyperparams and configuration data to the legacy location for the old clients
     """
     for param_field in ("hyperparams", "configuration"):
-        params = safe_get(fields, param_field)
+        params = fields.get(param_field)
         if params:
             unescaped_params = {
                 ParameterKeyEscaper.unescape(key): {
@@ -150,18 +164,18 @@ def params_unprepare_from_saved(fields, copy_to_legacy=False):
                 else value
                 for key, value in params.items()
             }
-            dpath.set(fields, param_field, unescaped_params)
+            fields[param_field] = unescaped_params
 
     if copy_to_legacy:
         for new_params_field, old_params_field, use_sections in (
-            (f"hyperparams", "execution/parameters", True),
-            (f"configuration", "execution/model_desc", False),
+            ("hyperparams", ("execution", "parameters"), True),
+            ("configuration", ("execution", "model_desc"), False),
         ):
             legacy_params = _get_legacy_params(
-                safe_get(fields, new_params_field), with_sections=use_sections
+                fields.get(new_params_field), with_sections=use_sections
             )
             if legacy_params:
-                dpath.new(
+                nested_set(
                     fields,
                     old_params_field,
                     {_get_full_param_name(p): p["value"] for p in legacy_params},
@@ -174,7 +188,7 @@ def _process_path(path: str):
     Need to unescape and apply a full mongo escaping
     """
     parts = path.split(".")
-    if len(parts) < 2 or len(parts) > 3:
+    if len(parts) < 2 or len(parts) > 4:
         raise errors.bad_request.ValidationError("invalid task field", path=path)
     return ".".join(
         ParameterKeyEscaper.escape(ParameterKeyEscaper.unescape(p)) for p in parts
@@ -184,7 +198,8 @@ def _process_path(path: str):
 def escape_paths(paths: Sequence[str]) -> Sequence[str]:
     for old_prefix, new_prefix in (
         ("execution.parameters", f"hyperparams.{hyperparams_default_section}"),
-        ("execution.model_desc", f"configuration"),
+        ("execution.model_desc", "configuration"),
+        ("execution.docker_cmd", "container"),
     ):
         path: str
         paths = [path.replace(old_prefix, new_prefix) for path in paths]

apiserver/bll/task/task_bll.py

@@ -0,0 +1,488 @@
+from datetime import datetime
+from typing import Collection, Sequence, Tuple, Any, Optional, Dict
+
+import six
+from mongoengine import Q
+from redis import StrictRedis
+from six import string_types
+
+import apiserver.database.utils as dbutils
+from apiserver.apierrors import errors
+from apiserver.apimodels.tasks import TaskInputModel
+from apiserver.bll.queue import QueueBLL
+from apiserver.bll.organization import OrgBLL, Tags
+from apiserver.bll.project import ProjectBLL
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.model import Model
+from apiserver.database.model.project import Project
+from apiserver.database.model.task.metrics import EventStats, MetricEventStats
+from apiserver.database.model.task.output import Output
+from apiserver.database.model.task.task import (
+    Task,
+    TaskStatus,
+    TaskSystemTags,
+    ArtifactModes,
+    ModelItem,
+    Models,
+    DEFAULT_ARTIFACT_MODE,
+    TaskModelNames,
+    TaskModelTypes,
+)
+from apiserver.database.model import EntityVisibility
+from apiserver.database.utils import get_company_or_none_constraint, id as create_id
+from apiserver.es_factory import es_factory
+from apiserver.redis_manager import redman
+from apiserver.service_repo import APICall
+from apiserver.services.utils import validate_tags, escape_dict_field, escape_dict
+from apiserver.timing_context import TimingContext
+from .artifacts import artifacts_prepare_for_save
+from .param_utils import params_prepare_for_save
+from .utils import (
+    ChangeStatusRequest,
+    update_project_time,
+    deleted_prefix,
+)
+
+log = config.logger(__file__)
+org_bll = OrgBLL()
+queue_bll = QueueBLL()
+project_bll = ProjectBLL()
+
+
+class TaskBLL:
+    def __init__(self, events_es=None, redis=None):
+        self.events_es = events_es or es_factory.connect("events")
+        self.redis: StrictRedis = redis or redman.connection("apiserver")
+
+    @staticmethod
+    def get_task_with_access(
+        task_id, company_id, only=None, allow_public=False, requires_write_access=False
+    ) -> Task:
+        """
+        Gets a task that has a required write access
+        :except errors.bad_request.InvalidTaskId: if the task is not found
+        :except errors.forbidden.NoWritePermission: if write_access was required and the task cannot be modified
+        """
+        with translate_errors_context():
+            query = dict(id=task_id, company=company_id)
+            with TimingContext("mongo", "task_with_access"):
+                if requires_write_access:
+                    task = Task.get_for_writing(_only=only, **query)
+                else:
+                    task = Task.get(_only=only, **query, include_public=allow_public)
+
+            if not task:
+                raise errors.bad_request.InvalidTaskId(**query)
+
+            return task
+
+    @staticmethod
+    def get_by_id(
+        company_id, task_id, required_status=None, only_fields=None, allow_public=False,
+    ):
+        if only_fields:
+            if isinstance(only_fields, string_types):
+                only_fields = [only_fields]
+            else:
+                only_fields = list(only_fields)
+            only_fields = only_fields + ["status"]
+
+        with TimingContext("mongo", "task_by_id_all"):
+            tasks = Task.get_many(
+                company=company_id,
+                query=Q(id=task_id),
+                allow_public=allow_public,
+                override_projection=only_fields,
+                return_dicts=False,
+            )
+            task = None if not tasks else tasks[0]
+
+        if not task:
+            raise errors.bad_request.InvalidTaskId(id=task_id)
+
+        if required_status and not task.status == required_status:
+            raise errors.bad_request.InvalidTaskStatus(expected=required_status)
+
+        return task
+
+    @staticmethod
+    def assert_exists(
+        company_id, task_ids, only=None, allow_public=False, return_tasks=True
+    ) -> Optional[Sequence[Task]]:
+        task_ids = [task_ids] if isinstance(task_ids, six.string_types) else task_ids
+        with translate_errors_context(), TimingContext("mongo", "task_exists"):
+            ids = set(task_ids)
+            q = Task.get_many(
+                company=company_id,
+                query=Q(id__in=ids),
+                allow_public=allow_public,
+                return_dicts=False,
+            )
+            if only:
+                # Make sure to reset fields filters (some fields are excluded by default) since this
+                # is an internal call and specific fields were requested.
+                q = q.all_fields().only(*only)
+
+            if q.count() != len(ids):
+                raise errors.bad_request.InvalidTaskId(ids=task_ids)
+
+            if return_tasks:
+                return list(q)
+
+    @staticmethod
+    def create(call: APICall, fields: dict):
+        identity = call.identity
+        now = datetime.utcnow()
+        return Task(
+            id=create_id(),
+            user=identity.user,
+            company=identity.company,
+            created=now,
+            last_update=now,
+            last_change=now,
+            **fields,
+        )
+
+    @staticmethod
+    def validate_input_models(task, allow_only_public=False):
+        if not task.models.input:
+            return
+
+        company = None if allow_only_public else task.company
+        model_ids = set(m.model for m in task.models.input)
+        models = Model.objects(
+            Q(id__in=model_ids) & get_company_or_none_constraint(company)
+        ).only("id")
+        missing = model_ids - {m.id for m in models}
+        if missing:
+            raise errors.bad_request.InvalidModelId(models=missing)
+
+        return
+
+    @classmethod
+    def clone_task(
+        cls,
+        company_id: str,
+        user_id: str,
+        task_id: str,
+        name: Optional[str] = None,
+        comment: Optional[str] = None,
+        parent: Optional[str] = None,
+        project: Optional[str] = None,
+        tags: Optional[Sequence[str]] = None,
+        system_tags: Optional[Sequence[str]] = None,
+        hyperparams: Optional[dict] = None,
+        configuration: Optional[dict] = None,
+        container: Optional[dict] = None,
+        execution_overrides: Optional[dict] = None,
+        input_models: Optional[Sequence[TaskInputModel]] = None,
+        validate_references: bool = False,
+        new_project_name: str = None,
+    ) -> Tuple[Task, dict]:
+        validate_tags(tags, system_tags)
+        params_dict = {
+            field: value
+            for field, value in (
+                ("hyperparams", hyperparams),
+                ("configuration", configuration),
+            )
+            if value is not None
+        }
+
+        task = cls.get_by_id(company_id=company_id, task_id=task_id, allow_public=True)
+
+        now = datetime.utcnow()
+        if input_models:
+            input_models = [
+                ModelItem(model=m.model, name=m.name, updated=now) for m in input_models
+            ]
+
+        execution_dict = task.execution.to_proper_dict() if task.execution else {}
+        if execution_overrides:
+            execution_model = execution_overrides.pop("model", None)
+            if not input_models and execution_model:
+                input_models = [
+                    ModelItem(
+                        model=execution_model,
+                        name=TaskModelNames[TaskModelTypes.input],
+                        updated=now,
+                    )
+                ]
+
+            docker_cmd = execution_overrides.pop("docker_cmd", None)
+            if not container and docker_cmd:
+                image, _, arguments = docker_cmd.partition(" ")
+                container = {"image": image, "arguments": arguments}
+
+            artifacts_prepare_for_save({"execution": execution_overrides})
+
+            params_dict["execution"] = {}
+            for legacy_param in ("parameters", "configuration"):
+                legacy_value = execution_overrides.pop(legacy_param, None)
+                if legacy_value is not None:
+                    params_dict["execution"] = legacy_value
+
+            escape_dict_field(execution_overrides, "model_labels")
+
+            execution_dict.update(execution_overrides)
+
+        params_prepare_for_save(params_dict, previous_task=task)
+
+        artifacts = execution_dict.get("artifacts")
+        if artifacts:
+            execution_dict["artifacts"] = {
+                k: a
+                for k, a in artifacts.items()
+                if a.get("mode", DEFAULT_ARTIFACT_MODE) != ArtifactModes.output
+            }
+        execution_dict.pop("queue", None)
+
+        new_project_data = None
+        if not project and new_project_name:
+            # Use a project with the provided name, or create a new project
+            project = ProjectBLL.find_or_create(
+                project_name=new_project_name,
+                user=user_id,
+                company=company_id,
+                description="",
+            )
+            new_project_data = {"id": project, "name": new_project_name}
+
+        def clean_system_tags(input_tags: Sequence[str]) -> Sequence[str]:
+            if not input_tags:
+                return input_tags
+
+            return [
+                tag
+                for tag in input_tags
+                if tag
+                not in [TaskSystemTags.development, EntityVisibility.archived.value]
+            ]
+
+        with TimingContext("mongo", "clone task"):
+            parent_task = (
+                task.parent
+                if task.parent and not task.parent.startswith(deleted_prefix)
+                else None
+            )
+            new_task = Task(
+                id=create_id(),
+                user=user_id,
+                company=company_id,
+                created=now,
+                last_update=now,
+                last_change=now,
+                name=name or task.name,
+                comment=comment or task.comment,
+                parent=parent or parent_task,
+                project=project or task.project,
+                tags=tags or task.tags,
+                system_tags=system_tags or clean_system_tags(task.system_tags),
+                type=task.type,
+                script=task.script,
+                output=Output(destination=task.output.destination)
+                if task.output
+                else None,
+                models=Models(input=input_models or task.models.input),
+                container=escape_dict(container) or task.container,
+                execution=execution_dict,
+                configuration=params_dict.get("configuration") or task.configuration,
+                hyperparams=params_dict.get("hyperparams") or task.hyperparams,
+            )
+            cls.validate(
+                new_task,
+                validate_models=validate_references or input_models,
+                validate_parent=validate_references or parent,
+                validate_project=validate_references or project,
+            )
+            new_task.save()
+
+            if task.project == new_task.project:
+                updated_tags = tags
+                updated_system_tags = system_tags
+            else:
+                updated_tags = new_task.tags
+                updated_system_tags = new_task.system_tags
+            org_bll.update_tags(
+                company_id,
+                Tags.Task,
+                project=new_task.project,
+                tags=updated_tags,
+                system_tags=updated_system_tags,
+            )
+            update_project_time(new_task.project)
+
+        return new_task, new_project_data
+
+    @classmethod
+    def validate(
+        cls,
+        task: Task,
+        validate_models=True,
+        validate_parent=True,
+        validate_project=True,
+    ):
+        """
+        Validate task properties according to the flag
+        Task project is always checked for being writable
+        in order to disable the modification of public projects
+        """
+        if (
+            validate_parent
+            and task.parent
+            and not task.parent.startswith(deleted_prefix)
+            and not Task.get(
+                company=task.company, id=task.parent, _only=("id",), include_public=True
+            )
+        ):
+            raise errors.bad_request.InvalidTaskId("invalid parent", parent=task.parent)
+
+        if task.project:
+            project = Project.get_for_writing(company=task.company, id=task.project)
+            if validate_project and not project:
+                raise errors.bad_request.InvalidProjectId(id=task.project)
+
+        if validate_models:
+            cls.validate_input_models(task)
+
+    @staticmethod
+    def set_last_update(
+        task_ids: Collection[str],
+        company_id: str,
+        last_update: datetime,
+        **extra_updates,
+    ):
+        tasks = Task.objects(id__in=task_ids, company=company_id).only(
+            "status", "started"
+        )
+        count = 0
+        for task in tasks:
+            updates = extra_updates
+            if task.status == TaskStatus.in_progress and task.started:
+                updates = {
+                    "active_duration": (
+                        datetime.utcnow() - task.started
+                    ).total_seconds(),
+                    **extra_updates,
+                }
+            count += Task.objects(id=task.id, company=company_id).update(
+                upsert=False,
+                last_update=last_update,
+                last_change=last_update,
+                **updates,
+            )
+        return count
+
+    @staticmethod
+    def update_statistics(
+        task_id: str,
+        company_id: str,
+        last_update: datetime = None,
+        last_iteration: int = None,
+        last_iteration_max: int = None,
+        last_scalar_values: Sequence[Tuple[Tuple[str, ...], Any]] = None,
+        last_events: Dict[str, Dict[str, dict]] = None,
+        **extra_updates,
+    ):
+        """
+        Update task statistics
+        :param task_id: Task's ID.
+        :param company_id: Task's company ID.
+        :param last_update: Last update time. If not provided, defaults to datetime.utcnow().
+        :param last_iteration: Last reported iteration. Use this to set a value regardless of current
+            task's last iteration value.
+        :param last_iteration_max: Last reported iteration. Use this to conditionally set a value only
+            if the current task's last iteration value is smaller than the provided value.
+        :param last_scalar_values: Last reported metrics summary for scalar events (value, metric, variant).
+        :param last_events: Last reported metrics summary (value, metric, event type).
+        :param extra_updates: Extra task updates to include in this update call.
+        :return:
+        """
+        last_update = last_update or datetime.utcnow()
+
+        if last_iteration is not None:
+            extra_updates.update(last_iteration=last_iteration)
+        elif last_iteration_max is not None:
+            extra_updates.update(max__last_iteration=last_iteration_max)
+
+        if last_scalar_values is not None:
+
+            def op_path(op, *path):
+                return "__".join((op, "last_metrics") + path)
+
+            for path, value in last_scalar_values:
+                if path[-1] == "min_value":
+                    extra_updates[op_path("min", *path[:-1], "min_value")] = value
+                elif path[-1] == "max_value":
+                    extra_updates[op_path("max", *path[:-1], "max_value")] = value
+                else:
+                    extra_updates[op_path("set", *path)] = value
+
+        if last_events is not None:
+
+            def events_per_type(metric_data: Dict[str, dict]) -> Dict[str, EventStats]:
+                return {
+                    event_type: EventStats(last_update=event["timestamp"])
+                    for event_type, event in metric_data.items()
+                }
+
+            metric_stats = {
+                dbutils.hash_field_name(metric_key): MetricEventStats(
+                    metric=metric_key, event_stats_by_type=events_per_type(metric_data)
+                )
+                for metric_key, metric_data in last_events.items()
+            }
+            extra_updates["metric_stats"] = metric_stats
+
+        return TaskBLL.set_last_update(
+            task_ids=[task_id],
+            company_id=company_id,
+            last_update=last_update,
+            **extra_updates,
+        )
+
+    @classmethod
+    def dequeue_and_change_status(
+        cls, task: Task, company_id: str, status_message: str, status_reason: str,
+    ):
+        cls.dequeue(task, company_id)
+
+        return ChangeStatusRequest(
+            task=task,
+            new_status=task.enqueue_status or TaskStatus.created,
+            status_reason=status_reason,
+            status_message=status_message,
+        ).execute(enqueue_status=None)
+
+    @classmethod
+    def dequeue(cls, task: Task, company_id: str, silent_fail=False):
+        """
+        Dequeue the task from the queue
+        :param task: task to dequeue
+        :param company_id: task's company ID.
+        :param silent_fail: do not throw exceptions. APIError is still thrown
+        :raise errors.bad_request.InvalidTaskId: if the task's status is not queued
+        :raise errors.bad_request.MissingRequiredFields: if the task is not queued
+        :raise APIError or errors.server_error.TransactionError: if internal call to queues.remove_task fails
+        :return: the result of queues.remove_task call. None in case of silent failure
+        """
+        if task.status not in (TaskStatus.queued,):
+            if silent_fail:
+                return
+            raise errors.bad_request.InvalidTaskId(
+                status=task.status, expected=TaskStatus.queued
+            )
+
+        if not task.execution or not task.execution.queue:
+            if silent_fail:
+                return
+            raise errors.bad_request.MissingRequiredFields(
+                "task has no queue value", field="execution.queue"
+            )
+
+        return {
+            "removed": queue_bll.remove_task(
+                company_id=company_id, queue_id=task.execution.queue, task_id=task.id
+            )
+        }

apiserver/bll/task/task_cleanup.py

@@ -0,0 +1,225 @@
+from itertools import chain
+from operator import attrgetter
+from typing import Sequence, Set, Tuple
+
+import attr
+from boltons.iterutils import partition
+
+from apiserver.apierrors import errors
+from apiserver.bll.event import EventBLL
+from apiserver.bll.event.event_bll import PlotFields
+from apiserver.bll.task.utils import deleted_prefix
+from apiserver.database.model.model import Model
+from apiserver.database.model.task.task import Task, TaskStatus, ArtifactModes
+from apiserver.timing_context import TimingContext
+
+event_bll = EventBLL()
+
+
+@attr.s(auto_attribs=True)
+class TaskUrls:
+    model_urls: Sequence[str]
+    event_urls: Sequence[str]
+    artifact_urls: Sequence[str]
+
+    def __add__(self, other: "TaskUrls"):
+        if not other:
+            return self
+
+        return TaskUrls(
+            model_urls=list(set(self.model_urls) | set(other.model_urls)),
+            event_urls=list(set(self.event_urls) | set(other.event_urls)),
+            artifact_urls=list(set(self.artifact_urls) | set(other.artifact_urls)),
+        )
+
+
+@attr.s(auto_attribs=True)
+class CleanupResult:
+    """
+    Counts of objects modified in task cleanup operation
+    """
+
+    updated_children: int
+    updated_models: int
+    deleted_models: int
+    urls: TaskUrls = None
+
+    def __add__(self, other: "CleanupResult"):
+        if not other:
+            return self
+
+        return CleanupResult(
+            updated_children=self.updated_children + other.updated_children,
+            updated_models=self.updated_models + other.updated_models,
+            deleted_models=self.deleted_models + other.deleted_models,
+            urls=self.urls + other.urls if self.urls else other.urls,
+        )
+
+
+def collect_plot_image_urls(company: str, task: str) -> Set[str]:
+    urls = set()
+    next_scroll_id = None
+    with TimingContext("es", "collect_plot_image_urls"):
+        while True:
+            events, next_scroll_id = event_bll.get_plot_image_urls(
+                company_id=company, task_id=task, scroll_id=next_scroll_id
+            )
+            if not events:
+                break
+            for event in events:
+                event_urls = event.get(PlotFields.source_urls)
+                if event_urls:
+                    urls.update(set(event_urls))
+
+    return urls
+
+
+def collect_debug_image_urls(company: str, task: str) -> Set[str]:
+    """
+    Return the set of unique image urls
+    Uses DebugImagesIterator to make sure that we do not retrieve recycled urls
+    """
+    after_key = None
+    urls = set()
+    while True:
+        res, after_key = event_bll.get_debug_image_urls(
+            company_id=company,
+            task_id=task,
+            after_key=after_key,
+        )
+        urls.update(res)
+        if not after_key:
+            break
+
+    return urls
+
+
+def cleanup_task(
+    task: Task,
+    force: bool = False,
+    update_children=True,
+    return_file_urls=False,
+    delete_output_models=True,
+) -> CleanupResult:
+    """
+    Validate task deletion and delete/modify all its output.
+    :param task: task object
+    :param force: whether to delete task with published outputs
+    :return: count of delete and modified items
+    """
+    published_models, draft_models, in_use_model_ids = verify_task_children_and_ouptuts(
+        task, force
+    )
+
+    event_urls, artifact_urls, model_urls = set(), set(), set()
+    if return_file_urls:
+        event_urls = collect_debug_image_urls(task.company, task.id)
+        event_urls.update(collect_plot_image_urls(task.company, task.id))
+        if task.execution and task.execution.artifacts:
+            artifact_urls = {
+                a.uri
+                for a in task.execution.artifacts.values()
+                if a.mode == ArtifactModes.output and a.uri
+            }
+        model_urls = {
+            m.uri for m in draft_models if m.uri and m.id not in in_use_model_ids
+        }
+
+    deleted_task_id = f"{deleted_prefix}{task.id}"
+    updated_children = 0
+    if update_children:
+        with TimingContext("mongo", "update_task_children"):
+            updated_children = Task.objects(parent=task.id).update(
+                parent=deleted_task_id
+            )
+
+    deleted_models = 0
+    updated_models = 0
+    for models, allow_delete in ((draft_models, True), (published_models, False)):
+        if not models:
+            continue
+        if delete_output_models and allow_delete:
+            deleted_models += Model.objects(
+                id__in=[m.id for m in models if m.id not in in_use_model_ids]
+            ).delete()
+            if in_use_model_ids:
+                Model.objects(id__in=list(in_use_model_ids)).update(unset__task=1)
+            continue
+
+        if update_children:
+            updated_models += Model.objects(id__in=[m.id for m in models]).update(
+                task=deleted_task_id
+            )
+        else:
+            Model.objects(id__in=[m.id for m in models]).update(unset__task=1)
+
+    event_bll.delete_task_events(task.company, task.id, allow_locked=force)
+
+    return CleanupResult(
+        deleted_models=deleted_models,
+        updated_children=updated_children,
+        updated_models=updated_models,
+        urls=TaskUrls(
+            event_urls=list(event_urls),
+            artifact_urls=list(artifact_urls),
+            model_urls=list(model_urls),
+        )
+        if return_file_urls
+        else None,
+    )
+
+
+def verify_task_children_and_ouptuts(
+    task, force: bool
+) -> Tuple[Sequence[Model], Sequence[Model], Set[str]]:
+    if not force:
+        with TimingContext("mongo", "count_published_children"):
+            published_children_count = Task.objects(
+                parent=task.id, status=TaskStatus.published
+            ).count()
+            if published_children_count:
+                raise errors.bad_request.TaskCannotBeDeleted(
+                    "has children, use force=True",
+                    task=task.id,
+                    children=published_children_count,
+                )
+
+    model_fields = ["id", "ready", "uri"]
+    published_models, draft_models = partition(
+        Model.objects(task=task.id).only(*model_fields), key=attrgetter("ready"),
+    )
+    if not force and published_models:
+        raise errors.bad_request.TaskCannotBeDeleted(
+            "has output models, use force=True",
+            task=task.id,
+            models=len(published_models),
+        )
+
+    if task.models and task.models.output:
+        model_ids = [m.model for m in task.models.output]
+        for output_model in Model.objects(id__in=model_ids).only(*model_fields):
+            if output_model.ready:
+                if not force:
+                    raise errors.bad_request.TaskCannotBeDeleted(
+                        "has output model, use force=True",
+                        task=task.id,
+                        model=output_model.id,
+                    )
+                published_models.append(output_model)
+            else:
+                draft_models.append(output_model)
+
+    in_use_model_ids = {}
+    if draft_models:
+        model_ids = {m.id for m in draft_models}
+        dependent_tasks = Task.objects(models__input__model__in=list(model_ids)).only(
+            "id", "models"
+        )
+        in_use_model_ids = model_ids & {
+            m.model
+            for m in chain.from_iterable(
+                t.models.input for t in dependent_tasks if t.models
+            )
+        }
+
+    return published_models, draft_models, in_use_model_ids

apiserver/bll/task/task_operations.py

@@ -0,0 +1,438 @@
+from datetime import datetime
+from typing import Callable, Any, Tuple, Union, Sequence
+
+from apiserver.apierrors import errors, APIError
+from apiserver.bll.queue import QueueBLL
+from apiserver.bll.task import (
+    TaskBLL,
+    validate_status_change,
+    ChangeStatusRequest,
+    update_project_time,
+)
+from apiserver.bll.task.task_cleanup import cleanup_task, CleanupResult
+from apiserver.config_repo import config
+from apiserver.database.model import EntityVisibility
+from apiserver.database.model.model import Model
+from apiserver.database.model.task.output import Output
+from apiserver.database.model.task.task import (
+    TaskStatus,
+    Task,
+    TaskSystemTags,
+    TaskStatusMessage,
+    ArtifactModes,
+    Execution,
+    DEFAULT_LAST_ITERATION,
+)
+from apiserver.utilities.dicts import nested_set
+
+log = config.logger(__file__)
+queue_bll = QueueBLL()
+
+
+def archive_task(
+    task: Union[str, Task], company_id: str, status_message: str, status_reason: str,
+) -> int:
+    """
+    Deque and archive task
+    Return 1 if successful
+    """
+    if isinstance(task, str):
+        task = TaskBLL.get_task_with_access(
+            task,
+            company_id=company_id,
+            only=(
+                "id",
+                "execution",
+                "status",
+                "project",
+                "system_tags",
+                "enqueue_status",
+            ),
+            requires_write_access=True,
+        )
+    try:
+        TaskBLL.dequeue_and_change_status(
+            task, company_id, status_message, status_reason,
+        )
+    except APIError:
+        # dequeue may fail if the task was not enqueued
+        pass
+
+    return task.update(
+        status_message=status_message,
+        status_reason=status_reason,
+        add_to_set__system_tags=EntityVisibility.archived.value,
+        last_change=datetime.utcnow(),
+    )
+
+
+def unarchive_task(
+    task: str, company_id: str, status_message: str, status_reason: str,
+) -> int:
+    """
+    Unarchive task. Return 1 if successful
+    """
+    task = TaskBLL.get_task_with_access(
+        task, company_id=company_id, only=("id",), requires_write_access=True,
+    )
+    return task.update(
+        status_message=status_message,
+        status_reason=status_reason,
+        pull__system_tags=EntityVisibility.archived.value,
+        last_change=datetime.utcnow(),
+    )
+
+
+def dequeue_task(
+    task_id: str, company_id: str, status_message: str, status_reason: str,
+) -> Tuple[int, dict]:
+    query = dict(id=task_id, company=company_id)
+    task = Task.get_for_writing(**query)
+    if not task:
+        raise errors.bad_request.InvalidTaskId(**query)
+
+    res = TaskBLL.dequeue_and_change_status(
+        task, company_id, status_message=status_message, status_reason=status_reason,
+    )
+    return 1, res
+
+
+def enqueue_task(
+    task_id: str,
+    company_id: str,
+    queue_id: str,
+    status_message: str,
+    status_reason: str,
+    queue_name: str = None,
+    validate: bool = False,
+    force: bool = False,
+) -> Tuple[int, dict]:
+    if queue_id and queue_name:
+        raise errors.bad_request.ValidationError(
+            "Either queue id or queue name should be provided"
+        )
+
+    if queue_name:
+        queue = queue_bll.get_by_name(
+            company_id=company_id, queue_name=queue_name, only=("id",)
+        )
+        if not queue:
+            queue = queue_bll.create(company_id=company_id, name=queue_name)
+        queue_id = queue.id
+
+    if not queue_id:
+        # try to get default queue
+        queue_id = queue_bll.get_default(company_id).id
+
+    query = dict(id=task_id, company=company_id)
+    task = Task.get_for_writing(**query)
+    if not task:
+        raise errors.bad_request.InvalidTaskId(**query)
+
+    if validate:
+        TaskBLL.validate(task)
+
+    res = ChangeStatusRequest(
+        task=task,
+        new_status=TaskStatus.queued,
+        status_reason=status_reason,
+        status_message=status_message,
+        allow_same_state_transition=False,
+        force=force,
+    ).execute(enqueue_status=task.status)
+
+    try:
+        queue_bll.add_task(company_id=company_id, queue_id=queue_id, task_id=task.id)
+    except Exception:
+        # failed enqueueing, revert to previous state
+        ChangeStatusRequest(
+            task=task,
+            current_status_override=TaskStatus.queued,
+            new_status=task.status,
+            force=True,
+            status_reason="failed enqueueing",
+        ).execute(enqueue_status=None)
+        raise
+
+    # set the current queue ID in the task
+    if task.execution:
+        Task.objects(**query).update(execution__queue=queue_id, multi=False)
+    else:
+        Task.objects(**query).update(execution=Execution(queue=queue_id), multi=False)
+
+    nested_set(res, ("fields", "execution.queue"), queue_id)
+    return 1, res
+
+
+def move_tasks_to_trash(tasks: Sequence[str]) -> int:
+    try:
+        collection_name = Task._get_collection_name()
+        trash_collection_name = f"{collection_name}__trash"
+        Task.aggregate(
+            [
+                {"$match": {"_id": {"$in": tasks}}},
+                {
+                    "$merge": {
+                        "into": trash_collection_name,
+                        "on": "_id",
+                        "whenMatched": "replace",
+                        "whenNotMatched": "insert",
+                    }
+                },
+            ],
+            allow_disk_use=True,
+        )
+    except Exception as ex:
+        log.error(f"Error copying tasks to trash {str(ex)}")
+
+    return Task.objects(id__in=tasks).delete()
+
+
+def delete_task(
+    task_id: str,
+    company_id: str,
+    move_to_trash: bool,
+    force: bool,
+    return_file_urls: bool,
+    delete_output_models: bool,
+    status_message: str,
+    status_reason: str,
+) -> Tuple[int, Task, CleanupResult]:
+    task = TaskBLL.get_task_with_access(
+        task_id, company_id=company_id, requires_write_access=True
+    )
+
+    if (
+        task.status != TaskStatus.created
+        and EntityVisibility.archived.value not in task.system_tags
+        and not force
+    ):
+        raise errors.bad_request.TaskCannotBeDeleted(
+            "due to status, use force=True",
+            task=task.id,
+            expected=TaskStatus.created,
+            current=task.status,
+        )
+
+    try:
+        TaskBLL.dequeue_and_change_status(
+            task,
+            company_id=company_id,
+            status_message=status_message,
+            status_reason=status_reason,
+        )
+    except APIError:
+        # dequeue may fail if the task was not enqueued
+        pass
+
+    cleanup_res = cleanup_task(
+        task,
+        force=force,
+        return_file_urls=return_file_urls,
+        delete_output_models=delete_output_models,
+    )
+
+    if move_to_trash:
+        # make sure that whatever changes were done to the task are saved
+        # the task itself will be deleted later in the move_tasks_to_trash operation
+        task.save()
+    else:
+        task.delete()
+
+    update_project_time(task.project)
+    return 1, task, cleanup_res
+
+
+def reset_task(
+    task_id: str,
+    company_id: str,
+    force: bool,
+    return_file_urls: bool,
+    delete_output_models: bool,
+    clear_all: bool,
+) -> Tuple[dict, CleanupResult, dict]:
+    task = TaskBLL.get_task_with_access(
+        task_id, company_id=company_id, requires_write_access=True
+    )
+
+    if not force and task.status == TaskStatus.published:
+        raise errors.bad_request.InvalidTaskStatus(task_id=task.id, status=task.status)
+
+    dequeued = {}
+    updates = {}
+
+    try:
+        dequeued = TaskBLL.dequeue(task, company_id, silent_fail=True)
+    except APIError:
+        # dequeue may fail if the task was not enqueued
+        pass
+
+    cleaned_up = cleanup_task(
+        task,
+        force=force,
+        update_children=False,
+        return_file_urls=return_file_urls,
+        delete_output_models=delete_output_models,
+    )
+
+    updates.update(
+        set__last_iteration=DEFAULT_LAST_ITERATION,
+        set__last_metrics={},
+        set__metric_stats={},
+        set__models__output=[],
+        set__runtime={},
+        unset__output__result=1,
+        unset__output__error=1,
+        unset__last_worker=1,
+        unset__last_worker_report=1,
+    )
+
+    if clear_all:
+        updates.update(
+            set__execution=Execution(), unset__script=1,
+        )
+    else:
+        updates.update(unset__execution__queue=1)
+        if task.execution and task.execution.artifacts:
+            updates.update(
+                set__execution__artifacts={
+                    key: artifact
+                    for key, artifact in task.execution.artifacts.items()
+                    if artifact.mode == ArtifactModes.input
+                }
+            )
+
+    res = ChangeStatusRequest(
+        task=task,
+        new_status=TaskStatus.created,
+        force=force,
+        status_reason="reset",
+        status_message="reset",
+    ).execute(
+        started=None,
+        completed=None,
+        published=None,
+        active_duration=None,
+        enqueue_status=None,
+        **updates,
+    )
+
+    return dequeued, cleaned_up, res
+
+
+def publish_task(
+    task_id: str,
+    company_id: str,
+    force: bool,
+    publish_model_func: Callable[[str, str], Any] = None,
+    status_message: str = "",
+    status_reason: str = "",
+) -> dict:
+    task = TaskBLL.get_task_with_access(
+        task_id, company_id=company_id, requires_write_access=True
+    )
+    if not force:
+        validate_status_change(task.status, TaskStatus.published)
+
+    previous_task_status = task.status
+    output = task.output or Output()
+    publish_failed = False
+
+    try:
+        # set state to publishing
+        task.status = TaskStatus.publishing
+        task.save()
+
+        # publish task models
+        if task.models and task.models.output and publish_model_func:
+            model_id = task.models.output[-1].model
+            model = (
+                Model.objects(id=model_id, company=company_id)
+                .only("id", "ready")
+                .first()
+            )
+            if model and not model.ready:
+                publish_model_func(model.id, company_id)
+
+        # set task status to published, and update (or set) it's new output (view and models)
+        return ChangeStatusRequest(
+            task=task,
+            new_status=TaskStatus.published,
+            force=force,
+            status_reason=status_reason,
+            status_message=status_message,
+        ).execute(published=datetime.utcnow(), output=output)
+
+    except Exception as ex:
+        publish_failed = True
+        raise ex
+    finally:
+        if publish_failed:
+            task.status = previous_task_status
+            task.save()
+
+
+def stop_task(
+    task_id: str, company_id: str, user_name: str, status_reason: str, force: bool,
+) -> dict:
+    """
+    Stop a running task. Requires task status 'in_progress' and
+    execution_progress 'running', or force=True. Development task or
+    task that has no associated worker is stopped immediately.
+    For a non-development task with worker only the status message
+    is set to 'stopping' to allow the worker to stop the task and report by itself
+    :return: updated task fields
+    """
+
+    task = TaskBLL.get_task_with_access(
+        task_id,
+        company_id=company_id,
+        only=(
+            "status",
+            "project",
+            "tags",
+            "system_tags",
+            "last_worker",
+            "last_update",
+            "execution.queue",
+        ),
+        requires_write_access=True,
+    )
+
+    def is_run_by_worker(t: Task) -> bool:
+        """Checks if there is an active worker running the task"""
+        update_timeout = config.get("apiserver.workers.task_update_timeout", 600)
+        return (
+            t.last_worker
+            and t.last_update
+            and (datetime.utcnow() - t.last_update).total_seconds() < update_timeout
+        )
+
+    is_queued = task.status == TaskStatus.queued
+    set_stopped = (
+        is_queued
+        or TaskSystemTags.development in task.system_tags
+        or not is_run_by_worker(task)
+    )
+
+    if set_stopped:
+        if is_queued:
+            try:
+                TaskBLL.dequeue(task, company_id=company_id, silent_fail=True)
+            except APIError:
+                # dequeue may fail if the task was not enqueued
+                pass
+
+        new_status = TaskStatus.stopped
+        status_message = f"Stopped by {user_name}"
+    else:
+        new_status = task.status
+        status_message = TaskStatusMessage.stopping
+
+    return ChangeStatusRequest(
+        task=task,
+        new_status=new_status,
+        status_reason=status_reason,
+        status_message=status_message,
+        force=force,
+    ).execute()

apiserver/bll/task/utils.py

@@ -1,18 +1,19 @@
 from datetime import datetime
-from typing import TypeVar, Callable, Tuple, Sequence
+from typing import Sequence, Union
 
 import attr
 import six
 
-from apierrors import errors
-from database.errors import translate_errors_context
-from database.model.project import Project
-from database.model.task.task import Task, TaskStatus, TaskSystemTags
-from database.utils import get_options
-from timing_context import TimingContext
-from utilities.attrs import typed_attrs
+from apiserver.apierrors import errors
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.project import Project
+from apiserver.database.model.task.task import Task, TaskStatus, TaskSystemTags
+from apiserver.database.utils import get_options
+from apiserver.timing_context import TimingContext
+from apiserver.utilities.attrs import typed_attrs
 
 valid_statuses = get_options(TaskStatus)
+deleted_prefix = "__DELETED__"
 
 
 @typed_attrs
@@ -43,6 +44,7 @@ class ChangeStatusRequest(object):
             status_message=self.status_message,
             status_changed=now,
             last_update=now,
+            last_change=now,
         )
 
         if self.new_status == TaskStatus.queued:
@@ -104,7 +106,7 @@ def validate_status_change(current_status, new_status):
 
 state_machine = {
     TaskStatus.created: {TaskStatus.queued, TaskStatus.in_progress},
-    TaskStatus.queued: {TaskStatus.created, TaskStatus.in_progress},
+    TaskStatus.queued: {TaskStatus.created, TaskStatus.in_progress, TaskStatus.stopped},
     TaskStatus.in_progress: {
         TaskStatus.stopped,
         TaskStatus.failed,
@@ -115,6 +117,7 @@ state_machine = {
         TaskStatus.closed,
         TaskStatus.created,
         TaskStatus.failed,
+        TaskStatus.queued,
         TaskStatus.in_progress,
         TaskStatus.published,
         TaskStatus.publishing,
@@ -152,22 +155,42 @@ def get_possible_status_changes(current_status):
     return possible
 
 
-def update_project_time(project_id):
-    if project_id:
-        Project.objects(id=project_id).update(last_update=datetime.utcnow())
+def update_project_time(project_ids: Union[str, Sequence[str]]):
+    if not project_ids:
+        return
+
+    if isinstance(project_ids, str):
+        project_ids = [project_ids]
+
+    return Project.objects(id__in=project_ids).update(last_update=datetime.utcnow())
 
 
-T = TypeVar("T")
-
-
-def split_by(
-    condition: Callable[[T], bool], items: Sequence[T]
-) -> Tuple[Sequence[T], Sequence[T]]:
+def get_task_for_update(
+    company_id: str, task_id: str, allow_all_statuses: bool = False, force: bool = False
+) -> Task:
     """
-    split "items" to two lists by "condition"
+    Loads only task id and return the task only if it is updatable (status == 'created')
     """
-    applied = zip(map(condition, items), items)
-    return (
-        [item for cond, item in applied if cond],
-        [item for cond, item in applied if not cond],
+    task = Task.get_for_writing(company=company_id, id=task_id, _only=("id", "status"))
+    if not task:
+        raise errors.bad_request.InvalidTaskId(id=task_id)
+
+    if allow_all_statuses:
+        return task
+
+    allowed_statuses = (
+        [TaskStatus.created, TaskStatus.in_progress] if force else [TaskStatus.created]
     )
+    if task.status not in allowed_statuses:
+        raise errors.bad_request.InvalidTaskStatus(
+            expected=TaskStatus.created, status=task.status
+        )
+    return task
+
+
+def update_task(task: Task, update_cmds: dict, set_last_update: bool = True):
+    now = datetime.utcnow()
+    last_updates = dict(last_change=now)
+    if set_last_update:
+        last_updates.update(last_update=now)
+    return task.update(**update_cmds, **last_updates)

apiserver/bll/user/__init__.py

@@ -1,7 +1,7 @@
-from apierrors import errors
-from apimodels.users import CreateRequest
-from database.errors import translate_errors_context
-from database.model.user import User
+from apiserver.apierrors import errors
+from apiserver.apimodels.users import CreateRequest
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.user import User
 
 
 class UserBLL:

apiserver/bll/util.py

@@ -0,0 +1,134 @@
+import functools
+import itertools
+from concurrent.futures.thread import ThreadPoolExecutor
+from typing import (
+    Optional,
+    Callable,
+    Dict,
+    Any,
+    Set,
+    Iterable,
+    Tuple,
+    Sequence,
+    TypeVar,
+)
+
+from boltons import iterutils
+
+from apiserver.apierrors import APIError
+from apiserver.database.model import AttributedDocument
+from apiserver.database.model.settings import Settings
+
+
+class SetFieldsResolver:
+    """
+    The class receives set fields dictionary
+    and for the set fields that require 'min' or 'max'
+    operation replace them with a simple set in case the
+    DB document does not have these fields set
+    """
+
+    SET_MODIFIERS = ("min", "max")
+
+    def __init__(self, set_fields: Dict[str, Any]):
+        self.orig_fields = {}
+        self.fields = {}
+        self.add_fields(**set_fields)
+
+    def add_fields(self, **set_fields: Any):
+        self.orig_fields.update(set_fields)
+        self.fields.update(
+            {
+                f: fname
+                for f, modifier, dunder, fname in (
+                    (f,) + f.partition("__") for f in set_fields.keys()
+                )
+                if dunder and modifier in self.SET_MODIFIERS
+            }
+        )
+
+    def _get_updated_name(self, doc: AttributedDocument, name: str) -> str:
+        if name in self.fields and doc.get_field_value(self.fields[name]) is None:
+            return self.fields[name]
+        return name
+
+    def get_fields(self, doc: AttributedDocument):
+        """
+        For the given document return the set fields instructions
+        with min/max operations replaced with a single set in case
+        the document does not have the field set
+        """
+        return {
+            self._get_updated_name(doc, name): value
+            for name, value in self.orig_fields.items()
+        }
+
+    def get_names(self) -> Set[str]:
+        """
+        Returns the names of the fields that had min/max modifiers
+        in the format suitable for projection (dot separated)
+        """
+        return set(name.replace("__", ".") for name in self.fields.values())
+
+
+@functools.lru_cache()
+def get_server_uuid() -> Optional[str]:
+    return Settings.get_by_key("server.uuid")
+
+
+def parallel_chunked_decorator(func: Callable = None, chunk_size: int = 100):
+    """
+    Decorates a method for parallel chunked execution. The method should have
+    one positional parameter (that is used for breaking into chunks)
+    and arbitrary number of keyword params. The return value should be iterable
+    The results are concatenated in the same order as the passed params
+    """
+    if func is None:
+        return functools.partial(parallel_chunked_decorator, chunk_size=chunk_size)
+
+    @functools.wraps(func)
+    def wrapper(self, iterable: Iterable, **kwargs):
+        assert iterutils.is_collection(
+            iterable
+        ), "The positional parameter should be an iterable for breaking into chunks"
+
+        func_with_params = functools.partial(func, self, **kwargs)
+        with ThreadPoolExecutor() as pool:
+            return list(
+                itertools.chain.from_iterable(
+                    filter(
+                        None,
+                        pool.map(
+                            func_with_params,
+                            iterutils.chunked_iter(iterable, chunk_size),
+                        ),
+                    )
+                ),
+            )
+
+    return wrapper
+
+
+T = TypeVar("T")
+
+
+def run_batch_operation(
+    func: Callable[[str], T], ids: Sequence[str]
+) -> Tuple[Sequence[Tuple[str, T]], Sequence[dict]]:
+    results = list()
+    failures = list()
+    for _id in ids:
+        try:
+            results.append((_id, func(_id)))
+        except APIError as err:
+            failures.append(
+                {
+                    "id": _id,
+                    "error": {
+                        "codes": [err.code, err.subcode],
+                        "msg": err.msg,
+                        "data": err.error_data,
+                    },
+                }
+            )
+    return results, failures

apiserver/bll/workers/__init__.py

@@ -5,10 +5,10 @@ from typing import Sequence, Set, Optional
 import attr
 import elasticsearch.helpers
 
-import es_factory
-from apierrors import APIError
-from apierrors.errors import bad_request, server_error
-from apimodels.workers import (
+from apiserver.es_factory import es_factory
+from apiserver.apierrors import APIError
+from apiserver.apierrors.errors import bad_request, server_error
+from apiserver.apimodels.workers import (
     DEFAULT_TIMEOUT,
     IdNameEntry,
     WorkerEntry,
@@ -17,16 +17,16 @@ from apimodels.workers import (
     QueueEntry,
     MachineStats,
 )
-from config import config
-from database.errors import translate_errors_context
-from database.model.auth import User
-from database.model.company import Company
-from database.model.project import Project
-from database.model.queue import Queue
-from database.model.task.task import Task
-from redis_manager import redman
-from timing_context import TimingContext
-from tools import safe_get
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.database.model.auth import User
+from apiserver.database.model.company import Company
+from apiserver.database.model.project import Project
+from apiserver.database.model.queue import Queue
+from apiserver.database.model.task.task import Task
+from apiserver.redis_manager import redman
+from apiserver.timing_context import TimingContext
+from apiserver.tools import safe_get
 from .stats import WorkerStats
 
 log = config.logger(__file__)
@@ -76,7 +76,7 @@ class WorkerBLL:
                 raise bad_request.InvalidUserId(**query)
             company = Company.objects(id=company_id).only("id", "name").first()
             if not company:
-                raise server_error.InternalError("invalid company", company=company_id)
+                raise bad_request.InvalidId("invalid company", company=company_id)
 
             queue_objs = Queue.objects(company=company_id, id__in=queues).only("id")
             if len(queue_objs) < len(queues):
@@ -113,7 +113,7 @@ class WorkerBLL:
             res = self.redis.delete(
                 company_id, self._get_worker_key(company_id, user_id, worker)
             )
-        if not res:
+        if not res and not config.get("apiserver.workers.auto_unregister", False):
             raise bad_request.WorkerNotRegistered(worker=worker)
 
     def status_report(
@@ -143,7 +143,7 @@ class WorkerBLL:
                 self._log_stats_to_es(
                     company_id=company_id,
                     company_name=entry.company.name,
-                    worker=report.worker,
+                    worker=entry.key,
                     timestamp=report.timestamp,
                     task=report.task,
                     machine_stats=report.machine_stats,
@@ -164,6 +164,7 @@ class WorkerBLL:
                         last_worker=report.worker,
                         last_worker_report=now,
                         last_update=now,
+                        last_change=now,
                     )
                     # modify(new=True, ...) returns the modified object
                     task = Task.objects(**query).modify(new=True, **update)
@@ -188,7 +189,10 @@ class WorkerBLL:
             self._save_worker(entry)
 
     def get_all(
-        self, company_id: str, last_seen: Optional[int] = None
+        self,
+        company_id: str,
+        last_seen: Optional[int] = None,
+        tags: Sequence[str] = None,
     ) -> Sequence[WorkerEntry]:
         """
         Get all the company workers that were active during the last_seen period
@@ -209,16 +213,26 @@ class WorkerBLL:
                 if w.last_activity_time.replace(tzinfo=None) >= ref_time
             ]
 
+        if tags:
+            include = {t for t in tags if not t.startswith("-")}
+            exclude = {t[1:] for t in tags if t.startswith("-")}
+            workers = [
+                w
+                for w in workers
+                if (not include or any(t in include for t in w.tags))
+                and (not exclude or all(t not in exclude for t in w.tags))
+            ]
+
         return workers
 
     def get_all_with_projection(
-        self, company_id: str, last_seen: int
+        self, company_id: str, last_seen: int, tags: Sequence[str] = None
     ) -> Sequence[WorkerResponseEntry]:
 
         helpers = list(
             map(
                 WorkerConversionHelper.from_worker_entry,
-                self.get_all(company_id=company_id, last_seen=last_seen),
+                self.get_all(company_id=company_id, last_seen=last_seen, tags=tags),
             )
         )
 
@@ -257,7 +271,7 @@ class WorkerBLL:
             tasks_info = {
                 task.id: task
                 for task in Task.objects(id__in=task_ids).only(
-                    "name", "started", "last_iteration"
+                    "name", "started", "last_iteration", "active_duration"
                 )
             }
 
@@ -282,11 +296,7 @@ class WorkerBLL:
             if helper.task_id:
                 task = tasks_info.get(helper.task_id, None)
                 if task:
-                    worker.task.running_time = (
-                        int((datetime.utcnow() - task.started).total_seconds() * 1000)
-                        if task.started
-                        else 0
-                    )
+                    worker.task.running_time = (task.active_duration or 0) * 1000
                     worker.task.last_iteration = task.last_iteration
 
             update_queue_entries(worker.queue)
@@ -355,10 +365,15 @@ class WorkerBLL:
         self, company: str, user: str = "*", worker_id: str = "*"
     ) -> Sequence[WorkerEntry]:
         """Get worker entries matching the company and user, worker patterns"""
+        entries = []
         match = self._get_worker_key(company, user, worker_id)
         with TimingContext("redis", "workers_get_all"):
-            res = self.redis.scan_iter(match)
-        return [WorkerEntry.from_json(self.redis.get(r)) for r in res]
+            for r in self.redis.scan_iter(match):
+                data = self.redis.get(r)
+                if data:
+                    entries.append(WorkerEntry.from_json(data))
+
+        return entries
 
     @staticmethod
     def _get_es_index_suffix():

apiserver/bll/workers/stats.py

@@ -3,12 +3,12 @@ from typing import Optional, Sequence
 
 from boltons.iterutils import bucketize
 
-from apierrors.errors import bad_request
-from apimodels.workers import AggregationType, GetStatsRequest, StatItem
-from bll.query import Builder as QueryBuilder
-from config import config
-from database.errors import translate_errors_context
-from timing_context import TimingContext
+from apiserver.apierrors.errors import bad_request
+from apiserver.apimodels.workers import AggregationType, GetStatsRequest, StatItem
+from apiserver.bll.query import Builder as QueryBuilder
+from apiserver.config_repo import config
+from apiserver.database.errors import translate_errors_context
+from apiserver.timing_context import TimingContext
 
 log = config.logger(__file__)
 
@@ -20,7 +20,7 @@ class WorkerStats:
     @staticmethod
     def worker_stats_prefix_for_company(company_id: str) -> str:
         """Returns the es index prefix for the company"""
-        return f"worker_stats_{company_id}_"
+        return f"worker_stats_{company_id.lower()}_"
 
     def _search_company_stats(self, company_id: str, es_req: dict) -> dict:
         return self.es.search(

apiserver/config/__init__.py

@@ -0,0 +1 @@
+from .basic import BasicConfig, ConfigurationError

apiserver/config/basic.py

@@ -0,0 +1,217 @@
+import logging
+import logging.config
+import os
+import platform
+from functools import reduce
+from os import getenv
+from os.path import expandvars
+from pathlib import Path
+from typing import List, Any, TypeVar, Sequence
+
+from boltons.iterutils import first
+from pyhocon import ConfigTree, ConfigFactory, ConfigValues
+from pyparsing import (
+    ParseFatalException,
+    ParseException,
+    RecursiveGrammarException,
+    ParseSyntaxException,
+)
+
+from apiserver.utilities import json
+
+EXTRA_CONFIG_PATHS = ("/opt/trains/config", "/opt/clearml/config")
+DEFAULT_PREFIXES = ("clearml", "trains")
+EXTRA_CONFIG_PATH_SEP = ":" if platform.system() != "Windows" else ";"
+
+
+class BasicConfig:
+    NotSet = object()
+
+    extra_config_values_env_key_sep = "__"
+    default_config_dir = "default"
+
+    def __init__(
+        self,
+        folder: str = None,
+        verbose: bool = True,
+        prefix: Sequence[str] = DEFAULT_PREFIXES,
+    ):
+        folder = (
+            Path(folder)
+            if folder
+            else Path(__file__).with_name(self.default_config_dir)
+        )
+        if not folder.is_dir():
+            raise ValueError("Invalid configuration folder")
+
+        self.verbose = verbose
+
+        self.extra_config_path_override_var = [
+            f"{p.upper()}_CONFIG_DIR" for p in prefix
+        ]
+
+        self.prefix = prefix[0]
+        self.extra_config_values_env_key_prefix = [
+            f"{p.upper()}{self.extra_config_values_env_key_sep}"
+            for p in reversed(prefix)
+        ]
+
+        self._paths = [folder, *self._get_paths()]
+        self._config = self._reload()
+
+    def __getitem__(self, key):
+        return self._config[key]
+
+    def get(self, key: str, default: Any = NotSet) -> Any:
+        value = self._config.get(key, default)
+        if value is self.NotSet:
+            raise KeyError(
+                f"Unable to find value for key '{key}' and default value was not provided."
+            )
+        return value
+
+    def to_dict(self) -> dict:
+        return self._config.as_plain_ordered_dict()
+
+    def as_json(self) -> str:
+        return json.dumps(self.to_dict(), indent=2)
+
+    def logger(self, name: str) -> logging.Logger:
+        if Path(name).is_file():
+            name = Path(name).stem
+            if name == "__init__" and Path(name).parent.stem:
+                name = Path(name).parent.stem
+        path = ".".join((self.prefix, name))
+        return logging.getLogger(path)
+
+    def _read_extra_env_config_values(self) -> ConfigTree:
+        """ Loads extra configuration from environment-injected values """
+        result = ConfigTree()
+
+        for prefix in self.extra_config_values_env_key_prefix:
+            keys = sorted(k for k in os.environ if k.startswith(prefix))
+            for key in keys:
+                path = (
+                    key[len(prefix) :]
+                    .replace(self.extra_config_values_env_key_sep, ".")
+                    .lower()
+                )
+                result = self._merge_configs(
+                    result, ConfigFactory.parse_string(f"{path}: {os.environ[key]}")
+                )
+
+        return result
+
+    def _get_paths(self) -> List[Path]:
+        default_paths = EXTRA_CONFIG_PATH_SEP.join(EXTRA_CONFIG_PATHS)
+        value = first(map(getenv, self.extra_config_path_override_var), default_paths)
+
+        paths = [
+            Path(expandvars(v)).expanduser() for v in value.split(EXTRA_CONFIG_PATH_SEP)
+        ]
+
+        if value is not default_paths:
+            invalid = [path for path in paths if not path.is_dir()]
+            if invalid:
+                print(
+                    f"WARNING: Invalid paths in {self.extra_config_path_override_var} env var: {' '.join(map(str, invalid))}"
+                )
+
+        return [path for path in paths if path.is_dir()]
+
+    def reload(self):
+        self._config = self._reload()
+
+    def _reload(self) -> ConfigTree:
+        extra_config_values = self._read_extra_env_config_values()
+
+        configs = [self._read_recursive(path) for path in self._paths]
+
+        return reduce(
+            lambda last, config: self._merge_configs(
+                last, config, copy_trees=True
+            ),
+            configs + [extra_config_values],
+            ConfigTree(),
+        )
+
+    @classmethod
+    def _merge_configs(cls, a, b, copy_trees=False, override_prefix="-"):
+        """Based on pyhocon.ConfigTree.merge_configs, with dict override support using a `-` key prefix"""
+        for key, value in b.items():
+            override = key.startswith(override_prefix)
+            if override:
+                key = key[len(override_prefix):]
+            # if key is in both a and b and both values are dictionary then merge it otherwise override it
+            if not override and key in a and isinstance(a[key], ConfigTree) and isinstance(b[key], ConfigTree):
+                if copy_trees:
+                    a[key] = a[key].copy()
+                cls._merge_configs(a[key], b[key], copy_trees=copy_trees)
+            else:
+                if isinstance(value, ConfigValues):
+                    value.parent = a
+                    value.key = key
+                    if key in a:
+                        value.overriden_value = a[key]
+                a[key] = value
+                if a.root:
+                    if b.root:
+                        a.history[key] = a.history.get(key, []) + b.history.get(key, [value])
+                    else:
+                        a.history[key] = a.history.get(key, []) + [value]
+
+        return a
+
+    def _read_recursive(self, conf_root) -> ConfigTree:
+        conf = ConfigTree()
+
+        if not conf_root:
+            return conf
+
+        if not conf_root.is_dir():
+            if self.verbose:
+                if not conf_root.exists():
+                    print(f"No config in {conf_root}")
+                else:
+                    print(f"Not a directory: {conf_root}")
+            return conf
+
+        if self.verbose:
+            print(f"Loading config from {conf_root}")
+
+        for file in conf_root.rglob("*.conf"):
+            key = ".".join(file.relative_to(conf_root).with_suffix("").parts)
+            conf.put(key, self._read_single_file(file))
+
+        return conf
+
+    def _read_single_file(self, file_path):
+        if self.verbose:
+            print(f"Loading config from file {file_path}")
+
+        try:
+            return ConfigFactory.parse_file(file_path)
+        except ParseSyntaxException as ex:
+            msg = f"Failed parsing {file_path} ({ex.__class__.__name__}): (at char {ex.loc}, line:{ex.lineno}, col:{ex.column})"
+            raise ConfigurationError(msg, file_path=file_path) from ex
+        except (ParseException, ParseFatalException, RecursiveGrammarException) as ex:
+            msg = f"Failed parsing {file_path} ({ex.__class__.__name__}): {ex}"
+            raise ConfigurationError(msg) from ex
+        except Exception as ex:
+            print(f"Failed loading {file_path}: {ex}")
+            raise
+
+    def initialize_logging(self):
+        logging_config = self.get("logging", None)
+        if not logging_config:
+            return
+        logging.config.dictConfig(logging_config)
+
+
+class ConfigurationError(Exception):
+    def __init__(self, msg, file_path=None, *args):
+        super().__init__(msg, *args)
+        self.file_path = file_path
+
+
+ConfigType = TypeVar("ConfigType", bound=BasicConfig)

apiserver/config/default/apiserver.conf

@@ -3,7 +3,7 @@
     debug: false            # Debug mode
     pretty_json: false      # prettify json response
     return_stack: true      # return stack trace on error
-    log_calls: true         # Log API Calls
+    return_stack_to_caller: true # top-level control on whether to return stack trace in an API response
 
     # if 'return_stack' is true and error contains a status code, return stack trace only for these status codes
     # valid values are:
@@ -69,7 +69,7 @@
         default_expiration_sec: 2592000
 
         # cookie containing auth token, for requests arriving from a web-browser
-        session_auth_cookie_name: "trains_token_basic"
+        session_auth_cookie_name: "clearml_token_basic"
 
         # cookie configuration for authorization cookies generated by auth.login
         cookies {
@@ -79,9 +79,16 @@
             max_age: 99999999999
         }
 
+        # provide a cookie domain override per company
+#        cookies_domain_override {
+#            <company-id>: <domain>
+#        }
+
 #        # A list of fixed users
+#        # Note: password may be bcrypt-hashed (generate using `python -c 'import bcrypt; print(bcrypt.hashpw("password", bcrypt.gensalt()))'`)
 #        fixed_users {
 #            enabled: true
+#            pass_hashed: false
 #            users: [
 #                {
 #                    username: "john"
@@ -105,6 +112,8 @@
     workers {
         # Auto-register unknown workers on status reports and other calls
         auto_register: true
+        # Assume unknow workers have unregistered (i.e. do not raise unregistered error)
+        auto_unregister: true
         # Timeout in seconds on task status update. If exceeded
         # then task can be stopped without communicating to the worker
         task_update_timeout: 600
@@ -116,9 +125,9 @@
         # Check for updates every 24 hours
         check_interval_sec: 86400
 
-        url: "https://updates.trains.allegro.ai/updates"
+        url: "https://updates.clear.ml/updates"
 
-        component_name: "trains-server"
+        component_name: "clearml-server"
 
         # GET request timeout
         request_timeout_sec: 3.0
@@ -128,7 +137,7 @@
         # Note: statistics are sent ONLY if the user has actively opted-in
         supported: true
 
-        url: "https://updates.trains.allegro.ai/stats"
+        url: "https://updates.clear.ml/stats"
 
         report_interval_hours: 24
         agent_relevant_threshold_days: 30
@@ -137,4 +146,11 @@
         max_backoff_sec: 5
     }
 
+    getting_started_info {
+        "agentName": "clearml",
+        "configure": "clearml-init",
+        "install": "pip install clearml",
+        "packageName": "clearml"
+    }
+
 }

apiserver/config/default/hosts.conf

@@ -1,6 +1,6 @@
 elastic {
   events {
-    hosts: [{host: "127.0.0.1", port: 9211}]
+    hosts: [{host: "127.0.0.1", port: 9200}]
     args {
       timeout: 60
       dead_timeout: 10
@@ -11,7 +11,7 @@ elastic {
   }
 
   workers {
-    hosts: [{host:"127.0.0.1", port:9211}]
+    hosts: [{host:"127.0.0.1", port:9200}]
     args {
       timeout: 60
       dead_timeout: 10

apiserver/config/default/logging.conf

@@ -16,7 +16,7 @@
             backupCount: 3
             maxBytes: 10240000,
             class: "logging.handlers.RotatingFileHandler",
-            filename: "/var/log/trains/apiserver.log"
+            filename: "/var/log/clearml/apiserver.log"
         }
     }
     root {

apiserver/config/default/secure.conf

@@ -28,6 +28,7 @@
             display_name: "Default User"
             user_key: "EGRTCO8JMSIGI6S39GTP43NFWXDQOW"
             user_secret: "x!XTov_G-#vspE*Y(h$Anm&DIc5Ou-F)jsl$PdOyj5wG1&E!Z8"
+            revoke_in_fixed_mode: true
         }
     }
 }

apiserver/config/default/services/_mongo.conf

@@ -0,0 +1,4 @@
+max_page_size: 500
+
+# expiration time in seconds for the redis scroll states in get_many family of apis
+scroll_state_expiration_seconds: 600

apiserver/config/default/services/events.conf

@@ -0,0 +1,42 @@
+es_index_prefix: "events"
+
+ignore_iteration {
+    metrics: [":monitor:machine", ":monitor:gpu"]
+}
+
+
+events_retrieval {
+    state_expiration_sec: 3600
+
+    # max number of concurrent queries to ES when calculating events metrics
+    # should not exceed the amount of concurrent connections set in the ES driver
+    max_metrics_concurrency: 4
+
+    # If set then max_metrics_count and max_variants_count are calculated dynamically on user data
+    dynamic_metrics_count: true
+
+    # The percentage from the ES aggs limit (10000) to use for the max_metrics and max_variants calculation
+    dynamic_metrics_count_threshold: 80
+
+    # the max amount of metrics to aggregate on
+    max_metrics_count: 100
+
+    # the max amount of variants to aggregate on
+    max_variants_count: 100
+
+    debug_images {
+        # Allow to return the debug images for the variants with uninitialized valid iterations border
+        allow_uninitialized_variants: true
+    }
+
+    max_raw_scalars_size: 200000
+
+    scroll_id_key: "cTN5VEtWEC6QrHvUl0FTx9kNyO0CcCK1p57akxma"
+}
+
+# if set then plot str will be checked for the valid json on plot add
+# and the result of the check is written to the db
+validate_plot_str: false
+
+# If not 0 then the plots equal or greater to the size will be stored compressed in the DB
+plot_compression_threshold: 100000

apiserver/config/default/services/models.conf

@@ -0,0 +1,7 @@
+metadata_values {
+    # maximal amount of distinct model values to retrieve
+    max_count: 100
+
+    # cache ttl sec
+    cache_ttl_sec: 86400
+}

apiserver/config/default/services/projects.conf

@@ -0,0 +1,18 @@
+# Order of featured projects, by name or ID
+featured {
+  order: [
+    #  {id: "<project-id>"}
+    #  OR
+    #  {name: "<project-name>"}
+    #  OR
+    #  {name_regex: "<python-regex>"}
+  ]
+
+  # default featured index for public projects not specified in the order
+  public_default: 9999
+}
+
+sub_projects {
+    # the max sub project depth
+    max_depth: 10
+}

apiserver/config/default/services/queues.conf

@@ -0,0 +1,5 @@
+{
+    metrics_before_from_date: 3600
+    # interval in seconds to update queue metrics. Put 0 to disable
+    metrics_refresh_interval_sec: 300
+}

apiserver/config/default/services/tasks.conf

@@ -8,9 +8,15 @@ non_responsive_tasks_watchdog {
     watch_interval_sec: 900
 }
 
-artifacts {
-    update_attempts: 10
-    update_retry_msec: 500
-}
-
 multi_task_histogram_limit: 100
+
+hyperparam_values {
+    # maximal amount of distinct hyperparam values to retrieve
+    max_count: 100
+
+    # max allowed outdate time for the cashed result
+    cache_allowed_outdate_sec: 60
+
+    # cache ttl sec
+    cache_ttl_sec: 86400
+}

apiserver/config/info.py

@@ -1,15 +1,19 @@
 from functools import lru_cache
 from os import getenv
 from pathlib import Path
-from version import __version__
 
-from config import config
+from boltons.iterutils import first
+
+from apiserver.config_repo import config
+from apiserver.version import __version__
 
 root = Path(__file__).parent.parent
 
 
 def _get(prop_name, env_suffix=None, default=""):
-    value = getenv(f"TRAINS_SERVER_{env_suffix or prop_name}")
+    suffix = env_suffix or prop_name
+    keys = [f"{p}_SERVER_{suffix}" for p in ("CLEARML", "TRAINS")]
+    value = first(map(getenv, keys))
     if value:
         return value
 

apiserver/config_repo.py

@@ -0,0 +1,4 @@
+from apiserver.config import BasicConfig
+
+config = BasicConfig()
+config.initialize_logging()

apiserver/database/__init__.py

@@ -0,0 +1,117 @@
+from os import getenv
+
+from boltons.iterutils import first
+from furl import furl
+from jsonmodels import models
+from jsonmodels.errors import ValidationError
+from jsonmodels.fields import StringField
+from mongoengine import register_connection
+from mongoengine.connection import get_connection, disconnect
+
+from apiserver.config_repo import config
+from .defs import Database
+from .utils import get_items
+
+log = config.logger("database")
+
+strict = config.get("apiserver.mongo.strict", True)
+
+OVERRIDE_HOST_ENV_KEY = (
+    "CLEARML_MONGODB_SERVICE_HOST",
+    "TRAINS_MONGODB_SERVICE_HOST",
+    "MONGODB_SERVICE_HOST",
+    "MONGODB_SERVICE_SERVICE_HOST",
+)
+OVERRIDE_PORT_ENV_KEY = (
+    "CLEARML_MONGODB_SERVICE_PORT",
+    "TRAINS_MONGODB_SERVICE_PORT",
+    "MONGODB_SERVICE_PORT",
+)
+
+OVERRIDE_CONNECTION_STRING_ENV_KEY = "CLEARML_MONGODB_SERVICE_CONNECTION_STRING"
+
+
+class DatabaseEntry(models.Base):
+    host = StringField(required=True)
+    alias = StringField()
+
+
+class DatabaseFactory:
+    _entries = []
+
+    @classmethod
+    def _create_db_entry(cls, alias: str, settings: dict) -> DatabaseEntry:
+        return DatabaseEntry(alias=alias, **settings)
+
+    @classmethod
+    def initialize(cls):
+        db_entries = config.get("hosts.mongo", {})
+        missing = []
+        log.info("Initializing database connections")
+
+        override_connection_string = getenv(OVERRIDE_CONNECTION_STRING_ENV_KEY)
+        override_hostname = first(map(getenv, OVERRIDE_HOST_ENV_KEY), None)
+        override_port = first(map(getenv, OVERRIDE_PORT_ENV_KEY), None)
+
+        if override_connection_string:
+            log.info(f"Using override mongodb connection string {override_connection_string}")
+        else:
+            if override_hostname:
+                log.info(f"Using override mongodb host {override_hostname}")
+            if override_port:
+                log.info(f"Using override mongodb port {override_port}")
+
+        for key, alias in get_items(Database).items():
+            if key not in db_entries:
+                missing.append(key)
+                continue
+
+            entry = cls._create_db_entry(alias=alias, settings=db_entries.get(key))
+
+            if override_connection_string:
+                entry.host = override_connection_string
+            else:
+                if override_hostname:
+                    entry.host = furl(entry.host).set(host=override_hostname).url
+                if override_port:
+                    entry.host = furl(entry.host).set(port=override_port).url
+
+            try:
+                entry.validate()
+                log.info(
+                    "Registering connection to %(alias)s (%(host)s)" % entry.to_struct()
+                )
+                register_connection(**entry.to_struct())
+
+                cls._entries.append(entry)
+            except ValidationError as ex:
+                raise Exception("Invalid database entry `%s`: %s" % (key, ex.args[0]))
+        if missing:
+            raise ValueError(
+                "Missing database configuration for %s" % ", ".join(missing)
+            )
+
+    @classmethod
+    def get_entries(cls):
+        return cls._entries
+
+    @classmethod
+    def get_hosts(cls):
+        return [entry.host for entry in cls.get_entries()]
+
+    @classmethod
+    def get_aliases(cls):
+        return [entry.alias for entry in cls.get_entries()]
+
+    @classmethod
+    def reconnect(cls):
+        for entry in cls.get_entries():
+            # there is bug in the current implementation that prevents
+            # reconnection from work so workaround this
+            # get_connection(entry.alias, reconnect=True)
+            disconnect(entry.alias)
+            register_connection(**entry.to_struct())
+            get_connection(entry.alias)
+
+
+db = DatabaseFactory()

apiserver/database/errors.py

@@ -1,6 +1,7 @@
 import re
 from contextlib import contextmanager
 from functools import wraps
+from textwrap import shorten
 
 import dpath
 from dpath.exceptions import InvalidKeyName
@@ -17,7 +18,7 @@ from mongoengine.errors import (
 )
 from pymongo.errors import PyMongoError, NotMasterError
 
-from apierrors import errors
+from apiserver.apierrors import errors
 
 
 class MakeGetAllQueryError(Exception):
@@ -33,7 +34,7 @@ class ParseCallError(Exception):
         self.params = kwargs
 
 
-def throws_default_error(err_cls):
+def throws_default_error(err_cls, shorten_width: int = None):
     """
     Used to make functions (Exception, str) -> Optional[str] searching for specialized error messages raise those
     messages in ``err_cls``. If the decorated function does not find a suitable error message,
@@ -45,25 +46,49 @@ def throws_default_error(err_cls):
         @wraps(func)
         def wrapper(self, e, message, **kwargs):
             extra_info = func(self, e, message, **kwargs)
-            raise err_cls(message, err=e, extra_info=extra_info)
+            err = str(e)
+            if shorten_width:
+                err = shorten(err, shorten_width, placeholder="...")
+            raise err_cls(message, err=err, extra_info=extra_info)
 
         return wrapper
 
     return decorator
 
 
+# noinspection RegExpRedundantEscape
 class ElasticErrorsHandler(object):
     @classmethod
-    @throws_default_error(errors.server_error.DataError)
+    def _bulk_meta_error(cls, error):
+        try:
+            _, err_type = next(dpath.search(error, "*/error/type", yielded=True))
+            _, reason = next(dpath.search(error, "*/error/reason", yielded=True))
+            if err_type == "cluster_block_exception":
+                raise errors.server_error.LowDiskSpace(
+                    "metrics, logs and all indexed data is in read-only mode!",
+                    reason=re.sub(r"^index\s\[.*?\]\s", "", reason) if reason else ""
+                )
+            return
+        except StopIteration:
+            pass
+
+    @classmethod
+    @throws_default_error(errors.server_error.DataError, shorten_width=200)
     def bulk_error(cls, e, _, **__):
         if not e.errors:
             return
 
+        # Currently we only handle the first error
+        error = e.errors[0]
+
+        cls._bulk_meta_error(error)
+
         # Else try returning a better error string
         for _, reason in dpath.search(e.errors[0], "*/error/reason", yielded=True):
             return reason
 
 
+# noinspection RegExpRedundantEscape
 class MongoEngineErrorsHandler(object):
     # NotUniqueError
     __not_unique_regex = re.compile(
@@ -81,6 +106,7 @@ class MongoEngineErrorsHandler(object):
     def validation_error(cls, e: ValidationError, message, **_):
         # Thrown when a document is validated. Documents are validated by default on save and on update
         err_dict = e.errors or {e.field_name: e.message}
+        err_dict = {key: str(value) for key, value in err_dict.items()}
         raise errors.bad_request.DataValidationError(message, **err_dict)
 
     @classmethod
@@ -140,7 +166,10 @@ class MongoEngineErrorsHandler(object):
     @classmethod
     @throws_default_error(errors.server_error.InternalError)
     def invalid_query_error(cls, e, message, **_):
-        pass
+        if e.args:
+            inner = e.args[0]
+            if isinstance(inner, LookUpError):
+                cls.lookup_error(inner, message)
 
 
 @contextmanager

apiserver/database/fields.py

@@ -14,7 +14,7 @@ from mongoengine import (
     DictField,
     DynamicField,
 )
-from mongoengine.fields import key_not_string, key_starts_with_dollar
+from mongoengine.fields import key_not_string, key_starts_with_dollar, EmailField
 
 NoneType = type(None)
 
@@ -93,6 +93,24 @@ class CustomFloatField(FloatField):
             self.error("Float value must be greater than %s" % str(self.greater_than))
 
 
+class CanonicEmailField(EmailField):
+    """email field that is always lower cased"""
+    def __set__(self, instance, value: str):
+        if value is not None:
+            try:
+                value = value.lower()
+            except AttributeError:
+                pass
+        super().__set__(instance, value)
+
+    def prepare_query_value(self, op, value):
+        if not isinstance(op, six.string_types):
+            return value
+        if value is not None:
+            value = value.lower()
+        return super().prepare_query_value(op, value)
+
+
 class StrippedStringField(StringField):
     def __init__(
         self, regex=None, max_length=None, min_length=None, strip_chars=None, **kwargs
@@ -158,6 +176,13 @@ class SafeMapField(MapField, DictValidationMixin):
             self.error("Empty keys are not allowed in a MapField")
 
 
+class NullableStringField(StringField):
+    def validate(self, value):
+        if value is None:
+            return
+        super(NullableStringField, self).validate(value)
+
+
 class SafeDictField(DictField, DictValidationMixin):
     def validate(self, value):
         self._safe_validate(value)

apiserver/database/model/__init__.py

@@ -2,10 +2,10 @@ from enum import Enum
 
 from mongoengine import Document, StringField
 
-from apierrors import errors
-from database.model.base import DbModelMixin, ABSTRACT_FLAG
-from database.model.company import Company
-from database.model.user import User
+from apiserver.apierrors import errors
+from apiserver.database.model.base import DbModelMixin, ABSTRACT_FLAG
+from apiserver.database.model.company import Company
+from apiserver.database.model.user import User
 
 
 class AttributedDocument(DbModelMixin, Document):
@@ -60,3 +60,4 @@ def validate_id(cls, company, **kwargs):
 class EntityVisibility(Enum):
     active = "active"
     archived = "archived"
+    hidden = "hidden"

apiserver/database/model/auth.py

@@ -6,10 +6,10 @@ from mongoengine import (
     DateTimeField,
 )
 
-from database import Database, strict
-from database.model import DbModelMixin
-from database.model.base import AuthDocument
-from database.utils import get_options
+from apiserver.database import Database, strict
+from apiserver.database.model import DbModelMixin
+from apiserver.database.model.base import AuthDocument
+from apiserver.database.utils import get_options
 
 
 class Entities(object):
@@ -48,7 +48,9 @@ class Credentials(EmbeddedDocument):
     meta = {"strict": False}
     key = StringField(required=True)
     secret = StringField(required=True)
+    label = StringField()
     last_used = DateTimeField()
+    last_used_from = StringField()
 
 
 class User(DbModelMixin, AuthDocument):
@@ -72,5 +74,5 @@ class User(DbModelMixin, AuthDocument):
     credentials = EmbeddedDocumentListField(Credentials, default=list)
     """ Credentials generated for this user """
 
-    email = EmailField(unique=True, required=True)
+    email = EmailField(unique=True, sparse=True)
     """ Email uniquely identifying the user """

apiserver/database/model/base.py

@@ -1,26 +1,41 @@
 import re
 from collections import namedtuple
-from functools import reduce
-from typing import Collection, Sequence, Union, Optional, Type, Tuple
+from functools import reduce, partial
+from typing import (
+    Collection,
+    Sequence,
+    Union,
+    Optional,
+    Type,
+    Tuple,
+    Mapping,
+    Any,
+    Callable,
+    Dict,
+    List,
+)
 
-from boltons.iterutils import first, bucketize, partition
+from boltons.iterutils import first, partition
 from dateutil.parser import parse as parse_datetime
-from mongoengine import Q, Document, ListField, StringField
+from mongoengine import Q, Document, ListField, StringField, IntField
 from pymongo.command_cursor import CommandCursor
 
-from apierrors import errors
-from apierrors.base import BaseError
-from config import config
-from database.errors import MakeGetAllQueryError
-from database.projection import project_dict, ProjectionHelper
-from database.props import PropsMixin
-from database.query import RegexQ, RegexWrapper
-from database.utils import (
+from apiserver.apierrors import errors
+from apiserver.apierrors.base import BaseError
+from apiserver.bll.redis_cache_manager import RedisCacheManager
+from apiserver.config_repo import config
+from apiserver.database import Database
+from apiserver.database.errors import MakeGetAllQueryError
+from apiserver.database.projection import project_dict, ProjectionHelper
+from apiserver.database.props import PropsMixin
+from apiserver.database.query import RegexQ, RegexWrapper, RegexQCombination
+from apiserver.database.utils import (
     get_company_or_none_constraint,
     get_fields_choices,
     field_does_not_exist,
     field_exists,
 )
+from apiserver.redis_manager import redman
 
 log = config.logger("dbmodel")
 
@@ -70,6 +85,9 @@ class GetMixin(PropsMixin):
     _ordering_key = "order_by"
     _search_text_key = "search_text"
 
+    _start_key = "start"
+    _size_key = "size"
+
     _multi_field_param_sep = "__"
     _multi_field_param_prefix = {
         ("_any_", "_or_"): lambda a, b: a | b,
@@ -77,6 +95,7 @@ class GetMixin(PropsMixin):
     }
     MultiFieldParameters = namedtuple("MultiFieldParameters", "pattern fields")
 
+    _numeric_locale = {"locale": "en_US", "numericOrdering": True}
     _field_collation_overrides = {}
 
     class QueryParameterOptions(object):
@@ -86,6 +105,7 @@ class GetMixin(PropsMixin):
             list_fields=("tags", "system_tags", "id"),
             datetime_fields=None,
             fields=None,
+            range_fields=None,
         ):
             """
             :param pattern_fields: Fields for which a "string contains" condition should be generated
@@ -97,41 +117,111 @@ class GetMixin(PropsMixin):
             self.fields = fields
             self.datetime_fields = datetime_fields
             self.list_fields = list_fields
+            self.range_fields = range_fields
             self.pattern_fields = pattern_fields
 
     class ListFieldBucketHelper:
         op_prefix = "__$"
-        legacy_exclude_prefix = "-"
+        _legacy_exclude_prefix = "-"
+        _legacy_exclude_mongo_op = "nin"
 
-        _default = "in"
-        _ops = {"not": "nin"}
-        _next = _default
+        default_mongo_op = "in"
+        _ops = {
+            # op -> (mongo_op, sticky)
+            "not": ("nin", False),
+            "nop": (default_mongo_op, False),
+            "all": ("all", True),
+            "and": ("all", True),
+            "any": (default_mongo_op, True),
+            "or": (default_mongo_op, True),
+        }
 
         def __init__(self, legacy=False):
-            self._legacy = legacy
+            self._current_op = None
+            self._sticky = False
+            self._support_legacy = legacy
+            self.allow_empty = False
 
-        def key(self, v):
+        def _get_op(self, v: str, translate: bool = False) -> Optional[str]:
+            op = (
+                v[len(self.op_prefix) :] if v and v.startswith(self.op_prefix) else None
+            )
+            if translate:
+                tup = self._ops.get(op, None)
+                return tup[0] if tup else None
+            return op
+
+        def _key(self, v) -> Optional[Union[str, bool]]:
             if v is None:
-                self._next = self._default
-                return self._default
-            elif self._legacy and v.startswith(self.legacy_exclude_prefix):
-                self._next = self._default
-                return self._ops["not"]
-            elif v.startswith(self.op_prefix):
-                self._next = self._ops.get(v[len(self.op_prefix) :], self._default)
+                self.allow_empty = True
                 return None
 
-            next_ = self._next
-            self._next = self._default
-            return next_
+            op = self._get_op(v)
+            if op is not None:
+                # operator - set state and return None
+                self._current_op, self._sticky = self._ops.get(
+                    op, (self.default_mongo_op, self._sticky)
+                )
+                return None
+            elif self._current_op:
+                current_op = self._current_op
+                if not self._sticky:
+                    self._current_op = None
+                return current_op
+            elif self._support_legacy and v.startswith(self._legacy_exclude_prefix):
+                self._current_op = None
+                return False
 
-        def value_transform(self, v):
-            if self._legacy and v and v.startswith(self.legacy_exclude_prefix):
-                return v[len(self.legacy_exclude_prefix) :]
-            return v
+            return self.default_mongo_op
+
+        def get_global_op(self, data: Sequence[str]) -> int:
+            op_to_res = {
+                "in": Q.OR,
+                "all": Q.AND,
+            }
+            data = (x for x in data if x is not None)
+            first_op = (
+                self._get_op(next(data, ""), translate=True) or self.default_mongo_op
+            )
+            return op_to_res.get(first_op, self.default_mongo_op)
+
+        def get_actions(self, data: Sequence[str]) -> Dict[str, List[Union[str, None]]]:
+            actions = {}
+
+            for val in data:
+                key = self._key(val)
+                if key is None:
+                    continue
+                elif self._support_legacy and key is False:
+                    key = self._legacy_exclude_mongo_op
+                    val = val[len(self._legacy_exclude_prefix) :]
+                actions.setdefault(key, []).append(val)
+
+            return actions
 
     get_all_query_options = QueryParameterOptions()
 
+    class GetManyScrollState(ProperDictMixin, Document):
+        meta = {"db_alias": Database.backend, "strict": False}
+
+        id = StringField(primary_key=True)
+        position = IntField(default=0)
+
+    _cache_manager = None
+
+    @classmethod
+    def get_cache_manager(cls):
+        if not cls._cache_manager:
+            cls._cache_manager = RedisCacheManager(
+                state_class=cls.GetManyScrollState,
+                redis=redman.connection("apiserver"),
+                expiration_interval=config.get(
+                    "services._mongo.scroll_state_expiration_seconds", 600
+                ),
+            )
+
+        return cls._cache_manager
+
     @classmethod
     def get(
         cls: Union["GetMixin", Document],
@@ -175,6 +265,53 @@ class GetMixin(PropsMixin):
             parameters, parameters_options
         ) & cls._prepare_perm_query(company, allow_public=allow_public)
 
+    @staticmethod
+    def _pop_matching_params(
+        patterns: Sequence[str], parameters: dict
+    ) -> Mapping[str, Any]:
+        """
+        Pop the parameters that match the specified patterns and return
+        the dictionary of matching parameters
+        Pop None parameters since they are not the real queries
+        """
+        if not patterns:
+            return {}
+
+        fields = set()
+        for pattern in patterns:
+            if pattern.endswith("*"):
+                prefix = pattern[:-1]
+                fields.update(
+                    {field for field in parameters if field.startswith(prefix)}
+                )
+            elif pattern in parameters:
+                fields.add(pattern)
+
+        pairs = ((field, parameters.pop(field, None)) for field in fields)
+        return {k: v for k, v in pairs if v is not None}
+
+    @classmethod
+    def _try_convert_to_numeric(cls, value: Union[str, Sequence[str]]):
+        def convert_str(val: str) -> Union[float, str]:
+            try:
+                return float(val)
+            except ValueError:
+                return val
+
+        if isinstance(value, str):
+            return convert_str(value)
+
+        if isinstance(value, (list, tuple)):
+            return [convert_str(v) if isinstance(v, str) else v for v in value]
+
+        return value
+
+    @classmethod
+    def _get_fixed_field_value(cls, field: str, value):
+        if field.startswith("last_metrics."):
+            return cls._try_convert_to_numeric(value)
+        return value
+
     @classmethod
     def _prepare_query_no_company(
         cls, parameters=None, parameters_options=QueryParameterOptions()
@@ -183,7 +320,9 @@ class GetMixin(PropsMixin):
         Prepare a query object based on the provided query dictionary and various fields.
 
         NOTE: BE VERY CAREFUL WITH THIS CALL, as it allows creating queries that span across companies.
-
+        IMPLEMENTATION NOTE: Make sure that inside this function or the functions it depends on RegexQ is always
+        used instead of Q. Otherwise we can and up with some combination that is not processed according to
+        RegexQ rules
         :param parameters_options: Specifies options for parsing the parameters (see ParametersOptions)
         :param parameters: Query dictionary (relevant keys are these specified by the various field names parameters).
             Supported parameters:
@@ -197,22 +336,32 @@ class GetMixin(PropsMixin):
         dict_query = {}
         query = RegexQ()
         if parameters:
-            parameters = parameters.copy()
+            parameters = {
+                k: cls._get_fixed_field_value(k, v) for k, v in parameters.items()
+            }
             opts = parameters_options
             for field in opts.pattern_fields:
                 pattern = parameters.pop(field, None)
                 if pattern:
                     dict_query[field] = RegexWrapper(pattern)
 
-            for field in tuple(opts.list_fields or ()):
-                data = parameters.pop(field, None)
-                if data:
-                    query &= cls.get_list_field_query(field, data)
+            for field, data in cls._pop_matching_params(
+                patterns=opts.list_fields, parameters=parameters
+            ).items():
+                query &= cls.get_list_field_query(field, data)
 
-            for field in opts.fields or []:
-                data = parameters.pop(field, None)
-                if data is not None:
-                    dict_query[field] = data
+            for field, data in cls._pop_matching_params(
+                patterns=opts.range_fields, parameters=parameters
+            ).items():
+                query &= cls.get_range_field_query(field, data)
+
+            for field, data in cls._pop_matching_params(
+                patterns=opts.fields or [], parameters=parameters
+            ).items():
+                if "._" in field or "_." in field:
+                    query &= RegexQ(__raw__={field: data})
+                else:
+                    dict_query[field.replace(".", "__")] = data
 
             for field in opts.datetime_fields or []:
                 data = parameters.pop(field, None)
@@ -242,17 +391,64 @@ class GetMixin(PropsMixin):
                         raise MakeGetAllQueryError("incorrect field format", field)
                     if not data.fields:
                         break
-                    regex = RegexWrapper(data.pattern, flags=re.IGNORECASE)
-                    sep_fields = [f.replace(".", "__") for f in data.fields]
-                    q = reduce(
-                        lambda a, x: func(a, RegexQ(**{x: regex})), sep_fields, RegexQ()
-                    )
+                    if any("._" in f for f in data.fields):
+                        q = reduce(
+                            lambda a, x: func(
+                                a,
+                                RegexQ(
+                                    __raw__={
+                                        x: {"$regex": data.pattern, "$options": "i"}
+                                    }
+                                ),
+                            ),
+                            data.fields,
+                            RegexQ(),
+                        )
+                    else:
+                        regex = RegexWrapper(data.pattern, flags=re.IGNORECASE)
+                        sep_fields = [f.replace(".", "__") for f in data.fields]
+                        q = reduce(
+                            lambda a, x: func(a, RegexQ(**{x: regex})),
+                            sep_fields,
+                            RegexQ(),
+                        )
                     query = query & q
 
         return query & RegexQ(**dict_query)
 
     @classmethod
-    def get_list_field_query(cls, field: str, data: Sequence[Optional[str]]) -> Q:
+    def get_range_field_query(cls, field: str, data: Sequence[Optional[str]]) -> RegexQ:
+        """
+        Return a range query for the provided field. The data should contain min and max values
+        Both intervals are included. For open range queries either min or max can be None
+        In case the min value is None the records with missing or None value from db are included
+        """
+        if not isinstance(data, (list, tuple)) or len(data) != 2:
+            raise errors.bad_request.ValidationError(
+                f"Min and max values should be specified for range field {field}"
+            )
+
+        min_val, max_val = data
+        if min_val is None and max_val is None:
+            raise errors.bad_request.ValidationError(
+                f"At least one of min or max values should be provided for field {field}"
+            )
+
+        mongoengine_field = field.replace(".", "__")
+        query = {}
+        if min_val is not None:
+            query[f"{mongoengine_field}__gte"] = min_val
+        if max_val is not None:
+            query[f"{mongoengine_field}__lte"] = max_val
+
+        q = RegexQ(**query)
+        if min_val is None:
+            q |= RegexQ(**{mongoengine_field: None})
+
+        return q
+
+    @classmethod
+    def get_list_field_query(cls, field: str, data: Sequence[Optional[str]]) -> RegexQ:
         """
         Get a proper mongoengine Q object that represents an "or" query for the provided values
         with respect to the given list field, with support for "none of empty" in case a None value
@@ -260,36 +456,35 @@ class GetMixin(PropsMixin):
 
         - Exclusion can be specified by a leading "-" for each value (API versions <2.8)
             or by a preceding "__$not" value (operator)
+        - AND can be achieved using a preceding "__$all" or "__$and" value (operator)
         """
         if not isinstance(data, (list, tuple)):
-            raise MakeGetAllQueryError("expected list", field)
+            data = [data]
 
-        # TODO: backwards compatibility only for older API versions
         helper = cls.ListFieldBucketHelper(legacy=True)
-        actions = bucketize(
-            data, key=helper.key, value_transform=helper.value_transform
-        )
+        global_op = helper.get_global_op(data)
+        actions = helper.get_actions(data)
 
-        allow_empty = None in actions.get("in", {})
         mongoengine_field = field.replace(".", "__")
 
-        q = RegexQ()
-        for action in filter(None, actions):
-            q &= RegexQ(
-                **{
-                    f"{mongoengine_field}__{action}": list(
-                        set(filter(None, actions[action]))
-                    )
-                }
-            )
+        queries = [
+            RegexQ(**{f"{mongoengine_field}__{action}": list(set(actions[action]))})
+            for action in filter(None, actions)
+        ]
 
-        if not allow_empty:
+        if not queries:
+            q = RegexQ()
+        else:
+            q = RegexQCombination(operation=global_op, children=queries)
+
+        if not helper.allow_empty:
             return q
 
         return (
             q
-            | Q(**{f"{mongoengine_field}__exists": False})
-            | Q(**{mongoengine_field: []})
+            | RegexQ(**{f"{mongoengine_field}__exists": False})
+            | RegexQ(**{mongoengine_field: []})
+            | RegexQ(**{mongoengine_field: None})
         )
 
     @classmethod
@@ -317,27 +512,41 @@ class GetMixin(PropsMixin):
         return order_by
 
     @classmethod
-    def validate_paging(
-        cls, parameters=None, default_page=None, default_page_size=None
-    ):
-        """ Validate and extract paging info from from the provided dictionary. Supports default values. """
-        if parameters is None:
-            parameters = {}
-        default_page = parameters.get("page", default_page)
-        if default_page is None:
-            return None, None
-        default_page_size = parameters.get("page_size", default_page_size)
-        if not default_page_size:
-            raise errors.bad_request.MissingRequiredFields(
-                "page_size is required when page is requested", field="page_size"
-            )
-        elif default_page < 0:
+    def validate_paging(cls, parameters=None, default_page=0, default_page_size=None):
+        """
+        Validate and extract paging info from from the provided dictionary. Supports default values.
+        If page is specified then it should be non-negative, if page size is specified then it should be positive
+        If page size is specified and page is not then 0 page is assumed
+        If page is specified then page size should be specified too
+        """
+        parameters = parameters or {}
+
+        start = parameters.get(cls._start_key)
+        if start is not None:
+            return start, cls.validate_scroll_size(parameters)
+
+        max_page_size = config.get("services._mongo.max_page_size", 500)
+        page = parameters.get("page", default_page)
+        if page is not None and page < 0:
             raise errors.bad_request.ValidationError("page must be >=0", field="page")
-        elif default_page_size < 1:
+
+        page_size = parameters.get("page_size", default_page_size or max_page_size)
+        if page_size is not None and page_size < 1:
             raise errors.bad_request.ValidationError(
                 "page_size must be >0", field="page_size"
             )
-        return default_page, default_page_size
+
+        if page_size is not None:
+            page = page or 0
+            page_size = min(page_size, max_page_size)
+            return page * page_size, page_size
+
+        if page is not None:
+            raise errors.bad_request.MissingRequiredFields(
+                "page_size is required when page is requested", field="page_size"
+            )
+
+        return None, None
 
     @classmethod
     def get_projection(cls, parameters, override_projection=None, **__):
@@ -381,6 +590,54 @@ class GetMixin(PropsMixin):
     def set_default_ordering(cls, parameters: dict, value: Sequence[str]) -> None:
         cls.set_ordering(parameters, cls.get_ordering(parameters) or value)
 
+    @classmethod
+    def validate_scroll_size(cls, query_dict: dict) -> int:
+        size = query_dict.get(cls._size_key)
+        if not size or not isinstance(size, int) or size < 1:
+            raise errors.bad_request.ValidationError(
+                "Integer size parameter greater than 1 should be provided when working with scroll"
+            )
+        return size
+
+    @classmethod
+    def get_data_with_scroll_support(
+        cls,
+        query_dict: dict,
+        data_getter: Callable[[], Sequence[dict]],
+        ret_params: dict,
+    ) -> Sequence[dict]:
+        """
+        Retrieves the data by calling the provided data_getter api
+        If scroll parameters are specified then put the query_dict 'start' parameter to the last
+        scroll position and continue retrievals from that position
+        If refresh_scroll is requested then bring once more the data from the beginning
+        till the current scroll position
+        In the end the scroll position is updated and accumulated frames are returned
+        """
+        query_dict = query_dict or {}
+        state: Optional[cls.GetManyScrollState] = None
+        if "scroll_id" in query_dict:
+            size = cls.validate_scroll_size(query_dict)
+            state = cls.get_cache_manager().get_or_create_state_core(
+                query_dict.get("scroll_id")
+            )
+            if query_dict.get("refresh_scroll"):
+                query_dict[cls._size_key] = max(state.position, size)
+                state.position = 0
+            query_dict[cls._start_key] = state.position
+
+        data = data_getter()
+        if cls._start_key in query_dict:
+            query_dict[cls._start_key] = query_dict[cls._start_key] + len(data)
+
+        if state:
+            state.position = query_dict[cls._start_key]
+            cls.get_cache_manager().set_state(state)
+            if ret_params is not None:
+                ret_params["scroll_id"] = state.id
+
+        return data
+
     @classmethod
     def get_many_with_join(
         cls,
@@ -391,6 +648,8 @@ class GetMixin(PropsMixin):
         allow_public=False,
         override_projection=None,
         expand_reference_ids=True,
+        projection_fields: dict = None,
+        ret_params: dict = None,
     ):
         """
         Fetch all documents matching a provided query with support for joining referenced documents according to the
@@ -426,6 +685,8 @@ class GetMixin(PropsMixin):
             query=query,
             query_options=query_options,
             allow_public=allow_public,
+            projection_fields=projection_fields,
+            ret_params=ret_params,
         )
 
         def projection_func(doc_type, projection, ids):
@@ -439,6 +700,51 @@ class GetMixin(PropsMixin):
 
         return helper.project(results, projection_func)
 
+    @classmethod
+    def _get_collation_override(cls, field: str) -> Optional[dict]:
+        return first(
+            v for k, v in cls._field_collation_overrides.items() if field.startswith(k)
+        )
+
+    @classmethod
+    def get_count(
+        cls: Union["GetMixin", Document],
+        company,
+        query_dict: dict = None,
+        query_options: QueryParameterOptions = None,
+        query: Q = None,
+        allow_public=False,
+    ) -> int:
+        _query = cls._get_combined_query(
+            company=company,
+            query_dict=query_dict,
+            query_options=query_options,
+            query=query,
+            allow_public=allow_public,
+        )
+        return cls.objects(_query).count()
+
+    @classmethod
+    def _get_combined_query(
+        cls,
+        company,
+        query_dict: dict = None,
+        query_options: QueryParameterOptions = None,
+        query: Q = None,
+        allow_public=False,
+    ) -> Q:
+        if query_dict is not None:
+            q = cls.prepare_query(
+                parameters=query_dict,
+                company=company,
+                parameters_options=query_options,
+                allow_public=allow_public,
+            )
+        else:
+            q = cls._prepare_perm_query(company, allow_public=allow_public)
+
+        return (q & query) if query else q
+
     @classmethod
     def get_many(
         cls,
@@ -450,6 +756,8 @@ class GetMixin(PropsMixin):
         allow_public=False,
         override_projection: Collection[str] = None,
         return_dicts=True,
+        projection_fields: dict = None,
+        ret_params: dict = None,
     ):
         """
         Fetch all documents matching a provided query. Supported several built-in options
@@ -476,26 +784,40 @@ class GetMixin(PropsMixin):
         :param allow_public: If True, objects marked as public (no associated company) are also queried.
         :return: A list of objects matching the query.
         """
-        if query_dict is not None:
-            q = cls.prepare_query(
-                parameters=query_dict,
-                company=company,
-                parameters_options=query_options,
-                allow_public=allow_public,
-            )
-        else:
-            q = cls._prepare_perm_query(company, allow_public=allow_public)
-        _query = (q & query) if query else q
+        override_collation = None
+        if query_dict:
+            for field in query_dict:
+                override_collation = cls._get_collation_override(field)
+                if override_collation:
+                    break
+
+        _query = cls._get_combined_query(
+            company=company,
+            query_dict=query_dict,
+            query_options=query_options,
+            query=query,
+            allow_public=allow_public,
+        )
 
         if return_dicts:
-            return cls._get_many_override_none_ordering(
+            data_getter = partial(
+                cls._get_many_override_none_ordering,
                 query=_query,
                 parameters=parameters,
                 override_projection=override_projection,
+                override_collation=override_collation,
+                projection_fields=projection_fields,
+            )
+            return cls.get_data_with_scroll_support(
+                query_dict=query_dict, data_getter=data_getter, ret_params=ret_params,
             )
 
         return cls._get_many_no_company(
-            query=_query, parameters=parameters, override_projection=override_projection
+            query=_query,
+            parameters=parameters,
+            override_projection=override_projection,
+            override_collation=override_collation,
+            projection_fields=projection_fields,
         )
 
     @classmethod
@@ -519,6 +841,8 @@ class GetMixin(PropsMixin):
         query: Q,
         parameters=None,
         override_projection=None,
+        override_collation=None,
+        projection_fields: dict = None,
     ):
         """
         Fetch all documents matching a provided query.
@@ -538,12 +862,16 @@ class GetMixin(PropsMixin):
         parameters = parameters or {}
         search_text = parameters.get(cls._search_text_key)
         order_by = cls.validate_order_by(parameters=parameters, search_text=search_text)
-        page, page_size = cls.validate_paging(parameters=parameters)
+        if order_by and not override_collation:
+            override_collation = cls._get_collation_override(order_by[0])
+        start, size = cls.validate_paging(parameters=parameters)
         include, exclude = cls.split_projection(
             cls.get_projection(parameters, override_projection)
         )
 
         qs = cls.objects(query)
+        if override_collation:
+            qs = qs.collation(collation=override_collation)
         if search_text:
             qs = qs.search_text(search_text)
         if order_by:
@@ -557,18 +885,51 @@ class GetMixin(PropsMixin):
         if exclude:
             qs = qs.exclude(*exclude)
 
-        if page is not None and page_size:
+        if projection_fields:
+            qs = qs.fields(**projection_fields)
+
+        if start is not None and size:
             # add paging
-            qs = qs.skip(page * page_size).limit(page_size)
+            qs = qs.skip(start).limit(size)
 
         return qs
 
+    @classmethod
+    def _get_queries_for_order_field(
+        cls, query: Q, order_field: str
+    ) -> Union[None, Tuple[Q, Q]]:
+        """
+        In case the order_field is one of the cls fields and the sorting is ascending
+        then return the tuple of 2 queries:
+        1. original query with not empty constraint on the order_by field
+        2. original query with empty constraint on the order_by field
+        """
+        if not order_field or order_field.startswith("-") or "[" in order_field:
+            return
+
+        mongo_field_name = order_field.replace(".", "__")
+        mongo_field = first(
+            v for k, v in cls.get_all_fields_with_instance() if k == mongo_field_name
+        )
+
+        if isinstance(mongo_field, ListField):
+            params = {"is_list": True}
+        elif isinstance(mongo_field, StringField):
+            params = {"empty_value": ""}
+        else:
+            params = {}
+        non_empty = query & field_exists(mongo_field_name, **params)
+        empty = query & field_does_not_exist(mongo_field_name, **params)
+        return non_empty, empty
+
     @classmethod
     def _get_many_override_none_ordering(
         cls: Union[Document, "GetMixin"],
         query: Q = None,
         parameters: dict = None,
         override_projection: Collection[str] = None,
+        override_collation: dict = None,
+        projection_fields: dict = None,
     ) -> Sequence[dict]:
         """
         Fetch all documents matching a provided query. For the first order by field
@@ -591,7 +952,10 @@ class GetMixin(PropsMixin):
         parameters = parameters or {}
         search_text = parameters.get(cls._search_text_key)
         order_by = cls.validate_order_by(parameters=parameters, search_text=search_text)
-        page, page_size = cls.validate_paging(parameters=parameters)
+        start, size = cls.validate_paging(parameters=parameters)
+        if size is not None and size <= 0:
+            return []
+
         include, exclude = cls.split_projection(
             cls.get_projection(parameters, override_projection)
         )
@@ -601,32 +965,17 @@ class GetMixin(PropsMixin):
             order_field = first(
                 field for field in order_by if not field.startswith("$")
             )
-            if (
-                order_field
-                and not order_field.startswith("-")
-                and "[" not in order_field
-            ):
-                params = {}
-                mongo_field = order_field.replace(".", "__")
-                if mongo_field in cls.get_field_names_for_type(of_type=ListField):
-                    params["is_list"] = True
-                elif mongo_field in cls.get_field_names_for_type(of_type=StringField):
-                    params["empty_value"] = ""
-                non_empty = query & field_exists(mongo_field, **params)
-                empty = query & field_does_not_exist(mongo_field, **params)
-                query_sets = [cls.objects(non_empty), cls.objects(empty)]
-
+            res = cls._get_queries_for_order_field(query, order_field)
+            if res:
+                query_sets = [cls.objects(q) for q in res]
             query_sets = [qs.order_by(*order_by) for qs in query_sets]
-            if order_field:
-                collation_override = first(
-                    v
-                    for k, v in cls._field_collation_overrides.items()
-                    if order_field.startswith(k)
-                )
-                if collation_override:
-                    query_sets = [
-                        qs.collation(collation=collation_override) for qs in query_sets
-                    ]
+            if order_field and not override_collation:
+                override_collation = cls._get_collation_override(order_field)
+
+        if override_collation:
+            query_sets = [
+                qs.collation(collation=override_collation) for qs in query_sets
+            ]
 
         if search_text:
             query_sets = [qs.search_text(search_text) for qs in query_sets]
@@ -638,25 +987,31 @@ class GetMixin(PropsMixin):
         if exclude:
             query_sets = [qs.exclude(*exclude) for qs in query_sets]
 
-        if page is None or not page_size:
+        if projection_fields:
+            query_sets = [qs.fields(**projection_fields) for qs in query_sets]
+
+        if start is None or not size:
             return [obj.to_proper_dict(only=include) for qs in query_sets for obj in qs]
 
         # add paging
         ret = []
-        start = page * page_size
-        for qs in query_sets:
-            qs_size = qs.count()
-            if qs_size < start:
-                start -= qs_size
-                continue
+        last_set = len(query_sets) - 1
+        for i, qs in enumerate(query_sets):
+            last_size = len(ret)
             ret.extend(
                 obj.to_proper_dict(only=include)
-                for obj in qs.skip(start).limit(page_size)
+                for obj in (qs.skip(start) if start else qs).limit(size)
             )
-            if len(ret) >= page_size:
+            added = len(ret) - last_size
+
+            if added > 0:
+                start = 0
+                size = max(0, size - added)
+            elif i != last_set:
+                start -= min(start, qs.count())
+
+            if size <= 0:
                 break
-            start = 0
-            page_size -= len(ret)
 
         return ret
 
@@ -692,14 +1047,24 @@ class GetMixin(PropsMixin):
 
 
 class UpdateMixin(object):
+    __user_set_allowed_fields = None
+    __locked_when_published_fields = None
+
     @classmethod
     def user_set_allowed(cls):
-        res = getattr(cls, "__user_set_allowed_fields", None)
-        if res is None:
-            res = cls.__user_set_allowed_fields = get_fields_choices(
-                cls, "user_set_allowed"
+        if cls.__user_set_allowed_fields is None:
+            cls.__user_set_allowed_fields = dict(
+                get_fields_choices(cls, "user_set_allowed")
             )
-        return res
+        return cls.__user_set_allowed_fields
+
+    @classmethod
+    def locked_when_published(cls):
+        if cls.__locked_when_published_fields is None:
+            cls.__locked_when_published_fields = dict(
+                get_fields_choices(cls, "locked_when_published")
+            )
+        return cls.__locked_when_published_fields
 
     @classmethod
     def get_safe_update_dict(cls, fields):
@@ -773,7 +1138,7 @@ class DbModelMixin(GetMixin, ProperDictMixin, UpdateMixin):
     ):
         if enabled:
             items = list(cls.objects(id__in=ids, company=company_id).only("id"))
-            update = dict(set__company_origin=company_id, unset__company=1)
+            update = dict(set__company_origin=company_id, set__company="")
         else:
             items = list(
                 cls.objects(

apiserver/database/model/company.py

@@ -8,9 +8,9 @@ from mongoengine import (
     DateTimeField,
 )
 
-from database import Database, strict
-from database.fields import StrippedStringField
-from database.model import DbModelMixin
+from apiserver.database import Database, strict
+from apiserver.database.fields import StrippedStringField
+from apiserver.database.model import DbModelMixin
 
 
 class ReportStatsOption(EmbeddedDocument):
@@ -29,7 +29,7 @@ class Company(DbModelMixin, Document):
     meta = {"db_alias": Database.backend, "strict": strict}
 
     id = StringField(primary_key=True)
-    name = StrippedStringField(unique=True, min_length=3)
+    name = StrippedStringField(min_length=3)
     defaults = EmbeddedDocumentField(CompanyDefaults, default=CompanyDefaults)
 
     @classmethod