diff --git a/README.md b/README.md index 349e239d..96517a0a 100644 --- a/README.md +++ b/README.md @@ -1,143 +1,134 @@ -# Allegro Trains - new name is coming soon ;) -## Auto-Magical Experiment Manager, Version Control and ML-Ops for AI +
-## :confetti_ball: Now with Full ML/DL DevOps - See [TRAINS AGENT](https://github.com/allegroai/trains-agent) and [Services](https://github.com/allegroai/trains-server#trains-agent-services--) -## :station: [Documentation is here!](https://allegro.ai/docs) `wubba lubba dub dub` and a [Slack Channel](https://join.slack.com/t/allegroai-trains/shared_invite/enQtOTQyMTI1MzQxMzE4LTY5NTUxOTY1NmQ1MzQ5MjRhMGRhZmM4ODE5NTNjMTg2NTBlZGQzZGVkMWU3ZDg1MGE1MjQxNDEzMWU2NmVjZmY) :train2: -## Features: [AWS autoscaler wizard](https://allegro.ai/docs/examples/services/aws_autoscaler/aws_autoscaler/) :robot: [Hyper-Parameter Optimization](https://allegro.ai/docs/examples/optimization/hyper-parameter-optimization/examples_hyperparam_opt/) and :electric_plug: [Pipeline Controllers](https://allegro.ai/docs/examples/pipeline/pipeline_controller/) + -"Because it’s a jungle out there" -[![GitHub license](https://img.shields.io/github/license/allegroai/trains.svg)](https://img.shields.io/github/license/allegroai/trains.svg) -[![PyPI pyversions](https://img.shields.io/pypi/pyversions/trains.svg)](https://img.shields.io/pypi/pyversions/trains.svg) -[![PyPI version shields.io](https://img.shields.io/pypi/v/trains.svg)](https://img.shields.io/pypi/v/trains.svg) -[![PyPI status](https://img.shields.io/pypi/status/trains.svg)](https://pypi.python.org/pypi/trains/) +**ClearML - Auto-Magical Suite of tools to streamline your ML workflow +Experiment Manager, ML-Ops and Data-Management** + +[![GitHub license](https://img.shields.io/github/license/allegroai/trains.svg)](https://img.shields.io/github/license/allegroai/clearml.svg) +[![PyPI pyversions](https://img.shields.io/pypi/pyversions/clearml.svg)](https://img.shields.io/pypi/pyversions/clearml.svg) +[![PyPI version shields.io](https://img.shields.io/pypi/v/clearml.svg)](https://img.shields.io/pypi/v/clearml.svg) +[![PyPI status](https://img.shields.io/pypi/status/clearml.svg)](https://pypi.python.org/pypi/clearml/) [![Optuna](https://img.shields.io/badge/Optuna-integrated-blue)](https://optuna.org) -[![Slack Channel](https://img.shields.io/badge/slack-%23trains--community-blueviolet?logo=slack)](https://join.slack.com/t/allegroai-trains/shared_invite/zt-c0t13pty-aVUZZW1TSSSg2vyIGVPBhg) +[![Slack Channel](https://img.shields.io/badge/slack-%23clearml--community-blueviolet?logo=slack)](https://join.slack.com/t/allegroai-trains/shared_invite/zt-c0t13pty-aVUZZW1TSSSg2vyIGVPBhg) -### :point_right: Help improve Trains by filling our 2-min [user survey](https://allegro.ai/lp/trains-user-survey/) +
-Trains is our solution to a problem we share with countless other researchers and developers in the machine +--- +### ClearML +#### *Formerly known as Allegro Trains* +ClearML is a ML/DL development and production suite, it contains three main modules: + +- [Experiment Manager](#clearml-experiment-management) - Automagical experiment tracking, environments and results +- [ML-Ops](https://github.com/allegroai/trains-agent) - Automation, Pipelines & Orchestration solution for ML/DL jobs (K8s / Cloud / bare-metal) +- [Data-Management](https://github.com/allegroai/clearml/doc/clearml-data.md) - Fully differentiable data management & version control solution on top of object-storage + (S3/GS/Azure/NAS) + + +Instrumenting these components is the **ClearML-server**, see [Self-Hosting]() & [Free tier Hosting]() + + +--- +
+ +**[Signup](https://app.community.clear.ml) & [Start using](https://allegro.ai/clearml/docs/getting_started/getting_started/) in under 2 minutes** + +
+ +--- + + +## ClearML Experiment Manager + +**Adding only 2 lines to your code gets you the following** + +* Complete experiment setup log + * Full source control info including non-committed local changes + * Execution environment (including specific packages & versions) + * Hyper-parameters + * ArgParser for command line parameters with currently used values + * Explicit parameters dictionary + * Tensorflow Defines (absl-py) + * Hydra configuration and overrides + * Initial model weights file +* Full experiment output automatic capture + * stdout and stderr + * Resource Monitoring (CPU/GPU utilization, temperature, IO, network, etc.) + * Model snapshots (With optional automatic upload to central storage: Shared folder, S3, GS, Azure, Http) + * Artifacts log & store (Shared folder, S3, GS, Azure, Http) + * Tensorboard/TensorboardX scalars, metrics, histograms, **images, audio and video samples** + * [Matplotlib & Seaborn](https://github.com/allegroai/trains/tree/master/examples/frameworks/matplotlib) + * [ClearML Explicit Logging](https://allegro.ai/clearml/docs/examples/reporting/) interface for complete flexibility. +* Extensive platform support and integrations + * Supported ML/DL frameworks: [PyTorch](https://github.com/allegroai/trains/tree/master/examples/frameworks/pytorch)(incl' ignite/lightning), [Tensorflow](https://github.com/allegroai/trains/tree/master/examples/frameworks/tensorflow), [Keras](https://github.com/allegroai/trains/tree/master/examples/frameworks/keras), [AutoKeras](https://github.com/allegroai/trains/tree/master/examples/frameworks/autokeras), [XGBoost](https://github.com/allegroai/trains/tree/master/examples/frameworks/xgboost) and [Scikit-Learn](https://github.com/allegroai/trains/tree/master/examples/frameworks/scikit-learn) + * Seamless integration (including version control) with **Jupyter Notebook** + and [*PyCharm* remote debugging](https://github.com/allegroai/trains-pycharm-plugin) + +#### [Start using ClearML](https://allegro.ai/clearml/docs/getting_started/getting_started/) + +```bash +pip install clearml +``` + +Add two lines to your code: +```python +from clearml import Task +task = Task(project_name='examples', task_name='hello world') +``` + +You are done, everything your process outputs is now automagically logged into ClearML. +
Next step automation! **Learn more on ClearML two clicks automation [here]()** + +## ClearML Architecture + +The ClearML run-time components: + +* The ClearML Python Package for integrating ClearML into your existing scripts by adding just two lines of code, and optionally extending your experiments and other workflows with ClearML powerful and versatile set of classes and methods. +* The ClearML Server storing experiment, model, and workflow data, and supporting the Web UI experiment manager, and ML-Ops automation for reproducibility and tuning. It is available as a hosted service and open source for you to deploy your own ClearML Server. +* The ClearML Agent for ML-Ops orchestration, experiment and workflow reproducibility, and scalability. + +clearml-architecture + +## Additional Modules + +- [clearml-session](https://github.com/allegroai/clearml-session) - **Launch remote JupyterLab / VSCode-server inside any docker, on Cloud/On-Prem machines** +- [clearml-task](https://github.com/allegroai/clearml/doc/clearml-task.md) - Run any codebase on remote machines with full remote logging of Tensorboard, Matplotlib & Console outputs +- [clearml-data](https://github.com/allegroai/clearml/doc/clearml-data.md) - **CLI for managing and versioning your datasets, including creating / uploading / downloading of data from S3/GS/Azure/NAS** +- [AWS Auto-Scaler](examples/services/aws-autoscaler/aws_autoscaler.py) - Automatically spin EC2 instances based on your workloads with preconfigured budget! No need for K8s! +- [Hyper-Parameter Optimization](examples/services/hyper-parameter-optimization/hyper_parameter_optimizer.py) - Optimize any code with black-box approach and state of the art Bayesian optimization algorithms +- [Automation Pipeline](examples/pipeline/pipeline_controller.py) - Build pipelines based on existing experiments / jobs, supports building pipelines of pipelines! +- [Slack Integration](examples/services/monitoring/slack_alerts.py) - Report experiments progress / failure directly to Slack (fully customizable!) + +## Why ClearML? + +ClearML is our solution to a problem we share with countless other researchers and developers in the machine learning/deep learning universe: Training production-grade deep learning models is a glorious but messy process. -Trains tracks and controls the process by associating code version control, research projects, +ClearML tracks and controls the process by associating code version control, research projects, performance metrics, and model provenance. -We designed Trains specifically to require effortless integration so that teams can preserve their existing methods -and practices. Use it on a daily basis to boost collaboration and visibility, or use it to automatically collect -your experimentation logs, outputs, and data to one centralized server. +We designed ClearML specifically to require effortless integration so that teams can preserve their existing methods +and practices. -**We have a demo server up and running at [https://demoapp.trains.allegro.ai](https://demoapp.trains.allegro.ai).** - -### :steam_locomotive: [Getting Started Tutorial](https://allegro.ai/blog/setting-up-allegro-ai-platform/) :rocket: - -**You can try out Trains and [test your code](#integrate-trains), with no additional setup.** - - - -## Trains Automatically Logs Everything -**With only two lines of code, this is what you are getting:** - -* Git repository, branch, commit id, entry point and local git diff -* Python environment (including specific packages & versions) -* stdout and stderr -* Resource Monitoring (CPU/GPU utilization, temperature, IO, network, etc.) -* Hyper-parameters - * ArgParser for command line parameters with currently used values - * Explicit parameters dictionary - * Tensorflow Defines (absl-py) -* Initial model weights file -* Model snapshots (With optional automatic upload to central storage: Shared folder, S3, GS, Azure, Http) -* Artifacts log & store (Shared folder, S3, GS, Azure, Http) -* Tensorboard/TensorboardX scalars, metrics, histograms, **images, audio and video** -* [Matplotlib & Seaborn](https://github.com/allegroai/trains/tree/master/examples/frameworks/matplotlib) -* Supported frameworks: [PyTorch](https://github.com/allegroai/trains/tree/master/examples/frameworks/pytorch), [Tensorflow](https://github.com/allegroai/trains/tree/master/examples/frameworks/tensorflow), [Keras](https://github.com/allegroai/trains/tree/master/examples/frameworks/keras), [AutoKeras](https://github.com/allegroai/trains/tree/master/examples/frameworks/autokeras), [XGBoost](https://github.com/allegroai/trains/tree/master/examples/frameworks/xgboost) and [Scikit-Learn](https://github.com/allegroai/trains/tree/master/examples/frameworks/scikit-learn) (MxNet is coming soon) -* Seamless integration (including version control) with **Jupyter Notebook** - and [*PyCharm* remote debugging](https://github.com/allegroai/trains-pycharm-plugin) - -**Additionally, log data explicitly using [Trains Explicit Logging](https://allegro.ai/docs/examples/reporting/).** - -## Using Trains - -Trains is a two part solution: - -1. Trains [python package](https://pypi.org/project/trains/) auto-magically connects with your code - - **Trains requires only two lines of code for full integration.** - - To connect your code with Trains: - - - Install Trains - - pip install trains -
- Add optional cloud storage support (S3/GoogleStorage/Azure): - - ```bash - pip install trains[s3] - pip install trains[gs] - pip install trains[azure] - ``` - -
- - - Add the following lines to your code - - from trains import Task - task = Task.init(project_name="my project", task_name="my task") - - * If project_name is not provided, the repository name will be used instead - * If task_name (experiment) is not provided, the current filename will be used instead - - - Run your code. When Trains connects to the server, a link is printed. For example - - Trains Results page: - https://demoapp.trains.allegro.ai/projects/76e5e2d45e914f52880621fe64601e85/experiments/241f06ae0f5c4b27b8ce8b64890ce152/output/log - - - Open the link and view your experiment parameters, model and tensorboard metrics - - **See examples [here](https://allegro.ai/docs/examples/examples_overview/)** - -2. [Trains Server](https://github.com/allegroai/trains-server) for logging, querying, control and UI ([Web-App](https://github.com/allegroai/trains-web)) - - **We already have a demo server up and running for you at [https://demoapp.trains.allegro.ai](https://demoapp.trains.allegro.ai).** - - **You can try out Trains without the need to install your own *trains-server*, just add the two lines of code, and it will automatically connect to the Trains demo-server.** - - *Note that the demo server resets every 24 hours and all of the logged data is deleted.* - - When you are ready to use your own Trains server, go ahead and [install *trains-server*](https://github.com/allegroai/trains-server). - - - - -## Configuring Your Own Trains server - -1. Install and run *trains-server* (see [Installing the Trains Server](https://github.com/allegroai/trains-server)) - -2. Run the initial configuration wizard for your Trains installation and follow the instructions to setup Trains package -(http://**_trains-server-ip_**:__port__ and user credentials) - - trains-init - -After installing and configuring, you can access your configuration file at `~/trains.conf` - -Sample configuration file available [here](https://github.com/allegroai/trains/blob/master/docs/trains.conf). + - Use it on a daily basis to boost collaboration and visibility in your team + - Create a remote job from any experiment with a click of a button + - Automate processes and create pipelines to collect your experimentation logs, outputs, and data + - Store all you data on any object-storage solution, with the simplest interface possible + - Make you data transparent by cataloging it all on the ClearML platform +We believe ClearML is ground-breaking. We wish to establish new standards of true seamless integration between +experiment management,ML-Ops and data management. ## Who We Are -Trains is supported by the same team behind *allegro.ai*, +ClearML is supported by the team behind *allegro.ai*, where we build deep learning pipelines and infrastructure for enterprise companies. -We built Trains to track and control the glorious but messy process of training production-grade deep learning models. -We are committed to vigorously supporting and expanding the capabilities of Trains. +We built ClearML to track and control the glorious but messy process of training production-grade deep learning models. +We are committed to vigorously supporting and expanding the capabilities of ClearML. -## Why Are We Releasing Trains? - -We believe Trains is ground-breaking. We wish to establish new standards of experiment management in -deep-learning and ML. Only the greater community can help us do that. - -We promise to always be backwardly compatible. If you start working with Trains today, -even though this project is currently in the beta stage, your logs and data will always upgrade with you. +We promise to always be backwardly compatible, making sure all your logs, data and pipelines +will always upgrade with you. ## License @@ -145,19 +136,19 @@ Apache License, Version 2.0 (see the [LICENSE](https://www.apache.org/licenses/L ## Documentation, Community & Support -More information in the [official documentation](https://allegro.ai/docs) and [on YouTube](https://www.youtube.com/c/AllegroAI). +More information in the [official documentation](https://allegro.ai//clearml/docs) and [on YouTube](https://www.youtube.com/c/AllegroAI). -For examples and use cases, check the [examples folder](https://github.com/allegroai/trains/tree/master/examples) and [corresponding documentation](https://allegro.ai/docs/examples/examples_overview/). +For examples and use cases, check the [examples folder](https://github.com/allegroai/trains/tree/master/examples) and [corresponding documentation](https://allegro.ai/clearml/docs/examples/examples_overview/). If you have any questions: post on our [Slack Channel](https://join.slack.com/t/allegroai-trains/shared_invite/enQtOTQyMTI1MzQxMzE4LTY5NTUxOTY1NmQ1MzQ5MjRhMGRhZmM4ODE5NTNjMTg2NTBlZGQzZGVkMWU3ZDg1MGE1MjQxNDEzMWU2NmVjZmY), or tag your questions on [stackoverflow](https://stackoverflow.com/questions/tagged/trains) with '**trains**' tag. For feature requests or bug reports, please use [GitHub issues](https://github.com/allegroai/trains/issues). -Additionally, you can always find us at *trains@allegro.ai* +Additionally, you can always find us at *clearml@allegro.ai* ## Contributing -See the Trains [Guidelines for Contributing](https://github.com/allegroai/trains/blob/master/docs/contributing.md). +See the ClearML [Guidelines for Contributing](https://github.com/allegroai/trains/blob/master/docs/contributing.md). _May the force (and the goddess of learning rates) be with you!_ diff --git a/clearml/__init__.py b/clearml/__init__.py index c05221e3..0ad97be4 100644 --- a/clearml/__init__.py +++ b/clearml/__init__.py @@ -1,4 +1,4 @@ -""" TRAINS open SDK """ +""" ClearML open SDK """ from .version import __version__ from .task import Task @@ -6,5 +6,7 @@ from .model import InputModel, OutputModel, Model from .logger import Logger from .storage import StorageManager from .errors import UsageError +from .datasets import Dataset -__all__ = ["__version__", "Task", "InputModel", "OutputModel", "Model", "Logger", "StorageManager", "UsageError"] +__all__ = ["__version__", "Task", "InputModel", "OutputModel", "Model", "Logger", + "StorageManager", "UsageError", "Dataset"] diff --git a/clearml/automation/__init__.py b/clearml/automation/__init__.py index d2ac6d46..71b68351 100644 --- a/clearml/automation/__init__.py +++ b/clearml/automation/__init__.py @@ -1,6 +1,7 @@ from .parameters import UniformParameterRange, DiscreteParameterRange, UniformIntegerParameterRange, ParameterSet from .optimization import GridSearch, RandomSearch, HyperParameterOptimizer, Objective from .job import TrainsJob +from .controller import PipelineController __all__ = ["UniformParameterRange", "DiscreteParameterRange", "UniformIntegerParameterRange", "ParameterSet", - "GridSearch", "RandomSearch", "HyperParameterOptimizer", "Objective", "TrainsJob"] + "GridSearch", "RandomSearch", "HyperParameterOptimizer", "Objective", "TrainsJob", "PipelineController"] diff --git a/clearml/automation/auto_scaler.py b/clearml/automation/auto_scaler.py index b35d0d64..66b51d4a 100644 --- a/clearml/automation/auto_scaler.py +++ b/clearml/automation/auto_scaler.py @@ -102,15 +102,15 @@ class AutoScaler(object): def spin_up_worker(self, resource, worker_id_prefix, queue_name): """ - Creates a new worker for trains (cloud-specific implementation). + Creates a new worker for clearml (cloud-specific implementation). First, create an instance in the cloud and install some required packages. - Then, define trains-agent environment variables and run trains-agent for the specified queue. + Then, define clearml-agent environment variables and run clearml-agent for the specified queue. NOTE: - Will wait until instance is running - This implementation assumes the instance image already has docker installed :param str resource: resource name, as defined in self.resource_configurations and self.queues. :param str worker_id_prefix: worker name prefix - :param str queue_name: trains queue to listen to + :param str queue_name: clearml queue to listen to """ pass @@ -137,17 +137,17 @@ class AutoScaler(object): minutes would be removed. """ - # Worker's id in trains would be composed from prefix, name, instance_type and cloud_id separated by ';' + # Worker's id in clearml would be composed from prefix, name, instance_type and cloud_id separated by ';' workers_pattern = re.compile( r"^(?P[^:]+):(?P[^:]+):(?P[^:]+):(?P[^:]+)" ) - # Set up the environment variables for trains - os.environ["TRAINS_API_HOST"] = self.api_server - os.environ["TRAINS_WEB_HOST"] = self.web_server - os.environ["TRAINS_FILES_HOST"] = self.files_server - os.environ["TRAINS_API_ACCESS_KEY"] = self.access_key - os.environ["TRAINS_API_SECRET_KEY"] = self.secret_key + # Set up the environment variables for clearml + os.environ["CLEARML_API_HOST"] = self.api_server + os.environ["CLEARML_WEB_HOST"] = self.web_server + os.environ["CLEARML_FILES_HOST"] = self.files_server + os.environ["CLEARML_API_ACCESS_KEY"] = self.access_key + os.environ["CLEARML_API_SECRET_KEY"] = self.secret_key api_client = APIClient() # Verify the requested queues exist and create those that doesn't exist @@ -234,7 +234,7 @@ class AutoScaler(object): # skip resource types that might be needed if resources in required_idle_resources: continue - # Remove from both aws and trains all instances that are idle for longer than MAX_IDLE_TIME_MIN + # Remove from both aws and clearml all instances that are idle for longer than MAX_IDLE_TIME_MIN if time() - timestamp > self.max_idle_time_min * 60.0: cloud_id = workers_pattern.match(worker.id)["cloud_id"] self.spin_down_worker(cloud_id) diff --git a/clearml/automation/aws_auto_scaler.py b/clearml/automation/aws_auto_scaler.py index 0c8c4b2d..13c142c8 100644 --- a/clearml/automation/aws_auto_scaler.py +++ b/clearml/automation/aws_auto_scaler.py @@ -31,15 +31,15 @@ class AwsAutoScaler(AutoScaler): def spin_up_worker(self, resource, worker_id_prefix, queue_name): """ - Creates a new worker for trains. + Creates a new worker for clearml. First, create an instance in the cloud and install some required packages. - Then, define trains-agent environment variables and run trains-agent for the specified queue. + Then, define clearml-agent environment variables and run clearml-agent for the specified queue. NOTE: - Will wait until instance is running - This implementation assumes the instance image already has docker installed :param str resource: resource name, as defined in BUDGET and QUEUES. :param str worker_id_prefix: worker name prefix - :param str queue_name: trains queue to listen to + :param str queue_name: clearml queue to listen to """ resource_conf = self.resource_configurations[resource] # Add worker type and AWS instance type to the worker name. @@ -50,7 +50,7 @@ class AwsAutoScaler(AutoScaler): ) # user_data script will automatically run when the instance is started. it will install the required packages - # for trains-agent configure it using environment variables and run trains-agent on the required queue + # for clearml-agent configure it using environment variables and run clearml-agent on the required queue user_data = """#!/bin/bash sudo apt-get update sudo apt-get install -y python3-dev @@ -60,22 +60,22 @@ class AwsAutoScaler(AutoScaler): sudo apt-get install -y build-essential python3 -m pip install -U pip python3 -m pip install virtualenv - python3 -m virtualenv trains_agent_venv - source trains_agent_venv/bin/activate - python -m pip install trains-agent - echo 'agent.git_user=\"{git_user}\"' >> /root/trains.conf - echo 'agent.git_pass=\"{git_pass}\"' >> /root/trains.conf - echo "{trains_conf}" >> /root/trains.conf - export TRAINS_API_HOST={api_server} - export TRAINS_WEB_HOST={web_server} - export TRAINS_FILES_HOST={files_server} + python3 -m virtualenv clearml_agent_venv + source clearml_agent_venv/bin/activate + python -m pip install clearml-agent + echo 'agent.git_user=\"{git_user}\"' >> /root/clearml.conf + echo 'agent.git_pass=\"{git_pass}\"' >> /root/clearml.conf + echo "{clearml_conf}" >> /root/clearml.conf + export CLEARML_API_HOST={api_server} + export CLEARML_WEB_HOST={web_server} + export CLEARML_FILES_HOST={files_server} export DYNAMIC_INSTANCE_ID=`curl http://169.254.169.254/latest/meta-data/instance-id` - export TRAINS_WORKER_ID={worker_id}:$DYNAMIC_INSTANCE_ID - export TRAINS_API_ACCESS_KEY='{access_key}' - export TRAINS_API_SECRET_KEY='{secret_key}' + export CLEARML_WORKER_ID={worker_id}:$DYNAMIC_INSTANCE_ID + export CLEARML_API_ACCESS_KEY='{access_key}' + export CLEARML_API_SECRET_KEY='{secret_key}' {bash_script} source ~/.bashrc - python -m trains_agent --config-file '/root/trains.conf' daemon --queue '{queue}' {docker} + python -m clearml_agent --config-file '/root/clearml.conf' daemon --queue '{queue}' {docker} shutdown """.format( api_server=self.api_server, @@ -87,7 +87,7 @@ class AwsAutoScaler(AutoScaler): queue=queue_name, git_user=self.git_user or "", git_pass=self.git_pass or "", - trains_conf='\\"'.join(self.extra_trains_conf.split('"')), + clearml_conf='\\"'.join(self.extra_trains_conf.split('"')), bash_script=self.extra_vm_bash_script, docker="--docker '{}'".format(self.default_docker_image) if self.default_docker_image diff --git a/clearml/automation/controller.py b/clearml/automation/controller.py index 0625b2b2..365e41ac 100644 --- a/clearml/automation/controller.py +++ b/clearml/automation/controller.py @@ -17,7 +17,7 @@ class PipelineController(object): """ Pipeline controller. Pipeline is a DAG of base tasks, each task will be cloned (arguments changed as required) executed and monitored - The pipeline process (task) itself can be executed manually or by the trains-agent services queue. + The pipeline process (task) itself can be executed manually or by the clearml-agent services queue. Notice: The pipeline controller lives as long as the pipeline itself is being executed. """ _tag = 'pipeline' @@ -601,7 +601,7 @@ class PipelineController(object): print('Parameters:\n{}'.format(self._nodes[name].job.task_parameter_override)) self._running_nodes.append(name) else: - getLogger('trains.automation.controller').error( + getLogger('clearml.automation.controller').error( 'ERROR: Failed launching step \'{}\': {}'.format(name, self._nodes[name])) # update current state (in configuration, so that we could later continue an aborted pipeline) diff --git a/clearml/automation/job.py b/clearml/automation/job.py index cc2d9a92..344b976b 100644 --- a/clearml/automation/job.py +++ b/clearml/automation/job.py @@ -8,7 +8,7 @@ from ..task import Task from ..backend_api.services import tasks as tasks_service -logger = getLogger('trains.automation.job') +logger = getLogger('clearml.automation.job') class TrainsJob(object): diff --git a/clearml/automation/monitor.py b/clearml/automation/monitor.py index 7c751500..dff7820a 100644 --- a/clearml/automation/monitor.py +++ b/clearml/automation/monitor.py @@ -22,7 +22,7 @@ class Monitor(object): self._project_ids = None self._projects = None self._projects_refresh_timestamp = None - self._trains_apiclient = None + self._clearml_apiclient = None def set_projects(self, project_names=None, project_names_re=None, project_ids=None): # type: (Optional[Sequence[str]], Optional[Sequence[str]], Optional[Sequence[str]]) -> () @@ -167,10 +167,10 @@ class Monitor(object): def _get_api_client(self): # type: () -> APIClient """ - Return an APIClient object to directly query the trains-server + Return an APIClient object to directly query the clearml-server :return: APIClient object """ - if not self._trains_apiclient: - self._trains_apiclient = APIClient() - return self._trains_apiclient + if not self._clearml_apiclient: + self._clearml_apiclient = APIClient() + return self._clearml_apiclient diff --git a/clearml/automation/optimization.py b/clearml/automation/optimization.py index ac78c42f..f860642c 100644 --- a/clearml/automation/optimization.py +++ b/clearml/automation/optimization.py @@ -15,7 +15,7 @@ from ..logger import Logger from ..backend_api.services import workers as workers_service, tasks as tasks_services from ..task import Task -logger = getLogger('trains.automation.optimization') +logger = getLogger('clearml.automation.optimization') try: @@ -878,9 +878,9 @@ class HyperParameterOptimizer(object): :linenos: :caption: Example - from trains import Task - from trains.automation import UniformParameterRange, DiscreteParameterRange - from trains.automation import GridSearch, RandomSearch, HyperParameterOptimizer + from clearml import Task + from clearml.automation import UniformParameterRange, DiscreteParameterRange + from clearml.automation import GridSearch, RandomSearch, HyperParameterOptimizer task = Task.init('examples', 'HyperParameterOptimizer example') an_optimizer = HyperParameterOptimizer( diff --git a/clearml/backend_api/api_proxy.py b/clearml/backend_api/api_proxy.py index 164392e0..22281cfd 100644 --- a/clearml/backend_api/api_proxy.py +++ b/clearml/backend_api/api_proxy.py @@ -8,7 +8,7 @@ from ..utilities.check_updates import Version class ApiServiceProxy(object): - _main_services_module = "trains.backend_api.services" + _main_services_module = "clearml.backend_api.services" _available_versions = None def __init__(self, module): diff --git a/clearml/backend_api/config/default/api.conf b/clearml/backend_api/config/default/api.conf index e4d88bee..721f6434 100644 --- a/clearml/backend_api/config/default/api.conf +++ b/clearml/backend_api/config/default/api.conf @@ -1,16 +1,16 @@ { version: 1.5 - # default api_server: https://demoapi.trains.allegro.ai + # default api_server: https://demoapi.clearml.allegro.ai api_server: "" - # default web_server: https://demoapp.trains.allegro.ai + # default web_server: https://demoapp.clearml.allegro.ai web_server: "" - # default files_server: https://demofiles.trains.allegro.ai + # default files_server: https://demofiles.clearml.allegro.ai files_server: "" # verify host ssl certificate, set to False only if you have a very good reason verify_certificate: True - # default demoapi.trains.allegro.ai credentials + # default demoapi.clearml.allegro.ai credentials credentials { access_key: "" secret_key: "" diff --git a/clearml/backend_api/session/client/client.py b/clearml/backend_api/session/client/client.py index 1eefa199..69ce9add 100644 --- a/clearml/backend_api/session/client/client.py +++ b/clearml/backend_api/session/client/client.py @@ -107,15 +107,15 @@ class StrictSession(Session): init() return - original = os.environ.get(LOCAL_CONFIG_FILE_OVERRIDE_VAR, None) + original = LOCAL_CONFIG_FILE_OVERRIDE_VAR.get() or None try: - os.environ[LOCAL_CONFIG_FILE_OVERRIDE_VAR] = str(config_file) + LOCAL_CONFIG_FILE_OVERRIDE_VAR.set(str(config_file)) init() finally: if original is None: - os.environ.pop(LOCAL_CONFIG_FILE_OVERRIDE_VAR, None) + LOCAL_CONFIG_FILE_OVERRIDE_VAR.pop() else: - os.environ[LOCAL_CONFIG_FILE_OVERRIDE_VAR] = original + LOCAL_CONFIG_FILE_OVERRIDE_VAR.set(original) def send(self, request, *args, **kwargs): result = super(StrictSession, self).send(request, *args, **kwargs) @@ -560,4 +560,4 @@ class APIClient(object): for name, module in services.items() }, ) - ) \ No newline at end of file + ) diff --git a/clearml/backend_api/session/defs.py b/clearml/backend_api/session/defs.py index ff78f812..8571ab5d 100644 --- a/clearml/backend_api/session/defs.py +++ b/clearml/backend_api/session/defs.py @@ -2,12 +2,14 @@ from ...backend_config import EnvEntry from ...backend_config.converters import safe_text_to_bool -ENV_HOST = EnvEntry("TRAINS_API_HOST", "ALG_API_HOST") -ENV_WEB_HOST = EnvEntry("TRAINS_WEB_HOST", "ALG_WEB_HOST") -ENV_FILES_HOST = EnvEntry("TRAINS_FILES_HOST", "ALG_FILES_HOST") -ENV_ACCESS_KEY = EnvEntry("TRAINS_API_ACCESS_KEY", "ALG_API_ACCESS_KEY") -ENV_SECRET_KEY = EnvEntry("TRAINS_API_SECRET_KEY", "ALG_API_SECRET_KEY") -ENV_VERBOSE = EnvEntry("TRAINS_API_VERBOSE", "ALG_API_VERBOSE", type=bool, default=False) -ENV_HOST_VERIFY_CERT = EnvEntry("TRAINS_API_HOST_VERIFY_CERT", "ALG_API_HOST_VERIFY_CERT", type=bool, default=True) -ENV_OFFLINE_MODE = EnvEntry("TRAINS_OFFLINE_MODE", "ALG_OFFLINE_MODE", type=bool, converter=safe_text_to_bool) -ENV_TRAINS_NO_DEFAULT_SERVER = EnvEntry("TRAINS_NO_DEFAULT_SERVER", "ALG_NO_DEFAULT_SERVER", type=bool, default=False) +ENV_HOST = EnvEntry("CLEARML_API_HOST", "TRAINS_API_HOST") +ENV_WEB_HOST = EnvEntry("CLEARML_WEB_HOST", "TRAINS_WEB_HOST") +ENV_FILES_HOST = EnvEntry("CLEARML_FILES_HOST", "TRAINS_FILES_HOST") +ENV_ACCESS_KEY = EnvEntry("CLEARML_API_ACCESS_KEY", "TRAINS_API_ACCESS_KEY") +ENV_SECRET_KEY = EnvEntry("CLEARML_API_SECRET_KEY", "TRAINS_API_SECRET_KEY") +ENV_VERBOSE = EnvEntry("CLEARML_API_VERBOSE", "TRAINS_API_VERBOSE", type=bool, default=False) +ENV_HOST_VERIFY_CERT = EnvEntry("CLEARML_API_HOST_VERIFY_CERT", "TRAINS_API_HOST_VERIFY_CERT", + type=bool, default=True) +ENV_OFFLINE_MODE = EnvEntry("CLEARML_OFFLINE_MODE", "TRAINS_OFFLINE_MODE", type=bool, converter=safe_text_to_bool) +ENV_TRAINS_NO_DEFAULT_SERVER = EnvEntry("CLEARML_NO_DEFAULT_SERVER", "TRAINS_NO_DEFAULT_SERVER", + type=bool, default=False) diff --git a/clearml/backend_api/session/session.py b/clearml/backend_api/session/session.py index 34a0efe7..e6da6650 100644 --- a/clearml/backend_api/session/session.py +++ b/clearml/backend_api/session/session.py @@ -36,12 +36,12 @@ class MaxRequestSizeError(Exception): class Session(TokenManager): - """ TRAINS API Session class. """ + """ ClearML API Session class. """ _AUTHORIZATION_HEADER = "Authorization" - _WORKER_HEADER = "X-Trains-Worker" - _ASYNC_HEADER = "X-Trains-Async" - _CLIENT_HEADER = "X-Trains-Client" + _WORKER_HEADER = ("X-ClearML-Worker", "X-Trains-Worker", ) + _ASYNC_HEADER = ("X-ClearML-Async", "X-Trains-Async", ) + _CLIENT_HEADER = ("X-ClearML-Client", "X-Trains-Client", ) _async_status_code = 202 _session_requests = 0 @@ -57,10 +57,10 @@ class Session(TokenManager): _client = [(__package__.partition(".")[0], __version__)] api_version = '2.1' - default_demo_host = "https://demoapi.trains.allegro.ai" + default_demo_host = "https://demoapi.demo.clear.ml" default_host = default_demo_host - default_web = "https://demoapp.trains.allegro.ai" - default_files = "https://demofiles.trains.allegro.ai" + default_web = "https://demoapp.demo.clear.ml" + default_files = "https://demofiles.demo.clear.ml" default_key = "EGRTCO8JMSIGI6S39GTP43NFWXDQOW" default_secret = "x!XTov_G-#vspE*Y(h$Anm&DIc5Ou-F)jsl$PdOyj5wG1&E!Z8" force_max_api_version = None @@ -177,8 +177,8 @@ class Session(TokenManager): if not api_version: api_version = '2.2' if token_dict.get('env', '') == 'prod' else Session.api_version if token_dict.get('server_version'): - if not any(True for c in Session._client if c[0] == 'trains-server'): - Session._client.append(('trains-server', token_dict.get('server_version'), )) + if not any(True for c in Session._client if c[0] == 'clearml-server'): + Session._client.append(('clearml-server', token_dict.get('server_version'), )) Session.api_version = str(api_version) except (jwt.DecodeError, ValueError): @@ -218,10 +218,13 @@ class Session(TokenManager): if self._offline_mode: return None + res = None host = self.host headers = headers.copy() if headers else {} - headers[self._WORKER_HEADER] = self.worker - headers[self._CLIENT_HEADER] = self.client + for h in self._WORKER_HEADER: + headers[h] = self.worker + for h in self._CLIENT_HEADER: + headers[h] = self.client token_refreshed_on_error = False url = ( @@ -308,7 +311,8 @@ class Session(TokenManager): headers.copy() if headers else {} ) if async_enable: - headers[self._ASYNC_HEADER] = "1" + for h in self._ASYNC_HEADER: + headers[h] = "1" return self._send_request( service=service, action=action, @@ -508,7 +512,7 @@ class Session(TokenManager): if parsed.port == 8008: return host.replace(':8008', ':8080', 1) - raise ValueError('Could not detect TRAINS web application server') + raise ValueError('Could not detect ClearML web application server') @classmethod def get_files_server_host(cls, config=None): @@ -624,7 +628,7 @@ class Session(TokenManager): # check if this is a misconfigured api server (getting 200 without the data section) if res and res.status_code == 200: raise ValueError('It seems *api_server* is misconfigured. ' - 'Is this the TRAINS API server {} ?'.format(self.host)) + 'Is this the ClearML API server {} ?'.format(self.host)) else: raise LoginError("Response data mismatch: No 'token' in 'data' value from res, receive : {}, " "exception: {}".format(res, ex)) diff --git a/clearml/backend_api/utils.py b/clearml/backend_api/utils.py index 84d19211..ab35d473 100644 --- a/clearml/backend_api/utils.py +++ b/clearml/backend_api/utils.py @@ -14,7 +14,7 @@ if six.PY3: from functools import lru_cache elif six.PY2: # python 2 support - from backports.functools_lru_cache import lru_cache + from backports.functools_lru_cache import lru_cache # noqa __disable_certificate_verification_warning = 0 @@ -139,7 +139,7 @@ def get_http_session_with_retry( if not session.verify and __disable_certificate_verification_warning < 2: # show warning __disable_certificate_verification_warning += 1 - logging.getLogger('trains').warning( + logging.getLogger('clearml').warning( msg='InsecureRequestWarning: Certificate verification is disabled! Adding ' 'certificate verification is strongly advised. See: ' 'https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings') diff --git a/clearml/backend_config/config.py b/clearml/backend_config/config.py index 52d344b2..b79cb314 100644 --- a/clearml/backend_config/config.py +++ b/clearml/backend_config/config.py @@ -88,7 +88,7 @@ class Config(object): self._folder_name = config_folder or DEFAULT_CONFIG_FOLDER self._roots = [] self._config = ConfigTree() - self._env = env or os.environ.get("TRAINS_ENV", Environment.default) + self._env = env or os.environ.get("CLEARML_ENV", os.environ.get("TRAINS_ENV", Environment.default)) self.config_paths = set() self.is_server = is_server @@ -139,7 +139,7 @@ class Config(object): else: env_config_paths = [] - env_config_path_override = os.environ.get(ENV_CONFIG_PATH_OVERRIDE_VAR) + env_config_path_override = ENV_CONFIG_PATH_OVERRIDE_VAR.get() if env_config_path_override: env_config_paths = [expanduser(env_config_path_override)] @@ -166,7 +166,7 @@ class Config(object): ) local_config_files = LOCAL_CONFIG_FILES - local_config_override = os.environ.get(LOCAL_CONFIG_FILE_OVERRIDE_VAR) + local_config_override = LOCAL_CONFIG_FILE_OVERRIDE_VAR.get() if local_config_override: local_config_files = [expanduser(local_config_override)] diff --git a/clearml/backend_config/defs.py b/clearml/backend_config/defs.py index f503602c..3d0e145e 100644 --- a/clearml/backend_config/defs.py +++ b/clearml/backend_config/defs.py @@ -1,7 +1,10 @@ from os.path import expanduser from pathlib2 import Path -ENV_VAR = 'TRAINS_ENV' +from .environment import EnvEntry + + +ENV_VAR = 'CLEARML_ENV' """ Name of system environment variable that can be used to specify the config environment name """ @@ -25,15 +28,16 @@ LOCAL_CONFIG_PATHS = [ LOCAL_CONFIG_FILES = [ expanduser('~/trains.conf'), # used for workstation configuration (end-users, workers) + expanduser('~/clearml.conf'), # used for workstation configuration (end-users, workers) ] """ Local config files (not paths) """ -LOCAL_CONFIG_FILE_OVERRIDE_VAR = 'TRAINS_CONFIG_FILE' +LOCAL_CONFIG_FILE_OVERRIDE_VAR = EnvEntry("CLEARML_CONFIG_FILE", "TRAINS_CONFIG_FILE") """ Local config file override environment variable. If this is set, no other local config files will be used. """ -ENV_CONFIG_PATH_OVERRIDE_VAR = 'TRAINS_CONFIG_PATH' +ENV_CONFIG_PATH_OVERRIDE_VAR = EnvEntry("CLEARML_CONFIG_PATH", "TRAINS_CONFIG_PATH") """ Environment-related config path override environment variable. If this is set, no other env config path will be used. """ diff --git a/clearml/backend_config/entry.py b/clearml/backend_config/entry.py index 2ca3251c..709d538f 100644 --- a/clearml/backend_config/entry.py +++ b/clearml/backend_config/entry.py @@ -85,9 +85,10 @@ class Entry(object): return self.get_pair(default=default, converter=converter)[1] def set(self, value): - # type: (Any, Any) -> (Text, Any) - key, _ = self.get_pair(default=None, converter=None) - self._set(key, str(value)) + # type: (Any) -> () + # key, _ = self.get_pair(default=None, converter=None) + for k in self.keys: + self._set(k, str(value)) def _set(self, key, value): # type: (Text, Text) -> None diff --git a/clearml/backend_config/environment.py b/clearml/backend_config/environment.py index 915bceb7..ca389b5d 100644 --- a/clearml/backend_config/environment.py +++ b/clearml/backend_config/environment.py @@ -15,6 +15,10 @@ class EnvEntry(Entry): super(EnvEntry, self).__init__(key, *more_keys, **kwargs) self._ignore_errors = kwargs.pop('ignore_errors', False) + def pop(self): + for k in self.keys: + environ.pop(k, None) + def _get(self, key): value = getenv(key, "").strip() return value or NotSet diff --git a/clearml/backend_config/log.py b/clearml/backend_config/log.py index 6dad47bc..5cd6643f 100644 --- a/clearml/backend_config/log.py +++ b/clearml/backend_config/log.py @@ -5,11 +5,11 @@ from pathlib2 import Path def logger(path=None): - name = "trains" + name = "clearml" if path: p = Path(path) module = (p.parent if p.stem.startswith('_') else p).stem - name = "trains.%s" % module + name = "clearml.%s" % module return logging.getLogger(name) diff --git a/clearml/backend_interface/metrics/events.py b/clearml/backend_interface/metrics/events.py index 1640af88..8b84fb29 100644 --- a/clearml/backend_interface/metrics/events.py +++ b/clearml/backend_interface/metrics/events.py @@ -50,6 +50,7 @@ class MetricsEventAdapter(object): url = attr.attrib(default=None) exception = attr.attrib(default=None) + retries = attr.attrib(default=None) delete_local_file = attr.attrib(default=True) """ Local file path, if exists, delete the file after upload completed """ @@ -198,6 +199,7 @@ class UploadEvent(MetricsEventAdapter): _format = '.' + str(config.get('metrics.images.format', 'JPEG')).upper().lstrip('.') _quality = int(config.get('metrics.images.quality', 87)) _subsampling = int(config.get('metrics.images.subsampling', 0)) + _upload_retries = 3 _metric_counters = {} _metric_counters_lock = Lock() @@ -253,7 +255,7 @@ class UploadEvent(MetricsEventAdapter): self._upload_filename += filename_ext self._override_storage_key_prefix = kwargs.pop('override_storage_key_prefix', None) - + self.retries = self._upload_retries super(UploadEvent, self).__init__(metric, variant, iter=iter, **kwargs) @classmethod @@ -334,6 +336,7 @@ class UploadEvent(MetricsEventAdapter): key_prop='key', upload_uri=self._upload_uri, delete_local_file=local_file if self._delete_after_upload else None, + retries=self.retries, ) def get_target_full_upload_uri(self, storage_uri, storage_key_prefix=None, quote_uri=True): diff --git a/clearml/backend_interface/metrics/interface.py b/clearml/backend_interface/metrics/interface.py index f3022e38..c37927f3 100644 --- a/clearml/backend_interface/metrics/interface.py +++ b/clearml/backend_interface/metrics/interface.py @@ -165,10 +165,11 @@ class Metrics(InterfaceBase): try: storage = self._get_storage(upload_uri) + retries = getattr(e, 'retries', None) or self._file_upload_retries if isinstance(e.stream, Path): - url = storage.upload(e.stream.as_posix(), e.url, retries=self._file_upload_retries) + url = storage.upload(e.stream.as_posix(), e.url, retries=retries) else: - url = storage.upload_from_stream(e.stream, e.url, retries=self._file_upload_retries) + url = storage.upload_from_stream(e.stream, e.url, retries=retries) e.event.update(url=url) except Exception as exp: log.warning("Failed uploading to {} ({})".format( diff --git a/clearml/backend_interface/metrics/reporter.py b/clearml/backend_interface/metrics/reporter.py index 949f6f09..7a4639e8 100644 --- a/clearml/backend_interface/metrics/reporter.py +++ b/clearml/backend_interface/metrics/reporter.py @@ -178,7 +178,7 @@ class Reporter(InterfaceBase, AbstractContextManager, SetupUploadMixin, AsyncMan self._report(ev) def report_matplotlib(self, title, series, figure, iter, force_save_as_image=False, logger=None): - from trains.binding.matplotlib_bind import PatchedMatplotlib + from clearml.binding.matplotlib_bind import PatchedMatplotlib PatchedMatplotlib.report_figure( title=title, series=series, diff --git a/clearml/backend_interface/task/log.py b/clearml/backend_interface/task/log.py index bda1bfb8..06ab451b 100644 --- a/clearml/backend_interface/task/log.py +++ b/clearml/backend_interface/task/log.py @@ -62,7 +62,7 @@ class TaskHandler(BufferingHandler): if self._connect_logger and not TaskHandler.__once: base_logger = getLogger() if len(base_logger.handlers) == 1 and isinstance(base_logger.handlers[0], TaskHandler): - if record.name != 'console' and not record.name.startswith('trains.'): + if record.name != 'console' and not record.name.startswith('clearml.'): base_logger.removeHandler(self) basicConfig() base_logger.addHandler(self) @@ -149,7 +149,7 @@ class TaskHandler(BufferingHandler): self._last_event = None batch_requests = events.AddBatchRequest(requests=[events.AddRequest(e) for e in record_events if e]) except Exception: - self.__log_stderr("WARNING: trains.log - Failed logging task to backend ({:d} lines)".format(len(buffer))) + self.__log_stderr("WARNING: clearml.log - Failed logging task to backend ({:d} lines)".format(len(buffer))) batch_requests = None if batch_requests and batch_requests.requests: @@ -253,7 +253,7 @@ class TaskHandler(BufferingHandler): write = sys.stderr._original_write if hasattr(sys.stderr, '_original_write') else sys.stderr.write write('{asctime} - {name} - {levelname} - {message}\n'.format( asctime=Formatter().formatTime(makeLogRecord({})), - name='trains.log', levelname=getLevelName(level), message=msg)) + name='clearml.log', levelname=getLevelName(level), message=msg)) @classmethod def report_offline_session(cls, task, folder): diff --git a/clearml/backend_interface/task/populate.py b/clearml/backend_interface/task/populate.py new file mode 100644 index 00000000..cd1239fb --- /dev/null +++ b/clearml/backend_interface/task/populate.py @@ -0,0 +1,317 @@ +import json +import os +from functools import reduce +from logging import getLogger +from typing import Optional, Sequence + +from six.moves.urllib.parse import urlparse + +from pathlib2 import Path + +from ...task import Task +from .repo import ScriptInfo + + +class CreateAndPopulate(object): + def __init__( + self, + project_name=None, # Optional[str] + task_name=None, # Optional[str] + task_type=None, # Optional[str] + repo=None, # Optional[str] + branch=None, # Optional[str] + commit=None, # Optional[str] + script=None, # Optional[str] + working_directory=None, # Optional[str] + packages=None, # Optional[Sequence[str]] + requirements_file=None, # Optional[Union[str, Path]] + docker=None, # Optional[str] + base_task_id=None, # Optional[str] + add_task_init_call=True, # bool + raise_on_missing_entries=False, # bool + ): + # type: (...) -> None + """ + Create a new Task from an existing code base. + If the code does not already contain a call to Task.init, pass add_task_init_call=True, + and the code will be patched in remote execution (i.e. when executed by `clearml-agent` + + :param project_name: Set the project name for the task. Required if base_task_id is None. + :param task_name: Set the name of the remote task. Required if base_task_id is None. + :param task_type: Optional, The task type to be created. Supported values: 'training', 'testing', 'inference', + 'data_processing', 'application', 'monitor', 'controller', 'optimizer', 'service', 'qc', 'custom' + :param repo: Remote URL for the repository to use, or path to local copy of the git repository + Example: 'https://github.com/allegroai/clearml.git' or '~/project/repo' + :param branch: Select specific repository branch/tag (implies the latest commit from the branch) + :param commit: Select specific commit id to use (default: latest commit, + or when used with local repository matching the local commit id) + :param script: Specify the entry point script for the remote execution. When used in tandem with + remote git repository the script should be a relative path inside the repository, + for example: './source/train.py' . When used with local repository path it supports a + direct path to a file inside the local repository itself, for example: '~/project/source/train.py' + :param working_directory: Working directory to launch the script from. Default: repository root folder. + Relative to repo root or local folder. + :param packages: Manually specify a list of required packages. Example: ["tqdm>=2.1", "scikit-learn"] + :param requirements_file: Specify requirements.txt file to install when setting the session. + If not provided, the requirements.txt from the repository will be used. + :param docker: Select the docker image to be executed in by the remote session + :param base_task_id: Use a pre-existing task in the system, instead of a local repo/script. + Essentially clones an existing task and overrides arguments/requirements. + :param add_task_init_call: If True, a 'Task.init()' call is added to the script entry point in remote execution. + :param raise_on_missing_entries: If True raise ValueError on missing entries when populating + """ + if len(urlparse(repo).scheme) <= 1: + folder = repo + repo = None + else: + folder = None + + if raise_on_missing_entries and not base_task_id: + if not script: + raise ValueError("Entry point script not provided") + if not repo and not folder and not Path(script).is_file(): + raise ValueError("Repository or script must be provided") + if raise_on_missing_entries and commit and branch: + raise ValueError( + "Specify either a branch/tag or specific commit id, not both (either --commit or --branch)") + if raise_on_missing_entries and not folder and working_directory and working_directory.startswith('/'): + raise ValueError("working directory \'{}\', must be relative to repository root") + + if requirements_file and not Path(requirements_file).is_file(): + raise ValueError("requirements file could not be found \'{}\'") + + self.folder = folder + self.commit = commit + self.branch = branch + self.repo = repo + self.script = script + self.cwd = working_directory + assert not packages or isinstance(packages, (tuple, list)) + self.packages = list(packages) if packages else None + self.requirements_file = Path(requirements_file) if requirements_file else None + self.base_task_id = base_task_id + self.docker = docker + self.add_task_init_call = add_task_init_call + self.project_name = project_name + self.task_name = task_name + self.task_type = task_type + self.task = None + self.raise_on_missing_entries = raise_on_missing_entries + + def create_task(self): + # type: () -> Task + """ + Create the new populated Task + + :return: newly created Task object + """ + local_entry_file = None + repo_info = None + if self.folder or (self.script and Path(self.script).is_file()): + self.folder = os.path.expandvars(os.path.expanduser(self.folder)) if self.folder else None + self.script = os.path.expandvars(os.path.expanduser(self.script)) if self.script else None + self.cwd = os.path.expandvars(os.path.expanduser(self.cwd)) if self.cwd else None + if Path(self.script).is_file(): + entry_point = self.script + else: + entry_point = (Path(self.folder) / self.script).as_posix() + entry_point = os.path.abspath(entry_point) + if not os.path.isfile(entry_point): + raise ValueError("Script entrypoint file \'{}\' could not be found".format(entry_point)) + + local_entry_file = entry_point + repo_info, requirements = ScriptInfo.get( + filepaths=[entry_point], + log=getLogger(), + create_requirements=False, uncommitted_from_remote=True) + + # check if we have no repository and no requirements raise error + if self.raise_on_missing_entries and not self.requirements_file and not self.repo and ( + not repo_info or not repo_info.script or not repo_info.script.get('repository')): + raise ValueError("Standalone script detected \'{}\', but no requirements provided".format(self.script)) + + if self.base_task_id: + print('Cloning task {}'.format(self.base_task_id)) + task = Task.clone(source_task=self.base_task_id, project=Task.get_project_id(self.project_name)) + else: + # noinspection PyProtectedMember + task = Task._create(task_name=self.task_name, project_name=self.project_name, task_type=self.task_type) + # if there is nothing to populate, return + if not any([ + self.folder, self.commit, self.branch, self.repo, self.script, self.cwd, + self.packages, self.requirements_file, self.base_task_id, self.docker + ]): + return task + + task_state = task.export_task() + if 'script' not in task_state: + task_state['script'] = {} + + if repo_info: + task_state['script']['repository'] = repo_info.script['repository'] + task_state['script']['version_num'] = repo_info.script['version_num'] + task_state['script']['branch'] = repo_info.script['branch'] + task_state['script']['diff'] = repo_info.script['diff'] or '' + task_state['script']['working_dir'] = repo_info.script['working_dir'] + task_state['script']['entry_point'] = repo_info.script['entry_point'] + task_state['script']['binary'] = repo_info.script['binary'] + task_state['script']['requirements'] = {} + if self.cwd: + self.cwd = self.cwd + cwd = self.cwd if Path(self.cwd).is_dir() else ( + Path(repo_info.script['repo_root']) / self.cwd).as_posix() + if not Path(cwd).is_dir(): + raise ValueError("Working directory \'{}\' could not be found".format(cwd)) + cwd = Path(cwd).relative_to(repo_info.script['repo_root']).as_posix() + entry_point = \ + Path(repo_info.script['repo_root']) / repo_info.script['working_dir'] / repo_info.script[ + 'entry_point'] + entry_point = entry_point.relative_to(cwd).as_posix() + task_state['script']['entry_point'] = entry_point + task_state['script']['working_dir'] = cwd + elif self.repo: + # normalize backslashes and remove first one + entry_point = '/'.join([p for p in self.script.split('/') if p and p != '.']) + cwd = '/'.join([p for p in (self.cwd or '.').split('/') if p and p != '.']) + if cwd and entry_point.startswith(cwd + '/'): + entry_point = entry_point[len(cwd) + 1:] + task_state['script']['repository'] = self.repo + task_state['script']['version_num'] = self.commit or None + task_state['script']['branch'] = self.branch or None + task_state['script']['diff'] = '' + task_state['script']['working_dir'] = cwd or '.' + task_state['script']['entry_point'] = entry_point + + # update requirements + reqs = [] + if self.requirements_file: + with open(self.requirements_file.as_posix(), 'rt') as f: + reqs = [line.strip() for line in f.readlines()] + if self.packages: + reqs += self.packages + if reqs: + # make sure we have clearml. + clearml_found = False + for line in reqs: + if line.strip().startswith('#'): + continue + package = reduce(lambda a, b: a.split(b)[0], "#;@=~<>", line).strip() + if package == 'clearml': + clearml_found = True + break + if not clearml_found: + reqs.append('clearml') + task_state['script']['requirements'] = {'pip': '\n'.join(reqs)} + elif not self.repo and repo_info: + # we are in local mode, make sure we have "requirements.txt" it is a must + reqs_txt_file = Path(repo_info.script['repo_root']) / "requirements.txt" + if self.raise_on_missing_entries and not reqs_txt_file.is_file(): + raise ValueError( + "requirements.txt not found [{}] " + "Use --requirements or --packages".format(reqs_txt_file.as_posix())) + + if self.add_task_init_call: + script_entry = os.path.abspath('/' + task_state['script']['working_dir'] + + '/' + task_state['script']['entry_point']) + idx_a = 0 + # find the right entry for the patch if we have a local file (basically after __future__ + if local_entry_file: + with open(local_entry_file, 'rt') as f: + lines = f.readlines() + future_found = -1 + for i, line in enumerate(lines): + tokens = [t.strip() for t in line.split(' ') if t.strip()] + if tokens and tokens[0] in ('import', 'from',): + if '__future__' in line: + future_found = i + else: + break + if future_found >= 0: + idx_a = future_found + 1 + + task_init_patch = '' + # if we do not have requirements, add clearml to the requirements.txt + if not reqs: + task_init_patch += \ + "diff --git a/requirements.txt b/requirements.txt\n" \ + "--- a/requirements.txt\n" \ + "+++ b/requirements.txt\n" \ + "@@ -0,0 +1,1 @@\n" \ + "+clearml\n" + + task_init_patch += \ + "diff --git a{script_entry} b{script_entry}\n" \ + "--- a{script_entry}\n" \ + "+++ b{script_entry}\n" \ + "@@ -{idx_a},0 +{idx_b},3 @@\n" \ + "+from clearml import Task\n" \ + "+Task.init()\n" \ + "+\n".format( + script_entry=script_entry, idx_a=idx_a, idx_b=idx_a + 1) + + task_state['script']['diff'] = task_init_patch + task_state['script']['diff'] + + # set base docker image if provided + if self.docker: + task.set_base_docker(self.docker) + + if task_state['script']['repository']: + repo_details = {k: v for k, v in task_state['script'].items() + if v and k not in ('diff', 'requirements', 'binary')} + print('Repository Detected\n{}'.format(json.dumps(repo_details, indent=2))) + else: + print('Standalone script detected\n Script: {}\n: Requirements: {}'.format( + self.script, task_state['script']['requirements'].get('pip', []))) + + if task_state['script'].get('requirements') and task_state['script']['requirements'].get('pip'): + print('Requirements:\n requirements.txt: {}\n Additional Packages:{}'.format( + self.requirements_file.as_posix().name if self.requirements_file else '', self.packages)) + if self.docker: + print('Base docker image: {}'.format(self.docker)) + + # update the Task + task.update_task(task_state) + self.task = task + return task + + def update_task_args(self, args=None): + # type: (Optional[Sequence[str]]) -> () + """ + Update the newly created Task argparse Arguments + If called before Task created, used for argument verification + + :param args: Arguments to pass to the remote execution, list of string pairs (argument, value) or + list of strings '='. Example: ['lr=0.003', (batch_size, 64)] + """ + if not args: + return + + # check args are in format = + args_list = [] + for a in args: + if isinstance(a, (list, tuple)): + assert len(a) == 2 + args_list.append(a) + continue + try: + parts = a.split('=', 1) + assert len(parts) == 2 + args_list.append(parts) + except Exception: + raise ValueError( + "Failed parsing argument \'{}\', arguments must be in \'=\' format") + + if not self.task: + return + + task_params = self.task.get_parameters() + args_list = {'Args/{}'.format(k): v for k, v in args_list} + task_params.update(args_list) + self.task.set_parameters(task_params) + + def get_id(self): + # type: () -> Optional[str] + """ + :return: Return the created Task id (str) + """ + return self.task.id if self.task else None diff --git a/clearml/backend_interface/task/repo/scriptinfo.py b/clearml/backend_interface/task/repo/scriptinfo.py index 977fa426..3f0dd2fc 100644 --- a/clearml/backend_interface/task/repo/scriptinfo.py +++ b/clearml/backend_interface/task/repo/scriptinfo.py @@ -52,21 +52,21 @@ class ScriptRequirements(object): try: # noinspection PyPackageRequirements,PyUnresolvedReferences import boto3 # noqa: F401 - modules.add('boto3', 'trains.storage', 0) + modules.add('boto3', 'clearml.storage', 0) except Exception: pass # noinspection PyBroadException try: # noinspection PyPackageRequirements,PyUnresolvedReferences from google.cloud import storage # noqa: F401 - modules.add('google_cloud_storage', 'trains.storage', 0) + modules.add('google_cloud_storage', 'clearml.storage', 0) except Exception: pass # noinspection PyBroadException try: # noinspection PyPackageRequirements,PyUnresolvedReferences from azure.storage.blob import ContentSettings # noqa: F401 - modules.add('azure_storage_blob', 'trains.storage', 0) + modules.add('azure_storage_blob', 'clearml.storage', 0) except Exception: pass @@ -100,7 +100,7 @@ class ScriptRequirements(object): from ..task import Task # noinspection PyProtectedMember for package, version in Task._force_requirements.items(): - modules.add(package, 'trains', 0) + modules.add(package, 'clearml', 0) except Exception: pass @@ -265,7 +265,7 @@ class _JupyterObserver(object): @classmethod def _daemon(cls, jupyter_notebook_filename): - from trains import Task + from clearml import Task # load jupyter notebook package # noinspection PyBroadException @@ -715,12 +715,12 @@ class ScriptInfo(object): jupyter_filepath=jupyter_filepath, ) - if repo_info.modified: - messages.append( - "======> WARNING! UNCOMMITTED CHANGES IN REPOSITORY {} <======".format( - script_info.get("repository", "") - ) - ) + # if repo_info.modified: + # messages.append( + # "======> WARNING! UNCOMMITTED CHANGES IN REPOSITORY {} <======".format( + # script_info.get("repository", "") + # ) + # ) if not any(script_info.values()): script_info = None diff --git a/clearml/backend_interface/task/task.py b/clearml/backend_interface/task/task.py index 96ad742c..c9944e88 100644 --- a/clearml/backend_interface/task/task.py +++ b/clearml/backend_interface/task/task.py @@ -27,6 +27,7 @@ from six.moves.urllib.parse import quote from ...utilities.locks import RLock as FileRLock from ...utilities.attrs import readonly +from ...utilities.proxy_object import verify_basic_type from ...binding.artifacts import Artifacts from ...backend_interface.task.development.worker import DevWorker from ...backend_api import Session @@ -144,9 +145,9 @@ class Task(IdObjectBase, AccessMixin, SetupUploadMixin): self.__reporter = None self._curr_label_stats = {} self._raise_on_validation_errors = raise_on_validation_errors - self._parameters_allowed_types = ( + self._parameters_allowed_types = tuple(set( six.string_types + six.integer_types + (six.text_type, float, list, tuple, dict, type(None)) - ) + )) self._app_server = None self._files_server = None self._initial_iteration_offset = 0 @@ -216,7 +217,7 @@ class Task(IdObjectBase, AccessMixin, SetupUploadMixin): ) else: self.get_logger().report_text( - 'TRAINS new version available: upgrade to v{} is recommended!'.format( + 'ClearML new version available: upgrade to v{} is recommended!'.format( latest_version[0]), ) except Exception: @@ -296,8 +297,8 @@ class Task(IdObjectBase, AccessMixin, SetupUploadMixin): if task_type.value not in (self.TaskTypes.training, self.TaskTypes.testing) and \ not Session.check_min_api_version('2.8'): print('WARNING: Changing task type to "{}" : ' - 'trains-server does not support task type "{}", ' - 'please upgrade trains-server.'.format(self.TaskTypes.training, task_type.value)) + 'clearml-server does not support task type "{}", ' + 'please upgrade clearml-server.'.format(self.TaskTypes.training, task_type.value)) task_type = self.TaskTypes.training project_id = None @@ -402,7 +403,7 @@ class Task(IdObjectBase, AccessMixin, SetupUploadMixin): # type: () -> str """ The Task's status. To keep the Task updated. - Trains reloads the Task status information only, when this value is accessed. + ClearML reloads the Task status information only, when this value is accessed. return str: TaskStatusEnum status """ @@ -445,7 +446,7 @@ class Task(IdObjectBase, AccessMixin, SetupUploadMixin): def reload(self): # type: () -> () """ - Reload current Task's state from trains-server. + Reload current Task's state from clearml-server. Refresh all task's fields, including artifacts / models / parameters etc. """ return super(Task, self).reload() @@ -628,9 +629,9 @@ class Task(IdObjectBase, AccessMixin, SetupUploadMixin): ): # type: (...) -> str """ - Update the Task's output model weights file. First, Trains uploads the file to the preconfigured output + Update the Task's output model weights file. First, ClearML uploads the file to the preconfigured output destination (see the Task's ``output.destination`` property or call the ``setup_upload`` method), - then Trains updates the model object associated with the Task an API call. The API call uses with the URI + then ClearML updates the model object associated with the Task an API call. The API call uses with the URI of the uploaded file, and other values provided by additional arguments. :param str model_file: The path to the updated model weights file. @@ -684,19 +685,19 @@ class Task(IdObjectBase, AccessMixin, SetupUploadMixin): Set a new input model for the Task. The model must be "ready" (status is ``Published``) to be used as the Task's input model. - :param model_id: The Id of the model on the **Trains Server** (backend). If ``model_name`` is not specified, + :param model_id: The Id of the model on the **ClearML Server** (backend). If ``model_name`` is not specified, then ``model_id`` must be specified. - :param model_name: The model name. The name is used to locate an existing model in the **Trains Server** + :param model_name: The model name. The name is used to locate an existing model in the **ClearML Server** (backend). If ``model_id`` is not specified, then ``model_name`` must be specified. :param update_task_design: Update the Task's design - - ``True`` - Trains copies the Task's model design from the input model. - - ``False`` - Trains does not copy the Task's model design from the input model. + - ``True`` - ClearML copies the Task's model design from the input model. + - ``False`` - ClearML does not copy the Task's model design from the input model. :param update_task_labels: Update the Task's label enumeration - - ``True`` - Trains copies the Task's label enumeration from the input model. - - ``False`` - Trains does not copy the Task's label enumeration from the input model. + - ``True`` - ClearML copies the Task's label enumeration from the input model. + - ``False`` - ClearML does not copy the Task's label enumeration from the input model. """ if model_id is None and not model_name: raise ValueError('Expected one of [model_id, model_name]') @@ -749,7 +750,7 @@ class Task(IdObjectBase, AccessMixin, SetupUploadMixin): i.e. {'Args/param': 'value'} is the argument "param" from section "Args" :param backwards_compatibility: If True (default) parameters without section name - (API version < 2.9, trains-server < 0.16) will be at dict root level. + (API version < 2.9, clearml-server < 0.16) will be at dict root level. If False, parameters without section name, will be nested under "Args/" key. :return: dict of the task parameters, all flattened to key/value. @@ -838,14 +839,15 @@ class Task(IdObjectBase, AccessMixin, SetupUploadMixin): not_allowed = { k: type(v).__name__ for k, v in new_parameters.items() - if not isinstance(v, self._parameters_allowed_types) + if not verify_basic_type(v, self._parameters_allowed_types) } if not_allowed: - raise ValueError( - "Only builtin types ({}) are allowed for values (got {})".format( - ', '.join(t.__name__ for t in self._parameters_allowed_types), - ', '.join('%s=>%s' % p for p in not_allowed.items())), + self.log.warning( + "Skipping parameter: {}, only builtin types are supported ({})".format( + ', '.join('%s[%s]' % p for p in not_allowed.items()), + ', '.join(t.__name__ for t in self._parameters_allowed_types)) ) + new_parameters = {k: v for k, v in new_parameters.items() if k not in not_allowed} use_hyperparams = Session.check_min_api_version('2.9') @@ -958,7 +960,7 @@ class Task(IdObjectBase, AccessMixin, SetupUploadMixin): :return: True if the parameter was deleted successfully """ if not Session.check_min_api_version('2.9'): - raise ValueError("Delete hyper parameter is not supported by your trains-server, " + raise ValueError("Delete hyper parameter is not supported by your clearml-server, " "upgrade to the latest version") with self._edit_lock: paramkey = tasks.ParamKey(section=name.split('/', 1)[0], name=name.split('/', 1)[1]) @@ -1011,7 +1013,7 @@ class Task(IdObjectBase, AccessMixin, SetupUploadMixin): # type: (str) -> () """ Set the base docker image for this experiment - If provided, this value will be used by trains-agent to execute this experiment + If provided, this value will be used by clearml-agent to execute this experiment inside the provided docker image. When running remotely the call is ignored """ @@ -1275,7 +1277,7 @@ class Task(IdObjectBase, AccessMixin, SetupUploadMixin): # type: () -> str """ Return the Task results & outputs web page address. - For example: https://demoapp.trains.allegro.ai/projects/216431/experiments/60763e04/output/log + For example: https://demoapp.demo.clear.ml/projects/216431/experiments/60763e04/output/log :return: http/s URL link. """ @@ -1428,7 +1430,7 @@ class Task(IdObjectBase, AccessMixin, SetupUploadMixin): def running_locally(): # type: () -> bool """ - Is the task running locally (i.e., ``trains-agent`` is not executing it) + Is the task running locally (i.e., ``clearml-agent`` is not executing it) :return: True, if the task is running locally. False, if the task is not running locally. @@ -1637,7 +1639,7 @@ class Task(IdObjectBase, AccessMixin, SetupUploadMixin): mutually_exclusive(config_dict=config_dict, config_text=config_text, _check_none=True) if not Session.check_min_api_version('2.9'): - raise ValueError("Multiple configurations is not supported with the current 'trains-server', " + raise ValueError("Multiple configurations is not supported with the current 'clearml-server', " "please upgrade to the latest version") if description: @@ -1661,7 +1663,7 @@ class Task(IdObjectBase, AccessMixin, SetupUploadMixin): return None if configuration name is not valid. """ if not Session.check_min_api_version('2.9'): - raise ValueError("Multiple configurations is not supported with the current 'trains-server', " + raise ValueError("Multiple configurations is not supported with the current 'clearml-server', " "please upgrade to the latest version") configuration = self.data.configuration or {} @@ -1725,6 +1727,22 @@ class Task(IdObjectBase, AccessMixin, SetupUploadMixin): """ session = session if session else cls._get_default_session() + use_clone_api = Session.check_min_api_version('2.9') + if use_clone_api: + res = cls._send( + session=session, log=log, + req=tasks.CloneRequest( + task=cloned_task_id, + new_task_name=name, + new_task_tags=tags, + new_task_comment=comment, + new_task_parent=parent, + new_task_project=project, + execution_overrides=execution_overrides, + ) + ) + cloned_task_id = res.response.id + return cloned_task_id res = cls._send(session=session, log=log, req=tasks.GetByIdRequest(task=cloned_task_id)) task = res.response.task @@ -1858,7 +1876,7 @@ class Task(IdObjectBase, AccessMixin, SetupUploadMixin): if not PROC_MASTER_ID_ENV_VAR.get() or len(PROC_MASTER_ID_ENV_VAR.get().split(':')) < 2: self.__edit_lock = RLock() elif PROC_MASTER_ID_ENV_VAR.get().split(':')[1] == str(self.id): - filename = os.path.join(gettempdir(), 'trains_{}.lock'.format(self.id)) + filename = os.path.join(gettempdir(), 'clearml_{}.lock'.format(self.id)) # no need to remove previous file lock if we have a dead process, it will automatically release the lock. # # noinspection PyBroadException # try: diff --git a/clearml/binding/artifacts.py b/clearml/binding/artifacts.py index 85f6588c..ffa043f5 100644 --- a/clearml/binding/artifacts.py +++ b/clearml/binding/artifacts.py @@ -187,7 +187,7 @@ class Artifact(object): :raise: Raises error if local copy not found. :return: A local path to a downloaded copy of the artifact. """ - from trains.storage import StorageManager + from clearml.storage import StorageManager local_copy = StorageManager.get_local_copy( remote_url=self.url, extract_archive=extract_archive and self.type == 'archive', @@ -308,7 +308,7 @@ class Artifacts(object): delete_after_upload=False, auto_pickle=True, wait_on_upload=False): # type: (str, Optional[object], Optional[dict], Optional[str], bool, bool, bool) -> bool if not Session.check_min_api_version('2.3'): - LoggerRoot.get_base_logger().warning('Artifacts not supported by your TRAINS-server version, ' + LoggerRoot.get_base_logger().warning('Artifacts not supported by your ClearML-server version, ' 'please upgrade to the latest server version') return False @@ -648,7 +648,7 @@ class Artifacts(object): return self._last_artifacts_upload[name] = current_sha2 - # If old trains-server, upload as debug image + # If old clearml-server, upload as debug image if not Session.check_min_api_version('2.3'): logger.report_image(title='artifacts', series=name, local_path=local_csv.as_posix(), delete_after_upload=True, iteration=self._task.get_last_iteration(), @@ -698,7 +698,7 @@ class Artifacts(object): """ Upload local file and return uri of the uploaded file (uploading in the background) """ - from trains.storage import StorageManager + from clearml.storage import StorageManager upload_uri = self._task.output_uri or self._task.get_logger().get_default_upload_destination() if not isinstance(local_file, Path): @@ -715,7 +715,7 @@ class Artifacts(object): # send for upload # noinspection PyProtectedMember if wait_on_upload: - StorageManager.upload_file(local_file.as_posix(), uri) + StorageManager.upload_file(local_file.as_posix(), uri, wait_for_upload=True, retries=ev.retries) if delete_after_upload: try: os.unlink(local_file.as_posix()) diff --git a/clearml/binding/environ_bind.py b/clearml/binding/environ_bind.py index b2902b7f..af52343c 100644 --- a/clearml/binding/environ_bind.py +++ b/clearml/binding/environ_bind.py @@ -44,7 +44,7 @@ class EnvironmentBind(object): match = match.strip() if match == '*': env_param.update({k: os.environ.get(k) for k in os.environ - if not k.startswith('TRAINS_') and not k.startswith('ALG_')}) + if not k.startswith('TRAINS_') and not k.startswith('CLEARML_')}) elif match.endswith('*'): match = match.strip('*') env_param.update({k: os.environ.get(k) for k in os.environ if k.startswith(match)}) diff --git a/clearml/binding/frameworks/__init__.py b/clearml/binding/frameworks/__init__.py index b658f8b2..2a24ab83 100644 --- a/clearml/binding/frameworks/__init__.py +++ b/clearml/binding/frameworks/__init__.py @@ -114,7 +114,7 @@ class WeightsFileHandler(object): Add a pre-save/load callback for weights files and return its handle. If the callback was already added, return the existing handle. - Use this callback to modify the weights filename registered in the Trains Server. In case Trains is + Use this callback to modify the weights filename registered in the ClearML Server. In case ClearML is configured to upload the weights file, this will affect the uploaded filename as well. Callback returning None will disable the tracking of the current call Model save, it will not disable saving it to disk, just the logging/tracking/uploading. @@ -422,7 +422,7 @@ class WeightsFileHandler(object): # HACK: if pytorch-lightning is used, remove the temp '.part' file extension if sys.modules.get('pytorch_lightning') and target_filename.lower().endswith('.part'): target_filename = target_filename[:-len('.part')] - fd, temp_file = mkstemp(prefix='.trains.upload_model_', suffix='.tmp') + fd, temp_file = mkstemp(prefix='.clearml.upload_model_', suffix='.tmp') os.close(fd) shutil.copy(files[0], temp_file) trains_out_model.update_weights( diff --git a/clearml/binding/frameworks/tensorflow_bind.py b/clearml/binding/frameworks/tensorflow_bind.py index a9b00267..8012636f 100644 --- a/clearml/binding/frameworks/tensorflow_bind.py +++ b/clearml/binding/frameworks/tensorflow_bind.py @@ -192,7 +192,7 @@ class WeightsGradientHistHelper(object): class EventTrainsWriter(object): """ TF SummaryWriter implementation that converts the tensorboard's summary into - Trains events and reports the events (metrics) for an Trains task (logger). + ClearML events and reports the events (metrics) for an ClearML task (logger). """ _add_lock = threading.RLock() _series_name_lookup = {} @@ -298,8 +298,8 @@ class EventTrainsWriter(object): def __init__(self, logger, logdir=None, report_freq=100, image_report_freq=None, histogram_update_freq_multiplier=10, histogram_granularity=50, max_keep_images=None): """ - Create a compatible Trains backend to the TensorFlow SummaryToEventTransformer - Everything will be serialized directly to the Trains backend, instead of to the standard TF FileWriter + Create a compatible ClearML backend to the TensorFlow SummaryToEventTransformer + Everything will be serialized directly to the ClearML backend, instead of to the standard TF FileWriter :param logger: The task.logger to use for sending the metrics (def: task.get_logger()) :param report_freq: How often to update the statistics values @@ -846,7 +846,7 @@ class PatchSummaryToEventTransformer(object): if PatchSummaryToEventTransformer.__original_getattribute is None: PatchSummaryToEventTransformer.__original_getattribute = SummaryToEventTransformer.__getattribute__ SummaryToEventTransformer.__getattribute__ = PatchSummaryToEventTransformer._patched_getattribute - setattr(SummaryToEventTransformer, 'trains', + setattr(SummaryToEventTransformer, 'clearml', property(PatchSummaryToEventTransformer.trains_object)) except Exception as ex: LoggerRoot.get_base_logger(TensorflowBinding).debug(str(ex)) @@ -859,7 +859,7 @@ class PatchSummaryToEventTransformer(object): from torch.utils.tensorboard.writer import FileWriter as FileWriterT # noqa PatchSummaryToEventTransformer._original_add_eventT = FileWriterT.add_event FileWriterT.add_event = PatchSummaryToEventTransformer._patched_add_eventT - setattr(FileWriterT, 'trains', None) + setattr(FileWriterT, 'clearml', None) except ImportError: # this is a new version of TensorflowX pass @@ -875,7 +875,7 @@ class PatchSummaryToEventTransformer(object): PatchSummaryToEventTransformer.__original_getattributeX = \ SummaryToEventTransformerX.__getattribute__ SummaryToEventTransformerX.__getattribute__ = PatchSummaryToEventTransformer._patched_getattributeX - setattr(SummaryToEventTransformerX, 'trains', + setattr(SummaryToEventTransformerX, 'clearml', property(PatchSummaryToEventTransformer.trains_object)) except ImportError: # this is a new version of TensorflowX @@ -890,7 +890,7 @@ class PatchSummaryToEventTransformer(object): from tensorboardX.writer import FileWriter as FileWriterX # noqa PatchSummaryToEventTransformer._original_add_eventX = FileWriterX.add_event FileWriterX.add_event = PatchSummaryToEventTransformer._patched_add_eventX - setattr(FileWriterX, 'trains', None) + setattr(FileWriterX, 'clearml', None) except ImportError: # this is a new version of TensorflowX pass @@ -899,38 +899,38 @@ class PatchSummaryToEventTransformer(object): @staticmethod def _patched_add_eventT(self, *args, **kwargs): - if not hasattr(self, 'trains') or not PatchSummaryToEventTransformer.__main_task: + if not hasattr(self, 'clearml') or not PatchSummaryToEventTransformer.__main_task: return PatchSummaryToEventTransformer._original_add_eventT(self, *args, **kwargs) - if not self.trains: + if not self.clearml: # noqa # noinspection PyBroadException try: logdir = self.get_logdir() except Exception: logdir = None - self.trains = EventTrainsWriter(PatchSummaryToEventTransformer.__main_task.get_logger(), + self.clearml = EventTrainsWriter(PatchSummaryToEventTransformer.__main_task.get_logger(), logdir=logdir, **PatchSummaryToEventTransformer.defaults_dict) # noinspection PyBroadException try: - self.trains.add_event(*args, **kwargs) + self.clearml.add_event(*args, **kwargs) except Exception: pass return PatchSummaryToEventTransformer._original_add_eventT(self, *args, **kwargs) @staticmethod def _patched_add_eventX(self, *args, **kwargs): - if not hasattr(self, 'trains') or not PatchSummaryToEventTransformer.__main_task: + if not hasattr(self, 'clearml') or not PatchSummaryToEventTransformer.__main_task: return PatchSummaryToEventTransformer._original_add_eventX(self, *args, **kwargs) - if not self.trains: + if not self.clearml: # noinspection PyBroadException try: logdir = self.get_logdir() except Exception: logdir = None - self.trains = EventTrainsWriter(PatchSummaryToEventTransformer.__main_task.get_logger(), + self.clearml = EventTrainsWriter(PatchSummaryToEventTransformer.__main_task.get_logger(), logdir=logdir, **PatchSummaryToEventTransformer.defaults_dict) # noinspection PyBroadException try: - self.trains.add_event(*args, **kwargs) + self.clearml.add_event(*args, **kwargs) except Exception: pass return PatchSummaryToEventTransformer._original_add_eventX(self, *args, **kwargs) @@ -947,17 +947,17 @@ class PatchSummaryToEventTransformer(object): @staticmethod def _patched_getattribute_(self, attr, get_base): - # no main task, zero chance we have an Trains event logger + # no main task, zero chance we have an ClearML event logger if PatchSummaryToEventTransformer.__main_task is None: return get_base(self, attr) - # check if we already have an Trains event logger + # check if we already have an ClearML event logger __dict__ = get_base(self, '__dict__') if 'event_writer' not in __dict__ or \ isinstance(__dict__['event_writer'], (ProxyEventsWriter, EventTrainsWriter)): return get_base(self, attr) - # patch the events writer field, and add a double Event Logger (Trains and original) + # patch the events writer field, and add a double Event Logger (ClearML and original) base_eventwriter = __dict__['event_writer'] # noinspection PyBroadException try: @@ -1062,7 +1062,7 @@ class PatchModelCheckPointCallback(object): if PatchModelCheckPointCallback.__original_getattribute is None and callbacks is not None: PatchModelCheckPointCallback.__original_getattribute = callbacks.ModelCheckpoint.__getattribute__ callbacks.ModelCheckpoint.__getattribute__ = PatchModelCheckPointCallback._patched_getattribute - setattr(callbacks.ModelCheckpoint, 'trains', + setattr(callbacks.ModelCheckpoint, 'clearml', property(PatchModelCheckPointCallback.trains_object)) except Exception as ex: @@ -1072,17 +1072,17 @@ class PatchModelCheckPointCallback(object): def _patched_getattribute(self, attr): get_base = PatchModelCheckPointCallback.__original_getattribute - # no main task, zero chance we have an Trains event logger + # no main task, zero chance we have an ClearML event logger if PatchModelCheckPointCallback.__main_task is None: return get_base(self, attr) - # check if we already have an Trains event logger + # check if we already have an ClearML event logger __dict__ = get_base(self, '__dict__') if 'model' not in __dict__ or \ isinstance(__dict__['model'], _ModelAdapter): return get_base(self, attr) - # patch the events writer field, and add a double Event Logger (Trains and original) + # patch the events writer field, and add a double Event Logger (ClearML and original) base_model = __dict__['model'] defaults_dict = __dict__.get('_trains_defaults') or PatchModelCheckPointCallback.defaults_dict output_model = OutputModel( diff --git a/clearml/cli/__init__.py b/clearml/cli/__init__.py new file mode 100644 index 00000000..8b137891 --- /dev/null +++ b/clearml/cli/__init__.py @@ -0,0 +1 @@ + diff --git a/clearml/cli/config/__init__.py b/clearml/cli/config/__init__.py new file mode 100644 index 00000000..8b137891 --- /dev/null +++ b/clearml/cli/config/__init__.py @@ -0,0 +1 @@ + diff --git a/clearml/config/default/__main__.py b/clearml/cli/config/__main__.py similarity index 87% rename from clearml/config/default/__main__.py rename to clearml/cli/config/__main__.py index 410bacf6..ae4c2abf 100644 --- a/clearml/config/default/__main__.py +++ b/clearml/cli/config/__main__.py @@ -1,4 +1,4 @@ -""" Trains configuration wizard""" +""" ClearML configuration wizard""" from __future__ import print_function import argparse @@ -8,22 +8,23 @@ from pathlib2 import Path from six.moves import input from six.moves.urllib.parse import urlparse -from trains.backend_api.session import Session -from trains.backend_api.session.defs import ENV_HOST -from trains.backend_config.defs import LOCAL_CONFIG_FILES, LOCAL_CONFIG_FILE_OVERRIDE_VAR -from trains.config import config_obj -from trains.utilities.pyhocon import ConfigFactory, ConfigMissingException +from clearml.backend_api.session import Session +from clearml.backend_api.session.defs import ENV_HOST +from clearml.backend_config.defs import LOCAL_CONFIG_FILES, LOCAL_CONFIG_FILE_OVERRIDE_VAR +from clearml.config import config_obj +from clearml.utilities.pyhocon import ConfigFactory, ConfigMissingException description = "\n" \ - "Please create new trains credentials through the profile page in " \ - "your trains web app (e.g. http://localhost:8080/profile)\n" \ + "Please create new clearml credentials through the profile page in " \ + "your clearml web app (e.g. http://localhost:8080/profile) \n"\ + "Or with the free hosted service at https://app.community.clear.ml/profile\n" \ "In the profile page, press \"Create new credentials\", then press \"Copy to clipboard\".\n" \ "\n" \ "Paste copied configuration here:\n" host_description = """ Editing configuration file: {CONFIG_FILE} -Enter the url of the trains-server's Web service, for example: {HOST} +Enter the url of the clearml-server's Web service, for example: {HOST} """ # noinspection PyBroadException @@ -40,7 +41,12 @@ def validate_file(string): def main(): - default_config_file = os.getenv(LOCAL_CONFIG_FILE_OVERRIDE_VAR) or LOCAL_CONFIG_FILES[0] + default_config_file = LOCAL_CONFIG_FILE_OVERRIDE_VAR.get() + if not default_config_file: + for f in LOCAL_CONFIG_FILES: + default_config_file = f + if os.path.exists(os.path.expanduser(os.path.expandvars(f))): + break p = argparse.ArgumentParser(description=__doc__) p.add_argument( @@ -51,16 +57,20 @@ def main(): args = p.parse_args() - print('TRAINS SDK setup process') + print('ClearML SDK setup process') - conf_file = Path(args.file).absolute() + conf_file = Path(os.path.expanduser(args.file)).absolute() if conf_file.exists() and conf_file.is_file() and conf_file.stat().st_size > 0: print('Configuration file already exists: {}'.format(str(conf_file))) print('Leaving setup, feel free to edit the configuration file.') return print(description, end='') sentinel = '' - parse_input = '\n'.join(iter(input, sentinel)) + parse_input = '' + for line in iter(input, sentinel): + parse_input += line+'\n' + if line.rstrip() == '}': + break credentials = None api_server = None web_server = None @@ -104,7 +114,7 @@ def main(): files_host = input_url('File Store Host', files_host) - print('\nTRAINS Hosts configuration:\nWeb App: {}\nAPI: {}\nFile Store: {}\n'.format( + print('\nClearML Hosts configuration:\nWeb App: {}\nAPI: {}\nFile Store: {}\n'.format( web_host, api_host, files_host)) retry = 1 @@ -121,7 +131,7 @@ def main(): # noinspection PyBroadException try: - default_sdk_conf = Path(__file__).parent.absolute() / 'sdk.conf' + default_sdk_conf = Path(__file__).absolute().parents[2] / 'config/default/sdk.conf' with open(str(default_sdk_conf), 'rt') as f: default_sdk = f.read() except Exception: @@ -130,14 +140,14 @@ def main(): # noinspection PyBroadException try: with open(str(conf_file), 'wt') as f: - header = '# TRAINS SDK configuration file\n' \ + header = '# ClearML SDK configuration file\n' \ 'api {\n' \ ' # Notice: \'host\' is the api server (default port 8008), not the web server.\n' \ ' api_server: %s\n' \ ' web_server: %s\n' \ ' files_server: %s\n' \ ' # Credentials are generated using the webapp, %s/profile\n' \ - ' # Override with os environment: TRAINS_API_ACCESS_KEY / TRAINS_API_SECRET_KEY\n' \ + ' # Override with os environment: CLEARML_API_ACCESS_KEY / CLEARML_API_SECRET_KEY\n' \ ' credentials {"access_key": "%s", "secret_key": "%s"}\n' \ '}\n' \ 'sdk ' % (api_host, web_host, files_host, @@ -149,7 +159,7 @@ def main(): return print('\nNew configuration stored in {}'.format(str(conf_file))) - print('TRAINS setup completed successfully.') + print('ClearML setup completed successfully.') def parse_host(parsed_host, allow_input=True): @@ -290,7 +300,7 @@ def verify_url(parse_input): parsed_host = None except Exception: parsed_host = None - print('Could not parse url {}\nEnter your trains-server host: '.format(parse_input), end='') + print('Could not parse url {}\nEnter your clearml-server host: '.format(parse_input), end='') return parsed_host diff --git a/clearml/cli/task/__init__.py b/clearml/cli/task/__init__.py new file mode 100644 index 00000000..8b137891 --- /dev/null +++ b/clearml/cli/task/__init__.py @@ -0,0 +1 @@ + diff --git a/clearml/cli/task/__main__.py b/clearml/cli/task/__main__.py new file mode 100644 index 00000000..61a6ec9e --- /dev/null +++ b/clearml/cli/task/__main__.py @@ -0,0 +1,119 @@ +from argparse import ArgumentParser + +from pathlib2 import Path + +from clearml.backend_interface.task.populate import CreateAndPopulate +from clearml import Task + + +def setup_parser(parser): + parser.add_argument('--version', action='store_true', default=None, + help='Display the clearml-task utility version') + parser.add_argument('--project', type=str, default=None, + help='Required: set the project name for the task. ' + 'If --base-task-id is used, this arguments is optional.') + parser.add_argument('--name', type=str, default=None, required=True, + help='Required: select a name for the remote task') + parser.add_argument('--repo', type=str, default=None, + help='remote URL for the repository to use. ' + 'Example: --repo https://github.com/allegroai/clearml.git') + parser.add_argument('--branch', type=str, default=None, + help='Select specific repository branch/tag (implies the latest commit from the branch)') + parser.add_argument('--commit', type=str, default=None, + help='Select specific commit id to use (default: latest commit, ' + 'or when used with local repository matching the local commit id)') + parser.add_argument('--folder', type=str, default=None, + help='Remotely execute the code in the local folder. ' + 'Notice! It assumes a git repository already exists. ' + 'Current state of the repo (commit id and uncommitted changes) is logged ' + 'and will be replicated on the remote machine') + parser.add_argument('--script', type=str, default=None, + help='Specify the entry point script for the remote execution. ' + 'When used in tandem with --repo the script should be a relative path inside ' + 'the repository, for example: --script source/train.py .' + 'When used with --folder it supports a direct path to a file inside the local ' + 'repository itself, for example: --script ~/project/source/train.py') + parser.add_argument('--cwd', type=str, default=None, + help='Working directory to launch the script from. Default: repository root folder. ' + 'Relative to repo root or local folder') + parser.add_argument('--args', default=None, nargs='*', + help='Arguments to pass to the remote execution, list of = strings.' + 'Currently only argparse arguments are supported. ' + 'Example: --args lr=0.003 batch_size=64') + parser.add_argument('--queue', type=str, default=None, + help='Select the queue to launch the task. ' + 'If not provided a Task will be created but it will not be launched.') + parser.add_argument('--requirements', type=str, default=None, + help='Specify requirements.txt file to install when setting the session. ' + 'If not provided, the requirements.txt from the repository will be used.') + parser.add_argument('--packages', default=None, nargs='*', + help='Manually specify a list of required packages. ' + 'Example: --packages "tqdm>=2.1" "scikit-learn"') + parser.add_argument('--docker', type=str, default=None, + help='Select the docker image to use in the remote session') + parser.add_argument('--skip-task-init', action='store_true', default=None, + help='If set, Task.init() call is not added to the entry point, and is assumed ' + 'to be called in within the script. Default: add Task.init() call entry point script') + parser.add_argument('--base-task-id', type=str, default=None, + help='Use a pre-existing task in the system, instead of a local repo/script. ' + 'Essentially clones an existing task and overrides arguments/requirements.') + + +def cli(): + title = 'ClearML launch - launch any codebase on remote machine running clearml-agent' + print(title) + parser = ArgumentParser(description=title) + setup_parser(parser) + + # get the args + args = parser.parse_args() + + if args.version: + from ...version import __version__ + print('Version {}'.format(__version__)) + exit(0) + + create_populate = CreateAndPopulate( + project_name=args.project, + task_name=args.name, + repo=args.repo or args.folder, + branch=args.branch, + commit=args.commit, + script=args.script, + working_directory=args.cwd, + packages=args.packages, + requirements_file=args.requirements, + base_task_id=args.base_task_id, + add_task_init_call=not args.skip_task_init, + raise_on_missing_entries=True, + ) + # verify args + create_populate.update_task_args(args.args) + + print('Creating new task') + create_populate.create_task() + # update Task args + create_populate.update_task_args(args.args) + + print('New task created id={}'.format(create_populate.get_id())) + if not args.queue: + print('Warning: No queue was provided, leaving task in draft-mode.') + exit(0) + + Task.enqueue(create_populate.task, queue_name=args.queue) + print('Task id={} sent for execution on queue {}'.format(create_populate.get_id(), args.queue)) + print('Execution log at: {}'.format(create_populate.task.get_output_log_web_page())) + + +def main(): + try: + cli() + except KeyboardInterrupt: + print('\nUser aborted') + except Exception as ex: + print('\nError: {}'.format(ex)) + exit(1) + + +if __name__ == '__main__': + main() diff --git a/clearml/config/__init__.py b/clearml/config/__init__.py index 02295e06..107f8135 100644 --- a/clearml/config/__init__.py +++ b/clearml/config/__init__.py @@ -135,7 +135,10 @@ def dev_worker_name(): def __set_is_master_node(): # noinspection PyBroadException try: - force_master_node = os.environ.pop('TRAINS_FORCE_MASTER_NODE', None) + # pop both set the first + env_a = os.environ.pop('CLEARML_FORCE_MASTER_NODE', None) + env_b = os.environ.pop('TRAINS_FORCE_MASTER_NODE', None) + force_master_node = env_a or env_b except Exception: force_master_node = None diff --git a/clearml/config/default/logging.conf b/clearml/config/default/logging.conf index 6b3cb071..458695fa 100644 --- a/clearml/config/default/logging.conf +++ b/clearml/config/default/logging.conf @@ -2,7 +2,7 @@ version: 1 disable_existing_loggers: 0 loggers { - trains { + clearml { level: INFO } boto { diff --git a/clearml/config/default/sdk.conf b/clearml/config/default/sdk.conf index 9f481049..20c19a07 100644 --- a/clearml/config/default/sdk.conf +++ b/clearml/config/default/sdk.conf @@ -1,10 +1,10 @@ { - # TRAINS - default SDK configuration + # ClearML - default SDK configuration storage { cache { # Defaults to system temp folder / cache - default_base_dir: "~/.trains/cache" + default_base_dir: "~/.clearml/cache" } direct_access: [ @@ -93,7 +93,7 @@ google.storage { # # Default project and credentials file # # Will be used when no bucket configuration is found - # project: "trains" + # project: "clearml" # credentials_json: "/path/to/credentials.json" # # Specific credentials per bucket and sub directory @@ -101,7 +101,7 @@ # { # bucket: "my-bucket" # subdir: "path/in/bucket" # Not required - # project: "trains" + # project: "clearml" # credentials_json: "/path/to/credentials.json" # }, # ] @@ -109,7 +109,7 @@ azure.storage { # containers: [ # { - # account_name: "trains" + # account_name: "clearml" # account_key: "secret" # # container_name: # } @@ -150,8 +150,8 @@ # do not analyze the entire repository. force_analyze_entire_repo: false - # If set to true, *trains* update message will not be printed to the console - # this value can be overwritten with os environment variable TRAINS_SUPPRESS_UPDATE_MESSAGE=1 + # If set to true, *clearml* update message will not be printed to the console + # this value can be overwritten with os environment variable CLEARML_SUPPRESS_UPDATE_MESSAGE=1 suppress_update_message: false # If this flag is true (default is false), instead of analyzing the code with Pigar, analyze with `pip freeze` @@ -161,7 +161,7 @@ # of the Hyper-Parameters. # multiple selected variables are supported including the suffix '*'. # For example: "AWS_*" will log any OS environment variable starting with 'AWS_'. - # This value can be overwritten with os environment variable TRAINS_LOG_ENVIRONMENT="[AWS_*, CUDA_VERSION]" + # This value can be overwritten with os environment variable CLEARML_LOG_ENVIRONMENT="[AWS_*, CUDA_VERSION]" # Example: log_os_environments: ["AWS_*", "CUDA_VERSION"] log_os_environments: [] diff --git a/clearml/config/defs.py b/clearml/config/defs.py index 87a9cb5e..8fe85ac4 100644 --- a/clearml/config/defs.py +++ b/clearml/config/defs.py @@ -5,28 +5,28 @@ from ..backend_config.converters import base64_to_text, or_ from pathlib2 import Path SESSION_CACHE_FILE = ".session.json" -DEFAULT_CACHE_DIR = str(Path(tempfile.gettempdir()) / "trains_cache") +DEFAULT_CACHE_DIR = str(Path(tempfile.gettempdir()) / "clearml_cache") -TASK_ID_ENV_VAR = EnvEntry("TRAINS_TASK_ID", "ALG_TASK_ID") -DOCKER_IMAGE_ENV_VAR = EnvEntry("TRAINS_DOCKER_IMAGE", "ALG_DOCKER_IMAGE") -LOG_TO_BACKEND_ENV_VAR = EnvEntry("TRAINS_LOG_TASK_TO_BACKEND", "ALG_LOG_TASK_TO_BACKEND", type=bool) -NODE_ID_ENV_VAR = EnvEntry("TRAINS_NODE_ID", "ALG_NODE_ID", type=int) -PROC_MASTER_ID_ENV_VAR = EnvEntry("TRAINS_PROC_MASTER_ID", "ALG_PROC_MASTER_ID", type=str) -LOG_STDERR_REDIRECT_LEVEL = EnvEntry("TRAINS_LOG_STDERR_REDIRECT_LEVEL", "ALG_LOG_STDERR_REDIRECT_LEVEL") -DEV_WORKER_NAME = EnvEntry("TRAINS_WORKER_NAME", "ALG_WORKER_NAME") -DEV_TASK_NO_REUSE = EnvEntry("TRAINS_TASK_NO_REUSE", "ALG_TASK_NO_REUSE", type=bool) -TASK_LOG_ENVIRONMENT = EnvEntry("TRAINS_LOG_ENVIRONMENT", "ALG_LOG_ENVIRONMENT", type=str) -TRAINS_CACHE_DIR = EnvEntry("TRAINS_CACHE_DIR", "ALG_CACHE_DIR") +TASK_ID_ENV_VAR = EnvEntry("CLEARML_TASK_ID", "TRAINS_TASK_ID") +DOCKER_IMAGE_ENV_VAR = EnvEntry("CLEARML_DOCKER_IMAGE", "TRAINS_DOCKER_IMAGE") +LOG_TO_BACKEND_ENV_VAR = EnvEntry("CLEARML_LOG_TASK_TO_BACKEND", "TRAINS_LOG_TASK_TO_BACKEND", type=bool) +NODE_ID_ENV_VAR = EnvEntry("CLEARML_NODE_ID", "TRAINS_NODE_ID", type=int) +PROC_MASTER_ID_ENV_VAR = EnvEntry("CLEARML_PROC_MASTER_ID", "TRAINS_PROC_MASTER_ID", type=str) +LOG_STDERR_REDIRECT_LEVEL = EnvEntry("CLEARML_LOG_STDERR_REDIRECT_LEVEL", "TRAINS_LOG_STDERR_REDIRECT_LEVEL") +DEV_WORKER_NAME = EnvEntry("CLEARML_WORKER_NAME", "TRAINS_WORKER_NAME") +DEV_TASK_NO_REUSE = EnvEntry("CLEARML_TASK_NO_REUSE", "TRAINS_TASK_NO_REUSE", type=bool) +TASK_LOG_ENVIRONMENT = EnvEntry("CLEARML_LOG_ENVIRONMENT", "TRAINS_LOG_ENVIRONMENT", type=str) +TRAINS_CACHE_DIR = EnvEntry("CLEARML_CACHE_DIR", "TRAINS_CACHE_DIR") -LOG_LEVEL_ENV_VAR = EnvEntry("TRAINS_LOG_LEVEL", "ALG_LOG_LEVEL", converter=or_(int, str)) +LOG_LEVEL_ENV_VAR = EnvEntry("CLEARML_LOG_LEVEL", "TRAINS_LOG_LEVEL", converter=or_(int, str)) -SUPPRESS_UPDATE_MESSAGE_ENV_VAR = EnvEntry("TRAINS_SUPPRESS_UPDATE_MESSAGE", "ALG_SUPPRESS_UPDATE_MESSAGE", type=bool) +SUPPRESS_UPDATE_MESSAGE_ENV_VAR = EnvEntry("CLEARML_SUPPRESS_UPDATE_MESSAGE", "TRAINS_SUPPRESS_UPDATE_MESSAGE", type=bool) # Repository detection -VCS_REPO_TYPE = EnvEntry("TRAINS_VCS_REPO_TYPE", "ALG_VCS_REPO_TYPE", default="git") -VCS_REPOSITORY_URL = EnvEntry("TRAINS_VCS_REPO_URL", "ALG_VCS_REPO_URL") -VCS_COMMIT_ID = EnvEntry("TRAINS_VCS_COMMIT_ID", "ALG_VCS_COMMIT_ID") -VCS_BRANCH = EnvEntry("TRAINS_VCS_BRANCH", "ALG_VCS_BRANCH") -VCS_ROOT = EnvEntry("TRAINS_VCS_ROOT", "ALG_VCS_ROOT") -VCS_STATUS = EnvEntry("TRAINS_VCS_STATUS", "ALG_VCS_STATUS", converter=base64_to_text) -VCS_DIFF = EnvEntry("TRAINS_VCS_DIFF", "ALG_VCS_DIFF", converter=base64_to_text) +VCS_REPO_TYPE = EnvEntry("CLEARML_VCS_REPO_TYPE", "TRAINS_VCS_REPO_TYPE", default="git") +VCS_REPOSITORY_URL = EnvEntry("CLEARML_VCS_REPO_URL", "TRAINS_VCS_REPO_URL") +VCS_COMMIT_ID = EnvEntry("CLEARML_VCS_COMMIT_ID", "TRAINS_VCS_COMMIT_ID") +VCS_BRANCH = EnvEntry("CLEARML_VCS_BRANCH", "TRAINS_VCS_BRANCH") +VCS_ROOT = EnvEntry("CLEARML_VCS_ROOT", "TRAINS_VCS_ROOT") +VCS_STATUS = EnvEntry("CLEARML_VCS_STATUS", "TRAINS_VCS_STATUS", converter=base64_to_text) +VCS_DIFF = EnvEntry("CLEARML_VCS_DIFF", "TRAINS_VCS_DIFF", converter=base64_to_text) diff --git a/clearml/datasets/__init__.py b/clearml/datasets/__init__.py new file mode 100644 index 00000000..84f8c2a8 --- /dev/null +++ b/clearml/datasets/__init__.py @@ -0,0 +1,6 @@ +from .dataset import FileEntry, Dataset + +__all__ = [ + "FileEntry", + "Dataset", +] diff --git a/clearml/datasets/dataset.py b/clearml/datasets/dataset.py new file mode 100644 index 00000000..a8de4c00 --- /dev/null +++ b/clearml/datasets/dataset.py @@ -0,0 +1,1244 @@ +import json +import os +import shutil +from copy import deepcopy, copy +from fnmatch import fnmatch +from multiprocessing import cpu_count +from multiprocessing.pool import ThreadPool +from tempfile import mkstemp, mkdtemp +from typing import Union, Optional, Sequence, List, Dict, Any, Mapping +from zipfile import ZipFile, ZIP_DEFLATED + +import humanfriendly +from attr import attrs, attrib +from pathlib2 import Path + +from .. import Task, StorageManager +from ..backend_api.session.client import APIClient +from ..backend_interface.util import mutually_exclusive, exact_match_regex +from ..debugging.log import LoggerRoot +from ..storage.helper import StorageHelper +from ..storage.cache import CacheManager +from ..storage.util import sha256sum, is_windows, md5text + +try: + from pathlib import Path as _Path # noqa +except ImportError: + _Path = None + + +@attrs +class FileEntry: + relative_path = attrib(default=None, type=str) + hash = attrib(default=None, type=str) + parent_dataset_id = attrib(default=None, type=str) + size = attrib(default=None, type=int) + # cleared when file is uploaded. + local_path = attrib(default=None, type=str) + + def as_dict(self): + # type: () -> Dict + state = dict(relative_path=self.relative_path, hash=self.hash, + parent_dataset_id=self.parent_dataset_id, size=self.size, + **dict([('local_path', self.local_path)] if self.local_path else ())) + return state + + +class Dataset(object): + __private_magic = 42 * 1337 + __state_entry_name = 'state' + __data_entry_name = 'data' + __cache_context = 'datasets' + __tag = 'dataset' + __cache_folder_prefix = 'ds_' + __dataset_folder_template = CacheManager.set_context_folder_lookup(__cache_context, "{0}_archive_{1}") + __preview_max_file_entries = 15000 + __preview_max_size = 5 * 1024 * 1024 + + def __init__(self, _private, task=None, dataset_project=None, dataset_name=None): + # type: (int, Optional[Task], Optional[str], Optional[str]) -> () + """ + Do not use directly! Use Dataset.create(...) or Dataset.get(...) instead. + """ + assert _private == self.__private_magic + # key for the dataset file entries are the relative path within the data + self._dataset_file_entries = {} # type: Dict[str, FileEntry] + # this will create a graph of all the dependencies we have, each entry lists it's own direct parents + self._dependency_graph = {} # type: Dict[str, List[str]] + if not task: + task = Task.create( + project_name=dataset_project, task_name=dataset_name, task_type=Task.TaskTypes.data_processing) + task.set_system_tags((task.get_system_tags() or []) + [self.__tag]) + task.mark_started() + # generate the script section + script = \ + 'from clearml import Dataset\n\n' \ + 'ds = Dataset.create(dataset_project=\'{dataset_project}\', dataset_name=\'{dataset_name}\')\n'.format( + dataset_project=dataset_project, dataset_name=dataset_name) + task.data.script.diff = script + task.data.script.working_dir = '.' + task.data.script.entry_point = 'register_dataset.py' + from clearml import __version__ + task.data.script.requirements = {'pip': 'clearml == {}\n'.format(__version__)} + # noinspection PyProtectedMember + task._edit(script=task.data.script) + + # store current dataset Task + self._task = task + # store current dataset id + self._id = task.id + # store the folder where the dataset was downloaded to + self._local_base_folder = None # type: Optional[Path] + # dirty flag, set True by any function call changing the dataset (regardless of weather it did anything) + self._dirty = False + + @property + def id(self): + # type: () -> str + return self._id + + @property + def file_entries(self): + # type: () -> List[FileEntry] + return list(self._dataset_file_entries.values()) + + @property + def file_entries_dict(self): + # type: () -> Mapping[str, FileEntry] + """ + Notice this call returns an internal representation, do not modify! + :return: dict with relative file path as key, and FileEntry as value + """ + return self._dataset_file_entries + + @property + def project(self): + # type: () -> str + return self._task.get_project_name() + + @property + def name(self): + # type: () -> str + return self._task.name + + @property + def tags(self): + # type: () -> List[str] + return self._task.get_tags() or [] + + @tags.setter + def tags(self, values): + # type: (List[str]) -> () + self._task.set_tags(values or []) + + def add_files(self, + path, # type: Union[str, Path, _Path] + wildcard=None, # type: Optional[Union[str, Sequence[str]]] + local_base_folder=None, # type: Optional[str] + dataset_path=None, # type: Optional[str] + recursive=True, # type: bool + verbose=False # type: bool + ): + # type: (...) -> () + """ + Add a folder into the current dataset. calculate file hash, + and compare against parent, mark files to be uploaded + + :param path: Add a folder/file to the dataset + :param wildcard: add only specific set of files. + Wildcard matching, can be a single string or a list of wildcards) + :param local_base_folder: files will be located based on their relative path from local_base_folder + :param dataset_path: where in the dataset the folder/files should be located + :param recursive: If True match all wildcard files recursively + :param verbose: If True print to console files added/modified + :return: number of files added + """ + self._dirty = True + self._task.get_logger().report_text( + 'Adding files to dataset: {}'.format( + dict(path=path, wildcard=wildcard, local_base_folder=local_base_folder, + dataset_path=dataset_path, recursive=recursive, verbose=verbose)), + print_console=False) + + num_added = self._add_files( + path=path, wildcard=wildcard, local_base_folder=local_base_folder, + dataset_path=dataset_path, recursive=recursive, verbose=verbose) + + # update the task script + self._add_script_call( + 'add_files', path=path, wildcard=wildcard, local_base_folder=local_base_folder, + dataset_path=dataset_path, recursive=recursive) + + self._serialize() + + return num_added + + def remove_files(self, dataset_path=None, recursive=True, verbose=False): + # type: (Optional[str], bool, bool) -> int + """ + Add a folder into the current dataset. calculate file hash, + and compare against parent, mark files to be uploaded + + :param dataset_path: Remove files from the dataset. + The path is always relative to the dataset (e.g 'folder/file.bin') + :param recursive: If True match all wildcard files recursively + :param verbose: If True print to console files removed + :return: Number of files removed + """ + self._dirty = True + self._task.get_logger().report_text( + 'Removing files from dataset: {}'.format( + dict(dataset_path=dataset_path, recursive=recursive, verbose=verbose)), + print_console=False) + + if dataset_path and dataset_path.startswith('/'): + dataset_path = dataset_path[1:] + + num_files = len(self._dataset_file_entries) + org_files = list(self._dataset_file_entries.keys()) if verbose else None + + if not recursive: + self._dataset_file_entries = { + k: v for k, v in self._dataset_file_entries.items() + if not fnmatch(k + '/', dataset_path + '/')} + else: + wildcard = dataset_path.split('/')[-1] + path = dataset_path[:-len(dataset_path)] + '*' + + self._dataset_file_entries = { + k: v for k, v in self._dataset_file_entries.items() + if not (fnmatch(k, path) and fnmatch(k, '*/' + wildcard))} + + if verbose and org_files: + for f in org_files: + if f not in self._dataset_file_entries: + self._task.get_logger().report_text('Remove {}'.format(f)) + + # update the task script + self._add_script_call( + 'remove_files', dataset_path=dataset_path, recursive=recursive) + + self._serialize() + + return num_files - len(self._dataset_file_entries) + + def sync_folder(self, local_path, dataset_path=None, verbose=False): + # type: (Union[Path, _Path, str], Union[Path, _Path, str], bool) -> (int, int) + """ + Synchronize the dataset with a local folder. The dataset is synchronized from the + relative_base_folder (default: dataset root) and deeper with the specified local path. + + :param local_path: Local folder to sync (assumes all files and recursive) + :param dataset_path: Target dataset path to sync with (default the root of the dataset) + :param verbose: If true print to console files added/modified/removed + :return: number of files removed, number of files modified/added + """ + def filter_f(f): + keep = (not f.relative_path.startswith(relative_prefix) or + (local_path / f.relative_path[len(relative_prefix):]).is_file()) + if not keep and verbose: + self._task.get_logger().report_text('Remove {}'.format(f.relative_path)) + return keep + + self._task.get_logger().report_text( + 'Syncing local copy with dataset: {}'.format( + dict(local_path=local_path, dataset_path=dataset_path, verbose=verbose)), + print_console=False) + + self._dirty = True + local_path = Path(local_path) + + # Path().as_posix() will never end with / + relative_prefix = (Path(dataset_path).as_posix() + '/') if dataset_path else '' + + # remove files + num_files = len(self._dataset_file_entries) + self._dataset_file_entries = { + k: f for k, f in self._dataset_file_entries.items() if filter_f(f)} + removed_files = num_files - len(self._dataset_file_entries) + + # add remaining files + added_files = self._add_files(path=local_path, dataset_path=dataset_path, recursive=True, verbose=verbose) + + if verbose: + self._task.get_logger().report_text( + 'Syncing folder {} : {} files removed, {} added / modified'.format( + local_path.as_posix(), removed_files, added_files)) + + # update the task script + self._add_script_call( + 'sync_folder', local_path=local_path, dataset_path=dataset_path) + + self._serialize() + return removed_files, added_files + + def upload(self, show_progress=True, verbose=False, output_url=None, compression=None): + # type: (bool, bool, Optional[str], Optional[str]) -> () + """ + Start file uploading, the function returns when all files are uploaded. + + :param show_progress: If True show upload progress bar + :param verbose: If True print verbose progress report + :param output_url: Target storage for the compressed dataset (default: file server) + Examples: `s3://bucket/data`, `gs://bucket/data` , `azure://bucket/data` , `/mnt/share/data` + :param compression: Compression algorithm for the Zipped dataset file (default: ZIP_DEFLATED) + """ + # set output_url + if output_url: + self._task.output_uri = output_url + + self._task.get_logger().report_text( + 'Uploading dataset files: {}'.format( + dict(show_progress=show_progress, verbose=verbose, output_url=output_url, compression=compression)), + print_console=False) + + fd, zip_file = mkstemp( + prefix='dataset.{}.'.format(self._id), suffix='.zip' + ) + archive_preview = '' + count = 0 + try: + with ZipFile(zip_file, 'w', allowZip64=True, compression=compression or ZIP_DEFLATED) as zf: + for file_entry in self._dataset_file_entries.values(): + if not file_entry.local_path: + # file is located in a different version + continue + filename = Path(file_entry.local_path) + if not filename.is_file(): + LoggerRoot.get_base_logger().warning( + "Could not store dataset file {}. File skipped".format(file_entry.local_path)) + # mark for removal + file_entry.relative_path = None + continue + if verbose: + self._task.get_logger().report_text('Compressing {}'.format(filename.as_posix())) + + relative_file_name = file_entry.relative_path + zf.write(filename.as_posix(), arcname=relative_file_name) + archive_preview += '{} - {}\n'.format( + relative_file_name, humanfriendly.format_size(filename.stat().st_size)) + file_entry.local_path = None + count += 1 + except Exception as e: + # failed uploading folder: + LoggerRoot.get_base_logger().warning( + 'Exception {}\nFailed zipping dataset.'.format(e)) + return False + finally: + os.close(fd) + + zip_file = Path(zip_file) + + if not count: + zip_file.unlink() + LoggerRoot.get_base_logger().warning('No pending files, uploaded aborted') + return False + + archive_preview = 'Dataset archive content [{} files]:\n'.format(count) + archive_preview + + # noinspection PyBroadException + try: + # let's try to rename it + new_zip_file = zip_file.parent / 'dataset.{}.zip'.format(self._id) + zip_file.rename(new_zip_file) + zip_file = new_zip_file + except Exception: + pass + # remove files that could not be zipped, containing Null relative Path + self._dataset_file_entries = {k: v for k, v in self._dataset_file_entries.items() + if v.relative_path is not None} + # start upload + zip_file_size = humanfriendly.format_size(Path(zip_file).stat().st_size) + self._task.get_logger().report_text( + 'Uploading compressed dataset changes ({} files, total {})'.format(count, zip_file_size)) + self._task.upload_artifact( + name=self.__data_entry_name, artifact_object=Path(zip_file), preview=archive_preview, + delete_after_upload=True, wait_on_upload=True) + self._task.get_logger().report_text('Upload completed ({})'.format(zip_file_size)) + + self._add_script_call( + 'upload', show_progress=show_progress, verbose=verbose, output_url=output_url, compression=compression) + + self._dirty = False + self._serialize() + + def finalize(self, verbose=False, raise_on_error=True): + # type: (bool, bool) -> bool + """ + Finalize the dataset (if upload was not called, it will be called automatically) publish dataset Task. + If files need to be uploaded, throw exception (or return False) + + :param verbose: If True print verbose progress report + :param raise_on_error: If True raise exception if dataset finalizing failed + """ + # check we do not have files waiting for upload. + if self._dirty: + if raise_on_error: + raise ValueError("Cannot finalize dataset, pending uploads. Call Dataset.upload(...)") + return False + + self._task.get_logger().report_text('Finalizing dataset', print_console=False) + + # make sure we have no redundant parent versions + self._serialize() + self._add_script_call('finalize') + if verbose: + print('Updating statistics and genealogy') + self._report_dataset_genealogy() + hashed_nodes = [self._get_dataset_id_hash(k) for k in self._dependency_graph.keys()] + self._task.comment = 'Dependencies: {}\n'.format(hashed_nodes) + self._task.close() + self._task.completed() + return True + + def is_final(self): + # type: () -> bool + """ + Return True if the dataset was finalized and cannot be changed any more. + + :return: True if dataset if final + """ + return self._task.get_status() not in ( + Task.TaskStatusEnum.in_progress, Task.TaskStatusEnum.created, Task.TaskStatusEnum.failed) + + def get_local_copy(self, use_soft_links=None, raise_on_error=True): + # type: (bool, bool) -> str + """ + return a base folder with a read-only (immutable) local copy of the entire dataset + download and copy / soft-link, files from all the parent dataset versions + + :param use_soft_links: If True use soft links, default False on windows True on Posix systems + :param raise_on_error: If True raise exception if dataset merging failed on any file + :return: A base folder for the entire dataset + """ + assert self._id + if not self._task: + self._task = Task.get_task(task_id=self._id) + + # now let's merge the parents + target_folder = self._merge_datasets(use_soft_links=use_soft_links, raise_on_error=raise_on_error) + return target_folder + + def get_mutable_local_copy(self, target_folder, overwrite=False, raise_on_error=True): + # type: (Union[Path, _Path, str], bool, bool) -> Optional[str] + """ + return a base folder with a writable (mutable) local copy of the entire dataset + download and copy / soft-link, files from all the parent dataset versions + + :param target_folder: Target folder for the writable copy + :param overwrite: If True, recursively delete the target folder before creating a copy. + If False (default) and target folder contains files, raise exception or return None + :param raise_on_error: If True raise exception if dataset merging failed on any file + :return: A the target folder containing the entire dataset + """ + assert self._id + target_folder = Path(target_folder) + target_folder.mkdir(parents=True, exist_ok=True) + # noinspection PyBroadException + try: + target_folder.rmdir() + except Exception: + if not overwrite: + if raise_on_error: + raise ValueError("Target folder {} already contains files".format(target_folder.as_posix())) + else: + return None + shutil.rmtree(target_folder.as_posix()) + + ro_folder = self.get_local_copy(raise_on_error=raise_on_error) + shutil.copytree(ro_folder, target_folder.as_posix()) + return target_folder.as_posix() + + def list_files(self, dataset_path=None, recursive=True, dataset_id=None): + # type: (Optional[str], bool, Optional[str]) -> List[str] + """ + returns a list of files in the current dataset + If dataset_id is give, return a list of files that remained unchanged since the specified dataset_version + + :param dataset_path: Only match files matching the dataset_path (including wildcards). + Example: folder/sub/*.json + :param recursive: If True (default) matching dataset_path recursively + :param dataset_id: Filter list based on the dataset id containing the latest version of the file. + Default: None, do not filter files based on parent dataset. + :return: List of files with relative path + (files might not be available locally until get_local_copy() is called) + """ + files = self._dataset_file_entries.keys() if not dataset_id else \ + [k for k, v in self._dataset_file_entries.items() if v.parent_dataset_id == dataset_id] + + if not dataset_path: + return sorted(files) + + if dataset_path.startswith('/'): + dataset_path = dataset_path[1:] + + if not recursive: + return sorted([k for k in files if fnmatch(k + '/', dataset_path + '/')]) + + wildcard = dataset_path.split('/')[-1] + path = dataset_path[:-len(wildcard)] + '*' + return sorted([k for k in files if fnmatch(k, path) and fnmatch(k, '*/' + wildcard)]) + + def list_removed_files(self, dataset_id=None): + # type: (str) -> List[str] + """ + return a list of files removed when comparing to a specific dataset_version + + :param dataset_id: dataset id (str) to compare against, if None is given compare against the parents datasets + :return: List of files with relative path + (files might not be available locally until get_local_copy() is called) + """ + datasets = self._dependency_graph[self._id] if not dataset_id else [dataset_id] + unified_list = set() + for ds_id in datasets: + dataset = self.get(dataset_id=ds_id) + unified_list |= set(dataset._dataset_file_entries.keys()) + + removed_list = [f for f in unified_list if f not in self._dataset_file_entries] + return sorted(removed_list) + + def list_modified_files(self, dataset_id=None): + # type: (str) -> List[str] + """ + return a list of files removed when comparing to a specific dataset_version + + :param dataset_id: dataset id (str) to compare against, if None is given compare against the parents datasets + :return: List of files with relative path + (files might not be available locally until get_local_copy() is called) + """ + datasets = self._dependency_graph[self._id] if not dataset_id else [dataset_id] + unified_list = dict() + for ds_id in datasets: + dataset = self.get(dataset_id=ds_id) + unified_list.update(dict((k, v.hash) for k, v in dataset._dataset_file_entries.items())) + + modified_list = [k for k, v in self._dataset_file_entries.items() + if k in unified_list and v.hash != unified_list[k]] + return sorted(modified_list) + + def list_added_files(self, dataset_id=None): + # type: (str) -> List[str] + """ + return a list of files removed when comparing to a specific dataset_version + + :param dataset_id: dataset id (str) to compare against, if None is given compare against the parents datasets + :return: List of files with relative path + (files might not be available locally until get_local_copy() is called) + """ + datasets = self._dependency_graph[self._id] if not dataset_id else [dataset_id] + unified_list = set() + for ds_id in datasets: + dataset = self.get(dataset_id=ds_id) + unified_list |= set(dataset._dataset_file_entries.keys()) + + added_list = [f for f in self._dataset_file_entries.keys() if f not in unified_list] + return sorted(added_list) + + def get_dependency_graph(self): + """ + return the DAG of the dataset dependencies (all previous dataset version and their parents/ + Example: + { + 'current_dataset_id': ['parent_1_id', 'parent_2_id'], + 'parent_2_id': ['parent_1_id'], + 'parent_1_id': [], + } + + :return: dict representing the genealogy dag graph of the current dataset + """ + return deepcopy(self._dependency_graph) + + def verify_dataset_hash(self, local_copy_path=None, skip_hash=False, verbose=False): + # type: (Optional[str], bool, bool) -> List[str] + """ + Verify the current copy of the dataset against the stored hash + + :param local_copy_path: Specify local path containing a copy of the dataset, + If not provide use the cached folder + :param skip_hash: If True, skip hash checks and verify file size only + :param verbose: If True print errors while testing dataset files hash + :return: List of files with unmatched hashes + """ + local_path = local_copy_path or self.get_local_copy() + + def compare(file_entry): + file_entry_copy = copy(file_entry) + file_entry_copy.local_path = (Path(local_path) / file_entry.relative_path).as_posix() + if skip_hash: + file_entry_copy.size = Path(file_entry_copy.local_path).stat().st_size + if file_entry_copy.size != file_entry.size: + if verbose: + print('Error: file size mismatch {} expected size {} current {}'.format( + file_entry.relative_path, file_entry.size, file_entry_copy.size)) + return file_entry + else: + self._calc_file_hash(file_entry_copy) + if file_entry_copy.hash != file_entry.hash: + if verbose: + print('Error: hash mismatch {} expected size/hash {}/{} recalculated {}/{}'.format( + file_entry.relative_path, + file_entry.size, file_entry.hash, + file_entry_copy.size, file_entry_copy.hash)) + return file_entry + + return None + + pool = ThreadPool(cpu_count() * 2) + matching_errors = pool.map(compare, self._dataset_file_entries.values()) + pool.close() + return [f.relative_path for f in matching_errors if f is not None] + + @classmethod + def create(cls, dataset_project, dataset_name, parent_datasets=None): + # type: (str, str, Optional[Sequence[Union[str, Dataset]]]) -> Dataset + """ + Create a new dataset. Multiple dataset parents are supported. + Merging of parent datasets is done based on the order, + where each one can override overlapping files in the previous parent + + :param dataset_project: Project containing the dataset + :param dataset_name: Naming the new dataset + :param parent_datasets: Expand a parent dataset by adding/removing files + :return: Newly created Dataset object + """ + parent_datasets = [cls.get(dataset_id=p) if isinstance(p, str) else p for p in (parent_datasets or [])] + if any(not p.is_final() for p in parent_datasets): + raise ValueError("Cannot inherit from a parent that was not finalized/closed") + + # merge datasets according to order + dataset_file_entries = {} + dependency_graph = {} + for p in parent_datasets: + dataset_file_entries.update(deepcopy(p._dataset_file_entries)) + dependency_graph.update(deepcopy(p._dependency_graph)) + instance = cls(_private=cls.__private_magic, dataset_project=dataset_project, dataset_name=dataset_name) + instance._task.get_logger().report_text('Dataset created', print_console=False) + instance._dataset_file_entries = dataset_file_entries + instance._dependency_graph = dependency_graph + instance._dependency_graph[instance._id] = [p._id for p in parent_datasets] + instance._serialize() + instance._task.flush(wait_for_uploads=True) + return instance + + @classmethod + def delete(cls, dataset_id=None, dataset_project=None, dataset_name=None, force=False): + # type: (Optional[str], Optional[str], Optional[str], bool) -> () + """ + Delete a dataset, raise exception if dataset is used by other dataset versions. + Use force=True to forcefully delete the dataset + + :param dataset_id: Dataset id to delete + :param dataset_project: Project containing the dataset + :param dataset_name: Naming the new dataset + :param force: If True delete even if other datasets depend on the specified dataset version + """ + mutually_exclusive(dataset_id=dataset_id, dataset_project=dataset_project) + mutually_exclusive(dataset_id=dataset_id, dataset_name=dataset_name) + if not dataset_id: + tasks = Task.get_tasks( + project_name=dataset_project, + task_name=exact_match_regex(dataset_name) if dataset_name else None, + task_filter=dict( + system_tags=[cls.__tag], + type=[str(Task.TaskTypes.data_processing)], + page_size=2, page=0,) + ) + if not tasks: + raise ValueError("Dataset project={} name={} could not be found".format(dataset_project, dataset_name)) + if len(tasks) > 1: + raise ValueError("Too many datasets matching project={} name={}".format(dataset_project, dataset_name)) + dataset_id = tasks[0].id + + # check if someone is using the datasets + if not force: + # noinspection PyProtectedMember + dependencies = Task._query_tasks( + system_tags=[cls.__tag], + type=[str(Task.TaskTypes.data_processing)], + only_fields=['created', 'id', 'name'], + search_text='{}'.format(cls._get_dataset_id_hash(dataset_id)) + ) + # filter us out + if dependencies: + dependencies = [d for d in dependencies if d.id != dataset_id] + if dependencies: + raise ValueError("Dataset id={} is used by datasets: {}".format( + dataset_id, [d.id for d in dependencies])) + + client = APIClient() + # notice the force here is a must, since the state is never draft + # noinspection PyBroadException + try: + t = client.tasks.get_by_id(dataset_id) + except Exception: + t = None + if not t: + raise ValueError("Dataset id={} could not be found".format(dataset_id)) + if str(t.type) != str(Task.TaskTypes.data_processing) or cls.__tag not in t.system_tags: + raise ValueError("Dataset id={} is not of type Dataset".format(dataset_id)) + + task = Task.get_task(task_id=dataset_id) + # first delete all the artifacts from the dataset + for artifact in task.artifacts.values(): + h = StorageHelper.get(artifact.url) + # noinspection PyBroadException + try: + h.delete(artifact.url) + except Exception as ex: + LoggerRoot.get_base_logger().warning('Failed deleting remote file \'{}\': {}'.format( + artifact.url, ex)) + + # now delete the actual task + client.tasks.delete(task=dataset_id, force=True) + + @classmethod + def get(cls, dataset_id=None, dataset_project=None, dataset_name=None, only_completed=False, only_published=False): + # type: (Optional[str], Optional[str], Optional[str], bool , bool) -> Dataset + """ + Get a specific Dataset. If only dataset_project is given, return the last Dataset in the Dataset project + + :param dataset_id: Requested Dataset ID + :param dataset_project: Requested Dataset project name + :param dataset_name: Requested Dataset name + :param only_completed: Return only if the requested dataset is completed or published + :param only_published: Return only if the requested dataset is published + :return: Dataset object + """ + mutually_exclusive(dataset_id=dataset_id, dataset_project=dataset_project) + mutually_exclusive(dataset_id=dataset_id, dataset_name=dataset_name) + tasks = Task.get_tasks( + task_ids=[dataset_id] if dataset_id else None, + project_name=dataset_project, + task_name=exact_match_regex(dataset_name) if dataset_name else None, + task_filter=dict( + system_tags=[cls.__tag, '-archived'], order_by=['-created'], + type=[str(Task.TaskTypes.data_processing)], + page_size=1, page=0, + status=['publish'] if only_published else + ['publish', 'stopped', 'completed', 'closed'] if only_completed else None) + ) + if not tasks: + raise ValueError('Could not find Dataset {} {}'.format( + 'id' if dataset_id else 'project/name', + dataset_id if dataset_id else (dataset_project, dataset_name))) + task = tasks[0] + if task.status == 'created': + raise ValueError('Dataset id={} is in draft mode, delete and recreate it'.format(task.id)) + force_download = False if task.status in ('stopped', 'published', 'closed', 'completed') else True + local_state_file = StorageManager.get_local_copy( + remote_url=task.artifacts[cls.__state_entry_name].url, cache_context=cls.__cache_context, + extract_archive=False, name=task.id, force_download=force_download) + if not local_state_file: + raise ValueError('Could not load Dataset id={} state'.format(task.id)) + + instance = cls._deserialize(local_state_file, task) + # remove the artifact, just in case + if force_download: + os.unlink(local_state_file) + + return instance + + @classmethod + def squash(cls, dataset_name, dataset_ids=None, dataset_project_name_pairs=None, output_url=None): + # type: (str, Optional[Sequence[Union[str, Dataset]]],Optional[Sequence[(str, str)]], Optional[str]) -> Dataset + """ + Generate a new dataset from the squashed set of dataset versions. + If a single version is given it will squash to the root (i.e. create single standalone version) + If a set of versions are given it will squash the versions diff into a single version + + :param dataset_name: Target name for the newly generated squashed dataset + :param dataset_ids: List of dataset Ids (or objects) to squash. Notice order does matter. + The versions are merged from first to last. + :param dataset_project_name_pairs: List of pairs (project_name, dataset_name) to squash. + Notice order does matter. The versions are merged from first to last. + :param output_url: Target storage for the compressed dataset (default: file server) + Examples: `s3://bucket/data`, `gs://bucket/data` , `azure://bucket/data` , `/mnt/share/data` + :return: Newly created dataset object. + """ + mutually_exclusive(dataset_ids=dataset_ids, dataset_project_name_pairs=dataset_project_name_pairs) + datasets = [cls.get(dataset_id=d) for d in dataset_ids] if dataset_ids else \ + [cls.get(dataset_project=pair[0], dataset_name=pair[1]) for pair in dataset_project_name_pairs] + # single dataset to squash, squash it all. + if len(datasets) == 1: + temp_folder = datasets[0].get_local_copy() + parents = set() + else: + parents = None + temp_folder = Path(mkdtemp(prefix='squash-datasets.')) + pool = ThreadPool() + for ds in datasets: + base_folder = Path(ds._extract_dataset_archive()) + files = [f.relative_path for f in ds.file_entries if f.parent_dataset_id == ds.id] + pool.map( + lambda x: + (temp_folder / x).parent.mkdir(parents=True, exist_ok=True) or + shutil.copy((base_folder / x).as_posix(), (temp_folder / x).as_posix(), follow_symlinks=True), + files) + parents = set(ds._get_parents()) if parents is None else (parents & set(ds._get_parents())) + pool.close() + + squashed_ds = cls.create( + dataset_project=datasets[0].project, dataset_name=dataset_name, parent_datasets=list(parents)) + squashed_ds._task.get_logger().report_text('Squashing dataset', print_console=False) + squashed_ds.add_files(temp_folder) + squashed_ds.upload(output_url=output_url) + squashed_ds.finalize() + return squashed_ds + + @classmethod + def list_datasets(cls, dataset_project=None, partial_name=None, tags=None, ids=None, only_completed=True): + # type: (Optional[str], Optional[str], Optional[Sequence[str]], Optional[Sequence[str]], bool) -> List[dict] + """ + Query list of dataset in the system + + :param dataset_project: Specify dataset project name + :param partial_name: Specify partial match to a dataset name + :param tags: Specify user tags + :param ids: List specific dataset based on IDs list + :param only_completed: If False return dataset that are still in progress (uploading/edited etc.) + :return: List of dictionaries with dataset information + Example: [{'name': name, 'project': project name, 'id': dataset_id, 'created': date_created},] + """ + # noinspection PyProtectedMember + datasets = Task._query_tasks( + task_ids=ids or None, project_name=dataset_project or None, + task_name=partial_name, + system_tags=[cls.__tag], + type=[str(Task.TaskTypes.data_processing)], + tags=tags or None, + status=['stopped', 'published', 'completed', 'closed'] if only_completed else None, + only_fields=['created', 'id', 'name', 'project'] + ) + project_ids = {d.project for d in datasets} + # noinspection PyProtectedMember + project_id_lookup = {d: Task._get_project_name(d) for d in project_ids} + return [ + {'name': d.name, 'created': d.created, 'project': project_id_lookup[d.project], 'id': d.id} + for d in datasets + ] + + def _add_files(self, + path, # type: Union[str, Path, _Path] + wildcard=None, # type: Optional[Union[str, Sequence[str]]] + local_base_folder=None, # type: Optional[str] + dataset_path=None, # type: Optional[str] + recursive=True, # type: bool + verbose=False # type: bool + ): + # type: (...) -> int + """ + Add a folder into the current dataset. calculate file hash, + and compare against parent, mark files to be uploaded + + :param path: Add a folder/file to the dataset + :param wildcard: add only specific set of files. + Wildcard matching, can be a single string or a list of wildcards) + :param local_base_folder: files will be located based on their relative path from local_base_folder + :param dataset_path: where in the dataset the folder/files should be located + :param recursive: If True match all wildcard files recursively + :param verbose: If True print to console added files + """ + if dataset_path and dataset_path.startswith('/'): + dataset_path = dataset_path[1:] + path = Path(path) + local_base_folder = Path(local_base_folder or path) + wildcard = wildcard or '*' + # single file, no need for threading + if path.is_file(): + file_entry = self._calc_file_hash( + FileEntry(local_path=path.absolute().as_posix(), + relative_path=Path(dataset_path or '.') / path.relative_to(local_base_folder), + parent_dataset_id=self._id)) + file_entries = [file_entry] + else: + # if not a folder raise exception + if not path.is_dir(): + raise ValueError("Could not find file/folder \'{}\'", path.as_posix()) + + # prepare a list of files + files = list(path.rglob(wildcard)) if recursive else list(path.glob(wildcard)) + file_entries = [ + FileEntry( + parent_dataset_id=self._id, + local_path=f.absolute().as_posix(), + relative_path=(Path(dataset_path or '.') / f.relative_to(local_base_folder)).as_posix()) + for f in files if f.is_file()] + self._task.get_logger().report_text('Generating SHA2 hash for {} files'.format(len(file_entries))) + pool = ThreadPool(cpu_count() * 2) + pool.map(self._calc_file_hash, file_entries) + pool.close() + self._task.get_logger().report_text('Hash generation completed') + + # merge back into the dataset + count = 0 + for f in file_entries: + ds_cur_f = self._dataset_file_entries.get(f.relative_path) + if not ds_cur_f: + if verbose: + self._task.get_logger().report_text('Add {}'.format(f.relative_path)) + self._dataset_file_entries[f.relative_path] = f + count += 1 + elif ds_cur_f.hash != f.hash: + if verbose: + self._task.get_logger().report_text('Modified {}'.format(f.relative_path)) + self._dataset_file_entries[f.relative_path] = f + count += 1 + elif f.parent_dataset_id == self._id and ds_cur_f.parent_dataset_id == self._id: + if verbose: + self._task.get_logger().report_text('Re-Added {}'.format(f.relative_path)) + self._dataset_file_entries[f.relative_path] = f + count += 1 + else: + if verbose: + self._task.get_logger().report_text('Unchanged {}'.format(f.relative_path)) + + return count + + def _update_dependency_graph(self): + """ + Update the dependency graph based on the current self._dataset_file_entries state + :return: + """ + # collect all dataset versions + used_dataset_versions = set(f.parent_dataset_id for f in self._dataset_file_entries.values()) + used_dataset_versions.add(self._id) + current_parents = self._dependency_graph.get(self._id) + # remove parent versions we no longer need from the main version list + # per version, remove unnecessary parent versions, if we do not need them + self._dependency_graph = {k: [p for p in parents if p in used_dataset_versions] + for k, parents in self._dependency_graph.items() if k in used_dataset_versions} + # make sure we do not remove our parents, for geology sake + self._dependency_graph[self._id] = current_parents + + def _serialize(self): + """ + store current state of the Dataset for later use + :return: object to be used for later deserialization + """ + self._update_dependency_graph() + + state = dict( + dataset_file_entries=[f.as_dict() for f in self._dataset_file_entries.values()], + dependency_graph=self._dependency_graph, + id=self._id, + dirty=self._dirty, + ) + modified_files = [f['size'] for f in state['dataset_file_entries'] if f.get('parent_dataset_id') == self._id] + preview = \ + 'Dataset state\n' \ + 'Files added/modified: {0} - total size {1}\n' \ + 'Current dependency graph: {2}\n'.format( + len(modified_files), humanfriendly.format_size(sum(modified_files)), + json.dumps(self._dependency_graph, indent=2, sort_keys=True)) + # store as artifact of the Task. + self._task.upload_artifact( + name=self.__state_entry_name, artifact_object=state, preview=preview, wait_on_upload=True) + + def _download_dataset_archive(self): + """ + Download the dataset archive, return a link to locally stored zip file + :return: Path to locally stored zip file + """ + pass # TODO: implement + + def _extract_dataset_archive(self): + """ + Download the dataset archive, and extract the zip content to a cached folder. + Notice no merging is done. + + :return: Path to a local storage extracted archive + """ + if not self._task: + self._task = Task.get_task(task_id=self._id) + local_zip = StorageManager.get_local_copy( + remote_url=self._task.artifacts[self.__data_entry_name].url, cache_context=self.__cache_context, + extract_archive=False, name=self._id) + if not local_zip: + raise ValueError("Could not download dataset id={}".format(self._id)) + local_folder = (Path(local_zip).parent / self._get_cache_folder_name()).as_posix() + # if we got here, we need to clear the target folder + shutil.rmtree(local_folder, ignore_errors=True) + # noinspection PyProtectedMember + local_folder = StorageManager._extract_to_cache( + cached_file=local_zip, name=self._id, + cache_context=self.__cache_context, target_folder=local_folder) + return local_folder + + def _merge_datasets(self, use_soft_links=None, raise_on_error=True): + # type: (bool, bool) -> str + """ + download and copy / soft-link, files from all the parent dataset versions + :param use_soft_links: If True use soft links, default False on windows True on Posix systems + :param raise_on_error: If True raise exception if dataset merging failed on any file + :return: the target folder + """ + if use_soft_links is None: + use_soft_links = False if is_windows() else True + + # check if we already have everything + target_base_folder, target_base_size = CacheManager.get_cache_manager( + cache_context=self.__cache_context).get_cache_file(local_filename=self._get_cache_folder_name()) + if target_base_folder and target_base_size is not None: + target_base_folder = Path(target_base_folder) + # check dataset file size, if we have a full match no need for parent dataset download / merge + verified = True + # noinspection PyBroadException + try: + for f in self._dataset_file_entries.values(): + if (target_base_folder / f.relative_path).stat().st_size != f.size: + verified = False + break + except Exception: + verified = False + + if verified: + return target_base_folder.as_posix() + else: + LoggerRoot.get_base_logger().info('Dataset needs refreshing, fetching all parent datasets') + + # first get our dataset + target_base_folder = Path(self._extract_dataset_archive()) + target_base_folder.touch() + + # create thread pool + pool = ThreadPool(cpu_count() * 2) + for dataset_version_id in self._get_dependencies_by_order(): + # make sure we skip over empty dependencies + if dataset_version_id not in self._dependency_graph: + continue + + ds = Dataset.get(dataset_id=dataset_version_id) + ds_base_folder = Path(ds._extract_dataset_archive()) + ds_base_folder.touch() + + def copy_file(file_entry): + if file_entry.parent_dataset_id != dataset_version_id: + return + source = (ds_base_folder / file_entry.relative_path).as_posix() + target = (target_base_folder / file_entry.relative_path).as_posix() + try: + # make sure we have can overwrite the target file + # noinspection PyBroadException + try: + os.unlink(target) + except Exception: + Path(target).parent.mkdir(parents=True, exist_ok=True) + + # copy / link + if use_soft_links: + if not os.path.isfile(source): + raise ValueError("Extracted file missing {}".format(source)) + os.symlink(source, target) + else: + shutil.copy2(source, target, follow_symlinks=True) + except Exception as ex: + LoggerRoot.get_base_logger().warning('{}\nFailed {} file {} to {}'.format( + ex, 'linking' if use_soft_links else 'copying', source, target)) + return ex + + return None + + errors = pool.map(copy_file, self._dataset_file_entries.values()) + if raise_on_error and any(errors): + raise ValueError("Dataset merging failed: {}".format([e for e in errors if e is not None])) + + pool.close() + return target_base_folder.as_posix() + + def _get_dependencies_by_order(self, include_unused=False): + # type: (bool) -> List[str] + """ + Return the dataset dependencies by order of application (from the last to the current) + :param bool include_unused: If True include unused datasets in the dependencies + :return: list of str representing the datasets id + """ + roots = [self._id] + dependencies = [] + # noinspection DuplicatedCode + while roots: + r = roots.pop(0) + dependencies.append(r) + # add the parents of the current node, only if the parents are in the general graph node list + if include_unused and r not in self._dependency_graph: + roots.extend(list(reversed( + [p for p in self.get(dataset_id=r)._get_parents() if p not in roots]))) + else: + roots.extend(list(reversed( + [p for p in self._dependency_graph.get(r, []) + if p not in roots and (include_unused or (p in self._dependency_graph))]))) + + # make sure we cover leftovers + leftovers = set(self._dependency_graph.keys()) - set(dependencies) + if leftovers: + roots = list(leftovers) + # noinspection DuplicatedCode + while roots: + r = roots.pop(0) + dependencies.append(r) + # add the parents of the current node, only if the parents are in the general graph node list + if include_unused and r not in self._dependency_graph: + roots.extend(list(reversed( + [p for p in self.get(dataset_id=r)._get_parents() if p not in roots]))) + else: + roots.extend(list(reversed( + [p for p in self._dependency_graph.get(r, []) + if p not in roots and (include_unused or (p in self._dependency_graph))]))) + + # skip our id + return list(reversed(dependencies[1:])) + + def _get_parents(self): + # type: () -> Sequence[str] + """ + Return a list of direct parent datasets (str) + :return: list of dataset ids + """ + return self._dependency_graph[self.id] + + @classmethod + def _deserialize(cls, stored_state, task): + # type: (Union[dict, str, Path, _Path], Task) -> Dataset + """ + reload a dataset state from the stored_state object + :param task: Task object associated with the dataset + :return: A Dataset object + """ + assert isinstance(stored_state, (dict, str, Path, _Path)) + + if isinstance(stored_state, (str, Path, _Path)): + stored_state_file = Path(stored_state).as_posix() + with open(stored_state_file, 'rt') as f: + stored_state = json.load(f) + + instance = cls(_private=cls.__private_magic, task=task) + # assert instance._id == stored_state['id'] # They should match + instance._dependency_graph = stored_state['dependency_graph'] + instance._dirty = stored_state.get('dirty', False) + instance._dataset_file_entries = { + s['relative_path']: FileEntry(**s) for s in stored_state['dataset_file_entries']} + return instance + + @staticmethod + def _calc_file_hash(file_entry): + # calculate hash + file_entry.hash, _ = sha256sum(file_entry.local_path) + file_entry.size = Path(file_entry.local_path).stat().st_size + return file_entry + + @classmethod + def _get_dataset_id_hash(cls, dataset_id): + # type: (str) -> str + """ + Return hash used to search for the dataset id in text fields. + This is not a strong hash and used for defining dependencies. + :param dataset_id: + :return: + """ + return 'dsh{}'.format(md5text(dataset_id)) + + def _get_cache_folder_name(self): + return '{}{}'.format(self.__cache_folder_prefix, self._id) + + def _add_script_call(self, func_name, **kwargs): + # type: (str, **Any) -> () + args = ', '.join('\n {}={}'.format(k, '\''+str(v)+'\'' if isinstance(v, (str, Path, _Path)) else v) + for k, v in kwargs.items()) + if args: + args += '\n' + line = 'ds.{}({})\n'.format(func_name, args) + self._task.data.script.diff += line + # noinspection PyProtectedMember + self._task._edit(script=self._task.data.script) + + def _report_dataset_genealogy(self): + sankey_node = dict( + label=[], + color=[], + customdata=[], + hovertemplate='%{customdata}', + hoverlabel={"align": "left"}, + ) + sankey_link = dict( + source=[], + target=[], + value=[], + hovertemplate='', + ) + # get DAG nodes + nodes = self._get_dependencies_by_order(include_unused=True) + [self.id] + # dataset name lookup + # noinspection PyProtectedMember + node_names = {t.id: t.name for t in Task._query_tasks(task_ids=nodes, only_fields=['id', 'name'])} + node_details = {} + # Generate table and details + table_values = [["Dataset id", "name", "removed", "modified", "added", "size"]] + for node in nodes: + count = 0 + size = 0 + for f in self._dataset_file_entries.values(): + if f.parent_dataset_id == node: + count += 1 + size += f.size + removed = len(self.list_removed_files(node)) + modified = len(self.list_modified_files(node)) + table_values += [[node, node_names.get(node, ''), + removed, modified, count-modified, humanfriendly.format_size(size)]] + node_details[node] = [removed, modified, count-modified, humanfriendly.format_size(size)] + + # create DAG + visited = [] + for idx, node in enumerate(nodes): + visited.append(node) + if node in self._dependency_graph: + parents = [visited.index(p) for p in self._dependency_graph[node] or [] if p in visited] + else: + parents = [visited.index(p) for p in self.get(dataset_id=node)._get_parents() or [] if p in visited] + + sankey_node['color'].append("mediumpurple" if node == self.id else "lightblue") + sankey_node['label'].append('{}'.format(node)) + sankey_node['customdata'].append( + "name {}
removed {}
modified {}
added {}
size {}".format( + node_names.get(node, ''), *node_details[node])) + + for p in parents: + sankey_link['source'].append(p) + sankey_link['target'].append(idx) + sankey_link['value'].append(max(1, node_details[visited[p]][-2])) + + # create the sankey graph + dag_flow = dict( + link=sankey_link, + node=sankey_node, + textfont=dict(color='rgba(0,0,0,255)', size=10), + type='sankey', + orientation='h' + ) + fig = dict(data=[dag_flow], layout={'xaxis': {'visible': False}, 'yaxis': {'visible': False}}) + + if len(nodes) > 1: + self._task.get_logger().report_plotly( + title='Dataset Genealogy', series='', iteration=0, figure=fig) + + # report detailed table + self._task.get_logger().report_table( + title='Dataset Summary', series='Details', iteration=0, table_plot=table_values, + extra_layout={"title": "Files by parent dataset id"}) + + # report the detailed content of the dataset as configuration, + # this allows for easy version comparison in the UI + dataset_details = None + if len(self._dataset_file_entries) < self.__preview_max_file_entries: + file_entries = sorted(self._dataset_file_entries.values(), key=lambda x: x.relative_path) + dataset_details = \ + 'File Name - File Size - Hash (SHA2)\n' +\ + '\n'.join('{} - {} - {}'.format(f.relative_path, f.size, f.hash) for f in file_entries) + # too large to store + if not dataset_details or len(dataset_details) > self.__preview_max_size: + dataset_details = 'Dataset content is too large to preview' + + # noinspection PyProtectedMember + self._task._set_configuration( + name='Dataset Content', description='Dataset content preview', + config_type='read-only', + config_text=dataset_details + ) diff --git a/clearml/debugging/log.py b/clearml/debugging/log.py index 3fbb8d6a..366735a3 100644 --- a/clearml/debugging/log.py +++ b/clearml/debugging/log.py @@ -56,7 +56,7 @@ class LoggerRoot(object): # avoid nested imports from ..config import get_log_redirect_level - LoggerRoot.__base_logger = logging.getLogger('trains') + LoggerRoot.__base_logger = logging.getLogger('clearml') level = level if level is not None else default_level LoggerRoot.__base_logger.setLevel(level) diff --git a/clearml/debugging/trace.py b/clearml/debugging/trace.py index e060a54c..5ca1be0b 100644 --- a/clearml/debugging/trace.py +++ b/clearml/debugging/trace.py @@ -145,11 +145,11 @@ def _patch_module(module, prefix='', basepath=None, basemodule=None, exclude_pre prefix += module.__name__.split('.')[-1] + '.' # Do not patch low level network layer - if prefix.startswith('trains.backend_api.session.') and prefix != 'trains.backend_api.session.': + if prefix.startswith('clearml.backend_api.session.') and prefix != 'clearml.backend_api.session.': if not prefix.endswith('.Session.') and '.token_manager.' not in prefix: # print('SKIPPING: {}'.format(prefix)) return - if prefix.startswith('trains.backend_api.services.'): + if prefix.startswith('clearml.backend_api.services.'): return for skip in exclude_prefixes: @@ -208,7 +208,7 @@ def _patch_module(module, prefix='', basepath=None, basemodule=None, exclude_pre def trace_trains(stream=None, level=1, exclude_prefixes=[], only_prefix=[]): """ - DEBUG ONLY - Add full Trains package code trace + DEBUG ONLY - Add full ClearML package code trace Output trace to filename or stream, default is sys.stderr Trace level -2: Trace function and arguments and returned call @@ -244,7 +244,7 @@ def trace_trains(stream=None, level=1, exclude_prefixes=[], only_prefix=[]): __stream_flush = None from ..version import __version__ - msg = 'Trains v{} - Starting Trace\n\n'.format(__version__) + msg = 'ClearML v{} - Starting Trace\n\n'.format(__version__) # print to actual stderr stderr_write(msg) # store to stream @@ -252,7 +252,7 @@ def trace_trains(stream=None, level=1, exclude_prefixes=[], only_prefix=[]): __stream_write('{:9}:{:5}:{:8}: {:14}\n'.format('seconds', 'pid', 'tid', 'self')) __stream_write('{:9}:{:5}:{:8}:{:15}\n'.format('-' * 9, '-' * 5, '-' * 8, '-' * 15)) __trace_start = time.time() - _patch_module('trains', exclude_prefixes=exclude_prefixes or [], only_prefix=only_prefix or []) + _patch_module('clearml', exclude_prefixes=exclude_prefixes or [], only_prefix=only_prefix or []) def trace_level(level=1): @@ -343,7 +343,7 @@ def end_of_program(): if __name__ == '__main__': - # from trains import Task + # from clearml import Task # task = Task.init(project_name="examples", task_name="trace test") # trace_trains('_trace.txt', level=2) print_traced_files('_trace_*.txt', lines_per_tid=10) diff --git a/clearml/errors.py b/clearml/errors.py index f3ef762d..8f1ce8c3 100644 --- a/clearml/errors.py +++ b/clearml/errors.py @@ -1,3 +1,3 @@ class UsageError(RuntimeError): - """ An exception raised for illegal usage of trains objects""" + """ An exception raised for illegal usage of clearml objects""" pass diff --git a/clearml/external/kerastuner.py b/clearml/external/kerastuner.py index ffbb30fe..60e1bf3b 100644 --- a/clearml/external/kerastuner.py +++ b/clearml/external/kerastuner.py @@ -14,7 +14,7 @@ try: except ImportError: pd = None from logging import getLogger - getLogger('trains.external.kerastuner').warning( + getLogger('clearml.external.kerastuner').warning( 'Pandas is not installed, summary table reporting will be skipped.') @@ -26,7 +26,7 @@ class TrainsTunerLogger(Logger): super(TrainsTunerLogger, self).__init__() self.task = task or Task.current_task() if not self.task: - raise ValueError("Trains Task could not be found, pass in TrainsTunerLogger or " + raise ValueError("ClearML Task could not be found, pass in TrainsTunerLogger or " "call Task.init before initializing TrainsTunerLogger") self._summary = pd.DataFrame() if pd else None diff --git a/clearml/logger.py b/clearml/logger.py index 69a8604b..11e447c6 100644 --- a/clearml/logger.py +++ b/clearml/logger.py @@ -36,14 +36,14 @@ if TYPE_CHECKING: class Logger(object): """ - The ``Logger`` class is the Trains console log and metric statistics interface, and contains methods for explicit + The ``Logger`` class is the ClearML console log and metric statistics interface, and contains methods for explicit reporting. - Explicit reporting extends Trains automagical capturing of inputs and output. Explicit reporting + Explicit reporting extends ClearML automagical capturing of inputs and output. Explicit reporting methods include scalar plots, line plots, histograms, confusion matrices, 2D and 3D scatter diagrams, text logging, tables, and image uploading and reporting. - In the **Trains Web-App (UI)**, ``Logger`` output appears in the **RESULTS** tab, **LOG**, **SCALARS**, + In the **ClearML Web-App (UI)**, ``Logger`` output appears in the **RESULTS** tab, **LOG**, **SCALARS**, **PLOTS**, and **DEBUG SAMPLES** sub-tabs. When you compare experiments, ``Logger`` output appears in the comparisons. @@ -90,7 +90,7 @@ class Logger(object): if self._connect_logging: StdStreamPatch.patch_logging_formatter(self) elif not self._connect_std_streams: - # make sure that at least the main trains logger is connect + # make sure that at least the main clearml logger is connect base_logger = LoggerRoot.get_base_logger() if base_logger and base_logger.handlers: StdStreamPatch.patch_logging_formatter(self, base_logger.handlers[0]) @@ -126,7 +126,7 @@ class Logger(object): logger.report_text('log some text', level=logging.DEBUG, print_console=False) - You can view the reported text in the **Trains Web-App (UI)**, **RESULTS** tab, **LOG** sub-tab. + You can view the reported text in the **ClearML Web-App (UI)**, **RESULTS** tab, **LOG** sub-tab. :param str msg: The text to log. :param int level: The log level from the Python ``logging`` package. The default value is ``logging.INFO``. @@ -151,7 +151,7 @@ class Logger(object): scalar_series = [random.randint(0,10) for i in range(10)] logger.report_scalar(title='scalar metrics','series', value=scalar_series[iteration], iteration=0) - You can view the scalar plots in the **Trains Web-App (UI)**, **RESULTS** tab, **SCALARS** sub-tab. + You can view the scalar plots in the **ClearML Web-App (UI)**, **RESULTS** tab, **SCALARS** sub-tab. :param str title: The title (metric) of the plot. Plot more than one scalar series on the same plot by using the same ``title`` for each call to this method. @@ -190,7 +190,7 @@ class Logger(object): logger.report_vector(title='vector example', series='vector series', values=vector_series, iteration=0, labels=['A','B'], xaxis='X axis label', yaxis='Y axis label') - You can view the vectors plots in the **Trains Web-App (UI)**, **RESULTS** tab, **PLOTS** sub-tab. + You can view the vectors plots in the **ClearML Web-App (UI)**, **RESULTS** tab, **PLOTS** sub-tab. :param str title: The title (metric) of the plot. :param str series: The series name (variant) of the reported histogram. @@ -237,7 +237,7 @@ class Logger(object): logger.report_histogram(title='histogram example', series='histogram series', values=vector_series, iteration=0, labels=['A','B'], xaxis='X axis label', yaxis='Y axis label') - You can view the reported histograms in the **Trains Web-App (UI)**, **RESULTS** tab, **PLOTS** sub-tab. + You can view the reported histograms in the **ClearML Web-App (UI)**, **RESULTS** tab, **PLOTS** sub-tab. :param str title: The title (metric) of the plot. :param str series: The series name (variant) of the reported histogram. @@ -305,7 +305,7 @@ class Logger(object): logger.report_table(title='table example',series='pandas DataFrame',iteration=0,table_plot=df) - You can view the reported tables in the **Trains Web-App (UI)**, **RESULTS** tab, **PLOTS** sub-tab. + You can view the reported tables in the **ClearML Web-App (UI)**, **RESULTS** tab, **PLOTS** sub-tab. :param str title: The title (metric) of the table. :param str series: The series name (variant) of the reported table. @@ -1022,8 +1022,8 @@ class Logger(object): The images are uploaded separately. A link to each image is reported. .. note:: - Credentials for the destination storage are specified in the Trains configuration file, - ``~/trains.conf``. + Credentials for the destination storage are specified in the ClearML configuration file, + ``~/clearml.conf``. :param str uri: example: 's3://bucket/directory/' or 'file:///tmp/debug/' @@ -1150,7 +1150,7 @@ class Logger(object): The values are: - ``True`` - Scalars without specific titles are grouped together in the "Scalars" plot, preserving - backward compatibility with Trains automagical behavior. + backward compatibility with ClearML automagical behavior. - ``False`` - TensorBoard scalars without titles get a title/series with the same tag. (default) :type group_scalars: bool """ @@ -1214,7 +1214,7 @@ class Logger(object): try: # make sure we are writing to the original stdout StdStreamPatch.stderr_original_write( - 'trains.Logger failed sending log [level {}]: "{}"\n'.format(level, msg)) + 'clearml.Logger failed sending log [level {}]: "{}"\n'.format(level, msg)) except Exception: pass else: diff --git a/clearml/model.py b/clearml/model.py index f13064d4..667ed0d9 100644 --- a/clearml/model.py +++ b/clearml/model.py @@ -472,9 +472,9 @@ class InputModel(Model): framework, and indicate whether to immediately set the model's status to ``Published``. The model is read-only. - The **Trains Server** (backend) may already store the model's URL. If the input model's URL is not - stored, meaning the model is new, then it is imported and Trains stores its metadata. - If the URL is already stored, the import process stops, Trains issues a warning message, and Trains + The **ClearML Server** (backend) may already store the model's URL. If the input model's URL is not + stored, meaning the model is new, then it is imported and ClearML stores its metadata. + If the URL is already stored, the import process stops, ClearML issues a warning message, and ClearML reuses the model. In your Python experiment script, after importing the model, you can connect it to the main execution @@ -482,12 +482,12 @@ class InputModel(Model): network. .. note:: - Using the **Trains Web-App** (user interface), you can reuse imported models and switch models in + Using the **ClearML Web-App** (user interface), you can reuse imported models and switch models in experiments. - :param str weights_url: A valid URL for the initial weights file. If the **Trains Web-App** (backend) + :param str weights_url: A valid URL for the initial weights file. If the **ClearML Web-App** (backend) already stores the metadata of a model with the same URL, that existing model is returned - and Trains ignores all other parameters. + and ClearML ignores all other parameters. For example: @@ -715,7 +715,7 @@ class InputModel(Model): def __init__(self, model_id): # type: (str) -> None """ - :param str model_id: The Trains Id (system UUID) of the input model whose metadata the **Trains Server** + :param str model_id: The ClearML Id (system UUID) of the input model whose metadata the **ClearML Server** (backend) stores. """ super(InputModel, self).__init__(model_id) @@ -731,16 +731,16 @@ class InputModel(Model): Connect the current model to a Task object, if the model is preexisting. Preexisting models include: - Imported models (InputModel objects created using the :meth:`Logger.import_model` method). - - Models whose metadata is already in the Trains platform, meaning the InputModel object is instantiated - from the ``InputModel`` class specifying the the model's Trains Id as an argument. - - Models whose origin is not Trains that are used to create an InputModel object. For example, + - Models whose metadata is already in the ClearML platform, meaning the InputModel object is instantiated + from the ``InputModel`` class specifying the the model's ClearML Id as an argument. + - Models whose origin is not ClearML that are used to create an InputModel object. For example, models created using TensorFlow models. When the experiment is executed remotely in a worker, the input model already specified in the experiment is used. .. note:: - The **Trains Web-App** allows you to switch one input model for another and then enqueue the experiment + The **ClearML Web-App** allows you to switch one input model for another and then enqueue the experiment to execute in a worker. :param object task: A Task object. @@ -789,7 +789,7 @@ class OutputModel(BaseModel): .. note:: When executing a Task (experiment) remotely in a worker, you can modify the model configuration and / or model's - label enumeration using the **Trains Web-App**. + label enumeration using the **ClearML Web-App**. """ @property @@ -990,7 +990,7 @@ class OutputModel(BaseModel): Connect the current model to a Task object, if the model is a preexisting model. Preexisting models include: - Imported models. - - Models whose metadata the **Trains Server** (backend) is already storing. + - Models whose metadata the **ClearML Server** (backend) is already storing. - Models from another source, such as frameworks like TensorFlow. :param object task: A Task object. @@ -1044,8 +1044,8 @@ class OutputModel(BaseModel): Using this method, files uploads are separate and then a link to each is stored in the model object. .. note:: - For storage requiring credentials, the credentials are stored in the Trains configuration file, - ``~/trains.conf``. + For storage requiring credentials, the credentials are stored in the ClearML configuration file, + ``~/clearml.conf``. :param str uri: The URI of the upload storage destination. diff --git a/clearml/storage/cache.py b/clearml/storage/cache.py index 3e1af1e1..3a80dc8f 100644 --- a/clearml/storage/cache.py +++ b/clearml/storage/cache.py @@ -58,10 +58,10 @@ class CacheManager(object): return cached_file @staticmethod - def upload_file(local_file, remote_url, wait_for_upload=True): + def upload_file(local_file, remote_url, wait_for_upload=True, retries=1): helper = StorageHelper.get(remote_url) result = helper.upload( - local_file, remote_url, async_enable=not wait_for_upload + local_file, remote_url, async_enable=not wait_for_upload, retries=retries, ) CacheManager._add_remote_url(remote_url, local_file) return result diff --git a/clearml/storage/helper.py b/clearml/storage/helper.py index f8804284..efaa9487 100644 --- a/clearml/storage/helper.py +++ b/clearml/storage/helper.py @@ -681,6 +681,7 @@ class StorageHelper(object): # try to get file size try: if isinstance(self._driver, _HttpDriver) and obj: + obj = self._driver._get_download_object(obj) total_size_mb = float(obj.headers.get('Content-Length', 0)) / (1024 * 1024) elif hasattr(obj, 'size'): size = obj.size @@ -785,12 +786,12 @@ class StorageHelper(object): def check_write_permissions(self, dest_path=None): # create a temporary file, then de;ete it base_url = dest_path or self._base_url - dest_path = base_url + '/.trains.test' + dest_path = base_url + '/.clearml.test' # do not check http/s connection permissions if dest_path.startswith('http'): return True try: - self.upload_from_stream(stream=six.BytesIO(b'trains'), dest_path=dest_path) + self.upload_from_stream(stream=six.BytesIO(b'clearml'), dest_path=dest_path) self.delete(path=dest_path) except Exception: raise ValueError('Insufficient permissions for {}'.format(base_url)) @@ -1024,6 +1025,11 @@ class _HttpDriver(_Driver): return self._default_backend_session.add_auth_headers({}) return None + class _HttpSessionHandle(object): + def __init__(self, url, is_stream, container_name, object_name): + self.url, self.is_stream, self.container_name, self.object_name = \ + url, is_stream, container_name, object_name + def __init__(self, retries=5): self._retries = retries self._containers = {} @@ -1055,24 +1061,39 @@ class _HttpDriver(_Driver): def list_container_objects(self, *args, **kwargs): raise NotImplementedError('List is not implemented for http protocol') - def delete_object(self, *args, **kwargs): - raise NotImplementedError('Delete is not implemented for http protocol') + def delete_object(self, obj, *args, **kwargs): + assert isinstance(obj, self._HttpSessionHandle) + container = self._containers[obj.container_name] + res = container.session.delete(obj.url, headers=container.get_headers(obj.url)) + if res.status_code != requests.codes.ok: + raise ValueError('Failed deleting object %s (%d): %s' % (obj.object_name, res.status_code, res.text)) + return res def get_object(self, container_name, object_name, *args, **kwargs): - container = self._containers[container_name] - # set stream flag before get request - container.session.stream = kwargs.get('stream', True) + is_stream = kwargs.get('stream', True) url = ''.join((container_name, object_name.lstrip('/'))) - res = container.session.get(url, timeout=self.timeout, headers=container.get_headers(url)) + return self._HttpSessionHandle(url, is_stream, container_name, object_name) + + def _get_download_object(self, obj): + # bypass for session result + if not isinstance(obj, self._HttpSessionHandle): + return obj + + container = self._containers[obj.container_name] + # set stream flag before we send the request + container.session.stream = obj.is_stream + res = container.session.get(obj.url, timeout=self.timeout, headers=container.get_headers(obj.url)) if res.status_code != requests.codes.ok: - raise ValueError('Failed getting object %s (%d): %s' % (object_name, res.status_code, res.text)) + raise ValueError('Failed getting object %s (%d): %s' % (obj.object_name, res.status_code, res.text)) return res def download_object_as_stream(self, obj, chunk_size=64 * 1024, **_): # return iterable object + obj = self._get_download_object(obj) return obj.iter_content(chunk_size=chunk_size) def download_object(self, obj, local_path, overwrite_existing=True, delete_on_failure=True, callback=None, **_): + obj = self._get_download_object(obj) p = Path(local_path) if not overwrite_existing and p.is_file(): log.warning('failed saving after download: overwrite=False and file exists (%s)' % str(p)) diff --git a/clearml/storage/manager.py b/clearml/storage/manager.py index 13d13f8f..c3426b30 100644 --- a/clearml/storage/manager.py +++ b/clearml/storage/manager.py @@ -48,8 +48,8 @@ class StorageManager(object): @classmethod def upload_file( - cls, local_file, remote_url, wait_for_upload=True - ): # type: (str, str, bool) -> str + cls, local_file, remote_url, wait_for_upload=True, retries=1 + ): # type: (str, str, bool, int) -> str """ Upload a local file to a remote location. remote url is the finale destination of the uploaded file. @@ -64,12 +64,14 @@ class StorageManager(object): :param str local_file: Full path of a local file to be uploaded :param str remote_url: Full path or remote url to upload to (including file name) :param bool wait_for_upload: If False, return immediately and upload in the background. Default True. + :param int retries: Number of retries before failing to upload file, default 1. :return: Newly uploaded remote URL. """ return CacheManager.get_cache_manager().upload_file( local_file=local_file, remote_url=remote_url, wait_for_upload=wait_for_upload, + retries=retries, ) @classmethod diff --git a/clearml/storage/util.py b/clearml/storage/util.py index ce53889b..9875e0d2 100644 --- a/clearml/storage/util.py +++ b/clearml/storage/util.py @@ -1,6 +1,6 @@ import hashlib import sys -from typing import Optional +from typing import Optional, Union from six.moves.urllib.parse import quote, urlparse, urlunparse import six @@ -72,8 +72,23 @@ def sha256sum(filename, skip_header=0, block_size=65536): return h.hexdigest(), file_hash.hexdigest() if skip_header else None +def md5text(text, seed=1337): + # type: (str, Union[int, str]) -> str + """ + Return md5 hash of a string + Do not use this hash for security, if needed use something stronger like SHA2 + + :param text: string to hash + :param seed: use prefix seed for hashing + :return: md5 string + """ + h = hashlib.md5() + h.update((str(seed) + str(text)).encode('utf-8')) + return h.hexdigest() + + def is_windows(): """ :return: True if currently running on windows OS """ - return sys.platform == 'win32' \ No newline at end of file + return sys.platform == 'win32' diff --git a/clearml/task.py b/clearml/task.py index ee70255b..12b3a9ef 100644 --- a/clearml/task.py +++ b/clearml/task.py @@ -77,8 +77,8 @@ class Task(_Task): configuration, label enumeration, models, and other artifacts. The term "main execution Task" refers to the Task context for current running experiment. Python experiment scripts - can create one, and only one, main execution Task. It is a traceable, and after a script runs and Trains stores - the Task in the **Trains Server** (backend), it is modifiable, reproducible, executable by a worker, and you + can create one, and only one, main execution Task. It is a traceable, and after a script runs and ClearML stores + the Task in the **ClearML Server** (backend), it is modifiable, reproducible, executable by a worker, and you can duplicate it for further experimentation. The ``Task`` class and its methods allow you to create and manage experiments, as well as perform @@ -93,7 +93,7 @@ class Task(_Task): - Create a new reproducible Task - :meth:`Task.init` .. important:: - In some cases, ``Task.init`` may return a Task object which is already stored in **Trains Server** (already + In some cases, ``Task.init`` may return a Task object which is already stored in **ClearML Server** (already initialized), instead of creating a new Task. For a detailed explanation of those cases, see the ``Task.init`` method. @@ -102,17 +102,17 @@ class Task(_Task): - Get another (different) Task - :meth:`Task.get_task` .. note:: - The **Trains** documentation often refers to a Task as, "Task (experiment)". + The **ClearML** documentation often refers to a Task as, "Task (experiment)". - "Task" refers to the class in the Trains Python Client Package, the object in your Python experiment script, - and the entity with which **Trains Server** and **Trains Agent** work. + "Task" refers to the class in the ClearML Python Client Package, the object in your Python experiment script, + and the entity with which **ClearML Server** and **ClearML Agent** work. "Experiment" refers to your deep learning solution, including its connected components, inputs, and outputs, - and is the experiment you can view, analyze, compare, modify, duplicate, and manage using the Trains + and is the experiment you can view, analyze, compare, modify, duplicate, and manage using the ClearML **Web-App** (UI). Therefore, a "Task" is effectively an "experiment", and "Task (experiment)" encompasses its usage throughout - the Trains. + the ClearML. The exception to this Task behavior is sub-tasks (non-reproducible Tasks), which do not use the main execution Task. Creating a sub-task always creates a new Task with a new Task ID. @@ -197,7 +197,7 @@ class Task(_Task): Creates a new Task (experiment) if: - The Task never ran before. No Task with the same ``task_name`` and ``project_name`` is stored in - **Trains Server**. + **ClearML Server**. - The Task has run before (the same ``task_name`` and ``project_name``), and (a) it stored models and / or artifacts, or (b) its status is Published , or (c) it is Archived. - A new Task is forced by calling ``Task.init`` with ``reuse_last_task_id=False``. @@ -215,7 +215,7 @@ class Task(_Task): .. code-block:: py - from trains import Task + from clearml import Task task = Task.init('myProject', 'myTask') If this code runs again, it will not create a new Task. It does not store a model or artifact, @@ -285,7 +285,7 @@ class Task(_Task): This is equivalent to `continue_last_task=True` and `reuse_last_task_id=a_task_id_string`. :param str output_uri: The default location for output models and other artifacts. In the default location, - Trains creates a subfolder for the output. The subfolder structure is the following: + ClearML creates a subfolder for the output. The subfolder structure is the following: / / .< Task ID> @@ -297,9 +297,9 @@ class Task(_Task): - Azure Storage: ``azure://company.blob.core.windows.net/folder/`` .. important:: - For cloud storage, you must install the **Trains** package for your cloud storage type, + For cloud storage, you must install the **ClearML** package for your cloud storage type, and then configure your storage credentials. For detailed information, see - `Trains Python Client Extras <./references/trains_extras_storage/>`_ in the "Trains Python Client + `ClearML Python Client Extras <./references/clearml_extras_storage/>`_ in the "ClearML Python Client Reference" section. :param auto_connect_arg_parser: Automatically connect an argparse object to the Task @@ -324,7 +324,7 @@ class Task(_Task): :param auto_connect_frameworks: Automatically connect frameworks This includes patching MatplotLib, XGBoost, scikit-learn, Keras callbacks, and TensorBoard/X to serialize plots, graphs, and the model location to - the **Trains Server** (backend), in addition to original output destination. + the **ClearML Server** (backend), in addition to original output destination. The values are: @@ -342,7 +342,7 @@ class Task(_Task): 'xgboost': True, 'scikit': True, 'fastai': True, 'lightgbm': True, 'hydra': True} :param bool auto_resource_monitoring: Automatically create machine resource monitoring plots - These plots appear in in the **Trains Web-App (UI)**, **RESULTS** tab, **SCALARS** sub-tab, + These plots appear in in the **ClearML Web-App (UI)**, **RESULTS** tab, **SCALARS** sub-tab, with a title of **:resource monitor:**. The values are: @@ -409,6 +409,7 @@ class Task(_Task): # create a new logger (to catch stdout/err) cls.__main_task._logger = None cls.__main_task.__reporter = None + # noinspection PyProtectedMember cls.__main_task._get_logger(auto_connect_streams=auto_connect_streams) cls.__main_task._artifacts_manager = Artifacts(cls.__main_task) # unregister signal hooks, they cause subprocess to hang @@ -569,10 +570,10 @@ class Task(_Task): # show the debug metrics page in the log, it is very convenient if not is_sub_process_task_id: if cls._offline_mode: - logger.report_text('TRAINS running in offline mode, session stored in {}'.format( + logger.report_text('ClearML running in offline mode, session stored in {}'.format( task.get_offline_mode_folder())) else: - logger.report_text('TRAINS results page: {}'.format(task.get_output_log_web_page())) + logger.report_text('ClearML results page: {}'.format(task.get_output_log_web_page())) # Make sure we start the dev worker if required, otherwise it will only be started when we write # something to the log. task._dev_mode_task_start() @@ -580,55 +581,76 @@ class Task(_Task): return task @classmethod - def create(cls, project_name=None, task_name=None, task_type=TaskTypes.training): - # type: (Optional[str], Optional[str], Task.TaskTypes) -> Task + def create( + cls, + project_name=None, # Optional[str] + task_name=None, # Optional[str] + task_type=None, # Optional[str] + repo=None, # Optional[str] + branch=None, # Optional[str] + commit=None, # Optional[str] + script=None, # Optional[str] + working_directory=None, # Optional[str] + packages=None, # Optional[Sequence[str]] + requirements_file=None, # Optional[Union[str, Path]] + docker=None, # Optional[str] + base_task_id=None, # Optional[str] + add_task_init_call=True, # bool + ): + # type: (...) -> Task """ - Create a new, non-reproducible Task (experiment). This is called a sub-task. + Manually create and populate a new Task (experiment) in the system. + If the code does not already contain a call to ``Task.init``, pass add_task_init_call=True, + and the code will be patched in remote execution (i.e. when executed by `clearml-agent` .. note:: - This method always creates a new, non-reproducible Task. To create a reproducible Task, call the - :meth:`Task.init` method. To reference another Task, call the :meth:`Task.get_task` method . + This method **always** creates a new Task. + Use :meth:`Task.init` method to automatically create and populate task for the running process. + To reference an existing Task, call the :meth:`Task.get_task` method . - :param str project_name: The name of the project in which the experiment will be created. - If ``project_name`` is ``None``, and the main execution Task is initialized (see :meth:`Task.init`), - then the main execution Task's project is used. Otherwise, if the project does - not exist, it is created. (Optional) - :param str task_name: The name of Task (experiment). - :param TaskTypes task_type: The task type. + :param project_name: Set the project name for the task. Required if base_task_id is None. + :param task_name: Set the name of the remote task. Required if base_task_id is None. + :param task_type: Optional, The task type to be created. Supported values: 'training', 'testing', 'inference', + 'data_processing', 'application', 'monitor', 'controller', 'optimizer', 'service', 'qc', 'custom' + :param repo: Remote URL for the repository to use, or path to local copy of the git repository + Example: 'https://github.com/allegroai/clearml.git' or '~/project/repo' + :param branch: Select specific repository branch/tag (implies the latest commit from the branch) + :param commit: Select specific commit id to use (default: latest commit, + or when used with local repository matching the local commit id) + :param script: Specify the entry point script for the remote execution. When used in tandem with + remote git repository the script should be a relative path inside the repository, + for example: './source/train.py' . When used with local repository path it supports a + direct path to a file inside the local repository itself, for example: '~/project/source/train.py' + :param working_directory: Working directory to launch the script from. Default: repository root folder. + Relative to repo root or local folder. + :param packages: Manually specify a list of required packages. Example: ["tqdm>=2.1", "scikit-learn"] + :param requirements_file: Specify requirements.txt file to install when setting the session. + If not provided, the requirements.txt from the repository will be used. + :param docker: Select the docker image to be executed in by the remote session + :param base_task_id: Use a pre-existing task in the system, instead of a local repo/script. + Essentially clones an existing task and overrides arguments/requirements. + :param add_task_init_call: If True, a 'Task.init()' call is added to the script entry point in remote execution. - Valid task types: - - - ``TaskTypes.training`` (default) - - ``TaskTypes.testing`` - - ``TaskTypes.inference`` - - ``TaskTypes.data_processing`` - - ``TaskTypes.application`` - - ``TaskTypes.monitor`` - - ``TaskTypes.controller`` - - ``TaskTypes.optimizer`` - - ``TaskTypes.service`` - - ``TaskTypes.qc`` - - ``TaskTypes.custom`` - - :return: A new experiment. + :return: The newly created Task (experiment) """ - if not project_name: + if not project_name and not base_task_id: if not cls.__main_task: raise ValueError("Please provide project_name, no global task context found " "(Task.current_task hasn't been called)") project_name = cls.__main_task.get_project_name() + from .backend_interface.task.populate import CreateAndPopulate + manual_populate = CreateAndPopulate( + project_name=project_name, task_name=task_name, task_type=task_type, + repo=repo, branch=branch, commit=commit, + script=script, working_directory=working_directory, + packages=packages, requirements_file=requirements_file, + docker=docker, + base_task_id=base_task_id, + add_task_init_call=add_task_init_call, + raise_on_missing_entries=False, + ) + task = manual_populate.create_task() - try: - task = cls( - private=cls.__create_protection, - project_name=project_name, - task_name=task_name, - task_type=task_type, - log_to_backend=False, - force_create=True, - ) - except Exception: - raise return task @classmethod @@ -721,7 +743,7 @@ class Task(_Task): helper = StorageHelper.get(value) if not helper: raise ValueError("Could not get access credentials for '{}' " - ", check configuration file ~/trains.conf".format(value)) + ", check configuration file ~/clearml.conf".format(value)) helper.check_write_permissions(value) self.storage_uri = value @@ -758,7 +780,7 @@ class Task(_Task): """ Get a Logger object for reporting, for this task context. You can view all Logger report output associated with the Task for which this method is called, including metrics, plots, text, tables, and images, in the - **Trains Web-App (UI)**. + **ClearML Web-App (UI)**. :return: The Logger object for the current Task (experiment). """ @@ -796,8 +818,8 @@ class Task(_Task): """ assert isinstance(source_task, (six.string_types, Task)) if not Session.check_min_api_version('2.4'): - raise ValueError("Trains-server does not support DevOps features, " - "upgrade trains-server to 0.12.0 or above") + raise ValueError("ClearML-server does not support DevOps features, " + "upgrade clearml-server to 0.12.0 or above") task_id = source_task if isinstance(source_task, six.string_types) else source_task.id if not parent: @@ -820,7 +842,7 @@ class Task(_Task): .. note:: A worker daemon must be listening at the queue for the worker to fetch the Task and execute it, - see `Use Case Examples <../trains_agent_ref/#use-case-examples>`_ on the "Trains Agent + see `Use Case Examples <../clearml_agent_ref/#use-case-examples>`_ on the "ClearML Agent Reference page. :param Task/str task: The Task to enqueue. Specify a Task object or Task ID. @@ -859,8 +881,8 @@ class Task(_Task): """ assert isinstance(task, (six.string_types, Task)) if not Session.check_min_api_version('2.4'): - raise ValueError("Trains-server does not support DevOps features, " - "upgrade trains-server to 0.12.0 or above") + raise ValueError("ClearML-server does not support DevOps features, " + "upgrade clearml-server to 0.12.0 or above") # make sure we have wither name ot id mutually_exclusive(queue_name=queue_name, queue_id=queue_id) @@ -923,8 +945,8 @@ class Task(_Task): """ assert isinstance(task, (six.string_types, Task)) if not Session.check_min_api_version('2.4'): - raise ValueError("Trains-server does not support DevOps features, " - "upgrade trains-server to 0.12.0 or above") + raise ValueError("ClearML-server does not support DevOps features, " + "upgrade clearml-server to 0.12.0 or above") task_id = task if isinstance(task, six.string_types) else task.id session = cls._get_default_session() @@ -990,7 +1012,7 @@ class Task(_Task): name = self._default_configuration_section_name if not multi_config_support and name and name != self._default_configuration_section_name: - raise ValueError("Multiple configurations is not supported with the current 'trains-server', " + raise ValueError("Multiple configurations is not supported with the current 'clearml-server', " "please upgrade to the latest version") for mutable_type, method in dispatch: @@ -1024,11 +1046,11 @@ class Task(_Task): :param configuration: The configuration. This is usually the configuration used in the model training process. Specify one of the following: - - A dictionary - A dictionary containing the configuration. Trains stores the configuration in - the **Trains Server** (backend), in a HOCON format (JSON-like format) which is editable. - - A ``pathlib2.Path`` string - A path to the configuration file. Trains stores the content of the file. + - A dictionary - A dictionary containing the configuration. ClearML stores the configuration in + the **ClearML Server** (backend), in a HOCON format (JSON-like format) which is editable. + - A ``pathlib2.Path`` string - A path to the configuration file. ClearML stores the content of the file. A local path must be relative path. When executing a Task remotely in a worker, the contents brought - from the **Trains Server** (backend) overwrites the contents of the file. + from the **ClearML Server** (backend) overwrites the contents of the file. :param str name: Configuration section name. default: 'General' Allowing users to store multiple configuration dicts/files @@ -1038,10 +1060,10 @@ class Task(_Task): :return: If a dictionary is specified, then a dictionary is returned. If pathlib2.Path / string is specified, then a path to a local configuration file is returned. Configuration object. """ - pathlib_Path = None + pathlib_Path = None # noqa if not isinstance(configuration, (dict, Path, six.string_types)): try: - from pathlib import Path as pathlib_Path + from pathlib import Path as pathlib_Path # noqa except ImportError: pass if not pathlib_Path or not isinstance(configuration, pathlib_Path): @@ -1053,7 +1075,7 @@ class Task(_Task): name = self._default_configuration_section_name if not multi_config_support and name and name != self._default_configuration_section_name: - raise ValueError("Multiple configurations is not supported with the current 'trains-server', " + raise ValueError("Multiple configurations is not supported with the current 'clearml-server', " "please upgrade to the latest version") # parameter dictionary @@ -1141,7 +1163,7 @@ class Task(_Task): return configuration configuration_path = Path(configuration) - fd, local_filename = mkstemp(prefix='trains_task_config_', + fd, local_filename = mkstemp(prefix='clearml_task_config_', suffix=configuration_path.suffixes[-1] if configuration_path.suffixes else '.txt') os.write(fd, configuration_text.encode('utf-8')) @@ -1187,7 +1209,7 @@ class Task(_Task): """ Get a Logger object for reporting, for this task context. You can view all Logger report output associated with the Task for which this method is called, including metrics, plots, text, tables, and images, in the - **Trains Web-App (UI)**. + **ClearML Web-App (UI)**. :return: The Logger for the Task (experiment). """ @@ -1247,7 +1269,7 @@ class Task(_Task): def reset(self, set_started_on_success=False, force=False): # type: (bool, bool) -> None """ - Reset a Task. Trains reloads a Task after a successful reset. + Reset a Task. ClearML reloads a Task after a successful reset. When a worker executes a Task remotely, the Task does not reset unless the ``force`` parameter is set to ``True`` (this avoids accidentally clearing logs and metrics). @@ -1290,16 +1312,16 @@ class Task(_Task): # type: (str, pandas.DataFrame, Dict, Union[bool, Sequence[str]]) -> None """ Register (add) an artifact for the current Task. Registered artifacts are dynamically sychronized with the - **Trains Server** (backend). If a registered artifact is updated, the update is stored in the - **Trains Server** (backend). Registered artifacts are primarily used for Data Audition. + **ClearML Server** (backend). If a registered artifact is updated, the update is stored in the + **ClearML Server** (backend). Registered artifacts are primarily used for Data Audition. The currently supported registered artifact object type is a pandas.DataFrame. See also :meth:`Task.unregister_artifact` and :meth:`Task.get_registered_artifacts`. .. note:: - Trains also supports uploaded artifacts which are one-time uploads of static artifacts that are not - dynamically sychronized with the **Trains Server** (backend). These static artifacts include + ClearML also supports uploaded artifacts which are one-time uploads of static artifacts that are not + dynamically sychronized with the **ClearML Server** (backend). These static artifacts include additional object types. For more information, see :meth:`Task.upload_artifact`. :param str name: The name of the artifact. @@ -1308,7 +1330,7 @@ class Task(_Task): If an artifact with the same name was previously registered, it is overwritten. :param object artifact: The artifact object. :param dict metadata: A dictionary of key-value pairs for any metadata. This dictionary appears with the - experiment in the **Trains Web-App (UI)**, **ARTIFACTS** tab. + experiment in the **ClearML Web-App (UI)**, **ARTIFACTS** tab. :param uniqueness_columns: A Sequence of columns for artifact uniqueness comparison criteria, or the default value of ``True``. If ``True``, the artifact uniqueness comparison criteria is all the columns, which is the same as ``artifact.columns``. @@ -1323,13 +1345,13 @@ class Task(_Task): def unregister_artifact(self, name): # type: (str) -> None """ - Unregister (remove) a registered artifact. This removes the artifact from the watch list that Trains uses - to synchronize artifacts with the **Trains Server** (backend). + Unregister (remove) a registered artifact. This removes the artifact from the watch list that ClearML uses + to synchronize artifacts with the **ClearML Server** (backend). .. important:: - - Calling this method does not remove the artifact from a Task. It only stops Trains from + - Calling this method does not remove the artifact from a Task. It only stops ClearML from monitoring the artifact. - - When this method is called, Trains immediately takes the last snapshot of the artifact. + - When this method is called, ClearML immediately takes the last snapshot of the artifact. """ self._artifacts_manager.unregister_artifact(name=name) @@ -1361,12 +1383,12 @@ class Task(_Task): The currently supported upload (static) artifact types include: - - string / pathlib2.Path - A path to artifact file. If a wildcard or a folder is specified, then Trains + - string / pathlib2.Path - A path to artifact file. If a wildcard or a folder is specified, then ClearML creates and uploads a ZIP file. - - dict - Trains stores a dictionary as ``.json`` file and uploads it. - - pandas.DataFrame - Trains stores a pandas.DataFrame as ``.csv.gz`` (compressed CSV) file and uploads it. - - numpy.ndarray - Trains stores a numpy.ndarray as ``.npz`` file and uploads it. - - PIL.Image - Trains stores a PIL.Image as ``.png`` file and uploads it. + - dict - ClearML stores a dictionary as ``.json`` file and uploads it. + - pandas.DataFrame - ClearML stores a pandas.DataFrame as ``.csv.gz`` (compressed CSV) file and uploads it. + - numpy.ndarray - ClearML stores a numpy.ndarray as ``.npz`` file and uploads it. + - PIL.Image - ClearML stores a PIL.Image as ``.png`` file and uploads it. - Any - If called with auto_pickle=True, the object will be pickled and uploaded. :param str name: The artifact name. @@ -1376,7 +1398,7 @@ class Task(_Task): :param object artifact_object: The artifact object. :param dict metadata: A dictionary of key-value pairs for any metadata. This dictionary appears with the - experiment in the **Trains Web-App (UI)**, **ARTIFACTS** tab. + experiment in the **ClearML Web-App (UI)**, **ARTIFACTS** tab. :param bool delete_after_upload: After the upload, delete the local copy of the artifact - ``True`` - Delete the local copy of the artifact. @@ -1416,7 +1438,7 @@ class Task(_Task): .. code-block:: py - {'input': [trains.Model()], 'output': [trains.Model()]} + {'input': [clearml.Model()], 'output': [clearml.Model()]} """ task_models = {'input': self._get_models(model_type='input'), @@ -1510,7 +1532,7 @@ class Task(_Task): .. note:: The maximum reported iteration is not in the local cache. This method - sends a request to the **Trains Server** (backend). + sends a request to the **ClearML Server** (backend). :return: The last reported iteration number. """ @@ -1704,7 +1726,7 @@ class Task(_Task): # type: (str) -> () """ Set the base docker image for this experiment - If provided, this value will be used by trains-agent to execute this experiment + If provided, this value will be used by clearml-agent to execute this experiment inside the provided docker image. """ if not self.running_locally() and self.is_main_task(): @@ -1732,12 +1754,12 @@ class Task(_Task): def execute_remotely(self, queue_name=None, clone=False, exit_process=True): # type: (Optional[str], bool, bool) -> Optional[Task] """ - If task is running locally (i.e., not by ``trains-agent``), then clone the Task and enqueue it for remote + If task is running locally (i.e., not by ``clearml-agent``), then clone the Task and enqueue it for remote execution; or, stop the execution of the current Task, reset its state, and enqueue it. If ``exit==True``, *exit* this process. .. note:: - If the task is running remotely (i.e., ``trains-agent`` is executing it), this call is a no-op + If the task is running remotely (i.e., ``clearml-agent`` is executing it), this call is a no-op (i.e., does nothing). :param queue_name: The queue name used for enqueueing the task. If ``None``, this call exits the process @@ -2006,12 +2028,12 @@ class Task(_Task): :param session_folder_zip: Path to a folder containing the session, or zip-file of the session folder. :return: Newly created task ID (str) """ - print('TRAINS: Importing offline session from {}'.format(session_folder_zip)) + print('ClearML: Importing offline session from {}'.format(session_folder_zip)) temp_folder = None if Path(session_folder_zip).is_file(): # unzip the file: - temp_folder = mkdtemp(prefix='trains-offline-') + temp_folder = mkdtemp(prefix='clearml-offline-') ZipFile(session_folder_zip).extractall(path=temp_folder) session_folder_zip = temp_folder @@ -2053,7 +2075,7 @@ class Task(_Task): # metrics Metrics.report_offline_session(task, session_folder) # print imported results page - print('TRAINS results page: {}'.format(task.get_output_log_web_page())) + print('ClearML results page: {}'.format(task.get_output_log_web_page())) task.completed() # close task task.close() @@ -2072,10 +2094,10 @@ class Task(_Task): def set_credentials(cls, api_host=None, web_host=None, files_host=None, key=None, secret=None, host=None): # type: (Optional[str], Optional[str], Optional[str], Optional[str], Optional[str], Optional[str]) -> () """ - Set new default **Trains Server** (backend) host and credentials. + Set new default **ClearML Server** (backend) host and credentials. - These credentials will be overridden by either OS environment variables, or the Trains configuration - file, ``trains.conf``. + These credentials will be overridden by either OS environment variables, or the ClearML configuration + file, ``clearml.conf``. .. warning:: Credentials must be set before initializing a Task object. @@ -2114,6 +2136,40 @@ class Task(_Task): Session.default_web = web_host or '' Session.default_files = files_host or '' + @classmethod + def _create(cls, project_name=None, task_name=None, task_type=TaskTypes.training): + # type: (Optional[str], Optional[str], Task.TaskTypes) -> Task + """ + Create a new unpopulated Task (experiment). + + :param str project_name: The name of the project in which the experiment will be created. + If ``project_name`` is ``None``, and the main execution Task is initialized (see :meth:`Task.init`), + then the main execution Task's project is used. Otherwise, if the project does + not exist, it is created. (Optional) + :param str task_name: The name of Task (experiment). + :param TaskTypes task_type: The task type. + + :return: The newly created task created. + """ + if not project_name: + if not cls.__main_task: + raise ValueError("Please provide project_name, no global task context found " + "(Task.current_task hasn't been called)") + project_name = cls.__main_task.get_project_name() + + try: + task = cls( + private=cls.__create_protection, + project_name=project_name, + task_name=task_name, + task_type=task_type, + log_to_backend=False, + force_create=True, + ) + except Exception: + raise + return task + def _set_model_config(self, config_text=None, config_dict=None): # type: (Optional[str], Optional[Mapping]) -> None """ @@ -2285,15 +2341,15 @@ class Task(_Task): # force update of base logger to this current task (this is the main logger task) logger = task._get_logger(auto_connect_streams=auto_connect_streams) if closed_old_task: - logger.report_text('TRAINS Task: Closing old development task id={}'.format(default_task.get('id'))) + logger.report_text('ClearML Task: Closing old development task id={}'.format(default_task.get('id'))) # print warning, reusing/creating a task if default_task_id and not continue_last_task: - logger.report_text('TRAINS Task: overwriting (reusing) task id=%s' % task.id) + logger.report_text('ClearML Task: overwriting (reusing) task id=%s' % task.id) elif default_task_id and continue_last_task: - logger.report_text('TRAINS Task: continuing previous task id=%s ' + logger.report_text('ClearML Task: continuing previous task id=%s ' 'Notice this run will not be reproducible!' % task.id) else: - logger.report_text('TRAINS Task: created new task id=%s' % task.id) + logger.report_text('ClearML Task: created new task id=%s' % task.id) # update current repository and put warning into logs if detect_repo: @@ -2567,8 +2623,7 @@ class Task(_Task): self._kill_all_child_processes(send_kill=False) time.sleep(2.0) self._kill_all_child_processes(send_kill=True) - # noinspection PyProtectedMember - os._exit(1) + os._exit(1) # noqa @staticmethod def _kill_all_child_processes(send_kill=False): @@ -2800,7 +2855,7 @@ class Task(_Task): if filename.is_file(): relative_file_name = filename.relative_to(offline_folder).as_posix() zf.write(filename.as_posix(), arcname=relative_file_name) - print('TRAINS Task: Offline session stored in {}'.format(zip_file)) + print('ClearML Task: Offline session stored in {}'.format(zip_file)) except Exception: pass @@ -3179,8 +3234,8 @@ class Task(_Task): task_data.get('type') not in (cls.TaskTypes.training, cls.TaskTypes.testing) and \ not Session.check_min_api_version(2.8): print('WARNING: Changing task type to "{}" : ' - 'trains-server does not support task type "{}", ' - 'please upgrade trains-server.'.format(cls.TaskTypes.training, task_data['type'].value)) + 'clearml-server does not support task type "{}", ' + 'please upgrade clearml-server.'.format(cls.TaskTypes.training, task_data['type'].value)) task_data['type'] = cls.TaskTypes.training compares = ( diff --git a/clearml/utilities/args.py b/clearml/utilities/args.py index 7900d1b9..0d5fa853 100644 --- a/clearml/utilities/args.py +++ b/clearml/utilities/args.py @@ -46,7 +46,7 @@ class PatchArgumentParser: from ..config import running_remotely, get_remote_task_id if running_remotely(): # this will cause the current_task() to set PatchArgumentParser._current_task - from trains import Task + from clearml import Task # noinspection PyBroadException try: current_task = Task.get_task(task_id=get_remote_task_id()) diff --git a/clearml/utilities/check_updates.py b/clearml/utilities/check_updates.py index 8bec7c46..23916c6b 100644 --- a/clearml/utilities/check_updates.py +++ b/clearml/utilities/check_updates.py @@ -27,10 +27,10 @@ class CheckPackageUpdates(object): cls._package_version_checked = True client, version = Session._client[0] version = Version(version) - is_demo = 'https://demoapi.trains.allegro.ai/'.startswith(Session.get_api_server_host()) + is_demo = 'https://demoapi.demo.clear.ml/'.startswith(Session.get_api_server_host()) update_server_releases = requests.get( - 'https://updates.trains.allegro.ai/updates', + 'https://updates.clear.ml/updates', json={"demo": is_demo, "versions": {c: str(v) for c, v in Session._client}, "CI": str(os.environ.get('CI', ''))}, @@ -62,13 +62,13 @@ class CheckPackageUpdates(object): @staticmethod def get_version_from_updates_server(cur_version): """ - Get the latest version for trains from updates server - :param cur_version: The current running version of trains + Get the latest version for clearml from updates server + :param cur_version: The current running version of clearml :type cur_version: Version """ try: - _ = requests.get('https://updates.trains.allegro.ai/updates', - data=json.dumps({"versions": {"trains": str(cur_version)}}), + _ = requests.get('https://updates.clear.ml/updates', + data=json.dumps({"versions": {"clearml": str(cur_version)}}), timeout=1.0) return except Exception: diff --git a/clearml/utilities/proxy_object.py b/clearml/utilities/proxy_object.py index 837f80e1..14ce49cf 100644 --- a/clearml/utilities/proxy_object.py +++ b/clearml/utilities/proxy_object.py @@ -77,6 +77,19 @@ class ProxyDictPreWrite(dict): return self._set_callback((prefix + '.' + key_value[0], key_value[1],)) +def verify_basic_type(a_dict_list, basic_types=None): + basic_types = (float, int, bool, six.string_types, ) if not basic_types else \ + tuple(b for b in basic_types if b not in (list, tuple, dict)) + + if isinstance(a_dict_list, basic_types): + return True + if isinstance(a_dict_list, (list, tuple)): + return all(verify_basic_type(v) for v in a_dict_list) + elif isinstance(a_dict_list, dict): + return all(verify_basic_type(k) for k in a_dict_list.keys()) and \ + all(verify_basic_type(v) for v in a_dict_list.values()) + + def flatten_dictionary(a_dict, prefix=''): flat_dict = {} sep = '/' @@ -88,7 +101,11 @@ def flatten_dictionary(a_dict, prefix=''): elif isinstance(v, (list, tuple)) and all([isinstance(i, basic_types) for i in v]): flat_dict[prefix + k] = v elif isinstance(v, dict): - flat_dict.update(flatten_dictionary(v, prefix=prefix + k + sep)) + nested_flat_dict = flatten_dictionary(v, prefix=prefix + k + sep) + if nested_flat_dict: + flat_dict.update(nested_flat_dict) + else: + flat_dict[k] = {} else: # this is a mixture of list and dict, or any other object, # leave it as is, we have nothing to do with it. diff --git a/clearml/utilities/resource_monitor.py b/clearml/utilities/resource_monitor.py index 15f4d9b2..f64c2710 100644 --- a/clearml/utilities/resource_monitor.py +++ b/clearml/utilities/resource_monitor.py @@ -41,7 +41,7 @@ class ResourceMonitor(object): self._last_process_pool = {} self._last_process_id_list = [] if not self._gpustat: - self._task.get_logger().report_text('TRAINS Monitor: GPU monitoring is not available') + self._task.get_logger().report_text('ClearML Monitor: GPU monitoring is not available') else: # if running_remotely(): try: active_gpus = os.environ.get('NVIDIA_VISIBLE_DEVICES', '') or \ @@ -105,13 +105,13 @@ class ResourceMonitor(object): if IsTensorboardInit.tensorboard_used(): fallback_to_sec_as_iterations = False elif seconds_since_started >= self.wait_for_first_iteration: - self._task.get_logger().report_text('TRAINS Monitor: Could not detect iteration reporting, ' + self._task.get_logger().report_text('ClearML Monitor: Could not detect iteration reporting, ' 'falling back to iterations as seconds-from-start') fallback_to_sec_as_iterations = True elif fallback_to_sec_as_iterations is True and seconds_since_started <= self.max_check_first_iteration: if self._check_logger_reported(): fallback_to_sec_as_iterations = False - self._task.get_logger().report_text('TRAINS Monitor: Reporting detected, ' + self._task.get_logger().report_text('ClearML Monitor: Reporting detected, ' 'reverting back to iteration based reporting') clear_readouts = True @@ -231,7 +231,7 @@ class ResourceMonitor(object): # something happened and we can't use gpu stats, self._gpustat_fail += 1 if self._gpustat_fail >= 3: - self._task.get_logger().report_text('TRAINS Monitor: GPU monitoring failed getting GPU reading, ' + self._task.get_logger().report_text('ClearML Monitor: GPU monitoring failed getting GPU reading, ' 'switching off GPU monitoring') self._gpustat = None diff --git a/clearml/utilities/seed.py b/clearml/utilities/seed.py index ca67e8e5..aa71007c 100644 --- a/clearml/utilities/seed.py +++ b/clearml/utilities/seed.py @@ -12,7 +12,7 @@ def make_deterministic(seed=1337, cudnn_deterministic=False): Ensure deterministic behavior across PyTorch using the provided random seed. This function makes sure that torch, numpy and random use the same random seed. - When using trains's task, call this function using the task's random seed like so: + When using clearml's task, call this function using the task's random seed like so: make_deterministic(task.get_random_seed()) :param int seed: Seed number diff --git a/clearml/version.py b/clearml/version.py index 538eb5d8..610aac28 100644 --- a/clearml/version.py +++ b/clearml/version.py @@ -1 +1 @@ -__version__ = '0.16.4' +__version__ = '0.17.0rc0' diff --git a/docs/clearml-task.md b/docs/clearml-task.md new file mode 100644 index 00000000..9bbe41d8 --- /dev/null +++ b/docs/clearml-task.md @@ -0,0 +1,136 @@ +# `clearml-task` - Execute ANY python code on a remote machine + +If you are already familiar with `clearml`, then you can think of `clearml-task` as a way to create a Task/experiment +from any script without the need to add even a single line of code to the original codebase. + +`clearml-task` allows a user to **take any python code/repository and launch it on a remote machine**. + +The remote execution is fully monitored, all outputs - including console / tensorboard / matplotlib +are logged in real-time into the ClearML UI + +## What does it do? + +`clearml-task` creates a new experiment on your `clearml-server`; it populates the experiment's environment with: + +* repository/commit/branch, as specified by the command-line invocation. +* optional: the base docker image to be used as underlying environment +* optional: alternative python requirements, in case `requirements.txt` is not found inside the repository. + +Once the new experiment is created and populated, it will enqueue the experiment to the selected execution queue. + +When the experiment is executed on the remote machine (performed by an available `clearml-agent`), all the console outputs +will be logged in real-time, alongside your TensorBoard and matplotlib. + +### Use-cases for `clearml-task` remote execution + +- You have an off-the-shelf code, and you want to launch it on a remote machine with a specific resource (i.e., GPU) +- You want to run the [hyper-parameter optimization]() on a codebase that is still not connected with `clearml` +- You want to create a [pipeline]() from an assortment of scripts, and you need to create Tasks for those scripts +- Sometimes, you just want to run some code on a remote machine, either using an on-prem cluster or on the cloud... + +### Prerequisites + +- A single python script, or an up-to-date repository containing the codebase. +- `clearml-agent` running on at least one machine (to execute the experiment) + +## Tutorial + +### Launching a job from a repository + +We will be launching this [script](https://github.com/allegroai/trains/blob/master/examples/frameworks/scikit-learn/sklearn_matplotlib_example.py) on a remote machine. The following are the command-line options we will be using: +- First, we have to give the experiment a name and select a project (`--project examples --name remote_test`) +- Then, we select the repository with our code. If we do not specify branch / commit, it will take the latest commit + from the master branch (`--repo https://github.com/allegroai/clearml.git`) +- Lastly, we need to specify which script in the repository needs to be run (`--script examples/frameworks/scikit-learn/sklearn_matplotlib_example.py`) +Notice that by default, the execution working directory will be the root of the repository. If we need to change it, add `--cwd ` + +If we additionally need to pass an argument to our scripts, use the `--args` switch. + The names of the arguments should match the argparse arguments, removing the '--' prefix + (e.g. instead of --key=value -> use `--args key=value` ) + +``` bash +clearml-task --project examples --name remote_test --repo https://github.com/allegroai/clearml.git +--script examples/frameworks/scikit-learn/sklearn_matplotlib_example.py +--queue single_gpu +``` + +### Launching a job from a local script + +We will be launching a single local script file (no git repo needed) on a remote machine. + +- First, we have to give the experiment a name and select a project (`--project examples --name remote_test`) +- Then, we select the script file on our machine, `--script /path/to/my/script.py` +- If we need specific packages, we can specify them manually with `--packages "tqdm>=4" "torch>1.0"` + or we can pass a requirements file `--requirements /path/to/my/requirements.txt` +- Same as in the repo case, if we need to pass arguments to `argparse` we can add `--args key=value` +- If we have a docker container with an entire environment we want our script to run inside, + add e.g., `--docker nvcr.io/nvidia/pytorch:20.11-py3` + +Note: In this example, the exact version of PyTorch to install will be resolved by the `clearml-agent` depending on the CUDA environment available at runtime. + +``` bash +clearml-task --project examples --name remote_test --script /path/to/my/script.py +--packages "tqdm>=4" "torch>1.0" --args verbose=true +--queue dual_gpu +``` + +### CLI options + +``` bash +clearml-task --help +``` + +``` console +ClearML launch - launch any codebase on remote machines running clearml-agent + +optional arguments: + -h, --help show this help message and exit + --version Display the Allegro.ai utility version + --project PROJECT Required: set the project name for the task. If + --base-task-id is used, this arguments is optional. + --name NAME Required: select a name for the remote task + --repo REPO remote URL for the repository to use. Example: --repo + https://github.com/allegroai/clearml.git + --branch BRANCH Select specific repository branch/tag (implies the + latest commit from the branch) + --commit COMMIT Select specific commit id to use (default: latest + commit, or when used with local repository matching + the local commit id) + --folder FOLDER Remotely execute the code in the local folder. Notice! + It assumes a git repository already exists. Current + state of the repo (commit id and uncommitted changes) + is logged and will be replicated on the remote machine + --script SCRIPT Specify the entry point script for the remote + execution. When used in tandem with --repo the script + should be a relative path inside the repository, for + example: --script source/train.py .When used with + --folder it supports a direct path to a file inside + the local repository itself, for example: --script + ~/project/source/train.py + --cwd CWD Working directory to launch the script from. Default: + repository root folder. Relative to repo root or local + folder + --args [ARGS [ARGS ...]] + Arguments to pass to the remote execution, list of + = strings.Currently only argparse + arguments are supported. Example: --args lr=0.003 + batch_size=64 + --queue QUEUE Select the queue to launch the task. If not provided a + Task will be created but it will not be launched. + --requirements REQUIREMENTS + Specify requirements.txt file to install when setting + the session. If not provided, the requirements.txt + from the repository will be used. + --packages [PACKAGES [PACKAGES ...]] + Manually specify a list of required packages. Example: + --packages "tqdm>=2.1" "scikit-learn" + --docker DOCKER Select the docker image to use in the remote session + --skip-task-init If set, Task.init() call is not added to the entry + point, and is assumed to be called in within the + script. Default: add Task.init() call entry point + script + --base-task-id BASE_TASK_ID + Use a pre-existing task in the system, instead of a local repo/script. + Essentially clones an existing task and overrides arguments/requirements. + +``` diff --git a/docs/clearml.conf b/docs/clearml.conf new file mode 100644 index 00000000..cda232ad --- /dev/null +++ b/docs/clearml.conf @@ -0,0 +1,196 @@ +# ClearML SDK configuration file +api { + # web_server on port 8080 + web_server: "http://localhost:8080" + + # Notice: 'api_server' is the api server (default port 8008), not the web server. + api_server: "http://localhost:8008" + + # file server on port 8081 + files_server: "http://localhost:8081" + + # Credentials are generated using the webapp, http://localhost:8080/profile + credentials {"access_key": "EGRTCO8JMSIGI6S39GTP43NFWXDQOW", "secret_key": "x!XTov_G-#vspE*Y(h$Anm&DIc5Ou-F)jsl$PdOyj5wG1&E!Z8"} + + # verify host ssl certificate, set to False only if you have a very good reason + verify_certificate: True +} +sdk { + # ClearML - default SDK configuration + + storage { + cache { + # Defaults to system temp folder / cache + default_base_dir: "~/.clearml/cache" + } + } + + metrics { + # History size for debug files per metric/variant. For each metric/variant combination with an attached file + # (e.g. debug image event), file names for the uploaded files will be recycled in such a way that no more than + # X files are stored in the upload destination for each metric/variant combination. + file_history_size: 100 + + # Max history size for matplotlib imshow files per plot title. + # File names for the uploaded images will be recycled in such a way that no more than + # X images are stored in the upload destination for each matplotlib plot title. + matplotlib_untitled_history_size: 100 + + # Limit the number of digits after the dot in plot reporting (reducing plot report size) + # plot_max_num_digits: 5 + + # Settings for generated debug images + images { + format: JPEG + quality: 87 + subsampling: 0 + } + + # Support plot-per-graph fully matching Tensorboard behavior (i.e. if this is set to true, each series should have its own graph) + tensorboard_single_series_per_graph: false + } + + network { + metrics { + # Number of threads allocated to uploading files (typically debug images) when transmitting metrics for + # a specific iteration + file_upload_threads: 4 + + # Warn about upload starvation if no uploads were made in specified period while file-bearing events keep + # being sent for upload + file_upload_starvation_warning_sec: 120 + } + + iteration { + # Max number of retries when getting frames if the server returned an error (http code 500) + max_retries_on_server_error: 5 + # Backoff factory for consecutive retry attempts. + # SDK will wait for {backoff factor} * (2 ^ ({number of total retries} - 1)) between retries. + retry_backoff_factor_sec: 10 + } + } + aws { + s3 { + # S3 credentials, used for read/write access by various SDK elements + + # default, used for any bucket not specified below + key: "" + secret: "" + region: "" + + credentials: [ + # specifies key/secret credentials to use when handling s3 urls (read or write) + # { + # bucket: "my-bucket-name" + # key: "my-access-key" + # secret: "my-secret-key" + # }, + # { + # # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket) + # host: "my-minio-host:9000" + # key: "12345678" + # secret: "12345678" + # multipart: false + # secure: false + # } + ] + } + boto3 { + pool_connections: 512 + max_multipart_concurrency: 16 + } + } + google.storage { + # # Default project and credentials file + # # Will be used when no bucket configuration is found + # project: "clearml" + # credentials_json: "/path/to/credentials.json" + + # # Specific credentials per bucket and sub directory + # credentials = [ + # { + # bucket: "my-bucket" + # subdir: "path/in/bucket" # Not required + # project: "clearml" + # credentials_json: "/path/to/credentials.json" + # }, + # ] + } + azure.storage { + # containers: [ + # { + # account_name: "clearml" + # account_key: "secret" + # # container_name: + # } + # ] + } + + log { + # debugging feature: set this to true to make null log propagate messages to root logger (so they appear in stdout) + null_log_propagate: false + task_log_buffer_capacity: 66 + + # disable urllib info and lower levels + disable_urllib3_info: true + } + + development { + # Development-mode options + + # dev task reuse window + task_reuse_time_window_in_hours: 72.0 + + # Run VCS repository detection asynchronously + vcs_repo_detect_async: true + + # Store uncommitted git/hg source code diff in experiment manifest when training in development mode + # This stores "git diff" or "hg diff" into the experiment's "script.requirements.diff" section + store_uncommitted_code_diff: true + store_code_diff_from_remote: false + + # Support stopping an experiment in case it was externally stopped, status was changed or task was reset + support_stopping: true + + # Default Task output_uri. if output_uri is not provided to Task.init, default_output_uri will be used instead. + default_output_uri: "" + + # Default auto generated requirements optimize for smaller requirements + # If True, analyze the entire repository regardless of the entry point. + # If False, first analyze the entry point script, if it does not contain other to local files, + # do not analyze the entire repository. + force_analyze_entire_repo: false + + # If set to true, *clearml* update message will not be printed to the console + # this value can be overwritten with os environment variable CLEARML_SUPPRESS_UPDATE_MESSAGE=1 + suppress_update_message: false + + # If this flag is true (default is false), instead of analyzing the code with Pigar, analyze with `pip freeze` + detect_with_pip_freeze: false + detect_with_conda_freeze: false + + # Log specific environment variables. OS environments are enlisted in the "Environment" section + # of the Hyper-Parameters. + # multiple selected variables are supported including the suffix '*'. + # For example: "AWS_*" will log any OS environment variable starting with 'AWS_'. + # This value can be overwritten with os environment variable CLEARML_LOG_ENVIRONMENT="[AWS_*, CUDA_VERSION]" + # Example: log_os_environments: ["AWS_*", "CUDA_VERSION"] + log_os_environments: [] + + # Development mode worker + worker { + # Status report period in seconds + report_period_sec: 2 + + # ping to the server - check connectivity + ping_period_sec: 30 + + # Log all stdout & stderr + log_stdout: true + + # compatibility feature, report memory usage for the entire machine + # default (false), report only on the running process and its sub-processes + report_global_mem_used: false + } + } +} diff --git a/docs/contributing.md b/docs/contributing.md index ff3efbae..5365b6a5 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -2,39 +2,40 @@ Firstly, we thank you for taking the time to contribute! -The following is a set of guidelines for contributing to TRAINS. +Contribution comes in many forms: +* Reporting [issues](https://github.com/allegroai/clearml/issues) you've come upon +* Participating in issue discussions in the [issue tracker](https://github.com/allegroai/clearml/issues) and the [ClearML community slack space](https://join.slack.com/t/allegroai-trains/shared_invite/enQtOTQyMTI1MzQxMzE4LTY5NTUxOTY1NmQ1MzQ5MjRhMGRhZmM4ODE5NTNjMTg2NTBlZGQzZGVkMWU3ZDg1MGE1MjQxNDEzMWU2NmVjZmY) +* Suggesting new features or enhancements +* Implementing new features or fixing outstanding issues + +The following is a set of guidelines for contributing to ClearML. These are primarily guidelines, not rules. Use your best judgment and feel free to propose changes to this document in a pull request. -## Reporting Bugs +## Reporting Issues -This section guides you through submitting a bug report for TRAINS. -By following these guidelines, you -help maintainers and the community understand your report, reproduce the behavior, and find related reports. +By following these guidelines, you help maintainers and the community understand your report, reproduce the behavior, and find related reports. -Before creating bug reports, please check whether the bug you want to report already appears [here](link to issues). -You may discover that you do not need to create a bug report. -When you are creating a bug report, please include as much detail as possible. +Before reporting an issue, please check whether it already appears [here](https://github.com/allegroai/clearml/issues). +If it does, join the on-going discussion instead. **Note**: If you find a **Closed** issue that may be the same issue which you are currently experiencing, then open a **New** issue and include a link to the original (Closed) issue in the body of your new one. -Explain the problem and include additional details to help maintainers reproduce the problem: +When reporting an issue, please include as much detail as possible: explain the problem and include additional details to help maintainers reproduce the problem: * **Use a clear and descriptive title** for the issue to identify the problem. * **Describe the exact steps necessary to reproduce the problem** in as much detail as possible. Please do not just summarize what you did. Make sure to explain how you did it. * **Provide the specific environment setup.** Include the `pip freeze` output, specific environment variables, Python version, and other relevant information. * **Provide specific examples to demonstrate the steps.** Include links to files or GitHub projects, or copy/paste snippets which you use in those examples. -* **If you are reporting any TRAINS crash,** include a crash report with a stack trace from the operating system. Make sure to add the crash report in the issue and place it in a [code block](https://help.github.com/en/articles/getting-started-with-writing-and-formatting-on-github#multiple-lines), +* **If you are reporting any ClearML crash,** include a crash report with a stack trace from the operating system. Make sure to add the crash report in the issue and place it in a [code block](https://help.github.com/en/articles/getting-started-with-writing-and-formatting-on-github#multiple-lines), a [file attachment](https://help.github.com/articles/file-attachments-on-issues-and-pull-requests/), or just put it in a [gist](https://gist.github.com/) (and provide link to that gist). * **Describe the behavior you observed after following the steps** and the exact problem with that behavior. * **Explain which behavior you expected to see and why.** * **For Web-App issues, please include screenshots and animated GIFs** which recreate the described steps and clearly demonstrate the problem. You can use [LICEcap](https://www.cockos.com/licecap/) to record GIFs on macOS and Windows, and [silentcast](https://github.com/colinkeenan/silentcast) or [byzanz](https://github.com/threedaymonk/byzanz) on Linux. -## Suggesting Enhancements +## Suggesting New Features and Enhancements -This section guides you through submitting an enhancement suggestion for TRAINS, including -completely new features and minor improvements to existing functionality. By following these guidelines, you help maintainers and the community understand your suggestion and find related suggestions. Enhancement suggestions are tracked as GitHub issues. After you determine which repository your enhancement suggestion is related to, create an issue on that repository and provide the following: @@ -43,12 +44,18 @@ Enhancement suggestions are tracked as GitHub issues. After you determine which * **A step-by-step description of the suggested enhancement** in as much detail as possible. * **Specific examples to demonstrate the steps.** Include copy/pasteable snippets which you use in those examples as [Markdown code blocks](https://help.github.com/articles/markdown-basics/#multiple-lines). * **Describe the current behavior and explain which behavior you expected to see instead and why.** -* **Include screenshots or animated GIFs** which help you demonstrate the steps or point out the part of TRAINS which the suggestion is related to. You can use [LICEcap](https://www.cockos.com/licecap/) to record GIFs on macOS and Windows, and [silentcast](https://github.com/colinkeenan/silentcast) or [byzanz](https://github.com/threedaymonk/byzanz) on Linux. - - - +* **Include screenshots or animated GIFs** which help you demonstrate the steps or point out the part of ClearML which the suggestion is related to. You can use [LICEcap](https://www.cockos.com/licecap/) to record GIFs on macOS and Windows, and [silentcast](https://github.com/colinkeenan/silentcast) or [byzanz](https://github.com/threedaymonk/byzanz) on Linux. +## Pull Requests +Before you submit a new PR: +* Verify the work you plan to merge addresses an existing [issue](https://github.com/allegroai/clearml/issues) (If not, open a new one) +* Check related discussions in the [ClearML slack community](https://join.slack.com/t/allegroai-trains/shared_invite/enQtOTQyMTI1MzQxMzE4LTY5NTUxOTY1NmQ1MzQ5MjRhMGRhZmM4ODE5NTNjMTg2NTBlZGQzZGVkMWU3ZDg1MGE1MjQxNDEzMWU2NmVjZmY) (Or start your own discussion on the `#clearml-dev` channel) +* Make sure your code conforms to the ClearML coding standards by running: + `flake8 --max-line-length=120 --statistics --show-source --extend-ignore=E501 ./trains*` +In your PR include: +* A reference to the issue it addresses +* A brief description of the approach you've taken for implementing diff --git a/docs/dataset_screenshots.gif b/docs/dataset_screenshots.gif new file mode 100644 index 00000000..9049c2d1 Binary files /dev/null and b/docs/dataset_screenshots.gif differ diff --git a/docs/datasets.md b/docs/datasets.md new file mode 100644 index 00000000..de4b34d0 --- /dev/null +++ b/docs/datasets.md @@ -0,0 +1,139 @@ +# ClearML introducing Dataset management! + +## Decoupling Data from Code - The Dataset Paradigm + +### The ultimate goal of `clearml-data` is to transform datasets into configuration parameters +Just like any other argument, the dataset argument should retrieve a full local copy of the +dataset to be used by the experiment. +This means datasets can be efficiently retrieved by any machine in a reproducible way. +Together it creates a full version control solution for all your data, +that is both machine and environment agnostic. + + +### Design Goals : Simple / Agnostic / File-based / Efficient + +## Key Concepts: +1) **Dataset** is a **collection of files** : e.g. folder with all subdirectories and files included in the dataset +2) **Differential storage** : Efficient storage / network +3) **Flexible**: support addition / removal / merge of files and datasets +4) **Descriptive, transparent & searchable**: support projects, names, descriptions, tags and searchable fields +5) **Simple interface** (CLI and programmatic) +6) **Accessible**: get a copy of the dataset files from anywhere on any machine + +### Workflow: + +#### Simple dataset creation with CLI: + +- Create a dataset +``` bash +clearml-data create --project --name +``` +- Add local files to the dataset +``` bashtrue +clearml-data add --id --files ~/datasets/best_dataset/ +``` +- Upload files (Optional: specify storage `--storage` `s3://bucket` or `gs://` or `azure://` or `/mnt/shared/`) +``` bash +clearml-data upload --id +``` +- Close dataset +``` bash +clearml-data close --id +``` + + +#### Integrating datasets into your code: +``` python +from argparse import ArgumentParser +from clearml import Dataset + +# adding command line interface, so it is easy to use +parser = ArgumentParser() +parser.add_argument('--dataset', default='aayyzz', type=str, help='Dataset ID to train on') +args = parser.parse_args() + +# creating a task, so that later we could override the argparse from UI +task = Task.init(project_name='examples', task_name='dataset demo') + +# getting a local copy of the dataset +dataset_folder = Datset.get(dataset_id=args.dataset).get_local_copy() + +# go over the files in `dataset_folder` and train your model +``` + + +#### Modifying a dataset with CLI: + +- Create a new dataset (specify the parent dataset id) +``` bash +clearml-data create --name --parents +``` +- Get a mutable copy of the current dataset +``` bash +clearml-data get --id --copy ~/datasets/working_dataset +``` +- Change / add / remove files from the dataset folder +``` bash +vim ~/datasets/working_dataset/everything.csv +``` +- Sync local changes +``` bash +clearml-data sync --id --folder ~/datasets/working_dataset +``` +- Upload files (Optional: specify storage `--storage` `s3://bucket` or `gs://` or `azure://` or `/mnt/shared/`) +``` bash +clearml-data upload --id +``` +- Close dataset +``` bash +clearml-data close --id +``` + + +#### Command Line Interface Summary: + +- **`search`** Search a dataset based on project / name / description / tag etc. +- **`list`** List the file directory content of a dataset (no need to download a copy pf the dataset) +- **`verify`** Verify a local copy of a dataset (verify the dataset files SHA2 hash) +- **`create`** Create a new dataset (support extending/inheriting multiple parents) +- **`delete`** Delete a dataset +- **`add`** Add local files to a dataset +- **`sync`** Sync dataset with a local folder (source-of-truth being the local folder) +- **`remove`** Remove files from dataset (no need to download a copy of the dataset) +- **`get`** Get a local copy of the dataset (either readonly --link, or writable --copy) +- **`upload`** Upload the dataset (use --storage to specify storage target such as S3/GS/Azure/Folder, default: file server) + + +#### Under the hood (how it all works): + +Each dataset instance stores the collection of files added/modified from the previous version (parent). + +When requesting a copy of the dataset all parent datasets on the graph are downloaded and a new folder +is merged with all changes introduced in the dataset DAG. + +Implementation details: + +Dataset differential snapshot is stored in a single zip file for efficiency in storage and network +bandwidth. Local cache is built into the process making sure datasets are downloaded only once. +Dataset contains SHA2 hash of all the files in the dataset. +In order to increase dataset fetching speed, only file size is verified automatically, +the SHA2 hash is verified only on user's request. + +The design supports multiple parents per dataset, essentially merging all parents based on order. +To improve deep dataset DAG storage and speed, dataset squashing was introduced. A user can squash +a dataset, merging down all changes introduced in the DAG, creating a new flat version without parent datasets. + + +### Datasets UI: + +A dataset is represented as a special `Task` in the system.
+It is of type `data-processing` with a special tag `dataset`. + +- Full log (calls / CLI) of the dataset creation process can be found in the "Execution" section. +- Listing of the dataset differential snapshot, summary of files added / modified / removed and details of files +in the differential snapshot (location / size / hash), is available in the Artifacts section you can find a +- The full dataset listing (all files included) is available in the Configuration section under `Dataset Content`. +This allows you to quickly compare two dataset contents and visually see the difference. +- The dataset genealogy DAG and change-set summary table is visualized in Results / Plots + + diff --git a/docs/logger.md b/docs/logger.md index 7572bc01..58842593 100644 --- a/docs/logger.md +++ b/docs/logger.md @@ -1,6 +1,6 @@ -# TRAINS Explicit Logging +# ClearML Explicit Logging -Using the **TRAINS** [Logger](https://github.com/allegroai/trains/blob/master/trains/logger.py) module and other **TRAINS** features, you can explicitly log any of the following: +Using the **ClearML** [Logger](https://github.com/allegroai/clearml/blob/master/clearml/logger.py) module and other **ClearML** features, you can explicitly log any of the following: * Report graphs and images * [Scalar metrics](#scalar-metrics) @@ -19,10 +19,10 @@ Using the **TRAINS** [Logger](https://github.com/allegroai/trains/blob/master/tr * Message logging * [Reporting text without formatting](#reporting-text-without-formatting) -Additionally, the **TRAINS** Logger module provides methods that allow you to do the following: +Additionally, the **ClearML** Logger module provides methods that allow you to do the following: * Get the [current logger]() - * Overrride the TRAINS configuration file with a [default upload destination]() for images and files + * Overrride the ClearML configuration file with a [default upload destination]() for images and files ## Graphs and Images @@ -30,7 +30,7 @@ Additionally, the **TRAINS** Logger module provides methods that allow you to do Use to report scalar metrics by iteration as a line plot. -First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/trains/blob/master/examples/reporting/scalar_reporting.py)) with the following method. +First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/clearml/blob/master/examples/reporting/scalar_reporting.py)) with the following method. **Method**: @@ -101,7 +101,7 @@ def report_scalar(self, title, series, value, iteration) Use to report any data by iteration as a histogram. -First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/trains/blob/master/examples/reporting/scatter_hist_confusion_mat_reporting.py)) with the following method. +First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/clearml/blob/master/examples/reporting/scatter_hist_confusion_mat_reporting.py)) with the following method. **Method**: @@ -199,7 +199,7 @@ def report_histogram(self, title, series, values, iteration, labels=None, xlabel Use to report any data by iteration as a single or multiple line plot. -First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/trains/blob/master/examples/reporting/scatter_hist_confusion_mat_reporting.py)) with the following method. +First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/clearml/blob/master/examples/reporting/scatter_hist_confusion_mat_reporting.py)) with the following method. **Method**: @@ -325,7 +325,7 @@ def report_line_plot(self, title, series, iteration, xaxis, yaxis, mode='lines', Use to report any vector data as a 2D scatter diagram. -First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/trains/blob/master/examples/reporting/scatter_hist_confusion_mat_reporting.py)) with the following method. +First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/clearml/blob/master/examples/reporting/scatter_hist_confusion_mat_reporting.py)) with the following method. **Method**: @@ -461,7 +461,7 @@ def report_scatter2d(self, title, series, scatter, iteration, xaxis=None, yaxis= Use to report any array data as a 3D scatter diagram. -First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/trains/blob/master/examples/reporting/3d_plots_reporting.py)) with the following method. +First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/clearml/blob/master/examples/reporting/3d_plots_reporting.py)) with the following method. **Method**: @@ -597,7 +597,7 @@ def report_scatter3d(self, title, series, scatter, iteration, labels=None, mode= Use to report a heat-map matrix as a confusion matrix. You can also plot a heat-map as a [surface diagram](#surface-diagrams). -First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/trains/blob/master/examples/reporting/scatter_hist_confusion_mat_reporting.py)) with the following method. +First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/clearml/blob/master/examples/reporting/scatter_hist_confusion_mat_reporting.py)) with the following method. **Method**: @@ -689,7 +689,7 @@ def report_confusion_matrix(self, title, series, matrix, iteration, xlabels=None Use to plot a heat-map matrix as a surface diagram. You can also plot a heat-map as a [confusion matrix](#confusion-matrices). -First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/trains/blob/master/examples/reporting/3d_plots_reporting.py)) with the following method. +First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/clearml/blob/master/examples/reporting/3d_plots_reporting.py)) with the following method. **Method**: @@ -818,10 +818,10 @@ def report_surface(self, title, series, matrix, iteration, xlabels=None, ylabels ### Images -Use to report an image and upload its contents to the bucket specified in the **TRAINS** configuration file, +Use to report an image and upload its contents to the bucket specified in the **ClearML** configuration file, or a [a default upload destination](#set-default-upload-destination), if you set a default. -First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/trains/blob/master/examples/manual_reporting.py)) with the following method. +First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/clearml/blob/master/examples/manual_reporting.py)) with the following method. **Method**: @@ -929,7 +929,7 @@ def report_image(self, title, series, iteration, local_path=None, matrix=None, m ### Logging Experiment Parameter Dictionaries -In order for **TRAINS** to log a dictionary of parameters, use the `Task.connect` method. +In order for **ClearML** to log a dictionary of parameters, use the `Task.connect` method. For example, to log the hyper-parameters learning_rate, batch_size, display_step, model_path, n_hidden_1, and n_hidden_2: @@ -938,27 +938,27 @@ For example, to log the hyper-parameters learning_rate, batch parameters_dict = { 'learning_rate': 0.001, 'batch_size': 100, 'display_step': 1, 'model_path': "/tmp/model.ckpt", 'n_hidden_1': 256, 'n_hidden_2': 256 } -# Connect the dictionary to your TRAINS Task +# Connect the dictionary to your ClearML Task parameters_dict = Task.current_task().connect(parameters_dict) ``` ### Specifying Environment Variables to Track -By setting the `TRAINS_LOG_ENVIRONMENT` environment variable, make **TRAINS** log either: +By setting the `CLEARML_LOG_ENVIRONMENT` environment variable, make **ClearML** log either: * All environment variables - export TRAINS_LOG_ENVIRONMENT="*" + export CLEARML_LOG_ENVIRONMENT="*" * Specific environment variables For example, log `PWD` and `PYTHONPATH` - export TRAINS_LOG_ENVIRONMENT="PWD,PYTHONPATH" + export CLEARML_LOG_ENVIRONMENT="PWD,PYTHONPATH" * No environment variables - export TRAINS_LOG_ENVIRONMENT= + export CLEARML_LOG_ENVIRONMENT= ## Logging Messages @@ -972,7 +972,7 @@ Use the methods in this section to log various types of messages. The method nam def debug(self, msg, *args, **kwargs) ``` -First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/trains/blob/master/examples/reporting/text_reporting.py)) with the following method. +First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/clearml/blob/master/examples/reporting/text_reporting.py)) with the following method. **Arguments**: @@ -1010,7 +1010,7 @@ First [get the current logger](#get-the-current-logger) and then use it (see an def info(self, msg, *args, **kwargs) ``` -First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/trains/blob/master/examples/reporting/text_reporting.py)) with the following method. +First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/clearml/blob/master/examples/reporting/text_reporting.py)) with the following method. **Arguments**: @@ -1048,7 +1048,7 @@ First [get the current logger](#get-the-current-logger) and then use it (see an def warn(self, msg, *args, **kwargs) ``` -First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/trains/blob/master/examples/reporting/text_reporting.py)) with the following method. +First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/clearml/blob/master/examples/reporting/text_reporting.py)) with the following method. **Arguments**: @@ -1087,7 +1087,7 @@ First [get the current logger](#get-the-current-logger) and then use it (see an def error(self, msg, *args, **kwargs) ``` -First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/trains/blob/master/examples/reporting/text_reporting.py)) with the following method. +First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/clearml/blob/master/examples/reporting/text_reporting.py)) with the following method. **Arguments**: @@ -1125,7 +1125,7 @@ First [get the current logger](#get-the-current-logger) and then use it (see an def critical(self, msg, *args, **kwargs) ``` -First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/trains/blob/master/examples/reporting/text_reporting.py)) with the following method. +First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/clearml/blob/master/examples/reporting/text_reporting.py)) with the following method. **Arguments**: @@ -1163,7 +1163,7 @@ First [get the current logger](#get-the-current-logger) and then use it (see an def fatal(self, msg, *args, **kwargs) ``` -First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/trains/blob/master/examples/reporting/text_reporting.py)) with the following method. +First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/clearml/blob/master/examples/reporting/text_reporting.py)) with the following method. **Arguments**: @@ -1201,7 +1201,7 @@ First [get the current logger](#get-the-current-logger) and then use it (see an def console(self, msg, level=logging.INFO, omit_console=False, *args, **kwargs) ``` -First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/trains/blob/master/examples/reporting/text_reporting.py)) with the following method. +First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/clearml/blob/master/examples/reporting/text_reporting.py)) with the following method. **Arguments**: @@ -1279,7 +1279,7 @@ First [get the current logger](#get-the-current-logger) and then use it (see an def report_text(self, msg, level=logging.INFO, print_console=False, *args, **_) ``` -First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/trains/blob/master/examples/reporting/text_reporting.py)) with the following method. +First [get the current logger](#get-the-current-logger) and then use it (see an [example script](https://github.com/allegroai/clearml/blob/master/examples/reporting/text_reporting.py)) with the following method. **Arguments**: @@ -1371,7 +1371,7 @@ None. Use to specify the default destination storage location used for uploading images. Images are uploaded and a link to the image is reported. -Credentials for the storage location are in the global configuration file (for example, on Linux, ~/trains.conf). +Credentials for the storage location are in the global configuration file (for example, on Linux, ~/clearml.conf). **Method**: diff --git a/docs/trains.conf b/docs/trains.conf index 9d5b2df3..552a4488 100644 --- a/docs/trains.conf +++ b/docs/trains.conf @@ -1,4 +1,4 @@ -# TRAINS SDK configuration file +# ClearML SDK configuration file - Please use ~/clearml.conf api { # web_server on port 8080 web_server: "http://localhost:8080" @@ -16,12 +16,12 @@ api { verify_certificate: True } sdk { - # TRAINS - default SDK configuration + # ClearML - default SDK configuration storage { cache { # Defaults to system temp folder / cache - default_base_dir: "~/.trains/cache" + default_base_dir: "~/.clearml/cache" } } @@ -103,7 +103,7 @@ sdk { google.storage { # # Default project and credentials file # # Will be used when no bucket configuration is found - # project: "trains" + # project: "clearml" # credentials_json: "/path/to/credentials.json" # # Specific credentials per bucket and sub directory @@ -111,7 +111,7 @@ sdk { # { # bucket: "my-bucket" # subdir: "path/in/bucket" # Not required - # project: "trains" + # project: "clearml" # credentials_json: "/path/to/credentials.json" # }, # ] @@ -119,7 +119,7 @@ sdk { azure.storage { # containers: [ # { - # account_name: "trains" + # account_name: "clearml" # account_key: "secret" # # container_name: # } @@ -161,8 +161,8 @@ sdk { # do not analyze the entire repository. force_analyze_entire_repo: false - # If set to true, *trains* update message will not be printed to the console - # this value can be overwritten with os environment variable TRAINS_SUPPRESS_UPDATE_MESSAGE=1 + # If set to true, *clearml* update message will not be printed to the console + # this value can be overwritten with os environment variable CLEARML_SUPPRESS_UPDATE_MESSAGE=1 suppress_update_message: false # If this flag is true (default is false), instead of analyzing the code with Pigar, analyze with `pip freeze` @@ -173,7 +173,7 @@ sdk { # of the Hyper-Parameters. # multiple selected variables are supported including the suffix '*'. # For example: "AWS_*" will log any OS environment variable starting with 'AWS_'. - # This value can be overwritten with os environment variable TRAINS_LOG_ENVIRONMENT="[AWS_*, CUDA_VERSION]" + # This value can be overwritten with os environment variable CLEARML_LOG_ENVIRONMENT="[AWS_*, CUDA_VERSION]" # Example: log_os_environments: ["AWS_*", "CUDA_VERSION"] log_os_environments: [] diff --git a/examples/automation/manual_random_param_search_example.py b/examples/automation/manual_random_param_search_example.py index f444edb9..2bd7ec96 100644 --- a/examples/automation/manual_random_param_search_example.py +++ b/examples/automation/manual_random_param_search_example.py @@ -1,8 +1,8 @@ from random import sample -from trains import Task +from clearml import Task -# Connecting TRAINS +# Connecting ClearML task = Task.init(project_name='examples', task_name='Random Hyper-Parameter Search Example', task_type=Task.TaskTypes.optimizer) # Create a hyper-parameter dictionary for the task diff --git a/examples/automation/requirements.txt b/examples/automation/requirements.txt index 3417fd38..6dac6a83 100644 --- a/examples/automation/requirements.txt +++ b/examples/automation/requirements.txt @@ -1 +1 @@ -trains \ No newline at end of file +clearml \ No newline at end of file diff --git a/examples/automation/task_piping_example.py b/examples/automation/task_piping_example.py index 96acc86e..6be25679 100644 --- a/examples/automation/task_piping_example.py +++ b/examples/automation/task_piping_example.py @@ -1,4 +1,4 @@ -from trains import Task +from clearml import Task from time import sleep # Initialize the Task Pipe's first Task used to start the Task Pipe @@ -35,7 +35,7 @@ cloned_task_parameters = cloned_task.get_parameters() cloned_task_parameters[param['param_name']] = param['param_name_new_value'] cloned_task.set_parameters(cloned_task_parameters) -# Enqueue the Task for execution. The enqueued Task must already exist in the trains platform +# Enqueue the Task for execution. The enqueued Task must already exist in the clearml platform print('Enqueue next step in pipeline to queue: {}'.format(param['execution_queue_name'])) Task.enqueue(cloned_task.id, queue_name=param['execution_queue_name']) diff --git a/examples/automation/toy_base_task.py b/examples/automation/toy_base_task.py index 7e1521be..f0f53a3b 100644 --- a/examples/automation/toy_base_task.py +++ b/examples/automation/toy_base_task.py @@ -1,7 +1,7 @@ # This Task is the base task that we will be executing as a second step (see task_piping.py) # In order to make sure this experiment is registered in the platform, you must execute it once. -from trains import Task +from clearml import Task # Initialize the task pipe's first task used to start the task pipe task = Task.init('examples', 'Toy Base Task') diff --git a/examples/distributed/requirements.txt b/examples/distributed/requirements.txt index 27c16b3a..5107c4eb 100644 --- a/examples/distributed/requirements.txt +++ b/examples/distributed/requirements.txt @@ -1,3 +1,3 @@ torch>=1.1.0 torchvision>=0.3.0 -trains \ No newline at end of file +clearml \ No newline at end of file diff --git a/examples/distributed/subprocess_example.py b/examples/distributed/subprocess_example.py index e0af0019..0726f5a2 100644 --- a/examples/distributed/subprocess_example.py +++ b/examples/distributed/subprocess_example.py @@ -1,4 +1,4 @@ -# TRAINS - example of multiple sub-processes interacting and reporting to a single master experiment +# ClearML - example of multiple sub-processes interacting and reporting to a single master experiment import multiprocessing import os @@ -7,7 +7,7 @@ import sys import time from argparse import ArgumentParser -from trains import Task +from clearml import Task # fake data for us to "process" data = ( @@ -51,7 +51,7 @@ if __name__ == '__main__': # We have to initialize the task in the master process, # it will make sure that any sub-process calling Task.init will get the master task object - # notice that we exclude the `counter` argument, so we can launch multiple sub-processes with trains-agent + # notice that we exclude the `counter` argument, so we can launch multiple sub-processes with clearml-agent # otherwise, the `counter` will always be set to the original value. task = Task.init('examples', 'Popen example', auto_connect_arg_parser={'counter': False}) diff --git a/examples/frameworks/autokeras/autokeras_imdb_example.py b/examples/frameworks/autokeras/autokeras_imdb_example.py index 4bef0a53..6d268891 100644 --- a/examples/frameworks/autokeras/autokeras_imdb_example.py +++ b/examples/frameworks/autokeras/autokeras_imdb_example.py @@ -3,7 +3,7 @@ import numpy as np import tensorflow as tf from tensorflow import keras -from trains import Task +from clearml import Task task = Task.init(project_name="autokeras", task_name="autokeras imdb example with scalars") diff --git a/examples/frameworks/autokeras/requirements.txt b/examples/frameworks/autokeras/requirements.txt index b50c4d62..6f74aa0d 100644 --- a/examples/frameworks/autokeras/requirements.txt +++ b/examples/frameworks/autokeras/requirements.txt @@ -1,5 +1,5 @@ # Plese read this https://github.com/keras-team/autokeras#installation before doing changes autokeras tensorflow>=2.3.0 -trains +clearml git+https://github.com/keras-team/keras-tuner.git@1.0.2rc2 diff --git a/examples/frameworks/fastai/fastai_with_tensorboard.py b/examples/frameworks/fastai/fastai_with_tensorboard.py index 63cdda9b..2fb2facd 100644 --- a/examples/frameworks/fastai/fastai_with_tensorboard.py +++ b/examples/frameworks/fastai/fastai_with_tensorboard.py @@ -1,10 +1,10 @@ -# TRAINS - Fastai with Tensorboard example code, automatic logging the model and Tensorboard outputs +# ClearML - Fastai with Tensorboard example code, automatic logging the model and Tensorboard outputs # from fastai.callbacks.tensorboard import LearnerTensorboardWriter from fastai.vision import * # Quick access to computer vision functionality -from trains import Task +from clearml import Task task = Task.init(project_name="example", task_name="fastai with tensorboard callback") diff --git a/examples/frameworks/fastai/requirements.txt b/examples/frameworks/fastai/requirements.txt index d36e6488..929405e1 100644 --- a/examples/frameworks/fastai/requirements.txt +++ b/examples/frameworks/fastai/requirements.txt @@ -1,4 +1,4 @@ fastai tensorboard tensorboardX -trains +clearml diff --git a/examples/frameworks/ignite/cifar_ignite.py b/examples/frameworks/ignite/cifar_ignite.py index de641cf9..0ebbfd89 100644 --- a/examples/frameworks/ignite/cifar_ignite.py +++ b/examples/frameworks/ignite/cifar_ignite.py @@ -15,12 +15,12 @@ from ignite.utils import setup_logger from torch.utils.tensorboard import SummaryWriter from tqdm import tqdm -from trains import Task, StorageManager +from clearml import Task, StorageManager -# Trains Initializations +# ClearML Initializations task = Task.init(project_name='Image Example', task_name='image classification CIFAR10') params = {'number_of_epochs': 20, 'batch_size': 64, 'dropout': 0.25, 'base_lr': 0.001, 'momentum': 0.9, 'loss_report': 100} -params = task.connect(params) # enabling configuration override by trains +params = task.connect(params) # enabling configuration override by clearml print(params) # printing actual configuration (after override in remote mode) manager = StorageManager() diff --git a/examples/frameworks/keras/keras_tensorboard.py b/examples/frameworks/keras/keras_tensorboard.py index cc17e05a..d24cf1ed 100644 --- a/examples/frameworks/keras/keras_tensorboard.py +++ b/examples/frameworks/keras/keras_tensorboard.py @@ -1,4 +1,4 @@ -# TRAINS - Keras with Tensorboard example code, automatic logging model and Tensorboard outputs +# ClearML - Keras with Tensorboard example code, automatic logging model and Tensorboard outputs # # Train a simple deep NN on the MNIST dataset. # Gets to 98.40% test accuracy after 20 epochs @@ -19,7 +19,7 @@ from tensorflow.keras.layers import Activation, Dense from tensorflow.keras.models import Sequential from tensorflow.keras.optimizers import RMSprop -from trains import Task +from clearml import Task class TensorBoardImage(TensorBoard): @@ -89,7 +89,7 @@ model.compile(loss='categorical_crossentropy', optimizer=RMSprop(), metrics=['accuracy']) -# Connecting TRAINS +# Connecting ClearML task = Task.init(project_name='examples', task_name='Keras with TensorBoard example') # To set your own configuration: diff --git a/examples/frameworks/keras/legacy/keras_tensorboard.py b/examples/frameworks/keras/legacy/keras_tensorboard.py index f36fc7bd..28f04d3d 100644 --- a/examples/frameworks/keras/legacy/keras_tensorboard.py +++ b/examples/frameworks/keras/legacy/keras_tensorboard.py @@ -1,4 +1,4 @@ -# TRAINS - Keras with Tensorboard example code, automatic logging model and Tensorboard outputs +# ClearML - Keras with Tensorboard example code, automatic logging model and Tensorboard outputs # # Train a simple deep NN on the MNIST dataset. # Gets to 98.40% test accuracy after 20 epochs @@ -18,7 +18,7 @@ from keras.layers.core import Dense, Activation from keras.optimizers import RMSprop from keras.utils import np_utils import tensorflow as tf -from trains import Task +from clearml import Task class TensorBoardImage(TensorBoard): @@ -88,7 +88,7 @@ model.compile(loss='categorical_crossentropy', optimizer=RMSprop(), metrics=['accuracy']) -# Connecting TRAINS +# Connecting ClearML task = Task.init(project_name='examples', task_name='Keras with TensorBoard example') task.connect_configuration({'test': 1337, 'nested': {'key': 'value', 'number': 1}}) diff --git a/examples/frameworks/keras/legacy/requirements.txt b/examples/frameworks/keras/legacy/requirements.txt index 7049a0a0..f37621b1 100644 --- a/examples/frameworks/keras/legacy/requirements.txt +++ b/examples/frameworks/keras/legacy/requirements.txt @@ -1,2 +1,2 @@ -trains +clearml Keras>=2.2.4 diff --git a/examples/frameworks/keras/manual_model_upload.py b/examples/frameworks/keras/manual_model_upload.py index 9aaec286..6b2a0b4f 100644 --- a/examples/frameworks/keras/manual_model_upload.py +++ b/examples/frameworks/keras/manual_model_upload.py @@ -1,11 +1,11 @@ -# TRAINS - Example of manual model configuration and uploading +# ClearML - Example of manual model configuration and uploading # import os from tempfile import gettempdir from keras import Input, layers, Model -from trains import Task +from clearml import Task task = Task.init(project_name='examples', task_name='Model configuration and upload') diff --git a/examples/frameworks/keras/requirements.txt b/examples/frameworks/keras/requirements.txt index cb700b34..4f7fcce4 100644 --- a/examples/frameworks/keras/requirements.txt +++ b/examples/frameworks/keras/requirements.txt @@ -1,3 +1,3 @@ Keras tensorflow>=2.0 -trains \ No newline at end of file +clearml \ No newline at end of file diff --git a/examples/frameworks/kerastuner/keras_tuner_cifar.py b/examples/frameworks/kerastuner/keras_tuner_cifar.py index eb0dcf8a..a2b5e019 100644 --- a/examples/frameworks/kerastuner/keras_tuner_cifar.py +++ b/examples/frameworks/kerastuner/keras_tuner_cifar.py @@ -3,9 +3,9 @@ import kerastuner as kt import tensorflow as tf import tensorflow_datasets as tfds -from trains.external.kerastuner import TrainsTunerLogger +from clearml.external.kerastuner import TrainsTunerLogger -from trains import Task +from clearml import Task physical_devices = tf.config.list_physical_devices('GPU') if physical_devices: diff --git a/examples/frameworks/kerastuner/requirements.txt b/examples/frameworks/kerastuner/requirements.txt index cd066fc5..4677be30 100644 --- a/examples/frameworks/kerastuner/requirements.txt +++ b/examples/frameworks/kerastuner/requirements.txt @@ -1,4 +1,4 @@ keras-tuner tensorflow>=2.0 tensorflow-datasets -trains \ No newline at end of file +clearml \ No newline at end of file diff --git a/examples/frameworks/lightgbm/requirements.txt b/examples/frameworks/lightgbm/requirements.txt index bc0c122d..ac7ee02e 100644 --- a/examples/frameworks/lightgbm/requirements.txt +++ b/examples/frameworks/lightgbm/requirements.txt @@ -1,4 +1,4 @@ lightgbm scikit-learn pandas -trains \ No newline at end of file +clearml \ No newline at end of file diff --git a/examples/frameworks/lightgbm/train_with_lightbgm.py b/examples/frameworks/lightgbm/train_with_lightbgm.py index c4613968..b6fad561 100644 --- a/examples/frameworks/lightgbm/train_with_lightbgm.py +++ b/examples/frameworks/lightgbm/train_with_lightbgm.py @@ -1,10 +1,10 @@ -# TRAINS - Example of LightGBM integration +# ClearML - Example of LightGBM integration # import lightgbm as lgb import pandas as pd from sklearn.metrics import mean_squared_error -from trains import Task +from clearml import Task task = Task.init(project_name="examples", task_name="LIGHTgbm") diff --git a/examples/frameworks/matplotlib/matplotlib_example.py b/examples/frameworks/matplotlib/matplotlib_example.py index f00443f8..47575937 100644 --- a/examples/frameworks/matplotlib/matplotlib_example.py +++ b/examples/frameworks/matplotlib/matplotlib_example.py @@ -1,9 +1,9 @@ -# TRAINS - Example of Matplotlib and Seaborn integration and reporting +# ClearML - Example of Matplotlib and Seaborn integration and reporting # import numpy as np import matplotlib.pyplot as plt import seaborn as sns -from trains import Task +from clearml import Task task = Task.init(project_name='examples', task_name='Matplotlib example') diff --git a/examples/frameworks/matplotlib/requirements.txt b/examples/frameworks/matplotlib/requirements.txt index de3c8632..fdcc6684 100644 --- a/examples/frameworks/matplotlib/requirements.txt +++ b/examples/frameworks/matplotlib/requirements.txt @@ -1,4 +1,4 @@ matplotlib >= 3.1.1 ; python_version >= '3.6' matplotlib >= 2.2.4 ; python_version < '3.6' seaborn -trains \ No newline at end of file +clearml \ No newline at end of file diff --git a/examples/frameworks/pytorch/manual_model_upload.py b/examples/frameworks/pytorch/manual_model_upload.py index dfd963e0..485d33fb 100644 --- a/examples/frameworks/pytorch/manual_model_upload.py +++ b/examples/frameworks/pytorch/manual_model_upload.py @@ -1,10 +1,10 @@ -# TRAINS - Example of manual model configuration and uploading +# ClearML - Example of manual model configuration and uploading # import os from tempfile import gettempdir import torch -from trains import Task +from clearml import Task task = Task.init(project_name='examples', task_name='Model configuration and upload') diff --git a/examples/frameworks/pytorch/pytorch_distributed_example.py b/examples/frameworks/pytorch/pytorch_distributed_example.py index 5a98e0c4..9130f3c4 100644 --- a/examples/frameworks/pytorch/pytorch_distributed_example.py +++ b/examples/frameworks/pytorch/pytorch_distributed_example.py @@ -1,4 +1,4 @@ -# TRAINS - example of TRAINS torch distributed support +# ClearML - example of ClearML torch distributed support # notice all nodes will be reporting to the master Task (experiment) import os @@ -15,7 +15,7 @@ import torch.nn.functional as F from torch import optim from torchvision import datasets, transforms -from trains import Task +from clearml import Task local_dataset_path = './MNIST_data' @@ -150,7 +150,7 @@ if __name__ == "__main__": # We have to initialize the task in the master process, # it will make sure that any sub-process calling Task.init will get the master task object - # notice that we exclude the `rank` argument, so we can launch multiple sub-processes with trains-agent + # notice that we exclude the `rank` argument, so we can launch multiple sub-processes with clearml-agent # otherwise, the `rank` will always be set to the original value. task = Task.init("examples", "test torch distributed", auto_connect_arg_parser={'rank': False}) diff --git a/examples/frameworks/pytorch/pytorch_matplotlib.py b/examples/frameworks/pytorch/pytorch_matplotlib.py index 933bb5cf..53a88b00 100644 --- a/examples/frameworks/pytorch/pytorch_matplotlib.py +++ b/examples/frameworks/pytorch/pytorch_matplotlib.py @@ -1,4 +1,4 @@ -# TRAINS - Example of Pytorch and matplotlib integration and reporting +# ClearML - Example of Pytorch and matplotlib integration and reporting # """ Neural Transfer Using PyTorch @@ -60,7 +60,7 @@ import torchvision.transforms as transforms import torchvision.models as models import copy -from trains import Task +from clearml import Task task = Task.init(project_name='examples', task_name='pytorch with matplotlib example', task_type=Task.TaskTypes.testing) diff --git a/examples/frameworks/pytorch/pytorch_mnist.py b/examples/frameworks/pytorch/pytorch_mnist.py index 0572336e..b1fac4e6 100644 --- a/examples/frameworks/pytorch/pytorch_mnist.py +++ b/examples/frameworks/pytorch/pytorch_mnist.py @@ -1,4 +1,4 @@ -# TRAINS - Example of Pytorch mnist training integration +# ClearML - Example of Pytorch mnist training integration # from __future__ import print_function import argparse @@ -11,7 +11,7 @@ import torch.nn.functional as F import torch.optim as optim from torchvision import datasets, transforms -from trains import Task, Logger +from clearml import Task, Logger class Net(nn.Module): diff --git a/examples/frameworks/pytorch/pytorch_tensorboard.py b/examples/frameworks/pytorch/pytorch_tensorboard.py index ece50a7b..58005848 100644 --- a/examples/frameworks/pytorch/pytorch_tensorboard.py +++ b/examples/frameworks/pytorch/pytorch_tensorboard.py @@ -1,4 +1,4 @@ -# TRAINS - Example of pytorch with tensorboard>=v1.14 +# ClearML - Example of pytorch with tensorboard>=v1.14 # from __future__ import print_function @@ -14,7 +14,7 @@ from torchvision import datasets, transforms from torch.autograd import Variable from torch.utils.tensorboard import SummaryWriter -from trains import Task +from clearml import Task class Net(nn.Module): @@ -99,7 +99,7 @@ def main(): parser.add_argument('--log-interval', type=int, default=10, metavar='N', help='how many batches to wait before logging training status') args = parser.parse_args() - task = Task.init(project_name='examples', task_name='pytorch with tensorboard') # noqa: F841 + Task.init(project_name='examples', task_name='pytorch with tensorboard') writer = SummaryWriter('runs') writer.add_text('TEXT', 'This is some text', 0) args.cuda = not args.no_cuda and torch.cuda.is_available() diff --git a/examples/frameworks/pytorch/requirements.txt b/examples/frameworks/pytorch/requirements.txt index 169f0b09..d888329b 100644 --- a/examples/frameworks/pytorch/requirements.txt +++ b/examples/frameworks/pytorch/requirements.txt @@ -3,4 +3,4 @@ tensorboardX tensorboard>=1.14.0 torch>=1.1.0 torchvision>=0.3.0 -trains \ No newline at end of file +clearml \ No newline at end of file diff --git a/examples/frameworks/pytorch/tensorboard_toy_pytorch.py b/examples/frameworks/pytorch/tensorboard_toy_pytorch.py index 71e1037d..8ff1dc75 100644 --- a/examples/frameworks/pytorch/tensorboard_toy_pytorch.py +++ b/examples/frameworks/pytorch/tensorboard_toy_pytorch.py @@ -5,7 +5,7 @@ import numpy as np from PIL import Image from torch.utils.tensorboard import SummaryWriter -from trains import Task +from clearml import Task task = Task.init(project_name='examples', task_name='pytorch tensorboard toy example') diff --git a/examples/frameworks/scikit-learn/requirements.txt b/examples/frameworks/scikit-learn/requirements.txt index dcb80536..07819fbf 100644 --- a/examples/frameworks/scikit-learn/requirements.txt +++ b/examples/frameworks/scikit-learn/requirements.txt @@ -2,4 +2,4 @@ joblib>=0.13.2 matplotlib >= 3.1.1 ; python_version >= '3.6' matplotlib >= 2.2.4 ; python_version < '3.6' scikit-learn -trains \ No newline at end of file +clearml \ No newline at end of file diff --git a/examples/frameworks/scikit-learn/sklearn_joblib_example.py b/examples/frameworks/scikit-learn/sklearn_joblib_example.py index 9a04660f..b29a6bef 100644 --- a/examples/frameworks/scikit-learn/sklearn_joblib_example.py +++ b/examples/frameworks/scikit-learn/sklearn_joblib_example.py @@ -10,7 +10,7 @@ import numpy as np import matplotlib.pyplot as plt -from trains import Task +from clearml import Task task = Task.init(project_name="examples", task_name="scikit-learn joblib example") diff --git a/examples/frameworks/scikit-learn/sklearn_matplotlib_example.py b/examples/frameworks/scikit-learn/sklearn_matplotlib_example.py index f56d1c20..de5a3a63 100644 --- a/examples/frameworks/scikit-learn/sklearn_matplotlib_example.py +++ b/examples/frameworks/scikit-learn/sklearn_matplotlib_example.py @@ -6,7 +6,7 @@ from sklearn.model_selection import learning_curve from sklearn.naive_bayes import GaussianNB from sklearn.svm import SVC -from trains import Task +from clearml import Task def plot_learning_curve(estimator, title, X, y, axes=None, ylim=None, cv=None, n_jobs=None, diff --git a/examples/frameworks/tensorboardx/pytorch_tensorboardX.py b/examples/frameworks/tensorboardx/pytorch_tensorboardX.py index b0a1901f..498c09fa 100644 --- a/examples/frameworks/tensorboardx/pytorch_tensorboardX.py +++ b/examples/frameworks/tensorboardx/pytorch_tensorboardX.py @@ -1,4 +1,4 @@ -# TRAINS - Example of pytorch with tensorboardX +# ClearML - Example of pytorch with tensorboardX # from __future__ import print_function @@ -14,7 +14,7 @@ from tensorboardX import SummaryWriter from torch.autograd import Variable from torchvision import datasets, transforms -from trains import Task +from clearml import Task class Net(nn.Module): diff --git a/examples/frameworks/tensorboardx/requirements.txt b/examples/frameworks/tensorboardx/requirements.txt index 1c860d52..302ac560 100644 --- a/examples/frameworks/tensorboardx/requirements.txt +++ b/examples/frameworks/tensorboardx/requirements.txt @@ -1,4 +1,4 @@ tensorboardX>=1.8 torch>=1.1.0 torchvision>=0.3.0 -trains \ No newline at end of file +clearml \ No newline at end of file diff --git a/examples/frameworks/tensorflow/legacy/tensorboard_pr_curve.py b/examples/frameworks/tensorflow/legacy/tensorboard_pr_curve.py index b490fdad..441aba41 100644 --- a/examples/frameworks/tensorflow/legacy/tensorboard_pr_curve.py +++ b/examples/frameworks/tensorflow/legacy/tensorboard_pr_curve.py @@ -1,4 +1,4 @@ -# TRAINS - Example of new tensorboard pr_curves model +# ClearML - Example of new tensorboard pr_curves model # # Copyright 2017 The TensorFlow Authors. All Rights Reserved. # @@ -37,7 +37,7 @@ from absl import flags from six.moves import xrange # pylint: disable=redefined-builtin import tensorflow as tf from tensorboard.plugins.pr_curve import summary -from trains import Task +from clearml import Task task = Task.init(project_name='examples', task_name='tensorboard pr_curve') diff --git a/examples/frameworks/tensorflow/legacy/tensorboard_toy.py b/examples/frameworks/tensorflow/legacy/tensorboard_toy.py index 5984eca9..f3b7832d 100644 --- a/examples/frameworks/tensorflow/legacy/tensorboard_toy.py +++ b/examples/frameworks/tensorflow/legacy/tensorboard_toy.py @@ -1,4 +1,4 @@ -# TRAINS - Example of tensorboard with tensorflow (without any actual training) +# ClearML - Example of tensorboard with tensorflow (without any actual training) # import os from tempfile import gettempdir @@ -7,7 +7,7 @@ import tensorflow as tf import numpy as np from PIL import Image -from trains import Task +from clearml import Task task = Task.init(project_name='examples', task_name='tensorboard toy example') diff --git a/examples/frameworks/tensorflow/legacy/tensorflow_eager.py b/examples/frameworks/tensorflow/legacy/tensorflow_eager.py index add5bc76..8f198991 100644 --- a/examples/frameworks/tensorflow/legacy/tensorflow_eager.py +++ b/examples/frameworks/tensorflow/legacy/tensorflow_eager.py @@ -1,4 +1,4 @@ -# TRAINS - Example of tensorflow eager mode, model logging and tensorboard +# ClearML - Example of tensorflow eager mode, model logging and tensorboard # # Copyright 2017 The TensorFlow Authors. All Rights Reserved. # @@ -30,7 +30,7 @@ from tempfile import gettempdir import tensorflow as tf from tensorflow.examples.tutorials.mnist import input_data -from trains import Task +from clearml import Task tf.compat.v1.enable_eager_execution() diff --git a/examples/frameworks/tensorflow/legacy/tensorflow_mnist_with_summaries.py b/examples/frameworks/tensorflow/legacy/tensorflow_mnist_with_summaries.py index 81b2b879..3e06fd80 100644 --- a/examples/frameworks/tensorflow/legacy/tensorflow_mnist_with_summaries.py +++ b/examples/frameworks/tensorflow/legacy/tensorflow_mnist_with_summaries.py @@ -31,7 +31,7 @@ import tempfile import tensorflow as tf from tensorflow.examples.tutorials.mnist import input_data -from trains import Task +from clearml import Task FLAGS = None task = Task.init(project_name='examples', task_name='Tensorflow mnist with summaries example') diff --git a/examples/frameworks/tensorflow/manual_model_upload.py b/examples/frameworks/tensorflow/manual_model_upload.py index ee0a6702..86f8dbe3 100644 --- a/examples/frameworks/tensorflow/manual_model_upload.py +++ b/examples/frameworks/tensorflow/manual_model_upload.py @@ -1,10 +1,10 @@ -# TRAINS - Example of manual model configuration and uploading +# ClearML - Example of manual model configuration and uploading # import os import tempfile import tensorflow as tf -from trains import Task +from clearml import Task task = Task.init(project_name='examples', task_name='Model configuration and upload') diff --git a/examples/frameworks/tensorflow/requirements.txt b/examples/frameworks/tensorflow/requirements.txt index 4c750c8d..a1f9ba5f 100644 --- a/examples/frameworks/tensorflow/requirements.txt +++ b/examples/frameworks/tensorflow/requirements.txt @@ -1,3 +1,3 @@ tensorboard>=2.0 tensorflow>=2.0 -trains \ No newline at end of file +clearml \ No newline at end of file diff --git a/examples/frameworks/tensorflow/tensorboard_pr_curve.py b/examples/frameworks/tensorflow/tensorboard_pr_curve.py index 14245c7f..686a3a06 100644 --- a/examples/frameworks/tensorflow/tensorboard_pr_curve.py +++ b/examples/frameworks/tensorflow/tensorboard_pr_curve.py @@ -36,7 +36,7 @@ from six.moves import xrange # pylint: disable=redefined-builtin import tensorflow as tf from tensorboard.plugins.pr_curve import summary -from trains import Task +from clearml import Task task = Task.init(project_name='examples', task_name='tensorboard pr_curve') diff --git a/examples/frameworks/tensorflow/tensorboard_toy.py b/examples/frameworks/tensorflow/tensorboard_toy.py index 2ccd74e1..4c8d5708 100644 --- a/examples/frameworks/tensorflow/tensorboard_toy.py +++ b/examples/frameworks/tensorflow/tensorboard_toy.py @@ -1,4 +1,4 @@ -# TRAINS - Example of tensorboard with tensorflow (without any actual training) +# ClearML - Example of tensorboard with tensorflow (without any actual training) # import os import tensorflow as tf @@ -6,7 +6,7 @@ import numpy as np from tempfile import gettempdir from PIL import Image -from trains import Task +from clearml import Task def generate_summary(k, step): diff --git a/examples/frameworks/tensorflow/tensorflow_mnist.py b/examples/frameworks/tensorflow/tensorflow_mnist.py index d5cc3564..f663648e 100644 --- a/examples/frameworks/tensorflow/tensorflow_mnist.py +++ b/examples/frameworks/tensorflow/tensorflow_mnist.py @@ -8,7 +8,7 @@ import tensorflow as tf from tensorflow.keras.layers import Dense, Flatten, Conv2D from tensorflow.keras import Model -from trains import Task +from clearml import Task task = Task.init(project_name='examples', diff --git a/examples/frameworks/xgboost/requirements.txt b/examples/frameworks/xgboost/requirements.txt index aa52d2d7..a54eb3ba 100644 --- a/examples/frameworks/xgboost/requirements.txt +++ b/examples/frameworks/xgboost/requirements.txt @@ -1,7 +1,7 @@ matplotlib >= 3.1.1 ; python_version >= '3.6' matplotlib >= 2.2.4 ; python_version < '3.6' sklearn -trains +clearml xgboost>=0.90 ; python_version >= '3' xgboost>=0.82 ; python_version < '3' # sudo apt-get install graphviz diff --git a/examples/frameworks/xgboost/xgboost_sample.py b/examples/frameworks/xgboost/xgboost_sample.py index cc8c2a96..8157ed6e 100644 --- a/examples/frameworks/xgboost/xgboost_sample.py +++ b/examples/frameworks/xgboost/xgboost_sample.py @@ -5,7 +5,7 @@ from sklearn.metrics import accuracy_score from sklearn.model_selection import train_test_split from xgboost import plot_tree -from trains import Task +from clearml import Task task = Task.init(project_name='examples', task_name='XGBoost simple example') iris = datasets.load_iris() diff --git a/examples/optimization/hyper-parameter-optimization/base_template_keras_simple.py b/examples/optimization/hyper-parameter-optimization/base_template_keras_simple.py index c2782063..cf97f1ff 100644 --- a/examples/optimization/hyper-parameter-optimization/base_template_keras_simple.py +++ b/examples/optimization/hyper-parameter-optimization/base_template_keras_simple.py @@ -1,4 +1,4 @@ -# TRAINS - Keras with Tensorboard example code, automatic logging model and Tensorboard outputs +# ClearML - Keras with Tensorboard example code, automatic logging model and Tensorboard outputs # # Train a simple deep NN on the MNIST dataset. # Gets to 98.40% test accuracy after 20 epochs @@ -17,10 +17,10 @@ from tensorflow.keras.layers import Activation, Dense from tensorflow.keras.models import Sequential from tensorflow.keras.optimizers import RMSprop -from trains import Task, Logger +from clearml import Task, Logger -# Connecting TRAINS +# Connecting ClearML task = Task.init(project_name='examples', task_name='Keras HP optimization base') diff --git a/examples/optimization/hyper-parameter-optimization/hyper_parameter_optimizer.py b/examples/optimization/hyper-parameter-optimization/hyper_parameter_optimizer.py index 964b717a..45a1eafe 100644 --- a/examples/optimization/hyper-parameter-optimization/hyper_parameter_optimizer.py +++ b/examples/optimization/hyper-parameter-optimization/hyper_parameter_optimizer.py @@ -1,7 +1,7 @@ import logging -from trains import Task -from trains.automation import ( +from clearml import Task +from clearml.automation import ( DiscreteParameterRange, HyperParameterOptimizer, RandomSearch, UniformIntegerParameterRange) @@ -9,14 +9,14 @@ aSearchStrategy = None if not aSearchStrategy: try: - from trains.automation.optuna import OptimizerOptuna + from clearml.automation.optuna import OptimizerOptuna aSearchStrategy = OptimizerOptuna except ImportError as ex: pass if not aSearchStrategy: try: - from trains.automation.hpbandster import OptimizerBOHB + from clearml.automation.hpbandster import OptimizerBOHB aSearchStrategy = OptimizerBOHB except ImportError as ex: pass @@ -40,7 +40,7 @@ def job_complete_callback( print('WOOT WOOT we broke the record! Objective reached {}'.format(objective_value)) -# Connecting TRAINS +# Connecting ClearML task = Task.init(project_name='Hyper-Parameter Optimization', task_name='Automatic Hyper-Parameter Optimization', task_type=Task.TaskTypes.optimizer, @@ -73,10 +73,11 @@ an_optimizer = HyperParameterOptimizer( UniformIntegerParameterRange('General/layer_2', min_value=128, max_value=512, step_size=128), DiscreteParameterRange('General/batch_size', values=[96, 128, 160]), DiscreteParameterRange('General/epochs', values=[30]), + DiscreteParameterRange('General/optimizer', values=['adam', 'sgd']), ], # this is the objective metric we want to maximize/minimize - objective_metric_title='epoch_accuracy', - objective_metric_series='epoch_accuracy', + objective_metric_title='accuracy', + objective_metric_series='accuracy', # now we decide if we want to maximize it or minimize it (accuracy we maximize) objective_metric_sign='max', # let us limit the number of concurrent experiments, @@ -109,7 +110,7 @@ an_optimizer = HyperParameterOptimizer( # if we are running as a service, just enqueue ourselves into the services queue and let it run the optimization if args['run_as_service']: - # if this code is executed by `trains-agent` the function call does nothing. + # if this code is executed by `clearml-agent` the function call does nothing. # if executed locally, the local process will be terminated, and a remote copy will be executed instead task.execute_remotely(queue_name='services', exit_process=True) diff --git a/examples/optimization/hyper-parameter-optimization/requirements.txt b/examples/optimization/hyper-parameter-optimization/requirements.txt index e1a47da3..770f214c 100644 --- a/examples/optimization/hyper-parameter-optimization/requirements.txt +++ b/examples/optimization/hyper-parameter-optimization/requirements.txt @@ -1,3 +1,3 @@ keras tensorflow -trains \ No newline at end of file +clearml \ No newline at end of file diff --git a/examples/pipeline/pipeline_controller.py b/examples/pipeline/pipeline_controller.py index ac00d024..3211c6c6 100644 --- a/examples/pipeline/pipeline_controller.py +++ b/examples/pipeline/pipeline_controller.py @@ -1,5 +1,5 @@ -from trains import Task -from trains.automation.controller import PipelineController +from clearml import Task +from clearml.automation.controller import PipelineController task = Task.init(project_name='examples', task_name='pipeline demo', diff --git a/examples/pipeline/requirements.txt b/examples/pipeline/requirements.txt index dcb80536..07819fbf 100644 --- a/examples/pipeline/requirements.txt +++ b/examples/pipeline/requirements.txt @@ -2,4 +2,4 @@ joblib>=0.13.2 matplotlib >= 3.1.1 ; python_version >= '3.6' matplotlib >= 2.2.4 ; python_version < '3.6' scikit-learn -trains \ No newline at end of file +clearml \ No newline at end of file diff --git a/examples/pipeline/step1_dataset_artifact.py b/examples/pipeline/step1_dataset_artifact.py index f1e7a6a3..154ab554 100644 --- a/examples/pipeline/step1_dataset_artifact.py +++ b/examples/pipeline/step1_dataset_artifact.py @@ -1,4 +1,4 @@ -from trains import Task, StorageManager +from clearml import Task, StorageManager # create an dataset experiment task = Task.init(project_name="examples", task_name="pipeline step 1 dataset artifact") diff --git a/examples/pipeline/step2_data_processing.py b/examples/pipeline/step2_data_processing.py index ff74c89e..dca47520 100644 --- a/examples/pipeline/step2_data_processing.py +++ b/examples/pipeline/step2_data_processing.py @@ -1,9 +1,9 @@ import pickle -from trains import Task, StorageManager +from clearml import Task, StorageManager from sklearn.model_selection import train_test_split -# Connecting TRAINS +# Connecting ClearML task = Task.init(project_name="examples", task_name="pipeline step 2 process dataset") # program arguments diff --git a/examples/pipeline/step3_train_model.py b/examples/pipeline/step3_train_model.py index 332532b4..a705588c 100644 --- a/examples/pipeline/step3_train_model.py +++ b/examples/pipeline/step3_train_model.py @@ -3,9 +3,9 @@ import matplotlib.pyplot as plt import numpy as np from sklearn.linear_model import LogisticRegression -from trains import Task +from clearml import Task -# Connecting TRAINS +# Connecting ClearML task = Task.init(project_name="examples", task_name="pipeline step 3 train model") # Arguments diff --git a/examples/reporting/3d_plots_reporting.py b/examples/reporting/3d_plots_reporting.py index 0c9b7e0f..9a41f249 100644 --- a/examples/reporting/3d_plots_reporting.py +++ b/examples/reporting/3d_plots_reporting.py @@ -1,8 +1,8 @@ -# TRAINS - Example of manual graphs and statistics reporting +# ClearML - Example of manual graphs and statistics reporting # import numpy as np -from trains import Task, Logger +from clearml import Task, Logger def report_plots(logger, iteration=0): diff --git a/examples/reporting/artifacts.py b/examples/reporting/artifacts.py index 6ba0168c..25fe22c2 100644 --- a/examples/reporting/artifacts.py +++ b/examples/reporting/artifacts.py @@ -4,7 +4,7 @@ from time import sleep import pandas as pd import numpy as np from PIL import Image -from trains import Task +from clearml import Task task = Task.init('examples', 'artifacts example') diff --git a/examples/reporting/html_reporting.py b/examples/reporting/html_reporting.py index b630a656..b6abd0bc 100644 --- a/examples/reporting/html_reporting.py +++ b/examples/reporting/html_reporting.py @@ -1,4 +1,4 @@ -# TRAINS - Example of manual graphs and statistics reporting +# ClearML - Example of manual graphs and statistics reporting # import math @@ -10,7 +10,7 @@ from bokeh.sampledata.autompg import autompg_clean as bokeh_df from bokeh.sampledata.periodic_table import elements from bokeh.transform import dodge, factor_cmap -from trains import Task, Logger +from clearml import Task, Logger def report_html_url(logger, iteration=0): diff --git a/examples/reporting/hyper_parameters.py b/examples/reporting/hyper_parameters.py index 89a336e0..9e05452e 100644 --- a/examples/reporting/hyper_parameters.py +++ b/examples/reporting/hyper_parameters.py @@ -1,4 +1,4 @@ -# TRAINS - example code, ArgumentParser parameter logging, absl parameter logging, and dictionary parameter logging +# ClearML - example code, ArgumentParser parameter logging, absl parameter logging, and dictionary parameter logging # from __future__ import absolute_import from __future__ import division @@ -11,7 +11,7 @@ from absl import app from absl import flags from absl import logging -from trains import Task +from clearml import Task FLAGS = flags.FLAGS @@ -32,7 +32,8 @@ parameters = { 'float': 2.2, 'string': 'my string', } -parameters = task.connect_configuration(parameters, name='test') +from clearml import Task +parameters = Task.current_task().connect(parameters, name='more_stuff_deep_inside_code') # adding new parameter after connect (will be logged as well) parameters['new_param'] = 'this is new' @@ -41,6 +42,19 @@ parameters['new_param'] = 'this is new' parameters['float'] = '9.9' print(parameters) +complex_nested_dict_configuration = { + 'list_of_dicts': [{'a': 1, 'b': 2}, {'c': 3, 'd': 4}, {'e': 5, 'f': 6}], + 'nested_dicts': {'nested': {'key': 'value', 'extra': 'value'}, 'number': 42}, + 'dict': {'simple': 'value', 'number': 2}, + 'list': [1, 2, 3], + 'int': 3, + 'float': 2.2, + 'string': 'additional string', +} +complex_nested_dict_configuration = task.connect_configuration( + complex_nested_dict_configuration, name='configuration dictionary') +print(complex_nested_dict_configuration) + def main(_): print('Running under Python {0[0]}.{0[1]}.{0[2]}'.format(sys.version_info), file=sys.stderr) diff --git a/examples/reporting/image_reporting.py b/examples/reporting/image_reporting.py index 7030401a..19cc6967 100644 --- a/examples/reporting/image_reporting.py +++ b/examples/reporting/image_reporting.py @@ -1,11 +1,11 @@ -# TRAINS - Example of manual graphs and statistics reporting +# ClearML - Example of manual graphs and statistics reporting # import os import numpy as np from PIL import Image -from trains import Task, Logger +from clearml import Task, Logger def report_debug_images(logger, iteration=0): diff --git a/examples/reporting/manual_matplotlib_reporting.py b/examples/reporting/manual_matplotlib_reporting.py deleted file mode 100644 index 19b932e3..00000000 --- a/examples/reporting/manual_matplotlib_reporting.py +++ /dev/null @@ -1,55 +0,0 @@ -# TRAINS - Example of Matplotlib and Seaborn integration and reporting -# -import numpy as np -import matplotlib.pyplot as plt -from trains import Task - -# Create a new task, disable automatic matplotlib connect -task = Task.init( - project_name='examples', - task_name='Manual Matplotlib example', - auto_connect_frameworks={'matplotlib': False} -) - -# Create plot and explicitly report as figure -N = 50 -x = np.random.rand(N) -y = np.random.rand(N) -colors = np.random.rand(N) -area = (30 * np.random.rand(N))**2 # 0 to 15 point radii -plt.scatter(x, y, s=area, c=colors, alpha=0.5) -task.logger.report_matplotlib_figure( - title="Manual Reporting", - series="Just a plot", - iteration=0, - figure=plt, -) - -# Show the plot -plt.show() - -# Create plot and explicitly report as an image -plt.scatter(x, y, s=area, c=colors, alpha=0.5) -task.logger.report_matplotlib_figure( - title="Manual Reporting", - series="Plot as an image", - iteration=0, - figure=plt, - report_image=True, -) - - -# Create an image plot and explicitly report (as an image) -m = np.eye(256, 256, dtype=np.uint8) -plt.imshow(m) -task.logger.report_matplotlib_figure( - title="Manual Reporting", - series="Image plot", - iteration=0, - figure=plt, - report_image=True, # Note this is required for image plots -) - -# Show the plot -plt.show() - diff --git a/examples/reporting/manual_matplotlib_reporting.py b/examples/reporting/manual_matplotlib_reporting.py new file mode 120000 index 00000000..294d4d39 --- /dev/null +++ b/examples/reporting/manual_matplotlib_reporting.py @@ -0,0 +1 @@ +matplotlib_manual_reporting.py \ No newline at end of file diff --git a/examples/reporting/matplotlib_automatic_reporting.py b/examples/reporting/matplotlib_automatic_reporting.py new file mode 120000 index 00000000..84a4cb08 --- /dev/null +++ b/examples/reporting/matplotlib_automatic_reporting.py @@ -0,0 +1 @@ +../frameworks/matplotlib/matplotlib_example.py \ No newline at end of file diff --git a/examples/reporting/matplotlib_manual_reporting.py b/examples/reporting/matplotlib_manual_reporting.py new file mode 100644 index 00000000..1d6b8e63 --- /dev/null +++ b/examples/reporting/matplotlib_manual_reporting.py @@ -0,0 +1,55 @@ +# ClearML - Example of Matplotlib and Seaborn integration and reporting +# +import numpy as np +import matplotlib.pyplot as plt +from clearml import Task + +# Create a new task, disable automatic matplotlib connect +task = Task.init( + project_name='examples', + task_name='Manual Matplotlib example', + auto_connect_frameworks={'matplotlib': False} +) + +# Create plot and explicitly report as figure +N = 50 +x = np.random.rand(N) +y = np.random.rand(N) +colors = np.random.rand(N) +area = (30 * np.random.rand(N))**2 # 0 to 15 point radii +plt.scatter(x, y, s=area, c=colors, alpha=0.5) +task.logger.report_matplotlib_figure( + title="Manual Reporting", + series="Just a plot", + iteration=0, + figure=plt, +) + +# Show the plot +plt.show() + +# Create plot and explicitly report as an image +plt.scatter(x, y, s=area, c=colors, alpha=0.5) +task.logger.report_matplotlib_figure( + title="Manual Reporting", + series="Plot as an image", + iteration=0, + figure=plt, + report_image=True, +) + + +# Create an image plot and explicitly report (as an image) +m = np.eye(256, 256, dtype=np.uint8) +plt.imshow(m) +task.logger.report_matplotlib_figure( + title="Manual Reporting", + series="Image plot", + iteration=0, + figure=plt, + report_image=True, # Note this is required for image plots +) + +# Show the plot +plt.show() + diff --git a/examples/reporting/media_reporting.py b/examples/reporting/media_reporting.py index 00f7ae70..7fb9007a 100644 --- a/examples/reporting/media_reporting.py +++ b/examples/reporting/media_reporting.py @@ -1,7 +1,7 @@ -# TRAINS - Example reporting video or audio links/file +# ClearML - Example reporting video or audio links/file # import os -from trains import Task, Logger +from clearml import Task, Logger task = Task.init(project_name="examples", task_name="audio and video reporting") diff --git a/examples/reporting/model_config.py b/examples/reporting/model_config.py index a0d401e6..bcfd3170 100644 --- a/examples/reporting/model_config.py +++ b/examples/reporting/model_config.py @@ -1,8 +1,8 @@ -# TRAINS - Example of manual model configuration +# ClearML - Example of manual model configuration # import os -from trains import Task, OutputModel +from clearml import Task, OutputModel task = Task.init(project_name='examples', task_name='Model configuration example') diff --git a/examples/reporting/pandas_reporting.py b/examples/reporting/pandas_reporting.py index 42ff8480..1792b254 100644 --- a/examples/reporting/pandas_reporting.py +++ b/examples/reporting/pandas_reporting.py @@ -1,9 +1,9 @@ -# TRAINS - Example of manual graphs and statistics reporting +# ClearML - Example of manual graphs and statistics reporting # import pandas as pd -from trains import Task, Logger +from clearml import Task, Logger def report_table(logger, iteration=0): diff --git a/examples/reporting/plotly_reporting.py b/examples/reporting/plotly_reporting.py index f70dea4e..dcbabc9b 100644 --- a/examples/reporting/plotly_reporting.py +++ b/examples/reporting/plotly_reporting.py @@ -1,6 +1,6 @@ -# TRAINS - Example of Plotly integration and reporting +# ClearML - Example of Plotly integration and reporting # -from trains import Task +from clearml import Task import plotly.express as px diff --git a/examples/reporting/requirements.txt b/examples/reporting/requirements.txt index 0b40bccf..a7621f15 100644 --- a/examples/reporting/requirements.txt +++ b/examples/reporting/requirements.txt @@ -6,4 +6,4 @@ numpy pandas pillow>=4.0 six -trains +clearml diff --git a/examples/reporting/scalar_reporting.py b/examples/reporting/scalar_reporting.py index 2be77854..adda0433 100644 --- a/examples/reporting/scalar_reporting.py +++ b/examples/reporting/scalar_reporting.py @@ -1,6 +1,6 @@ -# TRAINS - Example of manual graphs and statistics reporting +# ClearML - Example of manual graphs and statistics reporting # -from trains import Task, Logger +from clearml import Task, Logger def report_scalars(logger): diff --git a/examples/reporting/scatter_hist_confusion_mat_reporting.py b/examples/reporting/scatter_hist_confusion_mat_reporting.py index 184f5c8f..72b79318 100644 --- a/examples/reporting/scatter_hist_confusion_mat_reporting.py +++ b/examples/reporting/scatter_hist_confusion_mat_reporting.py @@ -1,8 +1,8 @@ -# TRAINS - Example of manual graphs and statistics reporting +# ClearML - Example of manual graphs and statistics reporting # import numpy as np -from trains import Task, Logger +from clearml import Task, Logger def report_plots(logger, iteration=0): @@ -117,6 +117,7 @@ def main(): # Get the task logger, # You can also call Task.current_task().get_logger() from anywhere in your code. logger = task.get_logger() + #logger.report_scatter2d() # report graphs report_plots(logger) diff --git a/examples/reporting/text_reporting.py b/examples/reporting/text_reporting.py index 2f91c4e6..99209ea9 100644 --- a/examples/reporting/text_reporting.py +++ b/examples/reporting/text_reporting.py @@ -1,4 +1,4 @@ -# TRAINS - Example of manual graphs and statistics reporting +# ClearML - Example of manual graphs and statistics reporting # from __future__ import print_function @@ -7,7 +7,7 @@ import sys import six -from trains import Logger, Task +from clearml import Logger, Task def report_logs(logger): diff --git a/examples/services/aws-autoscaler/aws_autoscaler.py b/examples/services/aws-autoscaler/aws_autoscaler.py index 4ff689e8..e9335bab 100644 --- a/examples/services/aws-autoscaler/aws_autoscaler.py +++ b/examples/services/aws-autoscaler/aws_autoscaler.py @@ -7,10 +7,10 @@ import yaml from pathlib2 import Path from six.moves import input -from trains import Task -from trains.automation.aws_auto_scaler import AwsAutoScaler -from trains.config import running_remotely -from trains.utilities.wizard.user_input import ( +from clearml import Task +from clearml.automation.aws_auto_scaler import AwsAutoScaler +from clearml.config import running_remotely +from clearml.utilities.wizard.user_input import ( get_input, input_int, input_bool, @@ -45,7 +45,7 @@ def main(): print("AWS Autoscaler setup wizard\n" "---------------------------\n" "Follow the wizard to configure your AWS auto-scaler service.\n" - "Once completed, you will be able to view and change the configuration in the trains-server web UI.\n" + "Once completed, you will be able to view and change the configuration in the clearml-server web UI.\n" "It means there is no need to worry about typos or mistakes :)\n") config_file = Path(CONF_FILE).absolute() @@ -85,7 +85,7 @@ def main(): if args.remote: # if we are running remotely enqueue this run, and leave the process - # the trains-agent services will pick it up and execute it for us. + # the clearml-agent services will pick it up and execute it for us. task.execute_remotely(queue_name='services') autoscaler = AwsAutoScaler(hyper_params, configurations) diff --git a/examples/services/aws-autoscaler/requirements.txt b/examples/services/aws-autoscaler/requirements.txt index 17225017..ae1e1883 100644 --- a/examples/services/aws-autoscaler/requirements.txt +++ b/examples/services/aws-autoscaler/requirements.txt @@ -1,5 +1,5 @@ boto3 pyYaml six -trains +clearml pathlib2 diff --git a/examples/services/cleanup/cleanup_service.py b/examples/services/cleanup/cleanup_service.py index c1cebad2..7939cebe 100644 --- a/examples/services/cleanup/cleanup_service.py +++ b/examples/services/cleanup/cleanup_service.py @@ -10,7 +10,7 @@ You can configure the run by changing the `args` dictionary: - force_delete (bool): Allows forcing the task deletion (for every task status). Default: False. Requirements: -- trains_agent installed -> pip install trains-agent +- clearml_agent installed -> pip install clearml-agent """ import logging @@ -20,11 +20,11 @@ from glob import glob from shutil import rmtree from time import sleep, time -from trains.backend_api.session.client import APIClient +from clearml.backend_api.session.client import APIClient -from trains import Task +from clearml import Task -# Connecting TRAINS +# Connecting ClearML task = Task.init( project_name="DevOps", task_name="Cleanup Service", @@ -54,7 +54,7 @@ if args["run_as_service"] and task.running_locally(): args["run_as_service"] = not verify or verify.startswith('y') if args["run_as_service"]: - # if this code is executed by `trains-agent` the function call does nothing. + # if this code is executed by `clearml-agent` the function call does nothing. # if executed locally, the local process will be terminated, and a remote copy will be executed instead task.execute_remotely(queue_name="services", exit_process=True) diff --git a/examples/services/cleanup/requirements.txt b/examples/services/cleanup/requirements.txt index 3417fd38..6dac6a83 100644 --- a/examples/services/cleanup/requirements.txt +++ b/examples/services/cleanup/requirements.txt @@ -1 +1 @@ -trains \ No newline at end of file +clearml \ No newline at end of file diff --git a/examples/services/jupyter-service/execute_jupyter_notebook_server.py b/examples/services/jupyter-service/execute_jupyter_notebook_server.py index b53bc21d..0c30ad6b 100644 --- a/examples/services/jupyter-service/execute_jupyter_notebook_server.py +++ b/examples/services/jupyter-service/execute_jupyter_notebook_server.py @@ -9,29 +9,30 @@ import psutil # make sure we have jupyter in the auto requirements import jupyter # noqa -from trains import Task +from clearml import Task -# initialize TRAINS +# initialize ClearML task = Task.init( project_name="DevOps", task_name="Allocate Jupyter Notebook Instance", task_type=Task.TaskTypes.service) -# get rid of all the runtime TRAINS +# get rid of all the runtime ClearML preserve = ( - "TRAINS_API_HOST", - "TRAINS_WEB_HOST", - "TRAINS_FILES_HOST", - "TRAINS_CONFIG_FILE", - "TRAINS_API_ACCESS_KEY", - "TRAINS_API_SECRET_KEY", - "TRAINS_API_HOST_VERIFY_CERT", - "TRAINS_DOCKER_IMAGE", + "_API_HOST", + "_WEB_HOST", + "_FILES_HOST", + "_CONFIG_FILE", + "_API_ACCESS_KEY", + "_API_SECRET_KEY", + "_API_HOST_VERIFY_CERT", + "_DOCKER_IMAGE", ) # setup os environment env = deepcopy(os.environ) for key in os.environ: - if key.startswith("TRAINS") and key not in preserve: + if (key.startswith("TRAINS") and key[6:] not in preserve) or \ + (key.startswith("CLEARML") and key[7:] not in preserve): env.pop(key, None) # Add jupyter server base folder @@ -44,7 +45,7 @@ param = { task.connect(param) # set default docker image, with network configuration -os.environ["TRAINS_DOCKER_IMAGE"] = param['default_docker_for_jupyter'] +os.environ["CLEARML_DOCKER_IMAGE"] = param['default_docker_for_jupyter'] task.set_base_docker("{} --network host".format(param['default_docker_for_jupyter'])) @@ -79,11 +80,11 @@ if param.get("ssh_server"): "sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config && " "sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd && " # noqa: W605 'echo "export VISIBLE=now" >> /etc/profile && ' - 'echo "export TRAINS_CONFIG_FILE={trains_config_file}" >> /etc/profile && ' + 'echo "export TRAINS_CONFIG_FILE={clearml_config_file}" >> /etc/profile && ' "/usr/sbin/sshd -p {port}".format( password=ssh_password, port=port, - trains_config_file=os.environ.get("TRAINS_CONFIG_FILE"), + clearml_config_file=os.environ.get("TRAINS_CONFIG_FILE"), ) ) diff --git a/examples/services/jupyter-service/requirements.txt b/examples/services/jupyter-service/requirements.txt index 340406f0..ad052a27 100644 --- a/examples/services/jupyter-service/requirements.txt +++ b/examples/services/jupyter-service/requirements.txt @@ -1,3 +1,3 @@ jupyter psutil -trains \ No newline at end of file +clearml \ No newline at end of file diff --git a/examples/services/monitoring/requirements.txt b/examples/services/monitoring/requirements.txt index 182dbbab..6972a82c 100644 --- a/examples/services/monitoring/requirements.txt +++ b/examples/services/monitoring/requirements.txt @@ -1,2 +1,2 @@ -trains +clearml slackclient > 2.0.0 diff --git a/examples/services/monitoring/slack_alerts.py b/examples/services/monitoring/slack_alerts.py index 59f12f4d..5ea5a89c 100644 --- a/examples/services/monitoring/slack_alerts.py +++ b/examples/services/monitoring/slack_alerts.py @@ -1,7 +1,7 @@ """ -Create a Trains Monitoring Service that posts alerts on Slack Channel groups based on some logic +Create a ClearML Monitoring Service that posts alerts on Slack Channel groups based on some logic -Creating a new Slack Bot (Allegro Trains Bot): +Creating a new Slack Bot (Allegro ClearML Bot): 1. Login to your Slack account 2. Go to https://api.slack.com/apps/new 3. Give the new App a name (For example "Allegro Train Bot") and select your workspace @@ -18,7 +18,7 @@ Creating a new Slack Bot (Allegro Trains Bot): 8. Now under "OAuth Tokens & Redirect URLs" press on "Install App to Workspace", then hit "Allow" on the confirmation dialog 9. Under "OAuth Tokens & Redirect URLs" copy the "Bot User OAuth Access Token" by clicking on "Copy" button -10. To use the copied API Token in the Allegro Trains Slack service, +10. To use the copied API Token in the Allegro ClearML Slack service, execute the script with --slack_api "" (notice the use of double quotes around the token) We are done! @@ -32,8 +32,8 @@ from typing import Optional from slack import WebClient from slack.errors import SlackApiError -from trains import Task -from trains.automation.monitor import Monitor +from clearml import Task +from clearml.automation.monitor import Monitor class SlackMonitor(Monitor): @@ -141,10 +141,10 @@ class SlackMonitor(Monitor): def main(): - print('TRAINS experiment monitor Slack service\n') + print('ClearML experiment monitor Slack service\n') # Slack Monitor arguments - parser = argparse.ArgumentParser(description='TRAINS monitor experiments and post Slack Alerts') + parser = argparse.ArgumentParser(description='ClearML monitor experiments and post Slack Alerts') parser.add_argument('--channel', type=str, help='Set the channel to post the Slack alerts') parser.add_argument('--slack_api', type=str, default=os.environ.get('SLACK_API_TOKEN', None), @@ -158,7 +158,7 @@ def main(): 'This will help eliminate unnecessary debug sessions that crashed right after starting ' '(default:0 alert on all)') parser.add_argument('--include_manual_experiments', action="store_true", default=False, - help='Include experiments running manually (i.e. not by trains-agent)') + help='Include experiments running manually (i.e. not by clearml-agent)') parser.add_argument('--include_completed_experiments', action="store_true", default=False, help='Include completed experiments (i.e. not just failed experiments)') parser.add_argument('--refresh_rate', type=float, default=10., @@ -202,7 +202,7 @@ def main(): # Let everyone know we are up and running start_message = \ - '{}Allegro Trains Slack monitoring service started\nMonitoring project \'{}\''.format( + '{}Allegro ClearML Slack monitoring service started\nMonitoring project \'{}\''.format( (args.message_prefix + ' ') if args.message_prefix else '', args.project or 'all') slack_monitor.post_message(start_message) diff --git a/requirements.txt b/requirements.txt index d3c3ba81..b840a10d 100644 --- a/requirements.txt +++ b/requirements.txt @@ -13,7 +13,7 @@ Pillow>=4.1.1 psutil>=3.4.2 pyparsing>=2.0.3 python-dateutil>=2.6.1 -pyjwt>=1.6.4 +pyjwt>=1.6.4,<2.0.0 PyYAML>=3.12 requests-file>=1.4.2 requests>=2.20.0 diff --git a/setup.py b/setup.py index 7bbbfe6e..7e34c524 100644 --- a/setup.py +++ b/setup.py @@ -1,5 +1,5 @@ """ -TRAINS - Artificial Intelligence Version Control +ClearML - Artificial Intelligence Version Control https://github.com/allegroai/trains """ @@ -27,20 +27,20 @@ def read_version_string(version_file): raise RuntimeError("Unable to find version string.") -version = read_version_string("trains/version.py") +version = read_version_string("clearml/version.py") requirements = read_text(os.path.join(here, 'requirements.txt')).splitlines() setup( - name='trains', + name='clearml', version=version, - description='TRAINS - Auto-Magical Experiment Manager & Version Control for AI', + description='ClearML - Auto-Magical Experiment Manager, Version Control, and MLOps for AI', long_description=long_description, long_description_content_type='text/markdown', # The project's main homepage. url='https://github.com/allegroai/trains', author='Allegroai', - author_email='trains@allegro.ai', + author_email='clearml@allegro.ai', license='Apache License 2.0', classifiers=[ # How mature is this project. Common values are @@ -65,7 +65,7 @@ setup( 'Programming Language :: Python :: 3.8', 'License :: OSI Approved :: Apache Software License', ], - keywords='trains development machine deep learning version control machine-learning machinelearning ' + keywords='clearml trains development machine deep learning version control machine-learning machinelearning ' 'deeplearning deep-learning experiment-manager experimentmanager', packages=find_packages(exclude=['contrib', 'docs', 'data', 'examples', 'tests']), install_requires=requirements, @@ -81,7 +81,7 @@ setup( ], }, package_data={ - 'trains': ['config/default/*.conf', 'backend_api/config/default/*.conf'] + 'clearml': ['config/default/*.conf', 'backend_api/config/default/*.conf'] }, include_package_data=True, # To provide executable scripts, use entry points in preference to the @@ -89,7 +89,9 @@ setup( # pip to create the appropriate form of executable for the target platform. entry_points={ 'console_scripts': [ - 'trains-init = trains.config.default.__main__:main', + 'clearml-init = clearml.cli.config.__main__:main', + 'clearml-data = clearml.cli.data.__main__:main', + 'clearml-launch = clearml.cli.task.__main__:main', ], }, )