ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-06-26 18:17:44 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Add Hyper-Datasets

Browse Source
This commit is contained in:
allegroai
2021-06-21 01:00:16 +03:00
parent 31b3b52cac
commit 81bcabcb10
76 changed files with 3319 additions and 168 deletions
Download Patch File Download Diff File Expand all files Collapse all files

56
docs/hyperdatasets/annotations.md Normal file
View File

@@ -0,0 +1,56 @@
---
title: Annotations
---
With **ClearML Enterprise**, annotations can be applied to video and image frames. [Frames](single_frames.md) support
two types of annotations: **Frame objects** and **Frame labels**.
Annotation Tasks can be used to efficiently organize the annotation of frames in Dataset versions (see
[Annotations Task Page](webapp/webapp_annotator.md)).
For information about how to view, create, and manage annotations using the WebApp, see [Annotating Images and Videos](#annotating-images-and-video).
## Frame objects
Frame objects are labeled Regions of Interest (ROIs), which can be bounded by polygons (including rectangles), ellipses,
or key points. These ROIs are useful for object detection, classification, or semantic segmentation.
Frame objects can include ROI labels, confidence levels, and masks for semantic segmentation. In **ClearML Enterprise**,
one or more labels and sources dictionaries can be associated with an ROI (although multiple source ROIs are not frequently used).
## Frame labels
Frame labels are applied to an entire frame, not a region in a frame.
## Usage
### Adding a frame object
To add a frame object annotation to a SingleFrame, use the [`SingleFrame.add_annotation`](google.com) method.
```python
# a bounding box labeled "test" at x=10,y=10 with width of 30px and height of 20px
frame.add_annotation(box2d_xywh=(10, 10, 30, 20), labels=['test'])
```
The argument `box2d_xywh` specifies the coordinates of the annotation's bounding box, and the argument `labels` specifies
a list of labels for the annotation.
When adding an annotation there are a few options for entering the annotation's boundaries, including:
* `poly2d_xy` - A list of floating points (x,y) to create for single polygon, or a list of Floating points lists for a
complex polygon
* `ellipse2d_xyrrt` - A List consisting of cx, cy, rx, ry, and theta for an ellipse
* And more! See [`SingleFrame.add_annotation`](google.com) for further options.
### Adding a Frame label
Adding a frame label is similar to creating a frame objects, except that coordinates don't need to be specified, since
the whole frame is being referenced.
Use the [`SingleFrame.add_annotation`](google.com) method, but use only the `labels` parameter.
```python
# labels for the whole frame
frame.add_annotation(labels=['frame level label one','frame level label two'])
```

40
docs/hyperdatasets/custom_metadata.md Normal file
View File

@@ -0,0 +1,40 @@
---
title: Custom Metadata
---
Metadata can be customized as needed using: **meta** dictionaries:
* As a top-level key for metadata applying to entire frame
* In `rois` dictionaries, for metadata applying to individual ROIs.
## Usage
### Adding Frame metadata
When instantiating a Frame, metadata that applies for the entire frame can be
added as an argument.
```python
from allegroai import SingleFrame
# create a frame with metadata
frame = SingleFrame(
source='https://allegro-datasets.s3.amazonaws.com/tutorials/000012.jpg',
preview_uri='https://allegro-datasets.s3.amazonaws.com/tutorials/000012.jpg',
# insert metadata dictionary
metadata={'alive':'yes'},
)
# add metadata to the frame
frame.metadata['dangerous'] = 'no'
```
### Adding ROI metadata
Metadata can be added to individual ROIs when adding an annotation to a `frame`, using the [`add_annotation`](google.com)
method.
```python
frame.add_annotation(box2d_xywh=(10, 10, 30, 20), labels=['tiger'],
# insert metadata dictionary
metadata={'dangerous':'yes'})
```

305
docs/hyperdatasets/dataset.md Normal file
View File

@@ -0,0 +1,305 @@
---
title: Datasets and Dataset Versions
---
ClearML Enterprise's **Datasets** and **Dataset versions** provide the internal data structure
and functionality for the following purposes:
* Connecting source data to the **ClearML Enterprise** platform
* Using **ClearML Enterprise**'s GIT-like [Dataset versioning](#dataset-versioning)
* Integrating the powerful features of [Dataviews](dataviews.md) with an experiment
* [Annotating](webapp/webapp_datasets_frames.md#annotations) images and videos
Datasets consist of versions with SingleFrames and / or FrameGroups. Each Dataset can contain multiple versions, where
each version can have multiple children that inherit their parent's SingleFrames and / or FrameGroups. This inheritance
includes the frame metadata and data connecting the source data to the ClearML Enterprise platform, as well as the other
metadata and data.
These parent-child version relationships can be represented as version trees with a root-level parent. A Dataset
can contain one or more trees.
## Dataset version state
Dataset versions can have either **Draft** or **Published** status.
A **Draft** version is editable, so frames can be added to and deleted and / or modified from the Dataset.
A **Published** version is read-only, which ensures reproducible experiments and preserves a version of a Dataset.
Child versions can only be created from *Published* versions. To create a child of a *Draft* Dataset version,
it must be published first.
## Example Datasets
**ClearML Enterprise** provides Example Datasets, available to in the **ClearML Enterprise** platform, with frames already built,
and ready for your experimentation. Find these example Datasets in the **ClearML Enterprise** WebApp (UI). They appear
with an "Example" banner in the WebApp (UI).
## Usage
### Creating Datasets
Use the [Dataset.create](google.com) method to create a Dataset. It will contain an empty version named `Current`.
```python
from allegroai import Dataset
myDataset = Dataset.create(dataset_name='myDataset')
```
Or, use the [DatasetVersion.create_new_dataset](google.com) method.
```python
from allegroai import DatasetVersion
myDataset = DatasetVersion.create_new_dataset(dataset_name='myDataset Two')
```
To raise a `ValueError` exception if the Dataset exists, specify the `raise_if_exists` parameters as `True`.
* With `Dataset.create`
```python
try:
myDataset = Dataset.create(dataset_name='myDataset One', raise_if_exists=True)
except ValueError:
print('Dataset exists.')
```
* Or with `DatasetVersion.create_new_dataset`
```python
try:
myDataset = DatasetVersion.create_new_dataset(dataset_name='myDataset Two', raise_if_exists=True)
except ValueError:
print('Dataset exists.')
```
Additionally, create a Dataset with tags and a description.
```python
myDataset = DatasetVersion.create_new_dataset(dataset_name='myDataset',
tags=['One Tag', 'Another Tag', 'And one more tag'],
description='some description text')
```
### Accessing current Dataset
To get the current Dataset, use the `DatasetVersion.get_current` method.
```python
myDataset = DatasetVersion.get_current(dataset_name='myDataset')
```
### Deleting Datasets
Use the `Dataset.delete` method to delete a Dataset.
Delete an empty Dataset (no versions).
```python
Dataset.delete(dataset_name='MyDataset', delete_all_versions=False, force=False)
```
Delete a Dataset containing only versions whose status is *Draft*.
```python
Dataset.delete(dataset_name='MyDataset', delete_all_versions=True, force=False)
```
Delete a Dataset even if it contains versions whose status is *Published*.
```python
Dataset.delete(dataset_name='MyDataset', delete_all_versions=True, force=True)
```
## Dataset Versioning
Dataset versioning refers to the group of **ClearML Enterprise** SDK and WebApp (UI) features for creating,
modifying, and deleting Dataset versions.
**ClearML Enterprise** supports simple and sophisticated Dataset versioning, including **simple version structures** and
**advanced version structures**.
In a **simple version structure**, a parent can have one and only one child, and the last child in the Dataset versions
tree must be a *Draft*. This simple structure allows working with a single set of versions of a Dataset. Create children
and publish versions to preserve data history. Each version whose status is *Published* in a simple version structure is
referred to as a **snapshot**.
In an **advanced version structure**, at least one parent has more than one child (this can include more than one parent
version at the root level), or the last child in the Dataset versions tree is *Published*.
Creating a version in a simple version structure may convert it to an advanced structure. This happens when creating
a Dataset version that yields a parent with two children, or when publishing the last child version.
## Versioning Usage
Manage Dataset versioning using the [DatasetVersion](google.com) class in the ClearML Enterprise SDK.
### Creating snapshots
If the Dataset contains only one version whose status is *Draft*, snapshots of the current version can be created.
When creating a snapshot, the current version becomes the snapshot (it keeps the same version ID),
and the newly created version (with its new version ID) becomes the current version.
To create a snapshot, use the [DatasetVersion.create_snapshot](google.com) method.
#### Snapshot naming
In the simple version structure, ClearML Enterprise supports two methods for snapshot naming:
* **Timestamp naming** - If only the Dataset name or ID is provided, the snapshot is named `snapshot` with a timestamp
appended.
The timestamp format is ISO 8601 (`YYYY-MM-DDTHH:mm:ss.SSSSSS`). For example, `snapshot 2020-03-26T16:55:38.441671`.
**Example:**
```python
from allegroai import DatasetVersion
myDataset = DatasetVersion.create_snapshot(dataset_name='MyDataset')
```
After the statement above runs, the previous current version keeps its existing version ID, and it becomes a
snapshot named `snapshot` with a timestamp appended. The newly created version with a new version ID becomes
the current version, and its name is `Current`.
* **User-specified snapshot naming** - If the `publish_name` parameter is provided, it will be the name of the snapshot name.
**Example:**
```python
myDataset = DatasetVersion.create_snapshot(dataset_name='MyDataset', publish_name='NewSnapshotName')
```
After the above statement runs, the previous current version keeps its existing version ID and becomes a snapshot named
`NewSnapshotName`.
The newly created version (with a new version ID) becomes the current version, and its name is `Current`.
#### Current version naming
In the simple version structure, ClearML Enterprise supports two methods for current version naming:
* **Default naming** - If the `child_name` parameter is not provided, `Current` is the current version name.
* **User-specified current version naming** - If the `child_name` parameter is provided, that child name becomes the current
version name.
For example, after the following statement runs, the previous current version keeps its existing version ID and becomes
a snapshot named `snapshot` with the timestamp appended.
The newly created version (with a new version ID) is the current version, and its name is `NewCurrentVersionName`.
```python
myDataset = DatasetVersion.create_snapshot(dataset_name='MyDataset',
child_name='NewCurrentVersionName')
```
#### Adding metadata and comments
Add a metadata dictionary and / or comment to a snapshot.
For example:
```python
myDataset = DatasetVersion.create_snapshot(dataset_name='MyDataset',
child_metadata={'abc':'1234','def':'5678'},
child_comment='some text comment')
```
### Creating child versions
Create a new version from any version whose status is *Published*.
To create a new version, call the [DatasetVersion.create_version](google.com) method, and
provide:
* Either the Dataset name or ID
* The parent version name or ID from which the child inherits frames
* The new version's name.
For example, create a new version named `NewChildVersion` from the existing version `PublishedVersion`,
where the new version inherits the frames of the existing version. If `NewChildVersion` already exists,
it is returned.
```python
myVersion = DatasetVersion.create_version(dataset_name='MyDataset',
parent_version_names=['PublishedVersion'],
version_name='NewChildVersion')
```
To raise a ValueError exception if `NewChildVersion` exists, set `raise_if_exists` to `True`.
```python
myVersion = DatasetVersion.create_version(dataset_name='MyDataset',
parent_version_names=['PublishedVersion'],
version_name='NewChildVersion',
raise_if_exists=True))
```
### Creating root-level parent versions
Create a new version at the root-level. This is a version without a parent, and it contains no frames.
```python
myDataset = DatasetVersion.create_version(dataset_name='MyDataset',
version_name='NewRootVersion')
```
### Getting versions
To get a version or versions, use the [DatasetVersion.get_version](google.com) and [DatasetVersion.get_versions](google.com)
methods, respectively.
**Getting a list of all versions**
```python
myDatasetversion = DatasetVersion.get_versions(dataset_name='MyDataset')
```
**Getting a list of all _published_ versions**
```python
myDatasetversion = DatasetVersion.get_versions(dataset_name='MyDataset',
only_published=True)
```
**Getting a list of all _drafts_ versions**
```python
myDatasetversion = DatasetVersion.get_versions(dataset_name='MyDataset',
only_draft=True)
```
**Getting the current version**
If more than one version exists, ClearML Enterprise outputs a warning.
```python
myDatasetversion = DatasetVersion.get_version(dataset_name='MyDataset')
```
**Getting a specific version**
```python
myDatasetversion = DatasetVersion.get_version(dataset_name='MyDataset',
version_name='VersionName')
```
### Deleting versions
Delete versions which are status *Draft* using the [Dataset.delete_version](google.com) method.
```python
from allegroai import Dataset
myDataset = Dataset.get(dataset_name='MyDataset')
myDataset.delete_version(version_name='VersionToDelete')
```
### Publishing versions
Publish (make read-only) versions which are status *Draft* using the [Dataset.publish_version](google.com) method. This includes the current version, if the Dataset is in
the simple version structure.
```python
myVersion = DatasetVersion.get_version(dataset_name='MyDataset',
version_name='VersionToPublish')
myVersion.publish_version()
```

389
docs/hyperdatasets/dataviews.md Normal file
View File

@@ -0,0 +1,389 @@
---
title: Dataviews
---
Dataviews is a powerful and easy-to-use **ClearML Enterprise** feature for creating and managing local views of remote
Datasets. Dataviews can use sophisticated queries to input data from a subset of a Dataset
or combinations of Datasets.
Dataviews support:
* Filtering by ROI labels, frame metadata, and data sources
* Data debiasing to adjust for imbalanced data
* ROI label mapping (label translation)
* Class label enumeration
* Controls for the frame iteration, such as sequential or random iteration, limited or infinite iteration, and reproducibility.
Dataviews are lazy and optimize processing. When an experiment script runs in a local environment, Dataview pointers
are initialized. If the experiment is cloned or extended, and that newly cloned or extended experiment is tuned and run,
only changed pointers are initialized. The pointers that did not change are reused.
## Filtering
A Dataview filters experiment input data, using one or more frame filters. A frame filter defines the criteria for the
selection of SingleFrames iterated by a Dataview.
A frame filter contains the following criteria:
* Dataset version - Choose whether the filter applies to one version or all versions of a Dataset.
* Any combination of the following rules:
* ROI rule - Include or exclude frames containing at least one ROI with any combination of labels in the Dataset version.
Optionally, limit the number of matching ROIs (instances) per frame, and / or limit the confidence level of the label.
For example: include frames containing two to four ROIs labeled `cat` and `dog`, with a confidence level from `0.8` to `1.0`.
* Frame rule - Filter by frame metadata key-value pairs, or ROI labels.
For example: if some frames contain the metadata
key `dangerous` with values of `yes` or `no`, filter `(meta.dangerous:'yes')`.
* Source rule - Filter by frame `source` dictionary key-value pairs.
For example: filter by source ID `(source.id:00)`.
* A ratio (weight) allowing to debias input data, to and adjust an imbalance in SingleFrames iterated by the Dataview (optional).
Use combinations of these frame filters to build sophisticated queries.
## Debiasing input data
Apply debiasing to each frame filter to adjust for an imbalance in input data. Ratios (weights) enable setting the proportion
of frames that are inputted, according to any of the criteria in a frame filter, including ROI labels, frame metadata,
and sources, as well as each Dataset version compared with the others.
For example, data may contain five times the number of frames labeled `daylight` as those labeled `nighttime`, but
you want to input the same number of both. To debias the data, create two frame filters, one for `daylight` with a ratio
of `1`, and the other for `nighttime` with a ratio of `5`. The Dataview will iterate approximately an equal number of
SingleFrames for each.
## ROI Label mapping (label translation)
ROI label mapping (label translation) applies to the new model. For example, apply mapping to:
* Combine different labels under another more generic label.
* Consolidate disparate datasets containing different names for the ROI.
* Hide labeled objects from the training process.
## Class label enumeration
Define class labels for the new model and assign integers to each in order to maintain data conformity across multiple
codebases and datasets. It is important to set enumeration values for all labels of importance.
## Data augmentation
On-the-fly data augmentation is applied to SingleFrames, transforming images without creating new data. Apply data augmentation
in steps, where each step is composed of a method, an operation, and a strength as follows:
* **Affine** augmentation method - Transform an image's geometric shape to another position on a 2-dimensional plane.
Use any of the following operations:
* Rotate
* Reflect-horiz - Flip images horizontally
* Reflect-vert - Flip images vertically
* Scale
* Shear - Skew
* No operation - Randomly select SingleFrames that are not transformed (skipped). If the experiment runs again, and
the random seed in [iteration control](#iteration-control) is unchanged, the same SingleFrames are not augmented.
* **Pixel** augmentation method - Transform images by modifying pixel values while retaining shape and perspective.
Use any of the following operations:
* Blur - Gaussian smoothing
* Noise - **ClearML Enterprise**'s own noise augmentation consisting of:
* **high** noise - like snow on analog televisions with a weak TV signal
* **low** noise - like a low resolution image magnified in localized areas on the image
* Recolor - using an internal RGB lookup-table
* No operation - Randomly select SingleFrames that are not transformed (skipped). If the experiment runs again, and
the random seed in [iteration control](#iteration-control) is unchanged, the same SingleFrames are not augmented.
* Strength - A number applied to adjust the degree of transformation. The recommended strengths are the following:
* 0.0 - No effect
* 0.5 - Low (weak)
* 1.0 - Medium (recommended)
* 2.0 - High (strong)
## Iteration control
The input data **iteration control** settings determine the order, number, timing, and reproducibility of the Dataview iterating
SingleFrames. Depending upon the combination of iteration control settings, all SingleFrames may not be iterated, and some
may repeat. The settings include the following:
* Order - Order of the SingleFrames returned by the iteration, which can be either:
* Sequential - Iterate SingleFrames in sorted order by context ID and timestamp.
* Random - Iterate SingleFrames randomly using a random seed that can be set (see Random Seed below).
* Repetition - The repetition of SingleFrames that, in conjunction with the order, determines whether all SingleFrames
are returned, and whether any may repeat. The repetition settings and their impact on iteration are the following:
* Use Each Frame Once - All SingleFrames are iterated. If the order is sequential, then no SingleFrames repeat. If
the order is random, then some SingleFrames may repeat.
* Limit Frames - The maximum number of SingleFrames to iterate, unless the actual number of SingleFrames is fewer than
the maximum, then the actual number of SingleFrames are iterated. If the order is sequential, then no SingleFrames
repeat. If the order is random, then some SingleFrames may repeat.
* Infinite Iterations - Iterate SingleFrames until the experiment is manually terminated. If the order is sequential,
then all SingleFrames are iterated (unless the experiment is manually terminated before all iterate) and SingleFrames
repeat. If the order is random, then all SingleFrames may not be iterated, and some SingleFrames may repeat.
* Random Seed - If the experiment is rerun and the seed remains unchanged, the SingleFrames iteration is the same.
* Clip Length - For video data sources, in the number of sequential SingleFrames from a clip to iterate.
## Usage
### Creating Dataviews
Use the [`allegroai.DataView`](google.com) class to create a DataView object. Instantiate DataView objects, specifying
iteration settings and additional iteration parameters that control query iterations.
```python
from allegroai import DataView, IterationOrder
# Create a DataView object that iterates randomly until terminated by the user
myDataView = DataView(iteration_order=IterationOrder.random, iteration_infinite=True)
```
### Adding queries
To add a query to a DataView, use the [`DataView.add_query`](google.com) method and specify Dataset versions,
ROI and / or frame queries, and other criteria.
The `dataset_name` and `version_name` arguments specify the Dataset Version. The `roi_query` and `frame_query` arguments
specify the queries.
* `roi_query` can be assigned ROI labels by label name or Lucene queries.
* `frame_query` must be assigned a Lucene query.
Multiple queries can be added to the same or different Dataset versions, each query with the same or different ROI
and / or frame queries.
#### ROI queries:
* ROI query for a single label
This example is an ROI query filtering for frames containing at least one ROI with the label `cat`.
```python
# Create a Dataview object for an iterator that randomly returns frames according to queries
myDataView = DataView(iteration_order=IterationOrder.random, iteration_infinite=True)
# Add a query for a Dataset version
myDataView.add_query(dataset_name='myDataset',
version_name='myVersion', roi_query='cat')
```
* ROI query for one label OR another
This example is an ROI query filtering for frames containing at least one ROI with the label `cat` OR `dog`:
```python
# Add a query for a Dataset version
myDataView.add_query(dataset_name='myDataset', version_name='myVersion',
roi_query='cat')
myDataView.add_query(dataset_name='myDataset', version_name='myVersion',
roi_query='dog')
```
* ROI query for one label AND another label
This example is an ROI query filtering for frames containing at least one ROI with the label `Car` AND `partly_occluded`.
```python
# Add a query for a Dataset version
myDataView.add_query(dataset_name='myDataset', version_name='training',
roi_query=['Car','partly_occluded'])
```
* ROI query for one label AND NOT another (Lucene query).
This example is an ROI query filtering for frames containing at least one ROI with the label `Car` AND NOT the label
`partly_occluded`.
```python
# Add a query for a Dataset version
# Use a Lucene Query
# "label" is a key in the rois dictionary of a frame
# In this Lucene Query, specify two values for the label key and use a Logical AND NOT
myDataView.add_query(dataset_name='myDataset', version_name='training',
roi_query='label.keyword:\"Car\" AND NOT label.keyword:\"partly_occluded\"')
```
#### Querying multiple Datasets and versions
This example demonstrates an ROI query filtering for frames containing the ROI labels `car`, `truck`, or `bicycle`
from two versions of one Dataset, and one version of another Dataset.
```python
# Add queries:
# The 1st Dataset version
myDataView.add_query(dataset_name='dataset_1',
version_name='version_1',
roi_query='label.keyword:\"car\" OR label.keyword:\"truck\" OR '
'label.keyword:\"bicycle\"')
# The 1st Dataset, but a different version
myDataView.add_query(dataset_name='dataset_1',
version_name='version_2',
roi_query='label.keyword:\"car\" OR label.keyword:\"truck\" OR '
'label.keyword:\"bicycle\"')
# A 2nd Dataset (version)
myDataView.add_query(dataset_name='dataset_2',
version_name='some_version',
roi_query='label.keyword:\"car\" OR label.keyword:\"truck\" OR '
'label.keyword:\"bicycle\"')
```
#### Frame queries
Use frame queries to filter frames by ROI labels and / or frame metadata key-value pairs that a frame must include or
exclude for the DataView to return the frame.
**Frame queries** match frame meta key-value pairs, ROI labels, or both.
They use the same logical OR, AND, NOT AND matching as ROI queries.
This example demonstrates a frame query filtering for frames containing the meta key `city` value of `bremen`.
```python
# Add a frame query for frames with the meta key "city" value of "bremen"
myDataView.add_query(dataset_name='myDataset',
version_name='version',
frame_query='meta.city:"bremen"')
```
### Controlling query iteration
Use [`DataView.set_iteration_parameters`](google.com) to manage the order, number, timing, and reproducibility of frames
for training.
#### Iterate frames infinitely
This example demonstrates creating a Dataview and setting its parameters to iterate infinitely until the script is
manually terminated.
```python
# Create a Dataview object for an iterator that returns frames
myDataView = DataView()
# Set Iteration Parameters (overrides parameters in constructing the DataView object
myDataView.set_iteration_parameters(order=IterationOrder.random, infinite=True)
```
#### Iterate all frames matching the query
This example demonstrates creating a DataView and setting its parameters to iterate and return all frames matching a query.
```python
# Create a Dataview object for an iterator for frames
myDataView = DataView(iteration_order=IterationOrder.random, iteration_infinite=True)
# Set Iteration Parameters (overrides parameters in constructing the DataView object
myDataView.set_iteration_parameters(
order=IterationOrder.random, infinite=False)
# Add a query for a Dataset version
myDataView.add_query(dataset_name='myDataset',
version_name='myVersion', roi_query='cat')
```
#### Iterate a maximum number of frames
This example demonstrates creating a DataView and setting its parameters to iterate a specific number of frames. If the
Dataset version contains fewer than that number of frames matching the query, then fewer are returned by the iterator.
```python
# Create a Dataview object for an iterator for frames
myDataView = DataView(iteration_order=IterationOrder.random, iteration_infinite=True)
# Set Iteration Parameters (overrides parameters in constructing the DataView object
myDataView.set_iteration_parameters(
order=IterationOrder.random, infinite=False,
maximum_number_of_frames=5000)
```
### Debiasing input data
Debias input data using the [DataView.add_query](google.com) method's [weight](google.com) argument to add weights. This
is the same [DataView.add_query](google.com) that can be used to specify Dataset versions, and ROI queries and frame queries.
This example adjusts an imbalance in the input data to improve training for `Car` ROIs that are also `largely occluded`
(obstructed). For every frame containing at least one ROI labeled `Car`, approximately five frames containing at least
one ROI labeled with both `Car` and `largely_occluded` will be input.
```python
myDataView = DataView(iteration_order=IterationOrder.random, iteration_infinite=True)
myDataView.add_query(dataset_name='myDataset', version_name='training',
roi_query='Car', weight = 1)
myDataView.add_query(dataset_name='myDataset', version_name='training',
roi_query='label.keyword:\"Car\" AND label.keyword:\"largely_occluded\"', weight = 5)
```
### Mapping ROI Labels
ROI label translation (label mapping) enables combining labels for training, combining disparate datasets, and hiding
certain labels for training.
This example demonstrates consolidating two disparate Datasets. Two Dataset versions use `car` (lower case "c"), but a
third uses `Car` (upper case "C").
The example maps `Car` (upper case "C") to `car` (lower case "c").
```python
# Create a Dataview object for an iterator that randomly returns frames according to queries
myDataView = DataView(iteration_order=IterationOrder.random, iteration_infinite=True)
# The 1st Dataset (version) - "car" with lowercase "c"
myDataView.add_query(dataset_name='myDataset', version_name='myVersion' roi_query='car')
# The 2nd Dataset (version) - "car" with lowercase "c"
myDataView.add_query(dataset_name='dataset_2', version_name='aVersion',
roi_query='car')
# A 3rd Dataset (version) - "Car" with uppercase "C"
myDataView.add_query(dataset_name='dataset_3', version_name='training',
roi_query='Car')
# Use a mapping rule to translate "Car" (uppercase) to "car" (lowercase)
myDataView.add_mapping_rule(dataset_name='dataset_3',
version_name='training',
from_labels=['Car'],
to_label='car')
```
### Setting Label Enumeration Values
Set label enumeration values to maintain data conformity across multiple codebases and datasets.
It is important to set enumeration values for all labels of importance.
The default value for labels that are not assigned values is `-1`.
To assign enumeration values for labels use the [`DataView.set_labels`](google.com) method, set a mapping of a label
(string) to an integer for ROI labels in a Dataview object.
If certain ROI labels are [mapped](#mapping-roi-labels) from certain labels **to** other labels,
then use the labels you map **to** when setting enumeration values.
For example, if the labels `truck`, `van`, and `car` are mapped **to** `vehicle`, then set enumeration for `vehicle`.
```python
# Create a Dataview object for an iterator that randomly returns frames according to queries
myDataView = DataView(iteration_order=IterationOrder.random, iteration_infinite=True)
# Add a query for a Dataset version
myDataView.add_query(dataset_name='myDataset', version_name='myVersion',
roi_query='cat')
myDataView.add_query(dataset_name='myDataset', version_name='myVersion',
roi_query='dog')
myDataView.add_query(dataset_name='myDataset', version_name='myVersion',
roi_query='bird')
myDataView.add_query(dataset_name='myDataset', version_name='myVersion',
roi_query='sheep')
myDataView.add_query(dataset_name='myDataset', version_name='myVersion',
roi_query='cow')
# Set the enumeration label values
myDataView.set_labels({"cat": 1, "dog": 2, "bird": 3, "sheep": 4, "cow": 5, "ignore": -1,})
```

118
docs/hyperdatasets/frame_groups.md Normal file
View File

@@ -0,0 +1,118 @@
---
title: FrameGroups
---
The ClearML Enterprise provides **FrameGroup**s as an easy-to-use type of frame supporting multiple sources.
Add a list of SingleFrames to a FrameGroup, and then register FrameGroups in a Dataset version.
[View and edit](webapp/webapp_datasets_frames.md) FrameGroups and the SingleFrames they contain
in the ClearML Enterprise WebApp (UI).
A SingleFrame is composed of metadata for raw data that is harvested at a specific point in time for a
specific spatial area, as well as additional metadata such as annotations and masks. Therefore, a **FrameGroup** combines
more than one set of raw data and its additional metadata for the same point in time.
For example, use FrameGroups for the following:
* Multiple cameras on an autonomous car - A FrameGroup composed of SingleFrames for each camera.
* Multiple sensors on a machine detecting defects - A FrameGroup composed of SingleFrames for each sensor.
## Usage
### Creating a FrameGroup
A FrameGroup is like a dictionary of SingleFrames. Instantiate a FrameGroup and a SingleFrame. Then add the SingleFrame
object into the FrameGroup, with the key being the name of the SingleFrame, and the value being the SingleFrame object.
```python
from allegroai import FrameGroup, SingleFrame
# Create a FrameGroup object
frame_group = FrameGroup()
# Create a SingleFrame
frame = SingleFrame(source='https://allegro-datasets.s3.amazonaws.com/tutorials/000012.jpg',
width=512, height=512, preview_uri='https://allegro-datasets.s3.amazonaws.com/tutorials/000012.jpg')
# Add the first SingleFrame to the FrameGroup.
frame_group['FrameOne'] = frame
```
### Adding FrameGroups to a Dataset Version
To add FrameGroups to a Dataset Version:
1. Create a FrameGroup object
1. Add SingleFrames to the FrameGroup, where the key of each SingleFrame in the FrameGroup is the SingleFrame's name
1. Append the FrameGroup object to a list of frames
1. Add that list to a DatasetVersion.
```python
# Create a FrameGroup object
frame_group = FrameGroup()
# Create SingleFrame
single_frame = SingleFrame(source='https://allegro-datasets.s3.amazonaws.com/tutorials/000012.jpg')
# Add the first SingleFrame to the FrameGroup.
frame_group['FrameOne'] = single_frame
# The DatasetVersion.add_frames requires a list of frames.
frames = []
frames.append(frame_group)
# Add the FrameGroup to the version
myVersion.add_frames(frames)
```
### Accessing a FrameGroup
To access a FrameGroup, use the [DatasetVersion.get_single_frame](google.com) method, just like when
[accessing a SingleFrame](single_frames.md#accessing-singleframes).
```python
# Get the FrameGroup
frame_group = DatasetVersion.get_single_frame(frame_id='f3ed0e09bf23fc947f426a0d254c652c',
dataset_name='MyDataset',
version_name='FrameGroup')
```
### Updating FrameGroups
Updating FrameGroups is similar to [updating SingleFrames](single_frames.md#updating-singleframes), except that each
SingleFrame needs to be referenced using its name as the key in the FrameGroup.
```python
frames = []
# Get the FrameGroup
frame_group = DatasetVersion.get_single_frame(frame_id='f3ed0e09bf23fc947f426a0d254c652c',
dataset_name='MyDataset', version_name='FrameGroup')
# Add metadata by referencing the name of the SingleFrame in the FrameGroup
frame_group['FrameOne'].metadata['new_key'] = 'new_value'
# Update change to the FrameGroup
frames.append(frame_group)
myVersion.update_frames(frames)
```
### Deleting frames
To delete a FrameGroup, use the [DatasetVersion.delete_frames](google.com) method, just like when deleting a
SingleFrame, except that a FrameGroup is being referenced.
```python
frames = []
# Get the FrameGroup
frame_group = DatasetVersion.get_single_frame(frame_id='f3ed0e09bf23fc947f426a0d254c652c',
dataset_name='MyDataset', version_name='FrameGroup')
# Delete the FrameGroup
frames.append(frame_group)
myVersion.delete_frames(frames)
```

14
docs/hyperdatasets/frames.md Normal file
View File

@@ -0,0 +1,14 @@
---
title: Frames Overview
---
The concept of a **Frame** represents the basic building block of data in ClearML Enterprise.
Two types of frames are supported:
* [SingleFrames](single_frames.md) - A frame with one source. For example, one image.
* [FrameGroups](frame_groups.md) - A frame with multiple sources. For example, multiple images.
**SingleFrames** and **FrameGroups** contain data sources, metadata, and other data. A Frame can be added to [Datasets](dataset.md)
and then modified or removed. [Versions](dataset.md#dataset-versioning) of the Datasets can be created, which enables
documenting changes and reproducing data for experiments.

241
docs/hyperdatasets/masks.md Normal file
View File

@@ -0,0 +1,241 @@
---
title: Masks
---
When applicable, [`sources`](sources.md) contains `masks`, a list of dictionaries used to connect a special type of
source data to the ClearML Enterprise platform. That source data is a **mask**.
Masks are used in deep learning for semantic segmentation.
Masks correspond to raw data where the objects to be detected are marked with colors in the masks. The colors
are RGB values and represent the objects, which are labeled for segmentation.
In frames used for semantic segmentation, the metadata connecting the mask files / images to the ClearML Enterprise platform,
and the RGB values and labels used for segmentation are separate. They are contained in two different dictionaries of
a SingleFrame:
* **`masks`** (plural) is in [`sources`](sources.md) and contains the mask files / images `URI` (in addition to other keys
and values).
* **`mask`** (singular) is in the `rois` array of a Frame.
Each `rois` dictionary contains:
* RGB values and labels of a **mask** (in addition to other keys and values)
* Metadata and data for the labeled area of an image
See [Example 1](#example-1), which shows `masks` in `sources`, `mask` in `rois`, and the key-value pairs used to relate
a mask to its source in a frame.
## Masks structure
The chart below explains the keys and values of the `masks` dictionary (in the [`sources`](sources.md)
section of a Frame).
|Key|Value Description|
|---|----|
|`id`|**Type**: integer. <ul><li> The ID is used to relate this mask data source to the `mask` dictionary containing the label and RGB value for the mask.</li><li> See the `mask` key in `rois`.</li></ul>|
|`content_type`| **Type**: string. <ul><li> Type of mask data. For example, image / png or video / mp4.</li></ul>|
|`timestamp`|**Type**: integer. <ul><li>For images from a video, indicates the absolute position of the frame from the source (video) </li><li> For still images, set this to 0 (for example, video from a camera on a car, at 30 frames per second, would have a timestamp of 0 for the first frame, and 33 for the second frame).</li></ul>|
|`uri`|**Type**: string. <ul><li> URI of the mask file / image.</li></ul>|
## Examples
### Example 1
This example demonstrates an original image, its masks, and its frame containing
the `sources` and ROI metadata.
<details className="cml-expansion-panel info">
<summary className="cml-expansion-panel-summary">Example 1: View the frame</summary>
<div className="cml-expansion-panel-content">
This frame contains the `masks` list of dictionaries in `sources`,
and the `rois` array, as well as several top-level key-value pairs.
```json
{
"timestamp": 1234567889,
"context_id": "car_1",
"meta": {
"velocity": "60"
},
"sources": [
{
"id": "front",
"content_type": "video/mp4",
"width": 800,
"height": 600,
"uri": "https://s3.amazonaws.com/my_cars/car_1/front.mp4",
"timestamp": 1234567889,
"meta" :{
"angle":45,
"fov":129
},
"masks": [
{
"id": "seg",
"content_type": "video/mp4",
"uri": "https://s3.amazonaws.com/seg_masks/car_1/front_seg.mp4",
"timestamp": 123456789
},
{
"id": "seg_instance",
"content_type": "video/mp4",
"uri": "https://s3.amazonaws.com/seg_masks/car_1/front_instance_seg.mp4",
"timestamp": 123456789
}
]
}
],
"rois": [
{
"sources":["front"],
"label": ["seg"],
"mask": {
"id": "car",
"value": [210,210,120]
}
},
{
"sources":["front"],
"label": ["seg"],
"mask": {
"id": "person",
"value": [147,44,209]
}
},
{
"sources":["front"],
"label": ["seg"],
"mask": {
"id": "road",
"value": [197,135,146]
}
},
{
"sources":["front"],
"label": ["seg"],
"mask": {
"id": "street",
"value": [135,198,145]
}
},
{
"sources":["front"],
"label": ["seg"],
"mask": {
"id": "building",
"value": [72,191,65]
}
}
]
}
```
</div>
</details>
<br/>
* In `sources`:
* The source ID is `front`.
* In the `masks` dictionary, the source contains mask sources with IDs of `seg` and `seg_instance`.
* In `rois`:
* Each ROI source is `front`, relating the ROI to its original source image.
* Each ROI has a label of `seg`, indicating segmentation.
* Each `mask` has an `id` (`car`, `person`, `road`, `street`, and `building`) and a unique RGB `value`
(color-coding).
<details className="cml-expansion-panel screenshot">
<summary className="cml-expansion-panel-summary">Example image and masks</summary>
<div className="cml-expansion-panel-content">
Original Image
![image](../img/hyperdatasets/concepts_masks_image_only.png)
Mask image
![image](../img/hyperdatasets/concepts_masks.png)
</div>
</details>
<br/>
### Example 2
This example shows two masks for video from a camera. The masks label cars and the road.
<details className="cml-expansion-panel info">
<summary className="cml-expansion-panel-summary">Example 2: View the frame</summary>
<div className="cml-expansion-panel-content">
```json
"sources": [
{
"id": "front",
"content_type": "video/mp4",
"width": 800,
"height": 600,
"uri": "https://s3.amazonaws.com/my_cars/car_1/front.mp4",
"timestamp": 1234567889,
"meta" :{
"angle":45,
"fov":129
},
"masks": [
{
"id": "car",
"content_type": "video/mp4",
"uri": "https://s3.amazonaws.com/seg_masks/car_1/front_seg.mp4",
"timestamp": 123456789
},
{
"id": "road",
"content_type": "video/mp4",
"uri": "https://s3.amazonaws.com/seg_masks/car_1/front_instance_seg.mp4",
"timestamp": 123456789
}
]
}
],
"rois": [
{
"sources":["front"],
"label": ["right_lane"],
"mask": {
"id": "car",
"value": [210,210,120]
}
},
{
"sources":["front"],
"label": ["right_lane"],
"mask": {
"id": "road",
"value": [197,135,146]
}
}
```
</div>
</details>
<br/>
* In `sources`:
* The source ID is `front`.
* The source contains mask sources with IDs of `car` and `road`.
* In `rois`:
* Each ROI source is `front` relating the ROI to its original source image.
* Each ROI has a label of `right_lane` indicating the ROI object.
* Each `mask` has an `id` (`car`, `person`) and a unique RGB `value` (color-coding).

34
docs/hyperdatasets/overview.md Normal file
View File

@@ -0,0 +1,34 @@
---
title: Hyper Datasets
---
ClearML's Hyper Datasets are an MLOps-oriented abstraction of your data, which facilitates traceable, reproducible model development
through parametrized data access and meta-data version control.
The basic premise is that a user-formed query is a full representation of the dataset used by the ML/DL process.
ClearML Enterprise's hyperdatasets supports rapid prototyping, creating new opportunities such as:
* Hyperparameter optimization of the data itself
* QA/QC pipelining
* CD/CT (continuous training) during deployment
* Enabling complex applications like collaborative (federated) learning.
## Hyperdataset Components
A hyperdataset is composed of the following components:
* [Frames](frames.md)
* [SingleFrames](single_frames.md)
* [FrameGroups](frame_groups.md)
* [Datasets and Dataset Versions](dataset.md)
* [Dataviews](dataviews.md)
These components interact in a way that enables revising data and tracking and accessing all of its version.
Frames are the basics units of data in ClearML Enterprise. SingleFrames and FrameGroups make up a Dataset version.
Dataset versions can be created, modified, and removed. The different version are recorded and available,
so experiments and their data are reproducible and traceable.
Lastly, Dataviews manage views of the dataset with queries, so the input data to an experiment can be defined from a
subset of a Dataset or combinations of Datasets.

105
docs/hyperdatasets/previews.md Normal file
View File

@@ -0,0 +1,105 @@
---
title: Previews
---
A `preview` is a dictionary containing metadata for optional thumbnail images that can be used in the ClearML Enterprise
WebApp (UI) to view selected images in a Dataset. `previews` includes the `uri` of the thumbnail image.
Previews are not mandatory. Their primary use is to view images with formats that cannot be displayed in a web browser
(such as TIFF and 3D formats).
## Example
The following is an example of preview metadata.
```json
"preview": {
"content_type": "image/jpg",
"uri": "https://s3.amazonaws.com/my_previews/car_1/front_preview.jpg",
"timestamp": 0
}
```
<details className="cml-expansion-panel info">
<summary className="cml-expansion-panel-summary">View an entire frame containing a preview</summary>
<div className="cml-expansion-panel-content">
```json
{
"timestamp": 1234567889,
"context_id": "car_1",
"meta": {
"velocity": "60"
},
"sources": [
{
"id": "front",
"content_type": "video/mp4",
"width": 800,
"height": 600,
"uri": "https://s3.amazonaws.com/my_cars/car_1/front.mp4",
"timestamp": 1234567889,
"meta" :{
"angle":45,
"fov":129
},
"preview": {
"content_type": "image/jpg",
"uri": "https://s3.amazonaws.com/my_previews/car_1/front_preview.jpg",
"timestamp": 0
},
"masks": [
{
"id": "seg",
"content_type": "video/mp4",
"uri": "https://s3.amazonaws.com/seg_masks/car_1/front_seg.mp4",
"timestamp": 1234567889
},
{
"id": "instances_seg",
"content_type": "video/mp4",
"uri": "https://s3.amazonaws.com/seg_masks/car_1/front_instance_seg.mp4",
"timestamp": 1234567889
}
]
},
{
"id": "rear",
"uri": "https://s3.amazonaws.com/my_cars/car_1/rear.mp4",
"content_type": "video/mp4",
"timestamp": 1234567889
}
],
"rois": [
{
"sources":["front"],
"label": ["right_lane"],
"mask": {
"id": "seg",
"value": [-1, 1, 255]
}
},
{
"sources": ["front"],
"label": ["bike"],
"poly":[30, 50, 50,50, 100,50, 100,100],
"meta": {
"velocity": 5.4
}
},
{
"sources": ["front", "rear"],
"label": ["car"],
"poly":[30, 50, 50,50, 100,50, 100,100]
}
]
}
```
</div>
</details>
<br/>
Here's an example of Previews in the ClearML Enterprise WebApp (UI). Each thumbnail is a Preview.
![image](../img/hyperdatasets/web-app/previews.png)

311
docs/hyperdatasets/single_frames.md Normal file
View File

@@ -0,0 +1,311 @@
---
title: SingleFrames
---
A `SingleFrame` contains metadata pointing to raw data, and other metadata and data, which supports experimentation and
**ClearML Enterprise**'s Git-like Dataset versioning.
## Frame Components
A `SingleFrame` contains the following components:
* [Sources](#sources)
* [Annotations](#annotation)
* [Masks](#masks)
* [Previews](#previews)
* [Metadata](#metadata)
### Sources
Every `SingleFrame` includes a [`sources`](sources.md) dictionary, which contains attributes of the raw data, including:
* URI pointing to the source data (image or video)
* Sources for masks used in semantic segmentation
* Image previews, which are thumbnails used in the WebApp (UI).
For more information, see [Sources](sources.md).
### Annotation
Each `SingleFrame` contains a list of dictionaries, where each dictionary includes information about a specific annotation.
Two types of annotations are supported:
* **FrameGroup objects** - label for Regions of Interest (ROIs)
* **FrameGroup labels** - labels for the entire frame
For more information, see [Annotations](annotations.md).
### Masks
A `SingleFrame` includes a URI link to a mask file if applicable. Masks correspond to raw data where the objects to be
detected in raw data are marked with colors in the masks.
For more information, see [Masks](masks.md).
### Previews
`previews` is a dictionary containing metadata for optional thumbnail images that can be used in the ClearML Enterprise WebApp (UI)
to view selected images in a Dataset. `previews` includes the `uri` of the thumbnail image.
For more information, see [Previews](previews.md).
### Metadata
`metadata` is a dictionary with general information about the `SingleFrame`.
For more information, see [Custom Metadata](custom_metadata.md).
## Frame structure
The panel below describes the details contained within a `frame`:
<details className="cml-expansion-panel info">
<summary className="cml-expansion-panel-summary">Frame Structure</summary>
<div className="cml-expansion-panel-content">
* `id` (*string*) - The unique ID of this frame.
* `blob` (*string*) - Raw data.
* `context_id` (*string*) - Source URL.
* `dataset` (*dict*) - The Dataset and version containing the frame.
* `id` - ID of the Dataset.
* `version` - ID of the version.
* `meta` (*dict*) - Frame custom metadata. Any custom key-value pairs (`sources` and `rois` can also contain a meta
dictionary for custom key-value pairs associated with individual sources and rois). See [Custom Metadata](custom_metadata.md).
* `num_frames`
* `rois` (*[dict]*) - Metadata for annotations, which can be Regions of Interest (ROIs) related to this frame's source data,
or frame labels applied to the entire frame (not a region). ROIs are labeled areas bounded by polygons or labeled RGB
values used for object detection and segmentation. See [Annotations](annotations.md).
* `id` - ID of the ROI.
* `confidence` (*float*) - Confidence level of the ROI label (between 0 and 1.0).
* `labels` (*[string]*)
* For [FrameGroup objects](#frame-objects) (Regions of Interest), these are the labels applied to the ROI.
* For [FrameGroup labels](#frame-labels), this is the label applied to the entire frame.
* `mask` (*dict*) - RGB value of the mask applied to the ROI, if a mask is used (for example, for semantic segmentation).
The ID points to the source of the mask.
* `id` - ID of the mask dictionary in `sources`.
* `value` - RGB value of the mask.
* `poly` (*[int]*) - Bounding area vertices.
* `sources` (*[string]*) - The `id` in the `sources` dictionary which relates an annotation to its raw data source.
* `sources` (*[dict]*) - Sources of the raw data in this frame. For a SingleFrame this is one source. For a FrameGroup,
this is multiple sources. See [Sources](sources.md).
* `id` - ID of the source.
* `uri` - URI of the raw data.
* `width` - Width of the image or video.
* `height` - Height of the image or video.
* `mask` - Sources of masks used in the `rois`.
* `id` - ID of the mask source. This relates a mask source to an ROI.
* `content_type` - The type of mask source. For example, `image/jpeg`.
* `uri` - URI of the mask source.
* `timestamp`
* `preview` - URI of the thumbnail preview image used in the ClearML Enterprise WebApp (UI)
* `timestamp` - For images from video, a timestamp that indicates the absolute position of this frame from the source (video).
For example, if video from a camera on a car is taken at 30 frames per second, it would have a timestamp of 0 for
the first frame, and 33 for the second frame. For still images, set this to 0.
* `saved_in_version` - The version in which the frame is saved.
* `saved` - The epoch time that the frame was saved.
* `timestamp` - For images from video, a timestamp that indicates the absolute position of this frame from the source (video).
</div>
</details>
<br/>
## WebApp
A frame that has been connected to the **ClearML Enterprise** platform is available to view and analyze on the
WebApp (UI).
When viewing a frame on the WebApp, all the information associated with it can be viewed, including its frame labels and
object annotations, its metadata, and other details.
<details className="cml-expansion-panel screenshot">
<summary className="cml-expansion-panel-summary">SingleFrame in the WebApp frame viewer</summary>
<div className="cml-expansion-panel-content">
This image shows a SingleFrame in the ClearML Enterprise WebApp (UI) [frame viewer](webapp/webapp_datasets_frames.md#frame-viewer).
![image](../img/hyperdatasets/frame_overview_01.png)
</div>
</details>
<br/>
<details className="cml-expansion-panel info">
<summary className="cml-expansion-panel-summary">SingleFrame details represented in the WebApp</summary>
<div className="cml-expansion-panel-content">
id : "287024"
timestamp : 0
rois : Array[2] [
{
"label":["tennis racket"],
"poly":[174,189,149,152,117,107,91,72,68,45,57,33,53,30,49,32,48,34,46,35,46,37,84,92,112,128,143,166,166,191,170,203,178,196,179,194,-999999999,194,238,204,250,212,250,221,250,223,249,206,230,205,230],
"confidence":1,
"sources":["default"],
"id":"f9fc8629d99b4e65aecacedd32ac356e"
},
{
"label":["person"],
"poly":[158,365,161,358,165,335,170,329,171,321,171,307,173,299,172,292,171,277,171,269,170,260,170,254,171,237,177,225,172,218,167,215,164,207,167,205,171,199,174,196,183,193,188,192,192,192,202,199,207,200,232,187,238,182,240,178,244,172,245,169,245,166,241,163,235,164,233,159,239,150,240,146,240,134,237,137,231,141,222,142,217,136,216,130,215,123,215,116,224,102,229,99,233,96,245,108,256,92,272,84,292,87,309,92,319,101,328,121,329,134,327,137,325,140,331,152,327,155,323,159,324,167,320,174,319,183,327,196,329,232,328,243,323,248,315,254,316,262,314,269,314,280,317,302,313,326,311,330,301,351,299,361,288,386,274,410,269,417,260,427,256,431,249,439,244,448,247,468,249,486,247,491,245,493,243,509,242,524,241,532,237,557,232,584,233,608,233,618,228,640,172,640,169,640,176,621,174,604,147,603,146,609,151,622,144,634,138,638,128,640,49,640,0,640,0,636,0,631,0,630,0,629,37,608,55,599,66,594,74,594,84,593,91,593,99,571,110,534,114,523,117,498,116,474,113,467,113,459,113,433,113,427,118,412,137,391,143,390,147,386,157,378,157,370],
"confidence":1,
"sources":["default"],
"id":"eda8c727fea24c49b6438e5e17c0a846"
}
]
sources : Array[1] [
{
"id":"default",
"uri":"https://s3.amazonaws.com/allegro-datasets/coco/train2017/000000287024.jpg",
"content_type":"image/jpeg",
"width":427,
"height":640,
"timestamp":0
}
]
dataset : Object
{
"id":"f7edb3399164460d82316fa5ab549d5b",
"version":"6ad8b10c668e419f9dd40422f667592c"
}
context_id : https://s3.amazonaws.com/allegro-datasets/coco/train2017/000000287024.jpg
saved : 1598982880693
saved_in_version : "6ad8b10c668e419f9dd40422f667592c"
num_frames : 1
</div>
</details>
<br/>
For more information about using Frames in the WebApp, see [Working with Frames](webapp/webapp_datasets_frames.md).
## Usage
### Creating a SingleFrame
To create a `SingleFrame`, instantiate a `SingleFrame` class and populate it with:
* The URI link to the source file of the data frame
* A preview URI that is accessible by browser, so you will be able to visualize the data frame in the web UI
```python
from allegroai import SingleFrame
frame = SingleFrame(
source='/home/user/woof_meow.jpg',
width=None,
height=None,
preview_uri='https://storage.googleapis.com/kaggle-competitions/kaggle/3362/media/woof_meow.jpg',
metadata=None,
annotations=None,
mask_source=None,
)
```
There are also options to populate the instance with:
* Dimensions - `width` and `height`
* General information about the frame - `metadata`
* A dictionary of annotation objects - `annotations`
* A URI link to a mask file for the frame - `mask_source`
For more information, see [SingleFrame](google.com).
### Adding SingleFrames to a Dataset Version
Use the [`DatasetVersion.add_frames`](google.com) method to add SingleFrames to a [Dataset version](dataset.md#dataset-versioning)
(see [Creating snapshots](dataset.md#creating-snapshots) or [Creating child versions](dataset.md#creating-child-versions)).
```python
from allegroai import DatasetVersion, SingleFrame
# a frames list is required for adding frames
frames = []
# create a frame
frame = SingleFrame(
source='https://allegro-datasets.s3.amazonaws.com/tutorials/000012.jpg',
width=512,
height=512,
preview_uri='https://allegro-datasets.s3.amazonaws.com/tutorials/000012.jpg',
metadata={'alive':'yes'},
)
frames.append(frame)
# add frame to the Dataset version
myDatasetversion.add_frames(frames)
```
### Accessing SingleFrames
To access a SingleFrame, use the [DatasetVersion.get_single_frame](google.com) method.
```python
from allegroai import DatasetVersion
frame = DatasetVersion.get_single_frame(frame_id='dcd81d094ab44e37875c13f2014530ae',
dataset_name='MyDataset', # OR dataset_id='80ccb3ae11a74b91b1c6f25f98539039'
version_name='SingleFrame' # OR version_id='b07b626e3b6f4be7be170a2f39e14bfb'
)
```
To access a SingleFrame, the following must be specified:
* `frame_id`, which can be found in the WebApp, in the frame's **FRAMEGROUP DETAILS**
* The frame's dataset - either with `dataset_name` or `dataset_id`
* The dataset version - either with `version_id` or `version_name`
### Updating SingleFrames
To update a SingleFrame:
* Access the SingleFrame by calling the [DatasetVersion.get_single_frame](google.com) method,
* Make changes to the frame
* Update the frame in a DatasetVersion using the [DatasetVersion.update_frames](google.com) method.
```python
frames = []
# get the SingleFrame
frame = DatasetVersion.get_single_frame(frame_id='dcd81d094ab44e37875c13f2014530ae',
dataset_name='MyDataset',
version_name='SingleFrame')
# make changes to the frame
## add a new annotation
frame.add_annotation(poly2d_xy=[154, 343, 209, 343, 209, 423, 154, 423],
labels=['tire'], metadata={'alive': 'no'},
confidence=0.5)
## add metadata
frame.meta['road_hazard'] = 'yes'
# update the SingeFrame
frames.append(frame)
myDatasetVersion.update_frames(frames)
```
### Deleting frames
To delete a SingleFrame, use the [DatasetVersion.delete_frames](google.com) method.
```python
frames = []
# get the SingleFrame
frame = DatasetVersion.get_single_frame(frame_id='f3ed0e09bf23fc947f426a0d254c652c',
dataset_name='MyDataset', version_name='FrameGroup')
# delete the SingleFrame
frames.append(frame)
myDatasetVersion.delete_frames(frames)
```

207
docs/hyperdatasets/sources.md Normal file
View File

@@ -0,0 +1,207 @@
---
title: Sources
---
Each frame contains `sources`, a list of dictionaries containing:
* Attributes of the source data (image raw data)
* A `URI` pointing to the source data (image or video)
* Sources for [masks](masks.md) used in semantic segmentation
* Image [previews](previews.md), which are thumbnails used in the ClearML Enterprise WebApp (UI).
`sources` does not contain:
* `rois` even though ROIs are directly associated with the images and `masks` in `sources`
* ROI metadata, because ROIs can be used over multiple frames.
Instead, frames contain a top-level `rois` array, which is a list of ROI dictionaries, where each dictionary contains a
list of source IDs. Those IDs connect `sources` to ROIs.
## Examples
The examples below demonstrate the `sources` section of a Frame for different types of content.
### Example 1: Video sources
This example demonstrates `sources` for video.
<details className="cml-expansion-panel info">
<summary className="cml-expansion-panel-summary">Example 1</summary>
<div className="cml-expansion-panel-content">
```json
/* video from one of four cameras on car */
"sources": [
{
"id": "front",
"content_type": "video/mp4",
"width": 800,
"height": 600,
"uri": "https://s3.amazonaws.com/my_cars/car_1/front.mp4",
"timestamp": 1234567889,
"meta" :{
"angle":45,
"fov":129
},
},
{
"id": "rear",
"uri": "https://s3.amazonaws.com/my_cars/car_1/rear.mp4",
"content_type": "video/mp4",
"timestamp": 1234567889
}
```
</div>
</details>
<br/>
The `sources` example above details a video from a car that has two cameras. One camera
is the source with the ID `front` and the other is the source with the ID `rear`.
`sources` includes the following information about the Frame:
* `content_type` - The video is an `mp4` file
* `width` and `height` - Each frame in the video is `800` pixels by `600` pixels,
* `uri` - The raw data is located in `s3.amazonaws.com/my_cars/car_1/front.mp4` and `s3.amazonaws.com/my_cars/car_1/rear.mp4`
(the front and rear camera, respectively)
* `timestamp` - This indicates the absolute position of the frame in the video
* `meta` - Additional metadata is included for the angle of the camera (`angle`) and its field of vision (`fov`).
:::note
Sources includes a variety of content types. This example shows mp4 video.
:::
### Example 2: Images sources
This example demonstrates `sources` images.
<details className="cml-expansion-panel info">
<summary className="cml-expansion-panel-summary">Example 2</summary>
<div className="cml-expansion-panel-content">
```json
/* camera images */
"sources": [
{
"id": "default",
"content_type": "png",
"width": 800,
"height": 600,
"uri": "https://s3.amazonaws.com/my_images/imag1000.png",
"timestamp": 0,
}
```
</div>
</details>
<br/>
The `sources` of this frame contains the following information:
* `content_type` - This frame contains an image in `png` format.
* `width` and `height` - It is `800` px by `600` px,
* `uri` - The raw data is located in `https://s3.amazonaws.com/my_images/imag1000.png`
* `timestamp` is 0 (timestamps are used for video).
### Example 3: Sources and regions of interest
This example demonstrates `sources` for video, `masks`, and `preview`.
<details className="cml-expansion-panel info">
<summary className="cml-expansion-panel-summary">Example 3</summary>
<div className="cml-expansion-panel-content">
```json
{
"timestamp": 1234567889,
"context_id": "car_1",
"meta": {
"velocity": "60"
},
"sources": [
{
"id": "front",
"content_type": "video/mp4",
"width": 800,
"height": 600,
"uri": "https://s3.amazonaws.com/my_cars/car_1/front.mp4",
"timestamp": 1234567889,
"meta" :{
"angle":45,
"fov":129
},
"preview": {
"content_type": "image/jpg",
"uri": "https://s3.amazonaws.com/my_previews/car_1/front_preview.jpg",
"timestamp": 0
},
"masks": [
{
"id": "seg",
"content_type": "video/mp4",
"uri": "https://s3.amazonaws.com/seg_masks/car_1/front_seg.mp4",
"timestamp": 1234567889
},
{
"id": "instances_seg",
"content_type": "video/mp4",
"uri": "https://s3.amazonaws.com/seg_masks/car_1/front_instance_seg.mp4",
"timestamp": 1234567889
}
]
},
{
"id": "rear",
"uri": "https://s3.amazonaws.com/my_cars/car_1/rear.mp4",
"content_type": "video/mp4",
"timestamp": 1234567889
}
],
"rois": [
{
"sources":["front"],
"label": ["right_lane"],
"mask": {
"id": "seg",
"value": [-1, 1, 255]
}
},
{
"sources": ["front"],
"label": ["bike"],
"poly":[30, 50, 50,50, 100,50, 100,100],
"meta": {
"velocity": 5.4
}
},
{
"sources": ["front", "rear"],
"label": ["car"],
"poly":[30, 50, 50,50, 100,50, 100,100]
}
]
}
```
</div>
</details>
<br/>
This frame shows the `masks` section in `sources`, and the top-level `rois` array.
In `sources`, the `masks` subsection contains the sources for the two masks associated with the raw data.
The raw mask data is located in:
* `https://s3.amazonaws.com/my_cars/car_1/front.mp4`
* `https://s3.amazonaws.com/my_previews/car_1/front_preview.jpg`
In `rois`, the `mask` section is associated with its `masks` source using the `id` key.
In this example:
* In the `rois` array, there is a region of interest that has a `mask` with the ID `seg` and an RGB
value
* The `masks` section in `sources` contains the location of each mask. The first dictionary of `masks`
details the mask with the ID `seg`. The ID connects it to the `seg` mask in `rois`
`sources` also contains the source of a preview. It is located in: `https://s3.amazonaws.com/my_previews/car_1/front_preview.jpg`.

31
docs/hyperdatasets/task.md Normal file
View File

@@ -0,0 +1,31 @@
---
title: Tasks
---
Hyper Datasets extend the **ClearML** [**Task**](../fundamentals/task.md) with [Dataviews](dataviews.md)
## Usage
Hyper datasets are supported through `allegroai` python package.
### Connecting Dataviews to a Task
Use [`Task.connect`](../references/sdk/task.md#connect) to connect a Dataview object to a Task:
```python
from allegroai import DataView
dataview = DataView()
task.connect(dataview)
```
### Accessing a Task's Dataviews
Use the [Task.get_dataviews](google.com) method to access the Dataviews that are connected to a Task.
```python
task.get_dataviews():
```
This returns a dictionary of Dataview objects and their names.

162
docs/hyperdatasets/webapp/webapp_annotator.md Normal file
View File

@@ -0,0 +1,162 @@
---
title: Annotation Tasks
---
Use the Annotations page to access and manage annotation Tasks.
Use annotation tasks to efficiently organize the annotation of frames in Dataset versions and manage the work of annotators
(see [Annotating Images and Videos](#annotating-images-and-video)).
## Managing Annotation Tasks
### Creating Annotation Tasks
![image](../../img/hyperdatasets/annotation_task_01.png)
**To create an annotation task:**
1. On the Annotator page, click **+ ADD NEW ANNOTATION**.
1. Enter a name for your new annotation task.
1. Choose a Dataset version to annotate. If the selected Dataset version's status is *Published*, then creating this
annotation task also creates a child version of the selected version. The new child version's status is *Draft*, and
its name is the same as the annotation task.
1. Set the filters for the frames this annotation task presents to the annotator.
* In the **SET FILTERS** list, choose either:
* **All Frames** - Include all frames in this task.
* **Empty Frames** - Include only frames without any annotations in this task.
* **By Label** - Include only frames with specific labels, and optionally filter these frames by confidence level and
the number of instances. You can also click <img src="/static/icons/ico-code.svg" className="icon size-md space-sm" /> and then add a Lucene query for this ROI label filter.
1. Choose the iteration parameters specifying how frames in this version are presented to the annotator.
1. In **ITERATION**, in the **ORDER** list, choose either:
* **Sequential** - Frames are sorted by the frame top-level `context_id` (primary sort key) and `timestamp` (secondary sort key) metadata key values, and returned by the iterator in the sorted order.
* **Random** - Frames are randomly returned using the value of the `random_seed` argument. The random seed is maintained with the experiments. Therefore, the random order is reproducible if the experiment is rerun.
1. In **REPETITION**, choose either **Use Each Frame Once** or **Limit Frames**. If you select **Limit Frames**, then in **Use Max. Frames**, type the number of frames to annotate.
1. If iterating randomly, in **RANDOM SEED** type your seed or leave blank, and the ClearML Enterprise platform generates a seed for you.
1. If annotating video, then in **CLIP LENGTH (FOR VIDEO)**, type of the number of sequential frames per iteration to annotate.
1. Click **Create**.
### Completing annotation tasks
To mark an annotation task as **Completed**:
* In the annotation task card, click <img src="/static/icons/ico-bars-menu.svg" className="icon size-md space-sm" /> (menu) **>** **Complete** **>** **CONFIRM**.
### Deleting annotation tasks
To delete an annotation task:
* In the annotation task card, click <img src="/static/icons/ico-bars-menu.svg" className="icon size-md space-sm" /> (menu) **>** **Delete** **>** **CONFIRM**.
### Filtering annotation tasks
There are two option for filtering annotation tasks:
* Active / Completed Filter - Toggle to show annotation tasks that are either **Active** or **Completed**
* Dataset Filter - Use to view only the annotation tasks for a specific Dataset.
### Sorting annotation tasks
Sort the annotation tasks by either using **RECENT** or **NAME** from the drop-down menu on the top left of the page.
### Viewing annotation task information
To View the Dataset version, filters, and iteration information:
* In the annotation task card, click <img src="/static/icons/ico-bars-menu.svg" className="icon size-md space-sm" /> (menu) **>** **Info**
## Annotating Images and Video
Annotate images and video by labeling regions of interest in Dataset version frames. The frames presented for annotation
depend upon the settings in the annotation task (see [Creating Annotation Tasks](#creating-annotation-tasks)).
### Annotating Frames
**To annotate frames:**
1. On the Annotator page, click the annotation task card, or click <img src="/static/icons/ico-bars-menu.svg" className="icon size-md space-sm" /> (menu)
and then click **Annotate**.
1. See instructions below about annotating frames.
#### Add FrameGroup objects
1. Select an annotation mode and add the bounded area to the frame image.
* Rectangle mode - Click <img src="/static/icons/ico-rectangle-icon-purple.svg" className="icon size-md space-sm" /> and then click the image, drag and release.
* Polygon mode - Click <img src="/static/icons/ico-polygon-icon-purple.svg" className="icon size-md space-sm" /> and then click the image for the first vertex,
move to another vertex and click, continue until closing the last vertex.
* Key points mode - Click <img src="/static/icons/ico-keypoint-icon-purple.svg" className="icon size-md space-sm" /> and then click each key point.
1. In the new label area, choose or enter a label.
1. Optionally, add metadata.
1. Optionally, lock the annotation.
#### Add frame labels
1. In **FRAME LABEL**, click **+ Add new**.
1. In the new label area, choose or enter a label.
1. Optionally, add metadata.
1. Optionally, lock the annotation.
#### Copy / paste an annotations
1. Click the annotation or bounded area in the image or video clip.
1. Optionally, navigate to a different frame.
1. Click **PASTE**. The new annotation appears in the same location as the one you copied.
1. Optionally, to paste the same annotation, again, click **PASTE**.
#### Copy / paste all annotations
1. Click **COPY ALL**.
1. Optionally, navigate to a different frame.
1. Click **PASTE**.
#### Move annotations
* Move a bounded area by clicking on it and dragging.
#### Resize annotations
* Resize a bounded area by clicking on a vertex and dragging.
#### Delete annotations
1. Click the annotation or bounded area in the image or video clip.
1. Press **DELETE** or in the annotation, click **>X**.
#### Add labels
* Click in the annotation and choose a label from the label list, or type a new label.
#### Modify labels
* In the annotation label textbox, choose a label from the list or type a new label.
#### Delete labels
* In the annotation, in the label area, click the label's **X**.
#### Modify annotation metadata
* In the label, click edit and then in the popup modify the metadata dictionary (in JSON format).
#### Modify annotation color
* Modify the color of an area by clicking the circle in the label name and select a new color.
#### Lock / unlock annotations
* Click the lock.
#### Modify frame metadata
* Expand the **FRAME METADATA** area, click edit, and then in the popup modify the metadata dictionary (in JSON format).

44
docs/hyperdatasets/webapp/webapp_datasets.md Normal file
View File

@@ -0,0 +1,44 @@
---
title: Datasets Page
---
The Datasets page offers the following functionalities:
* Managing the ClearML Enterprise **Datasets** and **versions**, which connect raw data to the ClearML Enterprise platform
* Using ClearML Enterprise's Git-like Dataset versioning features
* Managing SingleFrames and FrameGroups.
![image](../../img/hyperdatasets/datasets_01.png)
## Dataset cards
Dataset cards show summary information about versions, frames, and labels in a Dataset, and the elapsed time since the Dataset was last update and the user doing the update. Dataset cards allow you to open a specific Dataset to perform Dataset versioning and frames management.
* Dataset name
* Elapsed time since the last update. Hover over elapsed time and view date of last update.
* User updating the Dataset
* The number of versions in the Dataset
* The total number of frames in all versions of the Dataset. If an asterisk (\*) appears next to **FRAMES**, then you can hover it and see the name of the version whose frames were last updated appears.
* The percentage of frames annotated in all versions of the Dataset. If an asterisk (\*) appears next to **ANNOTATED**, then you can hover it and see the name of the version whose frames were last annotated appears.
* If the Dataset version's status is *Published*, then the top labels in the Dataset, color coded (colors are editable) appear. If the Dataset version is Draft, then no labels appear.
:::note
To change the label color coding, hover over a label color, click thr hand pointer, and then select a new color.
:::
## Creating new Datasets
Create a new Dataset which will contain one version named `Current`. The new version will not contain any frames.
* Click **+ NEW DATASET** **>** Enter a name and optionally a description **>** **CREATE DATASET**.
## Sort Datasets
* In **RECENT**, choose either:
* **RECENT** - Most recently update of the Datasets.
* **NAME** - Alphabetically sort by Dataset name.

310
docs/hyperdatasets/webapp/webapp_datasets_frames.md Normal file
View File

@@ -0,0 +1,310 @@
---
title: Working with Frames
---
View and edit SingleFrames in the Dataset page. After selecting a Dataset version, the **Version Browser** shows a sample
of frames and enables viewing SingleFrames and FramesGroups, and edit SingleFrames, in the [frame viewer](#frame-viewer).
Before opening the frame viewer, you can filter the frames by applying [simple](#simple-frame-filtering) or [advanced](#advanced-frame-filtering)
filtering logic.
![image](../../img/hyperdatasets/frames_01.png)
## Frame viewer
Frame viewer allows you to view and edit annotations which can be FrameGroup objects (Regions of Interest) and FrameGroup
labels applied to the entire frame not a region of the frame, the frame details (see [frames](../frames.md)),
frame metadata, the raw data source URI, as well as providing navigation and viewing tools.
![image](../../img/hyperdatasets/web-app/dataset_example_frame_editor.png)
### Frame viewer controls
Use frame viewer controls to navigate between frames in a Dataset Version, and control frame changes and viewing.
|Control Icon|Actions|
|-----|------|
|<img src="/static/icons/ico-skip-backward.svg" className="icon size-md space-sm" />|Jump backwards (CTRL + Left). Jumps backwards by five frames.|
|<img src="/static/icons/ico-skip-previous.svg" className="icon size-md space-sm" />|Go to the previous frame containing a non-filtered annotation. The filter is the minimum confidence level setting. If the confidence level filter is set to zero, any frame containing annotations matches the filter.|
|<img src="/static/icons/ico-arrow-left.svg" className="icon size-md space-sm" />|Go to the previous frame (Left Arrow).|
|<img src="/static/icons/ico-arrow-right.svg" className="icon size-md space-sm" />|Go to the next frame (Right Arrow).|
|<img src="/static/icons/ico-skip-next.svg" className="icon size-md space-sm" />|Go to the next frame containing a non-filtered annotation (same filter as <img src="/static/icons/ico-skip-previous.svg" className="icon size-md space-sm" />).|
|<img src="/static/icons/ico-skip-forward.svg" className="icon size-md space-sm" />|Jump forwards (CTRL + Right). Jumps 5 frames forwards.|
|<img src="/static/icons/ico-revert.svg" className="icon size-md space-sm" />|Reload the frame.|
|<img src="/static/icons/ico-undo.svg" className="icon size-md space-sm" />|Undo changes.|
|<img src="/static/icons/ico-redo.svg" className="icon size-md space-sm" />|Redo changes.|
|<img src="/static/icons/ico-reset_1.svg" className="icon size-md space-sm" />|Autofit|
|<img src="/static/icons/ico-zoom-in.svg" className="icon size-md space-sm" />|Zoom in|
|<img src="/static/icons/ico-zoom-out.svg" className="icon size-md space-sm" />|Zoom out|
|Percentage textbox|Zoom percentage|
### Viewing and editing frames
**To view / edit a frame in the frame editor**
1. Locate your frame by applying a [simple frame filter](#simple) or [advanced frame filter](#advanced), and clicking <span class="tr_gui">LOAD MORE</span>, if required.
1. Click the frame thumbnail. The frame editor appears.
1. Do any of the following:
* View frame details, including:
* Frame file path
* Dimensions of the image or video
* Frame details
* Frame metadata
* Annotations
* Frame objects - Labeled Regions of Interest, with confidence levels and custom metadata per frame object.
* Frame labels - Labels applied to the entire frame, not a region in the frame.
* Optionally, filter annotations by confidence level using the <span class="tr_gui">Minimum confidence</span> slider.
* Add, change, and delete [annotations](#annotations) and [frame metadata](#frame-metadata).
:::important
To save frames changes at any time, click **SAVE** (below the annotation list area).
:::
### Viewing FrameGroups
Viewing and editing frames in a FrameGroup is similar to viewing and editing SingleFrames.
Click the FrameGroup in the Dataset. In the frame viewer, select SingleFrame to view / modify from
a dropdown list in the **Current Source** section.
![image](../../img/hyperdatasets/framegroup_01.png)
## Filtering frames
### Simple frame filtering
Simple frame filtering applies one annotation object (ROI) label and returns frames containing at least one annotation
with that label.
**To apply a simple frame filter:**
* In the **Version Browser**, choose a label on the label list.
For example:
* Before filtering, the **Version Browser** in the image below contains seven frames.
<details className="cml-expansion-panel screenshot">
<summary className="cml-expansion-panel-summary">View a screenshot</summary>
<div className="cml-expansion-panel-content">
![image](../../img/hyperdatasets/frame_filtering_01.png)
</div>
</details>
<br/>
* A simple label filter for `person` shows three frames with each containing at least one ROI labeled `person`.
<details className="cml-expansion-panel screenshot">
<summary className="cml-expansion-panel-summary">View a screenshot</summary>
<div className="cml-expansion-panel-content">
![image](../../img/hyperdatasets/frame_filtering_02.png)
</div>
</details>
### Advanced frame filtering
Advanced frame filtering applies sophisticated filtering logic, which is composed of as many frame filters as needed,
where each frame filter can be a combination of ROI, frame, and source rules.
* ROI rules use include and exclude logic to match frames by ROI label; an ROI label can match frames containing at least
one annotation object (ROI) with all labels in the rule.
* Frame rules and source rules use Lucene queries with AND, OR, and NOT logic. Frame rules apply to frame metadata.
* Source rules apply to frame source information.
**To apply advanced filters:**
1. In the **Version Browser**, click **Switch to advanced filters**.
1. In a **FRAME FILTER**, create one of the following rules:
* ROI rule
* Choose **Include** or **Exclude**, select ROI labels, and optionally set the confidence level range.
* To switch from the ROI dropdown list to a Lucene query mode, click <img src="/static/icons/ico-edit.svg" className="icon size-md space-sm" />.
* Frame rule - Enter a Lucene query using frame metadata fields in the format `meta.<key>:<value>`.
* Source rule - Enter a Lucene query using frame metadata fields in the format `sources.<key>:<value>`.
### Examples
#### ROI rules
* Create one ROI rule for <code>person</code> shows the same three frames as the simple frame filter (above).
<details className="cml-expansion-panel screenshot">
<summary className="cml-expansion-panel-summary">View a screenshot</summary>
<div className="cml-expansion-panel-content">
![image](../../img/hyperdatasets/frame_filtering_03.png)
</div>
</details>
<br/>
* In the ROI rule, add a second label. Add `partially_occluded`. Only frames containing at least one ROI labeled as both <code>person</code> and <code>partially_occluded</code> match the filter.
<details className="cml-expansion-panel screenshot">
<summary className="cml-expansion-panel-summary">View a screenshot</summary>
<div className="cml-expansion-panel-content">
![image](../../img/hyperdatasets/frame_filtering_04.png)
</div>
</details>
<br/>
By opening a frame in the frame viewer, you can see an ROI labeled with both.
<details className="cml-expansion-panel screenshot">
<summary className="cml-expansion-panel-summary">View a screenshot</summary>
<div className="cml-expansion-panel-content">
![image](../../img/hyperdatasets/frame_filtering_05.png)
</div>
</details>
<br/>
#### Frame rules
Filter by metadata using Lucene queries.
* Add a frame rule to filter by the metadata key <code>dangerous</code> for the value of <code>no</code>.
<details className="cml-expansion-panel screenshot">
<summary className="cml-expansion-panel-summary">View a screenshot</summary>
<div className="cml-expansion-panel-content">
![image](../../img/hyperdatasets/frame_filtering_08.png)
</div>
</details>
<br/>
By opening a frame in the frame viewer, you can see the metadata.
<details className="cml-expansion-panel screenshot">
<summary className="cml-expansion-panel-summary">View a screenshot</summary>
<div className="cml-expansion-panel-content">
![image](../../img/hyperdatasets/frame_filtering_09.png)
</div>
</details>
<br/>
#### Source rules
Filter by sources using Lucene queries.
* Add a source rule to filter for sources URIs with a wildcards.
<details className="cml-expansion-panel screenshot">
<summary className="cml-expansion-panel-summary">View a screenshot</summary>
<div className="cml-expansion-panel-content">
![image](../../img/hyperdatasets/frame_filtering_10.png)
</div>
</details>
<br/>
Use Lucene queries in ROI label filters and frame rules.
## Annotations
### Frame objects (Regions of Interest)
You can add annotations by drawing new bounding areas, and copying existing annotations in the same or other frames.
**To draw a bounding area for a new annotation:**
1. Optionally, select a default label in the Default ROI Label(s) list. New annotations are automatically assigned this label.
1. Click one of the following modes and create a bounding area in the frame:
* <img src="/static/icons/ico-rectangle-icon-purple.svg" className="icon size-md space-sm" /> - Rectangle mode: Drag a
rectangle onto the frame.
* <img src="/static/icons/ico-ellipse-icon-purple.svg" className="icon size-md space-sm" /> - Ellipse mode: Drag an ellipse
onto the frame.
* <img src="/static/icons/ico-polygon-icon-purple.svg" className="icon size-md space-sm" /> - Polygon mode: Click the polygon
vertices onto the frame.
* <img src="/static/icons/ico-keypoint-icon-purple.svg" className="icon size-md space-sm" /> - Key points mode: Click each
keypoint onto the frame. After the clicking the last keypoint, click the first again to close the bounding area.
A new annotation is created.
1. In the newly created annotation, select or type a label.
1. Optionally, add metadata. This is metadata for the annotation, not the entire frame.
1. Optionally, lock the annotation.
1. If you move to another frame, the frame editor automatically saves changes. Otherwise, if you exit the frame editor,
you are prompted to save.
**To copy an annotation:**
1. Click the annotation or bounded area in the image or video clip.
1. Optionally, navigate to a different frame.
1. Click **PASTE**. The new annotation appears in the same location as the one you copied.
1. Optionally, to paste the same annotation, click **PASTE** again in the desired frame.
**To copy all annotations:**
1. Click **COPY ALL**.
1. Optionally, navigate to a different frame.
1. Click **PASTE**.
### Frame labels
**To add frame labels:**
1. Expand the **FRAME LABELS** area.
1. Click **+ Add new**.
1. Enter a label.
1. Optionally, click <img src="/static/icons/ico-edit.svg" className="icon size-md space-sm" />.
### Annotation management
**To move annotations:**
* Move a bounded area by clicking on it and dragging.
**To resize annotations:**
* Resize a bounded area by clicking on a vertex and dragging.
**To modify annotation metadata:**
* In the label, click edit and then in the popup modify the metadata dictionary (in JSON format).
**To modify annotation colors:**
* Modify the color of an area by clicking the circle in the label name and select a new color.
**To lock annotations:**
* All annotations - Above the annotations, click the lock / unlock.
* A specific annotation - In an annotation, click the lock / unlock.
**To delete annotations:**
1. Click the annotation or bounded area in the image or video clip.
1. Press the **DELETE** key, or in the annotation, click **X**.
**To add, change, or delete labels to annotations labels:**
* Add - Click in the annotation and choose a label from the label list, or type a new label.
* Change - In the annotation label textbox, choose a label from the list or type a new label.
* Delete - In the annotation, in the label area, click the label's **X**.
## Frame metadata
**To edit frame metadata:**
* Expand the **FRAME METADATA** area, click edit, and then in the popup modify the metadata dictionary (in JSON format).

113
docs/hyperdatasets/webapp/webapp_datasets_versioning.md Normal file
View File

@@ -0,0 +1,113 @@
---
title: Dataset Versioning
---
Use the Dataset versioning WebApp (UI) features for viewing, creating, modifying, and
deleting Dataset versions.
From the Datasets page, click on one of the Datasets in order to see and work with its versions.
### Viewing snapshots
View snapshots in the simple version structure using either:
* The simple view, a table of snapshots.
<details className="cml-expansion-panel screenshot">
<summary className="cml-expansion-panel-summary">Simple view (snapshot table)</summary>
<div className="cml-expansion-panel-content">
![image](../../img/hyperdatasets/web-app/dataset_simple_adv_01.png)
</div>
</details>
<br/>
* The advanced view, a tree of versions. The tree contains one version whose status is <i>Draft</i>, and snapshots appear in
chronological order, with oldest at the top, and the most recent at the bottom.
<details className="cml-expansion-panel screenshot">
<summary className="cml-expansion-panel-summary">Advanced view (version tree)</summary>
<div className="cml-expansion-panel-content">
![image](../../img/hyperdatasets/web-app/dataset_simple_adv_02.png)
</div>
</details>
### Creating snapshots
To create a snapshot, you must be in the simple (version table) view.
**To create a snapshot, do the following:**
1. If you are in the advanced view, click **Switch to Simple View** (In certain situations, this may not be possible,
see [Dataset Versioning](../dataset.md#dataset-versioning))
1. If the **DATASET HISTORY** section is not opened, click it.
1. If a snapshot is currently selected, click **RETURN TO CURRENT VERSION**.
1. Click **+ CREATE SNAPSHOT**.
1. Enter a version name, and optionally a description.
1. Click **CREATE**.
:::note
The WebApp (UI) does not currently support the automatic naming of snapshots with timestamps appended. You must provide a snapshot name.
:::
### Creating versions
To create a version, you must be in the advanced (version tree) view.
**To create a child version, do the following:**
1. If you are in the simple view, click **Switch to Advanced View**.
1. Click the (parent) version from which to create a child (inherit all frames).
1. Click **+ CREATE NEW VERSION**.
1. Enter a version name, and optionally a description.
1. Click **CREATE**.
### Publishing versions
Publish (make read-only) any Dataset version whose status is *Draft*. If the Dataset is in the simple structure,
and you publish the current version, then only the advanced view is available,
and you cannot create snapshots.
**To publish a version, do the following:**
1. If you are in the simple view, click **Switch to Advanced View**.
1. Click the version to publish.
1. Click **PUBLISH**.
1. Click **PUBLISH** again to confirm.
### Exporting frames
Frame exports downloaded filtered frames as a JSON file.
**To export frames, do the following:**
* In the Thumbnails area, click **EXPORT FRAMES**. The frames JSON file downloads.
### Modifying version names
**To modify a Dataset version name, do the following:**
* At the top right of the Dataset page, hover over the Dataset version name, click <img src="/static/icons/ico-edit.svg" className="icon size-md space-sm" /> , edit the name, and then click <img src="/static/icons/ico-save.svg" className="icon size-md space-sm" /> (check).
### Modifying version descriptions
**To modify a version description, do the following:**
* Expand the **INFO** area, hover over the **Description**, click <img src="/static/icons/ico-edit.svg" className="icon size-md space-sm" />,
edit the name, and then click <img src="/static/icons/ico-save.svg" className="icon size-md space-sm" /> (check).
### Deleting versions
You can delete versions whose status is *Draft*.
**To delete the current version, do the following:**
* If you are in the simple view, click **Switch to Advanced View**.
* Click the version to delete.
* Click **DELETE**.
* Click **DELETE** again to confirm.

95
docs/hyperdatasets/webapp/webapp_dataviews.md Normal file
View File

@@ -0,0 +1,95 @@
---
title: Dataviews Table
---
[Dataviews](../dataviews.mda) appear in the same Project as the experiment that stored the Dataview in the **ClearML Enterprise** platform,
as well as the **DATAVIEWS** tab in the **All Projects** page.
The **Dataviews table** is a [customizable](#customizing-the-dataviews-table) list of Dataviews associated with a project.
Use it to [view, create, and edit Dataviews](#viewing-adding-and-editing-dataviews) in the info panel. Dataview tables
can be filtered by name or name fragments and / or ID, by using the search bar.
![image](../../img/hyperdatasets/dataviews_table_01.png)
The Dataviews table columns in their default order are below. Dynamically order the columns by dragging a column heading
to a new position.
* **DATAVIEW** - Dataview name.
* **USER** - User creating the Dataview.
* **CREATED** - Elapsed time since the Dataview was created.
* **DESCRIPTION**
## Customizing the Dataviews table
The Dataviews table can be customized. Changes are persistent (cached in the browser), and represented in the URL.
Save customized settings in a browser bookmark, and share the URL with teammates.
Customize any combination of the following:
* Dynamic column ordering - Drag a column title to a different position.
* Filter by user
* Sort columns - By experiment name and / or elapsed time since creation.
* Column autofit - In the column heading, double click a resizer (column separator).
## Viewing, adding, and editing Dataviews
**To view, add, or edit a Dataview:**
1. Do one of the following:
* Create a new Dataview - Click **+ NEW DATAVIEW**.
* View or edit a Dataview - In the Dataview table, click the Dataview.
1. To edit sections of the Dataview, follow the steps on the "Modifying Dataviews" page for the following:
1. [Selecting Dataset versions](webapp_exp_modifying.md#selecting-dataset-versions)
1. [Filtering frames](webapp_exp_modifying.md#filtering-frames)
1. [Mapping labels (label translation)](webapp_exp_modifying.md#mapping-labels-label-translation) (if appropriate for
the data and experiment)
1. [Label enumeration](webapp_exp_modifying.md#label-enumeration)
1. [Data augmentation](webapp_exp_modifying.md#data-augmentation) (if appropriate for the data
and experiment)
1. [Iteration controls](webapp_exp_modifying.md#iteration-controls)
## Cloning Dataviews
Create an exact editable copy of a Dataview. For example, when tuning an experiment, clone a Dataview to apply the same
frame filters to different Dataset versions.
**To clone a Dataview:**
1. Do one of the following:
* In the Dataview table, right click a Dataview and then click **Clone**.
* If the info panel is opened, click <img src="/docs/img/svg/bars-menu.svg" alt="Menu" className="icon size-lg space-sm" />
(menu) and then click **Clone**.
1. Select a project or accept the current project, enter a name, and optionally enter a description
1. Click **CLONE**.
## Archiving Dataviews
Archive Dataviews to more easily manage current work. Archived Dataviews do not appear on the active Dataviews table.
They only appear in the archive. After archiving, the Dataview can be restored from the archive later.
**To archive a Dataview:**
* In the Dataview table:
* Archive one Dataview - Right click the Dataview **>** **Archive**.
* Archive multiple Dataviews - Select the Dataview checkboxes **>** In the footer menu that appears at the bottom of
the page, click **ARCHIVE**.
* In the Dataview info panel - Click <img src="/docs/img/svg/bars-menu.svg" alt="Menu" className="icon size-lg space-sm" />
(menu) **>** **ARCHIVE**.
**To restore a Dataview:**
1. Go to the Dataview table of the archived Dataview or of the **All Projects** page
1. Click **OPEN ARCHIVE**
1. Do any of the following:
* In the Dataview table:
* Restore one Dataview - Right click the Dataview **>** **Restore**.
* Restore multiple Dataviews - Select the Dataview checkboxes **>** **Restore**.
* In the info panel, restore one Dataview - Click <img src="/docs/img/svg/bars-menu.svg" alt="Menu" className="icon size-lg space-sm" />
(menu) **>** **Restore**.

37
docs/hyperdatasets/webapp/webapp_exp_comparing.md Normal file
View File

@@ -0,0 +1,37 @@
---
title: Comparing Dataviews
---
In addition to [**ClearML**'s comparison features](../../webapp/webapp_exp_comparing.md), the **ClearML Enterprise** WebApp
provides a deep comparison of input data selection criteria of experiment Dataviews, enabling to easily locate, visualize, and analyze differences.
## Selecting experiments
**To select experiments to compare:**
1. In the experiment's table, select the checkbox of each experiment to compare, or select the top checkbox for all experiments.
After selecting the second checkbox, the bottom bar appears.
1. In the bottom bar, click **COMPARE**. The comparison page appears, showing a column for each experiment and differences with
a highlighted background color. The experiment on the left is the base experiment. Other experiments compare to the base experiment.
## Dataviews (input data)
**To locate the input data differences:**
1. Click the **DETAILS** tab **>** Expand the **DATAVIEWS** section, or, in the header, click <img src="/static/icons/ico-previous-diff.svg" alt="Previous diff" className="icon size-md" />
(Previous diff) or <img src="/static/icons/ico-next-diff.svg" className="icon size-md space-sm" /> (Next diff).
1. Expand any of the following sections:
* **Augmentation** - On-the-fly data augmentation.
* **Filtering**
* Frame inclusion and exclusion rules based on ROI labels
* Frame metadata
* Frame sources
* Number of instances of a rule matching ROIs in each frame
* Confidence levels.
* **Iteration** - Iteration controls.
* **Labels Enumeration** - Class label enumeration.
* **Mapping** - ROI label translation.
* **View**
![image](../../img/hyperdatasets/web-app/compare_dataviews.png)

168
docs/hyperdatasets/webapp/webapp_exp_modifying.md Normal file
View File

@@ -0,0 +1,168 @@
---
title: Modifying Dataviews
---
An experiment that has been executed can be [cloned](../../webapp/webapp_exp_reproducing.md), then the cloned experiment's
execution details can be modified, and the modified experiment can be executed.
In addition to all the [**ClearML** tuning capabilities](../../webapp/webapp_exp_tuning.md), the **ClearML Enterprise WebApp** (UI)
enables modifying Dataviews, including:
* [Selected Dataview](#selected-dataview)
* [Dataset versions](#selecting-dataset-versions)
* [Frame filtering](#filtering-frames)
* [Label mapping](#mapping-labels-label-translation)
* [Class label enumeration](#label-enumeration)
* [Data augmentation](#data-augmentation)
* [Input frame iteration controls](#iteration-controls)
The selection and control of input data can be modified in *Draft* experiments that are not [development experiments](../task.md#development-experiments).
Do this by modifying the Dataview used by the experiment. The Dataview specifies the Dataset versions from which frames
are iterated and frame filters (see [Dataviews](webapp_dataviews.md).
**To choose a Dataview**, do any of the following:
* Create a new Dataview
* Click **+** and then follow the instructions below to select Dataset versions, filter frames, map labels (label translation),
and set label enumeration, data augmentation, and iteration controls.
* Select a different Dataview already associated with the experiment.
* In the **SELECTED DATAVIEW** list, choose a Dataview.
* Import a different Dataview associated with the same or another project.
* Click <img src="/static/icons/ico-import.svg" className="icon size-md space-sm" /> (**Import dataview**) and then
select **Import to current dataview** or **Import to aux dataview**.
:::note
After importing a Dataview, it can be renamed and / or removed.
:::
### Selecting Dataset versions
To input data from a different data source or different version of a data source, select a different Dataset version used
by the Dataview.
**To select Dataset versions for input data:**
1. In the **INPUT** area, click **EDIT**.
1. Do any of the following:
* Add a Dataset version - Input frames from another a version of another Dataset.
* Click **+**
* Select a Dataset and a Dataset version
* Remove a Dataset version - Do not input frames from a Dataset version.
Select frames from as many Dataset versions as are needed.
1. Click **SAVE**.
## Filtering frames
Filtering of SingleFrames iterated by a Dataview for input to the experiment is accomplished by frame filters.
For more detailed information, see [Filtering](../dataviews.md#filtering).
**To modify frame filtering:**
1. In the **FILTERING** area, click **EDIT**.
1. For each frame filter:
1. Select the Dataset version to which the frame filter applies.
1. Add, change, or remove any combination of the following rules:
* ROI rule - Include or exclude frames containing any single ROI with any combination of labels in the Dataset
version. Specify a range of the number of matching ROI (instances) per frame, and a range of confidence levels.
* Frame rule - Filter by frame metadata key-value pairs, or ROI labels.
* Source rule - Filter by frame `source` dictionary key-value pairs.
1. Optionally, debias input data by setting ratios for frames returned by the Dataview for each frame filter. These
ratios allow adjusting an imbalance in input data.
1. Click **SAVE**.
## Mapping labels (label translation)
Modify the ROI label mapping rules, which translate one or more input labels to another label for the output model. Labels
that are not mapped are ignored.
**To modify label mapping:**
1. In the **MAPPING** section, click **EDIT**
* Add (**+**) or edit a mapping:
1. Select the Dataset and version whose labels will be mapped.
1. Select one or more labels to map.
1. Select or enter the label to map to in the output model.
* Remove (<img src="/static/icons/ico-trash.svg" className="icon size-md space-sm" />) a mapping.
1. Click **SAVE**
## Label enumeration
Modify the label enumeration assigned to output models.
**To modify label enumeration:**
1. In the **LABELS ENUMERATION** section, click **EDIT**.
* Add (**+**) or edit an enumeration:
* Select a label and then enter an integer for it.
* Remove (<img src="/static/icons/ico-trash.svg" className="icon size-md space-sm" />) an enumeration.
1. Click **SAVE**.
## Data augmentation
Modify the on-the-fly data augmentation applied to frames input from the select Dataset versions and filtered by the frame filters. Data augmentation is applied in steps, where each step applies a method, operation, and strength.
For more detailed information, see [Data Augmentation](../dataviews.md#data-augmentation).
**To modify data augmentation**
1. In the **AUGMENTATION** section, click **EDIT**.
* Add (**+**) or edit an augmentation step - Select a **METHOD**, **OPERATION**, and **STRENGTH**.
* Remove (<img src="/static/icons/ico-trash.svg" className="icon size-md space-sm" />) an augmentation step.
1. Click **SAVE**.
## Iteration controls
Modify the frame iteration performed by the Dataview to control the order, number, timing, and reproducibility of frames
for training.
For more detailed information, see [Iteration Control](../dataviews.md#iteration-control).
**To modify iteration controls:**
1. In the **ITERATION** sections, click **EDIT**.
1. Select the **ORDER** of the SingleFrames returned by the iteration, either:
* **Sequential** - Iterate SingleFrames in sorted order by context ID and timestamp.
* **Random** - Iterate SingleFrames randomly using the random seed you can set (see Random Seed below).
1. Select the frame **REPETITION** option, either:
* **Use Each Frame Once**
* **Limit Frames**
* **Infinite Iterations**
1. Select the **RANDOM SEED** - If the experiment is rerun and the seed remains unchanged, the frames iteration is the same.
1. For video, enter a **CLIP LENGTH** - For video data sources, in the number of sequential frames from a clip to iterate.
1. Click **SAVE**.

69
docs/hyperdatasets/webapp/webapp_exp_track_visual.md Normal file
View File

@@ -0,0 +1,69 @@
---
title: Viewing Experiments
---
While an experiment is running, and any time after it finishes, results are tracked and can be visualized in the ClearML
Enterprise WebApp (UI).
In addition to all of **ClearML**'s offerings, ClearML Enterprise keeps track of the Dataviews associated with an
experiment, which can be viewed and [modified](webapp_exp_modifying.md) in the WebApp.
## Viewing an experiment's Dataviews
In an experiment's page, go to the **DATAVIEWS** tab to view all the experiment's Dataview details, including:
* Input data [selection](#dataset-versions) and [filtering](#filtering)
* ROI [mapping](#mapping) (label translation)
* [Label enumeration](#label-enumeration)
* On-the-fly [data augmentation](#augmentation)
* [Iteration controls](#iteration-control)
![image](../../img/hyperdatasets/web-app/dataview_tab.png)
### Input
SingleFrames are iterated from the Dataset versions specified in the **INPUT** area, in the **SELECTED DATAVIEW** drop-down
menu.
### Filtering
The **FILTERING** section lists the SingleFrame filters iterated by a Dataview, applied to the experiment data.
Each frame filter is composed of:
* A Dataset version to input from
* ROI Rules for SingleFrames to include and / or exclude certain criteria.
* Weights for debiasing input data.
Combinations of frame filters can implement complex querying.
For more detailed information, see [Filtering](../dataviews.md#filtering).
### Mapping
ROI label mapping (label translation) applies to the new model. For example, use ROI label mapping to accomplish the following:
* Combine several labels under another more generic label.
* Consolidate disparate datasets containing different names for the ROI.
* Hide labeled objects from the training process.
For detailed information, see [Mapping ROI labels](../dataviews.md#mapping-roi-labels).
### Label enumeration
Assign label enumeration in the **LABELS ENUMERATION** area.
### Augmentation
On-the-fly data augmentation applied to SingleFrames, which does not create new data. Apply data Augmentation in steps,
where each step is composed of a method, an operation, and a strength.
For detailed information, see [Data augmentation](../dataviews.md#data-augmentation).
### Iteration control
The input data iteration control settings determine the order, number, timing, and reproducibility of the Dataview iterating
SingleFrames. Depending upon the combination of iteration control settings, all SingleFrames may not be iterated, and some may repeat.
For detailed information, see [Iteration control](../dataviews.md#iteration-control).