Disambiguate dataset and hyper-dataset (#88)

This commit is contained in:
pollfly 2021-10-21 09:40:53 +03:00 committed by GitHub
parent 21071e9a07
commit f2162ba9e5
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
7 changed files with 18 additions and 18 deletions

View File

@ -3,7 +3,7 @@ title: Projects
--- ---
Projects are contextual containers for [tasks](task.md) and [models](artifacts.md#models) (as well as [dataviews](../hyperdatasets/dataviews.md) Projects are contextual containers for [tasks](task.md) and [models](artifacts.md#models) (as well as [dataviews](../hyperdatasets/dataviews.md)
when hyperdatasets are enabled), providing a logical structure similar to file system folders. when Hyper-Datasets are enabled), providing a logical structure similar to file system folders.
An often useful method is to categorize components into projects according to models or objectives. An often useful method is to categorize components into projects according to models or objectives.
Grouping into projects helps in identifying tasks, models, and dataviews when queried. Grouping into projects helps in identifying tasks, models, and dataviews when queried.

View File

@ -5,7 +5,7 @@ title: Annotations
With ClearML Enterprise, annotations can be applied to video and image frames. [Frames](single_frames.md) support With ClearML Enterprise, annotations can be applied to video and image frames. [Frames](single_frames.md) support
two types of annotations: **Frame objects** and **Frame labels**. two types of annotations: **Frame objects** and **Frame labels**.
Annotation Tasks can be used to efficiently organize the annotation of frames in Dataset versions (see Annotation Tasks can be used to efficiently organize the annotation of frames in Hyper-Dataset versions (see
[Annotations Task Page](webapp/webapp_annotator.md)). [Annotations Task Page](webapp/webapp_annotator.md)).
For information about how to view, create, and manage annotations using the WebApp, see [Annotating Images and Videos](webapp/webapp_annotator.md#annotating-images-and-video). For information about how to view, create, and manage annotations using the WebApp, see [Annotating Images and Videos](webapp/webapp_annotator.md#annotating-images-and-video).

View File

@ -1,8 +1,8 @@
--- ---
title: Hyper Datasets title: Hyper-Datasets
--- ---
ClearML's Hyper Datasets are an MLOps-oriented abstraction of your data, which facilitates traceable, reproducible model development ClearML's Hyper-Datasets are an MLOps-oriented abstraction of your data, which facilitates traceable, reproducible model development
through parametrized data access and meta-data version control. through parametrized data access and meta-data version control.
The basic premise is that a user-formed query is a full representation of the dataset used by the ML/DL process. The basic premise is that a user-formed query is a full representation of the dataset used by the ML/DL process.

View File

@ -4,7 +4,7 @@ title: SingleFrames
A `SingleFrame` contains metadata pointing to raw data, and other metadata and data, which supports experimentation and A `SingleFrame` contains metadata pointing to raw data, and other metadata and data, which supports experimentation and
ClearML Enterprise's Git-like Dataset versioning. ClearML Enterprise's Git-like Hyper-Dataset versioning.
## Frame Components ## Frame Components
@ -42,7 +42,7 @@ For more information, see [Masks](masks.md).
### Previews ### Previews
`previews` is a dictionary containing metadata for optional thumbnail images that can be used in the ClearML Enterprise WebApp (UI) `previews` is a dictionary containing metadata for optional thumbnail images that can be used in the ClearML Enterprise WebApp (UI)
to view selected images in a Dataset. `previews` includes the `uri` of the thumbnail image. to view selected images in a Hyper-Dataset. `previews` includes the `uri` of the thumbnail image.
For more information, see [Previews](previews.md). For more information, see [Previews](previews.md).
@ -63,9 +63,9 @@ The panel below describes the details contained within a `frame`:
* `id` (*string*) - The unique ID of this frame. * `id` (*string*) - The unique ID of this frame.
* `blob` (*string*) - Raw data. * `blob` (*string*) - Raw data.
* `context_id` (*string*) - Source URL. * `context_id` (*string*) - Source URL.
* `dataset` (*dict*) - The Dataset and version containing the frame. * `dataset` (*dict*) - The Hyper-Dataset and version containing the frame.
* `id` - ID of the Dataset. * `id` - ID of the Hyper-Dataset.
* `version` - ID of the version. * `version` - ID of the version.
* `meta` (*dict*) - Frame custom metadata. Any custom key-value pairs (`sources` and `rois` can also contain a meta * `meta` (*dict*) - Frame custom metadata. Any custom key-value pairs (`sources` and `rois` can also contain a meta

View File

@ -2,11 +2,11 @@
title: Tasks title: Tasks
--- ---
Hyper Datasets extend the **ClearML** [**Task**](../fundamentals/task.md) with [Dataviews](dataviews.md) Hyper-Datasets extend the **ClearML** [**Task**](../fundamentals/task.md) with [Dataviews](dataviews.md)
## Usage ## Usage
Hyper datasets are supported by the `allegroai` python package. Hyper-Datasets are supported by the `allegroai` python package.
### Connecting Dataviews to a Task ### Connecting Dataviews to a Task

View File

@ -2,7 +2,7 @@
title: Working with Frames title: Working with Frames
--- ---
View and edit SingleFrames in the Dataset page. After selecting a Dataset version, the **Version Browser** shows a sample View and edit SingleFrames in the Dataset page. After selecting a Hyper-Dataset version, the **Version Browser** shows a sample
of frames and enables viewing SingleFrames and FramesGroups, and edit SingleFrames, in the [frame viewer](#frame-viewer). of frames and enables viewing SingleFrames and FramesGroups, and edit SingleFrames, in the [frame viewer](#frame-viewer).
Before opening the frame viewer, you can filter the frames by applying [simple](#simple-frame-filtering) or [advanced](#advanced-frame-filtering) Before opening the frame viewer, you can filter the frames by applying [simple](#simple-frame-filtering) or [advanced](#advanced-frame-filtering)
filtering logic. filtering logic.
@ -19,7 +19,7 @@ frame metadata, the raw data source URI, as well as providing navigation and vie
### Frame Viewer Controls ### Frame Viewer Controls
Use frame viewer controls to navigate between frames in a Dataset Version, and control frame changes and viewing. Use frame viewer controls to navigate between frames in a Hyper-Dataset Version, and control frame changes and viewing.
|Control Icon|Actions|Shortcut| |Control Icon|Actions|Shortcut|
|-----|------|----| |-----|------|----|
@ -89,7 +89,7 @@ To save frames changes at any time, click **SAVE** (below the annotation list ar
### Viewing FrameGroups ### Viewing FrameGroups
Viewing and editing frames in a FrameGroup is similar to viewing and editing SingleFrames. Viewing and editing frames in a FrameGroup is similar to viewing and editing SingleFrames.
Click the FrameGroup in the Dataset. In the frame viewer, select SingleFrame to view / modify from Click the FrameGroup in the Hyper-Dataset. In the frame viewer, select SingleFrame to view / modify from
a dropdown list in the **Current Source** section. a dropdown list in the **Current Source** section.
![Frame dropdown menu in FrameGroup](../../img/hyperdatasets/framegroup_01.png) ![Frame dropdown menu in FrameGroup](../../img/hyperdatasets/framegroup_01.png)

View File

@ -17,14 +17,14 @@ enables modifying Dataviews, including:
The selection and control of input data can be modified in *Draft* experiments that are not [development experiments](../task.md#development-experiments). The selection and control of input data can be modified in *Draft* experiments that are not [development experiments](../task.md#development-experiments).
Do this by modifying the Dataview used by the experiment. The Dataview specifies the Dataset versions from which frames Do this by modifying the Dataview used by the experiment. The Dataview specifies the Hyper-Dataset versions from which frames
are iterated and frame filters (see [Dataviews](webapp_dataviews.md). are iterated and frame filters (see [Dataviews](webapp_dataviews.md).
**To choose a Dataview**, do any of the following: **To choose a Dataview**, do any of the following:
* Create a new Dataview * Create a new Dataview
* Click **+** and then follow the instructions below to select Dataset versions, filter frames, map labels (label translation), * Click **+** and then follow the instructions below to select Hyper-Dataset versions, filter frames, map labels (label translation),
and set label enumeration, data augmentation, and iteration controls. and set label enumeration, data augmentation, and iteration controls.
* Select a different Dataview already associated with the experiment. * Select a different Dataview already associated with the experiment.
@ -72,7 +72,7 @@ For more detailed information, see [Filtering](../dataviews.md#filtering).
1. In the **FILTERING** area, click **EDIT**. 1. In the **FILTERING** area, click **EDIT**.
1. For each frame filter: 1. For each frame filter:
1. Select the Dataset version to which the frame filter applies. 1. Select the Hyper-Dataset version to which the frame filter applies.
1. Add, change, or remove any combination of the following rules: 1. Add, change, or remove any combination of the following rules:
* ROI rule - Include or exclude frames containing any single ROI with any combination of labels in the Dataset * ROI rule - Include or exclude frames containing any single ROI with any combination of labels in the Dataset
@ -95,7 +95,7 @@ that are not mapped are ignored.
1. In the **MAPPING** section, click **EDIT** 1. In the **MAPPING** section, click **EDIT**
* Add (**+**) or edit a mapping: * Add (**+**) or edit a mapping:
1. Select the Dataset and version whose labels will be mapped. 1. Select the Hyper-Dataset and version whose labels will be mapped.
1. Select one or more labels to map. 1. Select one or more labels to map.
@ -123,7 +123,7 @@ Modify the label enumeration assigned to output models.
## Data Augmentation ## Data Augmentation
Modify the on-the-fly data augmentation applied to frames input from the select Dataset versions and filtered by the frame filters. Data augmentation is applied in steps, where each step applies a method, operation, and strength. Modify the on-the-fly data augmentation applied to frames input from the select Hyper-Dataset versions and filtered by the frame filters. Data augmentation is applied in steps, where each step applies a method, operation, and strength.
For more detailed information, see [Data Augmentation](../dataviews.md#data-augmentation). For more detailed information, see [Data Augmentation](../dataviews.md#data-augmentation).