ClearML/clearml
1
0
Fork 0
mirror of https://github.com/clearml/clearml synced 2025-01-31 09:07:00 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
1b474dc0b0
clearml/examples/hyperdatasets
History
allegroai beb0f5e607 Update examples dependencies
2024-02-22 12:25:32 +02:00
..
data-ingestion Add hyperdataset examples (#823) 2022-11-20 16:28:42 +02:00
data-registration Edit hyperdataset examples (#895) 2023-01-24 12:30:55 +02:00
README.md Edit hyperdataset examples (#895) 2023-01-24 12:30:55 +02:00
requirements.txt Update examples dependencies 2024-02-22 12:25:32 +02:00

README.md

ClearML HyperDatasets

Hyper-Datasets is a data management system that’s designed for unstructured data such as text, audio, or visual data. It is part of the ClearML Enterprise offering, which means it includes quite a few upgrades over the open source clearml-data.

The main conceptual difference between the two is that Hyper-Datasets decouple the metadata from the raw data files. This allows you to manipulate the metadata in all kinds of ways, while abstracting away the logistics of having to deal with large amounts of data.

To leverage Hyper-Datasets power, users define Dataviews, which are sophisticated queries connecting specific data from one or more datasets to an experiment in the Experiment Manager. Essentially it creates and manages local views of remote Datasets.

Dataview in the UI

Examples Overview

  • Hyper-Dataset registration into ClearML Enterprise
  • Hypder-Dataset usage examples, retrieving frames using the DataView Class and connecting to pytorch dataloader

Further Resources

Learn about ClearML HyperDataset

Watch video Tutorial