ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-06-26 18:17:44 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Fix top navigation bar highlighting (#1044)

Browse Source
This commit is contained in:
pollfly
2025-02-25 18:54:01 +02:00
committed by GitHub
parent 7bda34ca90
commit 02d24aff06
28 changed files with 64 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
docs/clearml_data/best_practices.md
View File

@@ -1,59 +0,0 @@
---
title: Best Practices
---
:::important
This page covers `clearml-data`, ClearML's file-based data management solution.
See [Hyper-Datasets](../hyperdatasets/overview.md) for ClearML's advanced queryable dataset management solution.
:::
The following are some recommendations for using ClearML Data.
![Dataset UI gif](../img/gif/dataset.gif)
## Versioning Datasets
Use ClearML Data to version your datasets. Once a dataset is finalized, it can no longer be modified. This makes clear
which version of the dataset was used with which task, enabling the accurate reproduction of your tasks.
Once you need to change the dataset's contents, you can create a new version of the dataset by specifying the previous
dataset as a parent. This makes the new dataset version inherit the previous version's contents, with the dataset's new
version contents ready to be updated.
## Organize Datasets for Easier Access
Organize the datasets according to use-cases and use tags. This makes managing multiple datasets and
accessing the most updated datasets for different use-cases easier.
Like any ClearML tasks, datasets can be organized into [projects (and subprojects)](../fundamentals/projects.md#creating-subprojects).
Additionally, when creating a dataset, tags can be applied to the dataset, which will make searching for the dataset easier.
Organizing your datasets into projects by use-case makes it easier to access the most recent dataset version for that use-case.
If only a project is specified when using [`Dataset.get()`](../references/sdk/dataset.md#datasetget), the method returns the
most recent dataset in a project. The same is true with tags; if a tag is specified, the method will return the most recent dataset that is labeled with that tag.
In cases where you use a dataset in a task (e.g. consuming a dataset), you can easily track which dataset the task is
using by using `Dataset.get()`'s `alias` parameter. Pass `alias=<dataset_alias_string>`, and the task using the dataset
will store the dataset's ID in the `dataset_alias_string` parameter under the task's **CONFIGURATION > HYPERPARAMETERS >
Datasets** section.
## Document your Datasets
Attach informative metrics or debug samples to the Dataset itself. Use [`Dataset.get_logger()`](../references/sdk/dataset.md#get_logger)
to access the dataset's logger object, then add any additional information to the dataset, using the methods
available with a [`Logger`](../references/sdk/logger.md) object.
You can add some dataset summaries (like [table reporting](../references/sdk/logger.md#report_table)) to create a preview
of the data stored for better visibility, or attach any statistics generated by the data ingestion process.
## Periodically Update Your Dataset
Your data probably changes from time to time. If the data is updated into the same local / network folder structure, which
serves as a dataset's single point of truth, you can schedule a script which uses the dataset `sync` functionality which
will update the dataset based on the modifications made to the folder. This way, there is no need to manually modify a dataset.
This functionality will also track the modifications made to a folder.
See the sync function with the [CLI](clearml_data_cli.md#sync) or [SDK](clearml_data_sdk.md#syncing-local-storage)
interface.

2
docs/clearml_data/clearml_data.md
View File

@@ -46,7 +46,7 @@ ClearML Data supports two interfaces:
- `clearml-data` - A CLI utility for creating, uploading, and managing datasets. See [CLI](clearml_data_cli.md) for a reference of `clearml-data` commands.
- `clearml.Dataset` - A Python interface for creating, retrieving, managing, and using datasets. See [SDK](clearml_data_sdk.md) for an overview of the basic methods of the `Dataset` module.
For an overview of recommendations for ClearML Data workflows and practices, see [Best Practices](best_practices.md).
For an overview of recommendations for ClearML Data workflows and practices, see [Best Practices](../best_practices/data_best_practices.md).
## Dataset Version States
The following table displays the possible states for a dataset version.