diff --git a/docs/faq.md b/docs/faq.md index 021d2c4e..81c740c7 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -37,7 +37,7 @@ title: FAQ **Graphs and Logs** * [The first log lines are missing from the experiment console tab. Where did they go?](#first-log-lines-missing) -* [Can I create a graph comparing hyperparameters vs model accuracy?](#compare-graph-parameters) +* [How do I create a graph comparing hyperparameters vs model accuracy?](#compare-graph-parameters) * [I want to add more graphs, not just with TensorBoard. Is this supported?](#more-graph-types) * [How can I report more than one scatter 2D series on the same plot?](#multiple-scatter2D) @@ -525,24 +525,39 @@ info panel > CONSOLE tab, use the *Download full log* feature.
-**Can I create a graph comparing hyperparameters vs. model accuracy?** +**How do I create a graph comparing hyperparameters vs. model accuracy?** -Yes! You can manually create a plot with a single point X-axis for the hyperparameter value, and Y-axis for the accuracy. -For example: +You can use the UI's [experiment comparison features](webapp/webapp_exp_comparing.md) to compare the logged hyperparameter +and accuracy values of several experiments. + +In an experiment comparison page, under the **HYPER PARAMETERS** tab, you can view the experiments' hyperparameter values +in relation to a specific metric (e.g. accuracy) in a parallel coordinates plot. + +The image below show a parallel coordinates plot which displays the values of selected hyperparameters (`base_lr`, +`batch_size`, and `number_of_epochs`) and a performance metric (`accuracy`) of three experiments. + +![Parallel Coordinates](img/clearml_faq_screenshots/compare_parallel_coordinates.png) + +You can also visualize the differences in a scatter plot. In each experiment whose values wil compared, report a plot +with a single point, x-axis for the hyperparameter value, and Y-axis for the accuracy. + +In the code below, the task reports a single-point scatter plot with `number_layers` as the x-axis and +`accuracy` as the Y-axis : ```python number_layers = 10 accuracy = 0.95 Task.current_task().get_logger().report_scatter2d( - "performance", - "accuracy", + title="performance", + series="accuracy", iteration=0, mode='markers', scatter=[(number_layers, accuracy)] ) ``` -Assuming the hyperparameter is `number_layers` with current value `10`, and the `accuracy` for the trained model is `0.95`. Then, the experiment comparison graph shows: +When these experiments are compared in the UI's experiment comparison, all the reported `performance/accuracy` values +are displayed in a single plot. ![image](img/clearml_faq_screenshots/compare_plots.png) @@ -552,14 +567,13 @@ Another option is a histogram chart: number_layers = 10 accuracy = 0.95 Task.current_task().get_logger().report_vector( - "performance", - "accuracy", + title="performance", + series="accuracy", iteration=0, labels=['accuracy'], values=[accuracy], xlabels=['number_layers %d' % number_layers] ) - ``` ![image](img/clearml_faq_screenshots/compare_plots_hist.png) diff --git a/docs/img/clearml_faq_screenshots/compare_parallel_coordinates.png b/docs/img/clearml_faq_screenshots/compare_parallel_coordinates.png new file mode 100644 index 00000000..71644c51 Binary files /dev/null and b/docs/img/clearml_faq_screenshots/compare_parallel_coordinates.png differ diff --git a/docs/img/clearml_faq_screenshots/compare_plots.png b/docs/img/clearml_faq_screenshots/compare_plots.png index cef2b457..370a45ed 100644 Binary files a/docs/img/clearml_faq_screenshots/compare_plots.png and b/docs/img/clearml_faq_screenshots/compare_plots.png differ diff --git a/docs/img/clearml_faq_screenshots/compare_plots_hist.png b/docs/img/clearml_faq_screenshots/compare_plots_hist.png index 7715e1a2..f37655cc 100644 Binary files a/docs/img/clearml_faq_screenshots/compare_plots_hist.png and b/docs/img/clearml_faq_screenshots/compare_plots_hist.png differ