diff --git a/docs/faq.md b/docs/faq.md index 50effc1e..96f80ee4 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -1,8 +1,12 @@ # TRAINS FAQ +* [How to change the location of TRAINS configuration file](#change-config-path) +* [How to override TRAINS credentials from OS environment](#credentials-os-env) +* [How to sort models by a certain metric?](#custom-columns) * [Can I store more information on the models?](#store-more-model-info) * [Can I store the model configuration file as well?](#store-model-configuration) * [I want to add more graphs, not just with Tensorboard. Is this supported?](#more-graph-types) +* [Is there a way to create a graph comparing hyper-parameters vs model accuracy?](#compare-graph-parameters) * [I noticed that all of my experiments appear as `Training`. Are there other options?](#other-experiment-types) * [I noticed I keep getting the message `warning: uncommitted code`. What does it mean?](#uncommitted-code-warning) * [Is there something TRAINS can do about uncommitted code running?](#help-uncommitted-code) @@ -19,6 +23,35 @@ * [The first log lines are missing from the experiment log tab. Where did they go?](#first-log-lines-missing) +## How to change the location of TRAINS configuration file? + +Set "TRAINS_CONFIG_FILE" OS environment variable to override the default configuration file location. + +```bash +export TRAINS_CONFIG_FILE="/home/user/mytrains.conf" +``` + + +## How to override TRAINS credentials from OS environment? + +Set the OS environment variables below, in order to override the configuration file / defaults. + +```bash +export TRAINS_API_ACCESS_KEY="key_here" +export TRAINS_API_SECRET_KEY="secret_here" +export TRAINS_API_HOST="http://localhost:8080" +``` + + +## How to sort models by a certain metric? + +Models are associated with the experiments that created them. +In order to sort experiments by a specific metric, add a custom column in the experiments table, + + + + + ## Can I store more information on the models? #### For example, can I store enumeration of classes? @@ -62,6 +95,34 @@ logger.report_scalar("loss", "classification", iteration=42, value=1.337) For a more detailed example, see [here](https://github.com/allegroai/trains/blob/master/examples/manual_reporting.py). +## Is there a way to create a graph comparing hyper-parameters vs model accuracy? + +Yes, You can manually create a plot with a single point X-axis for the hyper-parameter value, +and Y-Axis for the accuracy. For example: + +```python +number_layers = 10 +accuracy = 0.95 +Task.current_task().get_logger().report_scatter2d( + "performance", "accuracy", iteration=0, + mode='markers', scatter=[(number_layers, accuracy)]) +``` +Assuming the hyper-parameter is "number_layers" with current value 10, and the accuracy for the trained model is 0.95. +Then, the experiment comparison graph shows: + + + +Another option is a histogram chart: +```python +number_layers = 10 +accuracy = 0.95 +Task.current_task().get_logger().report_vector( + "performance", "accuracy", iteration=0, labels=['accuracy'], + values=[accuracy], xlabels=['number_layers %d' % number_layers]) +``` + + + ## I noticed that all of my experiments appear as `Training`. Are there other options? Yes! When creating experiments and calling `Task.init`, you can provide an experiment type. diff --git a/docs/screenshots/compare_plots.png b/docs/screenshots/compare_plots.png new file mode 100644 index 00000000..cef2b457 Binary files /dev/null and b/docs/screenshots/compare_plots.png differ diff --git a/docs/screenshots/compare_plots_hist.png b/docs/screenshots/compare_plots_hist.png new file mode 100644 index 00000000..7715e1a2 Binary files /dev/null and b/docs/screenshots/compare_plots_hist.png differ diff --git a/docs/screenshots/compare_values.png b/docs/screenshots/compare_values.png new file mode 100644 index 00000000..281eaa27 Binary files /dev/null and b/docs/screenshots/compare_values.png differ diff --git a/docs/screenshots/custom_column.png b/docs/screenshots/custom_column.png new file mode 100644 index 00000000..823dc320 Binary files /dev/null and b/docs/screenshots/custom_column.png differ diff --git a/docs/screenshots/set_custom_column.png b/docs/screenshots/set_custom_column.png new file mode 100644 index 00000000..e41990a7 Binary files /dev/null and b/docs/screenshots/set_custom_column.png differ