1.9 KiB
1. Introduction
We provide a test script for both zero-shot and few-shot evaluation on mathematical reasoning benchmarks used in our paper.
2. Setup
First configure the prefix in environment.yml and then run the following command
conda env create -f environment.yml
3. Evaluation
For chain-of-thought evaluation of DeepSeekMath-Instruct and DeepSeekMath-RL, our script (see def markup_question() in run_subset_parallel.py) processes each question as follows:
- English questions:
{question}\nPlease reason step by step, and put your final answer within \\boxed{}. - Chinese questions:
{question}\n请通过逐步推理来解答问题，并把最终答案放置于\\boxed{}中。
For tool-integrated reasoning, we process each question as follows:
- English questions:
{question}\nPlease integrate natural language reasoning with programs to solve the problem above, and put your final answer within \\boxed{}. - Chinese questions:
{question}\n请结合自然语言和Python程序语言来解答问题，并把最终答案放置于\\boxed{}中。
We provide an example of testing the DeepSeekMath-Base 7B using 8 GPUs.
If you wish to use a different model or dataset, you can modify the configs in submit_eval_jobs.py and configs/*test_configs.json
python submit_eval_jobs.py --n-gpus 8
Wait for all processes to finish, and then run the following command to aggregate results from all processes
python summarize_results.py [--eval-atp]
where the option --eval-atp will invoke unsafe_score_minif2f_isabelle.py to evaluate the informal-to-formal proving results. Please make sure you have set up the PISA server before using this option.
A summary of all evaluation results will be saved as evaluation_results.json
4. Model Outputs
We provide all model outputs in outputs.zip.