diff --git a/README.md b/README.md index 26985e6..d09850b 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ Bernhard Kerbl*, Georgios Kopanas*, Thomas Leimkühler, George Drettakis (* indi | [Webpage](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/) | [Full Paper](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/3d_gaussian_splatting_high.pdf) | [Video](https://youtu.be/T_kXY43VZnk) | [Other GRAPHDECO Publications](http://www-sop.inria.fr/reves/publis/gdindex.php) | [FUNGRAPH project page](https://fungraph.inria.fr) -[T&T+DB Datasets](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/datasets/input/tandt_db.zip) | [Pre-trained Models TODO](TODO)| [Evaluation Renderings TODO](TODO)|
+[T&T+DB Datasets (650MB)](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/datasets/input/tandt_db.zip) | [Pre-trained Models (14 GB) ](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/datasets/pretrained/models.zip)| [Evaluation Renderings TODO](TODO)|
![Teaser image](assets/teaser.png) This repository contains the code associated with the paper "3D Gaussian Splatting for Real-Time Radiance Field Rendering", which can be found [here](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/). We further provide the reference images used to create the error metrics reported in the paper, as well as recently created, pre-trained models. @@ -84,8 +84,6 @@ conda env create --file environment.yml --prefix //gaussian_spl conda activate //gaussian_splatting ``` -#### Custom Install - If you can afford the disk space, we recommend using our environment files for setting up a training environment identical to ours. If you want to make changes, please note that major version changes might affect the results of our method. However, our (limited) experiments suggest that the codebase works just fine inside a more up-to-date environment (Python 3.8, PyTorch 2.0.0, CUDA 11.8). ### Running @@ -174,11 +172,18 @@ python render.py -m # Generate renderings python metrics.py -m # Compute error metrics on renderings ``` +If you want to evaluate our pre-trained models, you will have to download the corresponding source data sets and indicate their location to ```render.py``` with an additional ```--source_path/-s``` flag. +```shell +python render.py -m -s +python metrics.py -m +``` +The pre-trained models were created with the release codebase. This code base has been cleaned up and includes bugfixes, hence the metrics you get from evaluating them will differ from those in the paper. +
Command Line Arguments for render.py #### --model_path / -m - Path where the trained model should be stored (```output/``` by default). + Path to the trained model directory you want to create renderings for. #### --skip_train Flag to skip rendering the training set. #### --skip_test @@ -217,29 +222,50 @@ We further provide the ```full_eval.py``` script. This script specifies the rout ```shell python full_eval.py -m360 -tat -db ``` -In the current version, this process takes about 7h on our reference machine containing an A6000. +In the current version, this process takes about 7h on our reference machine containing an A6000. If you want to do the full evaluation on our pre-trained models, you can specify their download location and skip training. +```shell +python full_eval.py -o --skip_training -m360 -tat -db +``` -## Network Viewer +
+Command Line Arguments for full_eval.py + + #### --skip_training + Flag to skip training stage. + #### --skip_rendering + Flag to skip rendering stage. + #### --skip_metrics + Flag to skip metrics calculation stage. + #### --output_path + Directory to put renderings and results in, ```./eval``` by default, set to pre-trained model location if evaluating them. + #### --mipnerf360 / -m360 + Path to MipNeRF360 source datasets, required if training or rendering. + #### --tanksandtemples / -tat + Path to Tanks&Temples source datasets, required if training or rendering. + #### --deepblending / -db + Path to Deep Blending source datasets, required if training or rendering. +
+
-The Network Viewer can be used to observe the training process and watch the model as it forms. It is not required for the basic workflow, but it is automatically set up when preparing SIBR for the Real-Time Viewer. +## Interactive Viewers +We provide two interactive iewers for our method: remote and real-time. Our viewing solutions are based on the SIBR framework. The setup is the same for both, the remote viewer (for observing the training) and the real-time viewer (for inspecting finalized models). ### Hardware Requirements - - OpenGL 4.5-ready GPU -- 8 GB VRAM +- CUDA-ready GPU with Compute Capability 7.0+ (only for Real-Time Viewer) +- 4 GB VRAM ### Software Requirements - C++ Compiler (we *recommend* Visual Studio 2019 for Windows) - CUDA 11 Developer SDK (we used 11.8) - CMake (recent version, we used 3.24) -- 7zip (Windows) +- 7zip (only on Windows) ### Setup - If you cloned with submodules (e.g., using ```--recursive```), the source code for the viewers is found in ```SIBR_viewers_(windows|linux)``` (choose whichever fits your OS). The network viewer runs within the SIBR framework for Image-based Rendering applications. #### Windows -On Windows, CMake should take care of your dependencies. +CMake should take care of your dependencies. ```shell cd SIBR_viewers_windows cmake -Bbuild . @@ -248,7 +274,7 @@ cmake --build build --target install --config RelWithDebInfo You may specify a different configuration, e.g. ```Debug``` if you need more control during development. #### Ubuntu -For Ubuntu, you will need to install a few dependencies before running the project setup. +You will need to install a few dependencies before running the project setup. ```shell # Dependencies sudo apt install -y libglew-dev libassimp-dev libboost-all-dev libgtk-3-dev libopencv-dev libglfw3-dev libavdevice-dev libavcodec-dev libeigen3-dev libxxf86vm-dev libembree-dev @@ -257,37 +283,19 @@ cd SIBR_viewers_linux cmake -Bbuild . cmake --build build --target install ``` -If you receive a build error related to ```libglfw```, locate the library directory and set up a symbolic link there ```libglfw3.so``` → ``````. +If you receive a build error related to ```libglfw```, locate its library directory on your machine and set up a symbolic link there ```libglfw3.so``` → ``````. -### Running +### Navigation in SIBR Viewers +The SIBR interface provides several methods of navigating the scene. By default, you will be started with an FPS navigator, which you can control with ```W, A, S, D``` for camera translation and ```Q, E, I, K, J, L``` for rotation. Alternatively, you may want to use a Trackball-style navigator (select from the floating menu). You can also snap to a camera from the data set with the ```Snap to``` button or find the closest camera with ```Snap to closest```. The floating menues also allow you to change the navigation speed. You can use the ```Scaling Modifier``` to control the size of the displayed Gaussians, or show the initial point cloud. + +### Running the Network Viewer You may run the compiled ```SIBR_remoteGaussian_app_``` either by opening the build in your C++ development IDE or by running the installed app in ```install/bin```, e.g.: ```shell ./SIBR_viewers_windows/install/bin/SIBR_remoteGaussian_app_rwdi.exe ``` - The network viewer allows you to connect to a running training process on the same or a different machine. If you are training on the same machine and OS, no command line parameters should be required: the optimizer communicates the location of the training data to the network viewer. By default, optimizer and network viewer will try to establish a connection on **localhost** on port **6009**. You can change this behavior by providing matching ```--ip``` and ```--port``` parameters to both the optimizer and the network viewer. If for some reason the path used by the optimizer to find the training data is not reachable by the network viewer (e.g., due to them running on different (virtual) machines), you may specify an override location to the viewer by using ```--path ```. -### Navigation -The SIBR interface provides several methods of navigating the scene. By default, you will be started with an FPS navigator, which you can control with ```W, A, S, D``` for camera translation and ```Q, E, I, K, J, L``` for rotation. Alternatively, you may want to use a Trackball-style navigator (select from the floating menu). You can also snap to a camera from the data set with the ```Snap to``` button or find the closest camera with ```Snap to closest```. The floating menues also allow you to change the navigation speed. You can use the ```Scaling Modifier``` to control the size of the displayed Gaussians, or show the initial point cloud. - -## Real-Time Viewer - -The Real-Time Viewer can be used to render trained models with real-time frame rates. - -### Hardware Requirements - -- CUDA-ready GPU with Compute Capability 7.0+ -- OpenGL 4.5-ready GPU -- 8 GB VRAM - -### Software Requirements -The requirements are the same as for the remote viewer. - -### Setup - -The setup is the same as for the remote viewer. - -### Running +### Running the Real-Time Viewer You may run the compiled ```SIBR_gaussianViewer_app_``` either by opening the build in your C++ development IDE or by running the installed app in ```install/bin```, e.g.: ```shell ./SIBR_viewers_windows/install/bin/SIBR_gaussianViewer_app_rwdi.exe --model-path @@ -295,9 +303,7 @@ You may run the compiled ```SIBR_gaussianViewer_app_``` either by openin It should suffice to provide the ```--model-path``` parameter pointing to a trained model directory. Alternatively, you can specify an override location for training input data using ```--path```. To use a specific resolution other than the auto-chosen one, specify ```--rendering-size ```. To unlock the full frame rate, please disable V-Sync on your machine and also in the application (Menu → Display). -### Navigation - -Navigation works exactly as it does in the network viewer. However, you also have the option to visualize the Gaussians by rendering them as ellipsoids from the floating menu. +In addition to the intial point cloud and the splats, you also have the option to visualize the Gaussians by rendering them as ellipsoids from the floating menu. ## Converting your own Scenes diff --git a/full_eval.py b/full_eval.py index c83efbd..6b5776e 100644 --- a/full_eval.py +++ b/full_eval.py @@ -21,43 +21,55 @@ parser = ArgumentParser(description="Full evaluation script parameters") parser.add_argument("--skip_training", action="store_true") parser.add_argument("--skip_rendering", action="store_true") parser.add_argument("--skip_metrics", action="store_true") +parser.add_argument("--output_path", default="./eval") args, _ = parser.parse_known_args() -if not args.skip_training: - parser.add_argument('--mipnerf360', "-m360", required=True, type=str) - parser.add_argument("--tanksandtemples", "-tat", required=True, type=str) - parser.add_argument("--deepblending", "-db", required=True, type=str) - args = parser.parse_args() - - common_args = " --quiet --eval --test_iterations -1" - for scene in tanks_and_temples_scenes: - source = args.tanksandtemples + "/" + scene - os.system("python train.py -s " + source + " -m ./eval/" + scene + common_args) - for scene in deep_blending_scenes: - source = args.deepblending + "/" + scene - os.system("python train.py -s " + source + " -m ./eval/" + scene + common_args) - for scene in mipnerf360_outdoor_scenes: - source = args.mipnerf360 + "/" + scene - os.system("python train.py -s " + source + " -i images_4 -m ./eval/" + scene + common_args) - for scene in mipnerf360_indoor_scenes: - source = args.mipnerf360 + "/" + scene - os.system("python train.py -s " + source + " -i images_2 -m ./eval/" + scene + common_args) - all_scenes = [] all_scenes.extend(mipnerf360_outdoor_scenes) all_scenes.extend(mipnerf360_indoor_scenes) all_scenes.extend(tanks_and_temples_scenes) all_scenes.extend(deep_blending_scenes) +if not args.skip_training or not args.skip_rendering: + parser.add_argument('--mipnerf360', "-m360", required=True, type=str) + parser.add_argument("--tanksandtemples", "-tat", required=True, type=str) + parser.add_argument("--deepblending", "-db", required=True, type=str) + args = parser.parse_args() + +if not args.skip_training: + common_args = " --quiet --eval --test_iterations -1" + for scene in tanks_and_temples_scenes: + source = args.tanksandtemples + "/" + scene + os.system("python train.py -s " + source + " -m " + args.output_path + "/" + scene + common_args) + for scene in deep_blending_scenes: + source = args.deepblending + "/" + scene + os.system("python train.py -s " + source + " -m " + args.output_path + "/" + scene + common_args) + for scene in mipnerf360_outdoor_scenes: + source = args.mipnerf360 + "/" + scene + os.system("python train.py -s " + source + " -i images_4 -m " + args.output_path + "/" + scene + common_args) + for scene in mipnerf360_indoor_scenes: + source = args.mipnerf360 + "/" + scene + os.system("python train.py -s " + source + " -i images_2 -m " + args.output_path + "/" + scene + common_args) + if not args.skip_rendering: - for scene in all_scenes: - os.system("python render.py --quiet --skip_train --eval --iteration 7000 -m ./eval/" + scene) - for scene in all_scenes: - os.system("python render.py --quiet --skip_train --eval --iteration 30000 -m ./eval/" + scene) + all_sources = [] + for scene in mipnerf360_outdoor_scenes: + all_sources.append(args.mipnerf360 + "/" + scene) + for scene in mipnerf360_indoor_scenes: + all_sources.append(args.mipnerf360 + "/" + scene) + for scene in tanks_and_temples_scenes: + all_sources.append(args.tanksandtemples + "/" + scene) + for scene in deep_blending_scenes: + all_sources.append(args.deepblending + "/" + scene) + + common_args = " --quiet --eval --skip_train" + for scene, source in zip(all_scenes, all_sources): + os.system("python render.py --iteration 7000 -s " + source + " -m " + args.output_path + "/" + scene + common_args) + os.system("python render.py --iteration 30000 -s " + source + " -m " + args.output_path + "/" + scene + common_args) if not args.skip_metrics: scenes_string = "" for scene in all_scenes: - scenes_string += "\"" + "./eval/" + scene + "\" " + scenes_string += "\"" + args.output_path + "/" + scene + "\" " os.system("python metrics.py -m " + scenes_string) \ No newline at end of file diff --git a/metrics.py b/metrics.py index 9eed113..f7393a4 100644 --- a/metrics.py +++ b/metrics.py @@ -39,10 +39,11 @@ def evaluate(model_paths): per_view_dict = {} full_dict_polytopeonly = {} per_view_dict_polytopeonly = {} + print("") for scene_dir in model_paths: try: - print("\nScene:", scene_dir) + print("Scene:", scene_dir) full_dict[scene_dir] = {} per_view_dict[scene_dir] = {} full_dict_polytopeonly[scene_dir] = {} @@ -74,7 +75,8 @@ def evaluate(model_paths): print(" SSIM : {:>12.7f}".format(torch.tensor(ssims).mean(), ".5")) print(" PSNR : {:>12.7f}".format(torch.tensor(psnrs).mean(), ".5")) - print(" LPIPS: {:>12.7f}".format(torch.tensor(lpipss).mean(), ".5"), "\n") + print(" LPIPS: {:>12.7f}".format(torch.tensor(lpipss).mean(), ".5")) + print("") full_dict[scene_dir][method].update({"SSIM": torch.tensor(ssims).mean().item(), "PSNR": torch.tensor(psnrs).mean().item(),