gaussian-splatting/xy_utils/learning.ipynb

542 lines
19 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

{
"cells": [
{
"cell_type": "markdown",
"id": "b66179c8",
"metadata": {},
"source": [
"# 1.预处理数据"
]
},
{
"cell_type": "markdown",
"id": "533db061",
"metadata": {},
"source": [
"## 1.1 对点云数据进行处理 - 滤除环视范围外过远的点"
]
},
{
"cell_type": "markdown",
"id": "864843b2",
"metadata": {},
"source": [
"1. 解析 images.txt计算所有相机在世界坐标系下的光心位姿。 \n",
"2. 使用 Open3D 加载 .pcd 点云,并基于与最近相机光心的距离进行滤除。 \n",
"3. 输出相机光心的范围(坐标轴最小/最大值)以及被滤除的点云数量。 \n",
"4. 将保留的点云按 COLMAP points3D.txt 格式保存,并额外生成一个 PLY 文件,其中红色标记被删除的点,原色保留未删除点。"
]
},
{
"cell_type": "markdown",
"id": "5dc3ac56",
"metadata": {},
"source": [
"#### ipynb内部函数 -- 未进行地面与boundingbox对齐"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cc2e8c18",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import open3d as o3d\n",
"from scipy.spatial.transform import Rotation as R\n",
"\n",
"def filter_point_cloud(images_txt, pcd_path, output_points_txt, output_points_ply, output_ply_path):\n",
" \"\"\"\n",
" 根据相机位姿过滤点云,并保存结果。\n",
"\n",
" Args:\n",
" images_txt (str): 包含相机位姿的 images.txt 文件路径。\n",
" pcd_path (str): 点云文件路径。\n",
" output_points_txt (str): 保存过滤后点云的 points3D.txt 文件路径。\n",
" output_ply_path (str): 保存带颜色标记的 PLY 文件路径。\n",
" \"\"\"\n",
" def load_camera_centers(images_txt):\n",
" centers = []\n",
" with open(images_txt, 'r') as f:\n",
" for line in f:\n",
" if line.startswith('#') or len(line.strip()) == 0:\n",
" continue\n",
" elems = line.strip().split()\n",
" if len(elems) < 10:\n",
" continue\n",
" qw, qx, qy, qz = map(float, elems[1:5])\n",
" tx, ty, tz = map(float, elems[5:8])\n",
" rot = R.from_quat([qx, qy, qz, qw]).as_matrix()\n",
" center = -rot.T @ np.array([tx, ty, tz])\n",
" centers.append(center)\n",
" return np.array(centers)\n",
"\n",
" # 1. 加载相机中心\n",
" camera_centers = load_camera_centers(images_txt)\n",
"\n",
" # 2. 计算平均中心和最大偏移\n",
" center_mean = camera_centers.mean(axis=0)\n",
" offsets = np.abs(camera_centers - center_mean)\n",
" max_offset = offsets.max(axis=0)\n",
"\n",
" # 3. 加载点云\n",
" pcd = o3d.io.read_point_cloud(pcd_path)\n",
" points = np.asarray(pcd.points)\n",
" colors = np.asarray(pcd.colors) if pcd.has_colors() else np.ones_like(points)\n",
"\n",
" # 4. 构建包围盒并裁剪点云\n",
" min_bound = center_mean - max_offset - 15\n",
" max_bound = center_mean + max_offset + 15\n",
" aabb = o3d.geometry.AxisAlignedBoundingBox(min_bound, max_bound)\n",
" pcd_cropped = pcd.crop(aabb)\n",
"\n",
" # 5. 统计信息\n",
" total_points = len(points)\n",
" kept_points = np.asarray(pcd_cropped.points)\n",
" kept_colors = np.asarray(pcd_cropped.colors) if pcd_cropped.has_colors() else np.ones_like(kept_points)\n",
" removed_mask = np.ones(total_points, dtype=bool)\n",
" kept_indices = aabb.get_point_indices_within_bounding_box(pcd.points)\n",
" removed_mask[kept_indices] = False\n",
" removed_points = points[removed_mask]\n",
" removed_colors = colors[removed_mask]\n",
"\n",
" print(\"相机中心范围:\")\n",
" print(f\" X: [{min_bound[0]:.3f}, {max_bound[0]:.3f}]\")\n",
" print(f\" Y: [{min_bound[1]:.3f}, {max_bound[1]:.3f}]\")\n",
" print(f\" Z: [{min_bound[2]:.3f}, {max_bound[2]:.3f}]\")\n",
" print(f\"总点数:{total_points}\")\n",
" print(f\"保留点数:{len(kept_points)}\")\n",
" print(f\"删除点数:{len(removed_points)}\")\n",
"\n",
" # 6. 保存 points3D.txt (速度太慢,直接使用 ply )\n",
" with open(output_points_txt, \"w\") as f:\n",
" f.write(\"# 3D point list with one line of data per point:\\n\")\n",
" f.write(\"# POINT3D_ID, X, Y, Z, R, G, B, ERROR, TRACK[]\\n\")\n",
" for idx, (pt, col) in enumerate(zip(kept_points, kept_colors), start=1):\n",
" r, g, b = (col * 255).astype(int)\n",
" f.write(f\"{idx} {pt[0]:.6f} {pt[1]:.6f} {pt[2]:.6f} {r} {g} {b} 0\\n\")\n",
" # 保存为 ply 文件\n",
" pcd_kept = o3d.geometry.PointCloud()\n",
" pcd_kept.points = o3d.utility.Vector3dVector(kept_points)\n",
" pcd_kept.colors = o3d.utility.Vector3dVector(kept_colors)\n",
" o3d.io.write_point_cloud(output_points_ply, pcd_kept)\n",
" print(\"保留的点云已保存为 {output_points_ply}\")\n",
" \n",
" # 7. 保存带颜色的 PLY 文件\n",
" all_points = np.vstack((kept_points, removed_points))\n",
" removed_colors_red = np.tile([1.0, 0.0, 0.0], (len(removed_points), 1))\n",
" all_colors = np.vstack((kept_colors, removed_colors_red))\n",
" pcd_all = o3d.geometry.PointCloud()\n",
" pcd_all.points = o3d.utility.Vector3dVector(all_points)\n",
" pcd_all.colors = o3d.utility.Vector3dVector(all_colors)\n",
" o3d.io.write_point_cloud(output_ply_path, pcd_all)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "80b924be",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"相机中心范围:\n",
" X: [-0.200, 28.270]\n",
" Y: [-16.082, 1.699]\n",
" Z: [-9.392, 0.714]\n",
"总点数16404321\n",
"保留点数10684495\n",
"删除点数5719826\n",
"保留的点云已保存为 {output_points_ply}\n"
]
}
],
"source": [
"import os\n",
"folder_path = '/home/qinllgroup/hongxiangyu/git_project/gaussian-splatting-xy/data/tree_01_livo2/livo2_results'\n",
"\n",
"images_txt = os.path.join(folder_path, 'sparse/0/images.txt')\n",
"pcd_path = os.path.join(folder_path, 'pcd/all_raw_points.pcd')\n",
"output_points_ply = os.path.join(folder_path, 'pcd/points3D.ply')\n",
"vis_ply_path = os.path.join(folder_path, 'pcd/vis_filter.ply')\n",
"output_points_txt = os.path.join(folder_path, 'pcd/points3D.txt')\n",
"filter_point_cloud(images_txt,pcd_path,output_points_txt, output_points_ply,vis_ply_path)"
]
},
{
"cell_type": "markdown",
"id": "1a4206e8",
"metadata": {},
"source": [
"#### 调用 crop_points.py 文件"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8d8c55ed",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"总点数16404321保留15254842删除1149479\n"
]
}
],
"source": [
"!python crop_points.py \\\n",
" --images_txt /home/qinllgroup/hongxiangyu/git_project/gaussian-splatting-xy/data/tree_01_livo2/livo2_results/sparse/0/images.txt \\\n",
" --pcd_path /home/qinllgroup/hongxiangyu/git_project/gaussian-splatting-xy/data/tree_01_livo2/livo2_results/pcd/all_raw_points.pcd \\\n",
" --output_points_txt /home/qinllgroup/hongxiangyu/git_project/gaussian-splatting-xy/data/tree_01_livo2/livo2_results/pcd/points3D.txt \\\n",
" --output_ply /home/qinllgroup/hongxiangyu/git_project/gaussian-splatting-xy/data/tree_01_livo2/livo2_results/pcd/filtered_colored.ply \\\n",
" --margin 10.0"
]
},
{
"cell_type": "markdown",
"id": "651ed72f",
"metadata": {},
"source": [
"## 1.2 利用体素化降采样"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "56f5cba7",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"raw points len : 15254842\n",
"downsample points len : 201667\n",
"降采样后的点云已保存到 /home/qinllgroup/hongxiangyu/git_project/gaussian-splatting-xy/data/tree_01_livo2/sparse/0/points3D_filter_0.2.ply体素大小0.2\n"
]
}
],
"source": [
"import os\n",
"from tools.points_utils import voxel_downsample_and_save\n",
"\n",
"voxel_size = 0.2\n",
"folder_path = '/home/qinllgroup/hongxiangyu/git_project/gaussian-splatting-xy/data/tree_01_livo2/livo2_results'\n",
"input_ply_path = os.path.join(folder_path,'pcd/points3D.ply')\n",
"output_ply_path = os.path.join(folder_path,f'pcd/points3D_{voxel_size}.ply')\n",
"input_ply_path = '/home/qinllgroup/hongxiangyu/git_project/gaussian-splatting-xy/data/tree_01_livo2/sparse/0/points3D_filter.ply'\n",
"output_ply_path = f'/home/qinllgroup/hongxiangyu/git_project/gaussian-splatting-xy/data/tree_01_livo2/sparse/0/points3D_filter_{voxel_size}.ply'\n",
"\n",
"voxel_downsample_and_save(voxel_size, input_ply_path, output_ply_path) # ply downsample to ply\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8544bb3e",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# pcd 2 ply\n",
"# 将原pcd点云转换为ply点云\n",
"from tools.points_utils import pcd_2_ply\n",
"pcd_file = '/home/qinllgroup/hongxiangyu/git_project/gaussian-splatting-xy/data/tree_01_livo2/livo2_results/pcd/all_raw_points.pcd'\n",
"ply_file = '/home/qinllgroup/hongxiangyu/git_project/gaussian-splatting-xy/data/tree_01_livo2/livo2_results/pcd/all_raw_points.ply'\n",
"\n",
"pcd_2_ply(pcd_file,ply_file)\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "2bf16bb1",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Jupyter environment detected. Enabling Open3D WebVisualizer.\n",
"[Open3D INFO] WebRTC GUI backend enabled.\n",
"[Open3D INFO] WebRTCWindowSystem: HTTP handshake server disabled.\n",
"正在写入COLMAP TXT文件: /home/qinllgroup/hongxiangyu/git_project/gaussian-splatting-xy/data/tree_01_livo2/livo2_results/pcd/points3D_filter_white.txt\n",
"Converted /home/qinllgroup/hongxiangyu/git_project/gaussian-splatting-xy/data/tree_01_livo2/sparse/0/points3D_filter.ply to Colmap TXT in: /home/qinllgroup/hongxiangyu/git_project/gaussian-splatting-xy/data/tree_01_livo2/livo2_results/pcd/points3D_filter_white.txt\n"
]
}
],
"source": [
"# pcd 2 txt\n",
"from tools.points_utils import pcd_2_colmap_txt\n",
"pcd_file = '/home/qinllgroup/hongxiangyu/git_project/gaussian-splatting-xy/data/tree_01_livo2/sparse/0/points3D_filter.ply'\n",
"txt_file = '/home/qinllgroup/hongxiangyu/git_project/gaussian-splatting-xy/data/tree_01_livo2/livo2_results/pcd/points3D_filter_white.txt'\n",
"\n",
"pcd_2_colmap_txt(pcd_file, txt_file, is_white=True)"
]
},
{
"cell_type": "markdown",
"id": "6fd0b306",
"metadata": {},
"source": [
"选择一个合适的 points3D.ply 文件复制到 sparse/0 下"
]
},
{
"cell_type": "markdown",
"id": "22e87bd3",
"metadata": {},
"source": [
"## 1.2 Resize for more images"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "91ce4cb8",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import cv2\n",
"from tqdm import tqdm\n",
"\n",
"def resize_images(input_dir, output_dir, extensions):\n",
" \"\"\"\n",
" 读取输入文件夹中的所有图片调整为1/2大小后保存到输出文件夹\n",
" \n",
" Args:\n",
" input_dir: 输入图片文件夹路径\n",
" output_dir: 输出图片文件夹路径\n",
" extensions: 支持的图片扩展名列表\n",
" \"\"\"\n",
" # 确保输出目录存在\n",
" os.makedirs(output_dir, exist_ok=True)\n",
" \n",
" # 获取所有图片文件\n",
" image_files = []\n",
" for file in os.listdir(input_dir):\n",
" if any(file.lower().endswith(ext) for ext in extensions):\n",
" image_files.append(file)\n",
" \n",
" if not image_files:\n",
" print(f\"在 {input_dir} 中未找到支持的图片文件\")\n",
" return\n",
" \n",
" print(f\"找到 {len(image_files)} 张图片\")\n",
" \n",
" # 处理每张图片\n",
" count = 0\n",
" for file in tqdm(image_files, desc=\"处理中\"):\n",
" input_path = os.path.join(input_dir, file)\n",
" output_path = os.path.join(output_dir, file)\n",
" \n",
" try:\n",
" # 读取图片\n",
" img = cv2.imread(input_path)\n",
" if img is None:\n",
" print(f\"警告: 无法读取图片 {input_path},跳过\")\n",
" continue\n",
" \n",
" # 获取原始尺寸\n",
" height, width = img.shape[:2]\n",
" \n",
" # 计算新尺寸\n",
" new_width = width // 2\n",
" new_height = height // 2\n",
" \n",
" # 调整尺寸\n",
" resized_img = cv2.resize(img, (new_width, new_height), interpolation=cv2.INTER_AREA)\n",
" \n",
" # 保存图片\n",
" cv2.imwrite(output_path, resized_img)\n",
" \n",
" # 输出尺寸信息\n",
" if count == 0:\n",
" print(f\"{file}: {width}x{height} -> {new_width}x{new_height}\")\n",
" count += 1\n",
" except Exception as e:\n",
" print(f\"错误: 处理图片 {input_path} 时出错: {str(e)}\")\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e9b4da36",
"metadata": {},
"outputs": [],
"source": [
"input_path = '/home/qinllgroup/hongxiangyu/git_project/gaussian-splatting-xy/data/tree_01/depth_maps'\n",
"output_path = '/home/qinllgroup/hongxiangyu/git_project/gaussian-splatting-xy/data/tree_01/depth_maps_2'\n",
"exts = ['jpg','jpeg','png']\n",
"\n",
"resize_images(input_path, output_path, exts)\n",
"print(\"所有图片处理完成!\") "
]
},
{
"cell_type": "markdown",
"id": "3830cb91",
"metadata": {},
"source": [
"# 2. LIVO2和Colmap的重建对比实验"
]
},
{
"cell_type": "markdown",
"id": "7dd6ff70",
"metadata": {},
"source": [
"colmap无法恢复相机位姿所以这里我们使用livo2恢复位姿后用colmap 进行三角测量获取关键点\n",
"https://www.cnblogs.com/Todd-Qi/p/15080968.html"
]
},
{
"cell_type": "markdown",
"id": "4ad9f127",
"metadata": {},
"source": [
"## 2.1 基于Livo2位姿进行稀疏重建"
]
},
{
"cell_type": "markdown",
"id": "40e8c7f1",
"metadata": {},
"source": [
"colmap无法恢复相机位姿所以这里我们使用livo2恢复位姿后用colmap 进行三角测量获取关键点\n",
"https://www.cnblogs.com/Todd-Qi/p/15080968.html"
]
},
{
"cell_type": "markdown",
"id": "497d715a",
"metadata": {},
"source": [
"1. 准备来自Livo2的位姿和相机数据 cameras.txt, images.txt\n",
" 将内参(camera intrinsics) 放入cameras.txt 外参(camera extrinsics)放入 images.txt , points3D.txt 为空 \n",
" - images.txt 中全部 0.0 0.0 -1 删除; \n",
" - points3D.txt 内容清空;\n",
" - cameras.txt 中的内参进行修改 (对输入图像全部进行了 resize 操作因此需要修改相机内参将fx, fy, cx, cy 都除以2)"
]
},
{
"cell_type": "markdown",
"id": "497596e0",
"metadata": {},
"source": [
"2. 特征匹配与特征提取 \n",
"``` bash\n",
" colmap feature_extractor \\\n",
" --database_path /path/to/project/database.db \\ \n",
" --image_path /path/to/project/images\n",
"```\n",
"``` bash\n",
" colmap exhaustive_matcher \\\n",
" --database_path /path/to/project/database.db\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "590c5ba8",
"metadata": {},
"source": [
"3. 三角化重建 (保存的点云和其他文件均为bin格式)\n",
"``` bash\n",
" colmap point_triangulator \\\n",
" --database_path /path/to/project/database.db \\\n",
" --image_path /path/to/project/images \\\n",
" --input_path /path/to/sparse_model \\\n",
" --output_path /path/to/triangulated_model\n",
"\n",
"```\n",
"\n",
"查看txt结果\n",
"``` bash\n",
" colmap model_converter \\\n",
" --input_path 0 \\\n",
" --output_path 0_txt_from_livo2 \\\n",
" --output_type TXT\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "6c2b853f",
"metadata": {},
"source": []
},
{
"cell_type": "markdown",
"id": "d359ac68",
"metadata": {},
"source": [
"4. 稠密重建(optional)"
]
},
{
"cell_type": "markdown",
"id": "77250b4b",
"metadata": {},
"source": [
"# 3.训练"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d44f71f7",
"metadata": {},
"outputs": [],
"source": [
"# baseline raw gs for training\n",
"!CUDA_VISIBLE_DEVICES=1 python train.py \\\n",
" -s data/tree_01_livo2 \\\n",
" -m data/tree_01_livo2/outputs/3dgs_baseline\n",
" \n",
"# render\n",
"!CUDA_VISIBLE_DEVICES=1 python render.py \\\n",
" -s data/tree_01_colmap \\\n",
" -m data/tree_01_colmap/outputs/3dgs_baseline "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.18"
}
},
"nbformat": 4,
"nbformat_minor": 5
}