Turning points into clouds: a 3D Gaussian splatting experiment
Creating point clouds from 2D images was something I experimented with during my time as a student. I completed multiple projects and tried various approaches. I bought a fisheye lens to test its performance on indoor cloud-map generation. I came across a tool called COLMAP. Creating point clouds and working with them fascinated me. This is the story of how I explored and discovered ways to create beautiful 3D point clouds and later embed them into a website.
The first baby steps with COLMAP
It all started with my discovery of COLMAP. I had quite a lot of time during the semester breaks as a student, so I quickly started experimenting with COLMAP. When I visited the Louvre in France, I took a lot of photos in the hope of generating a nice dense point cloud.
Luvre example
And this is the result of my first proper attempt to create point clouds. As you can see, only one of the walls with the big painted picture is really visible. The reason is that at this time I did not yet know how to properly take the pictures. This image is about 4 years old.
Drone Flight and Test of LumaAI Sparse Generation
To generate the points, you need to fly over the object with a central point. It does not need to be perfect, but one thing I learned while flying the drone is that the size of the object does not really matter. What matters is the framing of the object and that the camera angle is about 45° from the horizon to the ground. Rain and fog make generating the point cloud nearly impossible. All the raindrops and foggy parts interfere with the point cloud generation.
Now to the LumaAI part. I chose LumaAI for its simple web interface and easy point cloud generation. It's so easy, in fact, that it's like drag and drop. I zipped my two drone video footages and uploaded the zip into the web interface. After waiting some time for the generation process to complete, you can view the created/rendered point cloud in their 3D viewer. There is also the possibility to download the point cloud in different formats or even get a "Luma" file.
The results speak for themselves:
Lumaai Web problem with external resources
Lumaai was a good choice for generating beautiful splats from the drone footage. So my idea to integrate the luma-web node package into a custom component seemed to be the right choice.
Console spam from Lumaai Web
Sadly, the luma-web node package also spams the console output of the webpage. Until now, I have not found a way to disable those "debug" messages. This is also a reason why I did not end up using this viewer for 3D Gaussian splats, even though the loading of splats is really interesting and extremely user-friendly in my opinion.
If you want to try out the luma-web package, the GitHub repo and the hosted demo site have some good tutorials to explore.
Trying out nerfstudio
Lumaai was a good start to try out. But there are other free to use open source tools like nerfstudio wich looks realy promesing. So I did, but i was never able to go past the Installation point.
Dependency hell with Python
Nerfstudio is extremely finicky to build or even run. After about 6 hours of trying inside WSL or on Windows, I gave up. It was always some Python dependency that failed to install because another one was missing. Maybe I will try the Docker image from Nerfstudio without building it myself later.
It is quite interesting that with pip, it's possible to deprecate or even completely remove packages. If a dependency still relies on these deprecated or removed packages, an error is thrown. And heaven forbid you use the wrong pip or the wrong Python version. I will try to build Nerfstudio again once I become more familiar with Python.
LichtFeld Studio and OpenSplat
LichtFeld Studio and OpenSplat looks also realy promising. I will try those one out later too.
Generating a Mesh
I wasn't able to generate a Gaussian splat for my use case with Nerfstudio. My next step was to generate mesh data with COLMAP. After some tinkering, I managed to extract a mesh from the dense point cloud, but it looked similar to what you see in Google Maps when you enable 3D mode. I also didn't like the large polygon count of the mesh COLMAP produced. That meant I would need to fire up Blender to clean the mesh, which is a step I'd rather avoid.
Good looking goussian splatting
Back to using LumaAI to generate the Gaussian splat from the point clouds. It is possible to download the point cloud generated by LumaAI, but I am not interested in only the point cloud. If you use the Splat viewer from LumaAI, you will see that there is more than only the point clouds. There is a skybox that makes the horizon look more interesting and better instead of a generic one. There is also a JSON file with many camera parameters that I do not understand. There is also a meta.json file that defines all the needed data for the viewer.
As mentaiont above, i will not use this rendere. So i had to find a different solution.
Implementation with Spark
So this is the reason why I started to create my own component. But I did not want to create a 3D rendering from scratch, which is why I was searching for a solution that implements the basic setup. There I found Spark, a three.js renderer for splats. Super nice and exactly what I needed.
Final Implementation
This will be my final implementation of Spark for this custom component. I will make sure to update this component later with additional functions I have planned. I may keep this version as its own custom component for " archive" purposes.
This splat was created with two drone flybys; in one flyby, the camera was pointed about 45° down. This made the positioning of the camera with colmap easier. The 3D Gaussian Splat was generated with LumaAI, but I am still searching for a different tool to handle the splat generation and calculation for private data reasons.
Switchinge splat viewr
While writing and checking the sources of different viewers, I came across the Super Splat Viewer. It is a lightweight WebGPU-based renderer Gaussian Splatting and looks promesing to swithch to this viewer.