.png)
November 28, 2024
The rendering of 3D scenes for virtual reality, games, or even product design is experiencing rapid development. An important aspect is "Novel View Synthesis," which allows the generation of new views of a scene from a limited number of images. An established method for this is 3D Gaussian Splatting (3DGS), which represents the scene with a collection of Gaussian primitives. However, this method has limitations, particularly in the representation of sharp edges and flat surfaces. This is where the new 3D Convex Splatting (3DCS) method comes in.
3DGS uses Gaussian distributions to model the spatial extent and color of points in the 3D scene. While this works well for smooth transitions and organic shapes, the method reaches its limits with hard edges and flat surfaces. For example, to represent a sharp edge, 3DGS requires a high number of Gaussian primitives, which increases memory requirements and processing time. The situation is similar with flat surfaces, which are difficult to accurately represent due to the diffuse nature of the Gaussian primitives.
3DCS, on the other hand, uses three-dimensional, smooth convex shapes as basic primitives. These offer greater flexibility in shaping and allow for more precise representation of edges and surfaces. As a result, 3DCS can reconstruct scenes with significantly fewer primitives, reducing both memory requirements and processing time. Another advantage of 3DCS is the ability to represent both hard and soft transitions with the same primitives by adjusting the smoothness (δ) and sharpness (σ) parameters.
The implementation of 3DCS is based on an efficient, CUDA-based rasterizer that projects the convex shapes onto the image plane. A Signed Distance Function is calculated for each shape, indicating the distance of a point to the surface of the shape. This information is used to determine the color and transparency of each pixel in the rendered image. The entire process is differentiable, meaning that the parameters of the convex primitives can be optimized through machine learning to maximize the match with the training images.
In benchmarks on established datasets such as Mip-NeRF360, Tanks and Temples, and Deep Blending, 3DCS shows superior performance compared to 3DGS. The results show improvements of up to 0.81 PSNR and 0.026 LPIPS, two common metrics for image quality assessment. At the same time, 3DCS requires fewer primitives and maintains high rendering speeds. A lightweight version of 3DCS achieves comparable results with only 15% of the memory requirements of 3DGS.
3D Convex Splatting represents a promising advance in the field of 3D scene reconstruction. The ability to represent complex geometries with high accuracy and efficiency opens up new possibilities for various applications. From the creation of realistic virtual environments to the generation of training data for autonomous systems, 3DCS could make an important contribution. The research results suggest that 3D Convex Splatting has the potential to establish itself as a new standard for high-quality scene reconstruction and Novel View Synthesis.
Especially for companies like Mindverse, which specialize in AI-powered content creation, 3DCS offers interesting possibilities. The technology could be integrated into the existing product range to increase the quality and efficiency of 3D model generation and offer customers new, innovative solutions. The development of customized AI solutions, such as chatbots, voicebots, and AI search engines, could also benefit from the integration of 3DCS.
Held, J., Vandeghen, R., Hamdi, A., Deliege, A., Cioppa, A., Giancola, S., Vedaldi, A., Ghanem, B., & Van Droogenbroeck, M. (2024). 3D Convex Splatting: Radiance Field Rendering with 3D Smooth Convexes. arXiv preprint arXiv:2411.14974. https://convexsplatting.github.io/