The Basics of LIDAR
LiDAR (Light Detection and Ranging) is a remote sensing method that uses light in the form of a pulsed laser to measure distances to objects. Here’s the core process:
- Light Emission: The LiDAR sensor emits rapid pulses of infrared light.
- Reflection: These light pulses travel outward and bounce off objects in the environment.
- Time Measurement: The sensor measures the time it takes for the reflected light to return.
- Distance Calculation: By knowing the speed of light, the device calculates the distance to the object with high precision.
LiDAR on Mobile Phones
Mobile phone LiDAR systems are miniaturized versions of the technology used in other applications like autonomous vehicles or surveying. Here’s how it works on your phone:
- Integrated Sensor: A small LiDAR sensor is built into the phone, typically near the camera system.
- Point Cloud Creation: The LiDAR sensor rapidly scans the environment, capturing thousands of distance measurements. These measurements are used to create a “point cloud,” which is a 3D map of the surroundings composed of individual points in space.
- Data Processing: The phone’s processor analyzes the point cloud data to understand the shape, size, and distance of objects in the scene.
Key Differences from Traditional LiDAR
- Smaller Size and Range: Mobile phone LiDAR systems have a shorter range and lower power compared to larger LiDAR systems used in other applications. They are designed for indoor and close-range outdoor use.
- Integration with Other Sensors: Mobile phone LiDAR works in conjunction with other sensors like cameras, accelerometers, and gyroscopes to provide a more complete understanding of the environment.
Applications on Mobile Phones
- Augmented Reality (AR): LiDAR significantly enhances AR experiences by providing accurate depth sensing and object placement. This allows virtual objects to be realistically placed in the real world, interacting with the environment in a believable way.
- Improved Photography: LiDAR assists with autofocus, especially in low-light conditions, and improves portrait mode by accurately separating the subject from the background.
- 3D Scanning: Some apps use LiDAR to create 3D models of objects or spaces.
- Measurement Apps: LiDAR enables more accurate measurements in apps that use the camera to measure distances or dimensions.
Advantages of LiDAR on Mobile Phones
- Increased Accuracy: LiDAR provides much more accurate depth information compared to relying solely on cameras and software-based depth estimation.
- Improved Low-Light Performance: LiDAR works independently of ambient light, making it effective in low-light or even complete darkness.
- Faster and More Reliable AR: LiDAR enables faster and more stable AR experiences with more realistic object placement and interactions.
NERF
Neural Radiance Fields (NeRF) is a deep learning technique that uses neural networks to reconstruct a 3D scene from a set of 2D images, allowing the generation of photorealistic images from novel viewpoints, essentially creating a complete 3D representation of a scene based on multiple 2D perspectives; it’s particularly useful for applications like virtual reality, computer graphics, and 3D modeling where realistic rendering from any angle is desired.
Function: NeRF learns the geometry and appearance of a scene by analyzing multiple images taken from different angles, enabling it to predict the color and density of light at any point in 3D space.
How it works: A neural network is trained on the input images, where each pixel is associated with a 3D point in space and its corresponding color.
Applications: Novel view synthesis: Generating realistic images from viewpoints not captured in the original set of images
3D reconstruction: Creating a complete 3D model of a scene from 2D images
Augmented reality: Integrating virtual objects seamlessly into real-world scenes
NERF EXAMPLES:
https://www.matthewtancik.com/nerf
LUMALABS DREAM MACHINE: https://lumalabs.ai/dream-machine
RESEARCH
LIDAR on iPhone: https://developer.apple.com/videos/play/wwdc2020/10611/?time=1114%29,
Capturing DEPTH on iPhone: https://developer.apple.com/documentation/avfoundation/capturing-depth-using-the-lidar-camera
8 cool ways to use the LIDAR on iOS: https://www.zdnet.com/article/i-uncovered-8-cool-ways-to-use-lidar-on-an-iphone-and-ipad
LIDAR protocols: https://www.protocols.io/view/iphone-lidar-tutorial-yxmvm21w9g3p/v3
TOOLS
3D Scanner App
https://3dscannerapp.com/
Created with Record3D
Compression
A common compression format used by 3D scanner apps is “GLTF” (GL Transmission Format), particularly its binary version “GLB”, which offers efficient compression while maintaining interoperability with various 3D software and platforms across devices like Android and iOS; other formats like “USDZ” (for Apple devices) and “OBJ” are also frequently used depending on the specific app and desired level of detail.
GLTF/GLB: Considered a standard format for sharing 3D models on the web, offering good compression and wide compatibility.
USDZ: Primarily used for Apple devices, providing a compressed format with good quality for AR experiences.
OBJ: A widely supported format, often used for exporting detailed 3D models with texture information.
GAUSSIAN SPLATTER
Gaussian Splatting is a cutting-edge technique in computer graphics used for creating highly realistic 3D scenes from a set of images or videos. It’s gaining popularity due to its ability to render complex scenes in real-time with impressive quality.
1. Representing Scenes with 3D Gaussians:
- Instead of using traditional 3D models made of polygons or meshes, Gaussian Splatting represents a scene as a collection of millions of tiny 3D Gaussians.
- A Gaussian is a type of mathematical function that creates a bell-shaped curve. In 3D, it forms an ellipsoid (a 3D ellipse).
- Each Gaussian has properties like:
- 3D position: Where it’s located in space.
- Orientation: Its rotation in 3D space.
- 3D scale: Its size and shape (how stretched or compressed the ellipsoid is).
- Opacity: How transparent or opaque it is.
- View-dependent color: The color it appears to have from different viewing angles.
2. “Splatting” onto the 2D Image:
- To render an image of the scene from a particular viewpoint, these 3D Gaussians are “splatted” onto the 2D image plane.
- This means that each Gaussian is projected onto the 2D image as a 2D ellipse, with its size, shape, and color determined by its properties and the viewing angle.
- The contribution of each Gaussian to the final pixel color is weighted based on its opacity and how much it overlaps with that pixel.
3. Efficient Rendering:
- A key advantage of Gaussian Splatting is its efficient rendering process. Because the scene is represented by simple geometric shapes (ellipses), it can be rendered very quickly using standard graphics hardware.
- This allows for real-time rendering of complex scenes, which is crucial for applications like virtual reality, augmented reality, and interactive 3D environments.
Key Advantages over other methods:
- High Quality: Gaussian Splatting can produce very high-quality renderings with detailed textures and complex view-dependent effects.
- Real-time Performance: It enables real-time rendering, which is a significant advantage over other techniques like Neural Radiance Fields (NeRFs) that can be computationally intensive.
- Relatively Simple: Compared to some other advanced rendering techniques, Gaussian Splatting is conceptually simpler to understand and implement.
Applications:
- Virtual and Augmented Reality: Creating realistic virtual environments and AR experiences.
- 3D Reconstruction: Generating 3D models from photos or videos.
- Film and Visual Effects: Creating stunning visual effects for movies and TV shows.
- Gaming: Rendering realistic game environments.
This is probably one of the most beautiful examples of LIDAR scanning in super large scale with a clever usage of Gaussian Splatter
Basilica of St Peter Virtual 3D Tour
https://virtual.basilicasanpietro.va/en
Leica Lidar Scanners
https://shop.leica-geosystems.com/leica-blk/laser-scanners
ASSIGNMENT
13 • LIDAR Scan
Using any LIDAR enabled mobile equipment of your choice, create three accurate 3D LIDAR portraits of 3 different human beings. Export as both GLB and MP4 capture in high resolution. Upload the resulting files top DropBox.