Artificial Intelligence (AI) empowering the 3D Scanning revolution

3D scanning for motion pictures is undergoing exponential growth. What was once a niche technique reserved for big-budget productions is now becoming a critical tool employed across entire productions.

Key Applications Driving Adoption The increasingly widespread adoption is driven by the technology’s versatility and its ability to solve complex production challenges efficiently and cost-effectively. However this new found level of adoption comes with a new set of challenges, challenges that we are using AI to help us solve.

Let’s quickly delve into some of the key applications that are driving this adoption:

1.Pre-visualization: In the past, pre-visualization was often limited to rough sketches or basic 3D models. Now, production teams routinely use highly detailed 3D scans to support this crucial phase. This allows for virtual shot setup, remote collaboration, and critical decision-making long before a single crew member sets foot on location. For instance, directors and cinematographers can now explore virtual representations of actual locations, experimenting with camera angles and lighting setups from the comfort of their offices. This not only saves time and resources but also fosters a more collaborative and iterative creative process.

2. VFX Integration: The seamless integration of visual effects with live-action footage has always been a challenge in filmmaking. 3D scanning provides the foundational geometry for this integration, enhancing the overall production value. By capturing precise, real-world data, VFX artists can create digital environments and elements that match perfectly with the physical sets and locations. This level of accuracy not only improves the final look of the film but also reduces the time spent on manual adjustments and corrections in post-production.

3.Logistical Problem-Solving: Some of the most significant benefits of 3D scanning come from its ability to overcome logistical hurdles and reduce costs. We’ve seen this in action on projects like “Bridgerton” and the Netflix show “1899.” For these productions, 3D scanning allowed for the creation of highly detailed virtual backgrounds, eliminating the need to move large crews to difficult-to-access locations. This not only saved on travel and accommodation costs but also reduced the production’s carbon footprint. Moreover, it allowed for greater creative flexibility, as scenes could be “shot” in locations that would have been impractical or impossible to use traditionally.

4.Set and Prop Design: The marriage of traditional craftsmanship with digital technology is perhaps best exemplified by our scans of set and prop design. 3D scanning of props, such as the intricate carriages in “Bridgerton, or the stunning sets on Napoleon” allows for a unique blend of authenticity and efficiency. Designers can create physical props with all the detail and artistry of traditional methods, then scan these items to create digital versions. These digital assets can then be easily modified, replicated, or integrated into visual effects shots. This approach preserves the authentic look and feel that comes from real-world craftsmanship while leveraging the flexibility and cost-effectiveness of digital workflows.

The result of these applications is a dramatic shift in how productions approach visual storytelling. We’re seeing a move from hundreds of scans to thousands of scans per production, a trend that’s likely to continue due to its demonstrable benefits in both creative and budgetary terms.

As the demand for 3D scanning in film production has grown exponentially, so has the volume of data generated by these scans (up to a quarter of a petabyte per production). This presents both an opportunity and a challenge: while more data means greater detail and accuracy, it also requires more processing power and time. This is where Artificial Intelligence (AI) steps in, transforming the way we handle and utilize this wealth of information.

Below are a few examples of how its helping us now:

1. Automated Data Processing: One of the most time-consuming aspects of 3D scanning has traditionally been data processing, particularly color correction. With thousands of high-resolution images per scan, ensuring color consistency was once a painstaking manual process. Now, our machine learning algorithms can handle this task with remarkable speed and accuracy. By automatically detecting color charts in our datasets, we ensure consistent color accuracy while dramatically reducing manual labor. This not only saves time but also improves the overall quality of the final product, as the AI can process colors with a level of consistency that would be challenging for even the most skilled human operator.

2. Enhanced Geometry Reconstruction: Accurate geometry is the backbone of any 3D scan, but achieving this accuracy often required multiple scans from various angles. The Ai-driven methods we use now allow for much more robust reconstruction. In the near future we will be rolling out our own solutions that achieve equal quality with far fewer cameras or image inputs. These increased efficiencies don’t just save time and resources; they open up new possibilities for scanning in challenging environments where setting up too many cameras might be impractical.

3. Object Detection and Removal: One of the most exciting applications of AI in our workflow is its ability to recognize and remove unwanted elements from scans. Imagine scanning a period street scene, only to find modern cars or signage in the shot. In the past, removing these anachronisms would have been a painstaking process. Now, AI is helping us identify these elements and seamlessly remove them, helping to create a clean scan for production.

At the forefront of AI-powered solutions are entirely new rendering techniques known as radiance fields. These advancements in rendering and creating 3D scenes are not just incremental improvements; they represent a paradigm shift in how we capture, process, and visualize 3D environments.

1. Radiance Fields: Unlike traditional 3D scanning, which focuses on geometry and surface properties, the radiance fields process is more focussed on capturing the impression of how the light dynamically exists in the scene. This AI-powered technique allows for incredibly realistic faithful rendering of captured scenes, its far better at preserving the natural interaction of light with surfaces in a way that was previously impossible to achieve without extensive manual work. Equally impressive is the way these new techniques are capable of rendering complex textures and fine detail that have not really been possible before now.   The implications for filmmaking are profound. Imagine being able to capture a location not just in terms of its shape and colour, but in terms of how the light dynamically existed at the point of capture. This level of fidelity opens up new possibilities for virtual production and more faithful volumetric recordings of the sets, scenes and environments, allowing directors and cinematographers to work with virtual environments that maintain maximum visual fidelity to the real-world location it was captured from. Effectively promising to make the 3D scan a new form of photographic tool for the camera department.

2. Improved Real-time Visualization: One of the challenges in virtual production has been the trade-off between visual quality and real-time performance. AI-driven rendering techniques like Gaussian splatting are promising to change this equation also. These methods although very new are able to enable high-fidelity, real-time visualization, with a fraction of the demand on graphics processing. For filmmakers, this means being able to achieve a very high degree of fidelity between a real life set or environment and its virtual production counterpart. Capturing scenes in this way has great creative potential but could also offer a new way to mitigate against the impact of costly reshoots.

3. Integration with Professional Workflows: We are keenly aware that even the most advanced technology is only useful if it can be seamlessly integrated into existing production workflows. Teams like Jawset Postshot are at the forefront of doing this for radiance fields.

Some of these new techniques are barely a year old and Jawset are already bringing these tools to the professional industry. Their focus is on making sure these techniques sit inside a professional workflow that become a robust and dependable solution. Our excitement for these new techniques is equally tempered by our responsibility to our clients to make sure we only bring solutions to the table that work as expected. So we have been keen to share as much front line insights with Jawset as we can and help them continue to make the much needed industry solution for this exciting new era of  3D scanning. We see it as critical to help create a bridge between the cutting-edge capabilities of AI-driven rendering and the practical needs of film production.

“Our collaboration with Visualskies provides great insights into the production pipeline, allowing us to focus development of Postshot on what matters to users most.”

Jascha Wetzel/Jawset Postshot

The advent of radiance fields and related technologies is not just about improving visual quality; it’s about giving filmmakers new tools to tell their stories. It’s about creating virtual environments that are indistinguishable from reality, allowing for more creative freedom that was once limited by physical and budgetary constraints.

As we delve deeper into the world of AI-enhanced 3D scanning, we’re uncovering a fascinating symbiosis between the film industry and AI research.

In 2023 we began a working relationship with Bournemouth University and specifically their AI for Media department. We were keen to explore how the data we capture could be used to support further innovation in the AI field.

Our thesis being that motion picture industry’s insatiable demand for high-quality 3D scans is likely an invaluable resource for AI researchers, which in turn could lead to better tools for the industry. Its in our interest to help drive this virtuous cycle on behalf of the industry. Below are a few key areas where the data has value.

1. High-Quality Training Data: The motion picture industry produces some of the highest quality 3D scans available, often surpassing the fidelity required in other industries like architecture or manufacturing. This is because film production demands an unparalleled level of detail and accuracy to create convincing visual effects and immersive environments. For AI researchers, this wealth of high-quality data is invaluable. Machine learning models are only as good as the data they’re trained on, and the film industry is inadvertently creating a goldmine of training data. Every production that uses 3D scanning is contributing to this pool of high-fidelity, diverse data that can be used to train more sophisticated AI models.

2. Quantity Meets Quality: The sheer volume of scans produced for films is staggering. With up to 3,000 scans produced per production, the industry is generating vast datasets that rival or exceed existing 3D scan libraries. This combination of quantity and quality is unprecedented and provides an unparalleled resource for AI research. To put this in perspective, many existing 3D scan libraries top out at around 10,000 to 15,000 scans in total. A single film production can generate a dataset nearly a third of that size, and with much higher quality and diversity.

3. Reciprocal Benefits: By making this data available to researchers (with appropriate safeguards for intellectual property and privacy), the industry can drive innovations that directly benefit future productions. The AI models trained on this data can lead to more efficient scanning processes, better reconstruction algorithms, and new creative tools for filmmakers. This creates a feedback loop where the industry’s output feeds research, which in turn produces better tools for the industry. It’s a win-win situation that accelerates progress in both fields.

4. Efficiency Through Quality: Our collaboration with university researchers has yielded some exciting insights. We’ve found that high-quality scan data can train AI models that perform on par with or better than those trained on much larger, lower-quality datasets. In one instance, a model trained on just a fraction of one of our datasets achieved results comparable to models trained on much larger public datasets. This efficiency in training has significant implications. It means we can develop more sophisticated AI tools more quickly, and these tools can be more finely tuned to the specific needs of the film industry.

We’re looking at not just using AI to improve our scanning processes; we’re contributing to a broader ecosystem of research and development that promises to push the boundaries of what’s possible in both AI and filmmaking.

For the latest updates, behind-the-scenes insights, and a deeper dive into the world of visual effects and 3D scanning, we invite you to connect with us on our social media platforms. Follow us and be part of our community where we regularly share our experiences, challenges, and triumphs in the ever-evolving realm of film and VFX production. Don’t miss out on the opportunity to see how technology and creativity come together to create cinematic magic. 👉 Follow us on Instagram 👉 Like our Page on Facebook 👉 Subscribe to our YouTube Channel 👉 Connect with us on LinkedIn