The show must go on! Keeping 3D projection mapping on track

Machine vision expertise from STEMMER IMAGING helps disguise, one of the world’s leading providers of experience production software and media servers to deliver spectacular visual experiences at live events. The 3D camera/projector systems allow video content to be projected perfectly onto complex geometric surfaces such as stages, cars, buildings and domes.

STEMMER IMAGING takes care of the entire pre-configuration, including testing the camera systems, and handles the global logistics process all the way to the event locations. This ensures uniform quality standards and ready-to-use systems for the customer on site. The time and effort required to set up the projection is then significantly reduced.

The merging of creative talent with the latest projection technologies has brought numerous events to life with spectacular static and dynamic imagery to create simply breath-taking visual experiences.

The culmination of this process frequently requires distortion-free video projection onto complex 3 dimensional objects in the field of view with sub-pixel accuracy. The disguise software, media servers and services are used extensively across the live music, fixed installation, TV/Broadcast, film, esports and extended reality markets.

Examples include rock concert tours for U2, Beyoncé and Ariana Grande; theatre productions such as The Lehman Trilogy, The Cher Show and The House of Dancing Water; exhibitions such as the Renault booth at the Geneva Motor Show and other events such as the Apollo 11 live immersive experience to celebrate the 50th anniversary of the moon landings.

3D calibration

For many of these live events, both 2D and 3D projections will be required. In all cases, the projections need to be carefully aligned to accurately match the projection surfaces to ensure that imagery is not distorted in any way. 2D and 3D projections are usually calibrated separately.

For 3D projections, a 3D model representing the stage or show production environment is created in the disguise software and this needs to be accurately mapped to the real life environment. In most cases this requires the use of multiple projectors to achieve the coverage needed. It is rarely possible to build a stage set that does not have any physical disparities in the positioning of projection surfaces compared to the 3D model.

This is particularly true of touring productions where the set has to be dismantled then re-assembled at each new venue. The OmniCal calibration system uses two or more machine vision cameras mounted on the set rigging to image structured light patterns that have been projected onto the set.

Since the viewing angles and separation of the cameras is known, triangulation can be used to calculate the co-ordinates of each pixel in the image to create a 3D point cloud within the OmniCal calibration software. This allows correction for the position and orientation of projectors and projection surfaces, as well as the actual structure of the surfaces, achieving sub-pixel accuracy.

With respect to differences in projection surfaces, this translates to correcting for errors in the 3D model at large scale, as well as down to the millimetre range.

Disguise software actually includes a manual calibration system, which works very well, but time is a critical factor during a live show set up. Technicians working on the projections are competing for access to the set with a host of other personnel working on different aspects of the production. The manual process to calibrate a set of projectors and surfaces typically takes minutes per projector and has a roughly linear scale, meaning the larger the number of projectors the more people or more time would be needed to complete it. However, the machine-vision-based OmniCal system is completely scalable.

We can do calibrations in significantly less time because the point cloud generated can be used for all the projectors, enabling parallel calibrations rather than lots of individual manual processes. The OmniCal system enables higher quality calibration than a manually operated system. Not only that, it enables a significant new feature with the correction of differences between 3D models and the real world surfaces.

Peter Kirkup, Global Technical Solutions Manager, disguise

Perfecting the image

Disguise turned to STEMMER IMAGING for supply of the imaging components for OmniCal. Although stereo imaging for point cloud generation is a well-established machine vision application, there were some very specific technical requirements for OmniCal that STEMMER IMAGING was able to fulfil. The cameras are used to capture a short series of images of structured light patterns projected onto the scene to be calibrated. To prevent any areas of the scene being occluded, the sequences of patterns are generated from projectors in a variety of locations.

With OmniCal being used in a wide range of settings it is essential that the imaging system includes high quality optics with reliable behaviour with regards to focal length and distortion variance. GigE Vision cameras are required, since the positioning of the cameras will vary depending on the particular setting and they may be located at a significant distance from the control system. The GigE Vision standard allows high speed image data transmission in real time over distances up to 100 metres using standard Ethernet Cat 6 cables.

The cameras selected for OmniCal are 5 MPixel GigE Vision cameras from Allied Vision with Power over Ethernet simplifying set up. The cameras feature a high performance Sony Pregius CMOS sensor with a global shutter ensuring no smearing in the image as the projection patterns are switched.

Depending on the stage area set up, a mixture of 6 mm, 8 mm and 12 mm focal length C-mount lenses are used which match the resolution of the cameras and feature minimal and consistent distortion properties.

These lenses were chosen for their ease-of-use and reliability. This is of particular importance since the cameras are frequently installed by untrained staff in non-ideal locations such as theatres and arenas. Some support programming with the camera SDK was also provided by STEMMER IMAGING as well as help with integration with GigE Vision systems.

Value added by STEMMER IMAGING

Although the initial requirement was just for the supply of the cameras and lenses, disguise was able take advantage of STEMMER IMAGING’s extensive resources to configure each OmniCal equipment set in robust disguise branded flight cases for delivery directly to disguise customers anywhere in the world.

“The ability for us to outsource the complete equipment supply process including stringent pre-configuration and testing of components to a customer’s requirements brings considerable added value”, explains. Ed Plowman, Chief Technology Officer at disguise.

“Disguise systems are deployed globally and in general, there is very little time available during set-up in which to perform the calibration. It is essential, therefore, that every OmniCal camera and lens system performs in an identical fashion, so that once the camera is plugged in, we know it will work with the system, especially as installation is often carried out by staff who are not experts in machine vision.

Then all that needs to be done is to set up the camera parameters such as exposure time and focus at the particular event. STEMMER IMAGING have been able to address these requirements for us, which means that technicians on site can concentrate on the performance aspects of the projections.”

Ed Plowman, Chief Technology Officer at disguise

The OmniCal Wired kits come with 5MP Ethernet-based machine vision cameras, including filter and adaptor, and a selection of 6 mm, 8 mm and 12 mm lenses. Each camera is supplied pre-configured by STEMMER IMAGING, allowing for hassle-free use.
Source: disguise

STEMMER IMAGING services include:

The manufacturer’s camera casing is replaced with a camera casing with disguise branding. This shows that OmniCal is a disguise product and is particularly important for systems that are sold to rental suppliers for subsequent rental to end users as it ensures that the correct cameras are included in the package
The cameras are mounted on a tripod adapter ready for deployment at the venue
Each image sensor surface is cleaned; a filter fitted and images recorded to confirm that the sensor is completely clean before the camera is dispatched
All cameras are supplied with a set of matching lenses
All cameras are shipped with the original firmware version that was used to develop the system, even if a more recent version is available. This is important since any updated firmware may change the way the camera and software interact.
In addition, STEMMER IMAGING provided valuable support in both the programming of the camera SDK and the integration of the GigE Vision systems

In general, a minimum of four cameras is required per installation, and often eight are needed. Some scenes may require even more. “With such a diverse range of configurations and the need to deploy to anywhere in the world on demand, our value added services were a great match for disguise,” commented Mark Williamson, Managing Director of STEMMER IMAGING Ltd and Director of Corporate Marketing for STEMMER IMAGING AG.

In addition to pre-configuring the cameras, we are able to hold them in stock as kits complete with a selection of lenses, pre-packaged in flight-ready protective cases for immediate shipment to end users anywhere in the world, under instruction from disguise. Custom kits are also accommodated for a particular project with the appropriate number of cameras and the exact lenses required.

Mark Williamson, Managing Director, STEMMER IMAGING

Recalibrating after an earthquake

Calibration frequently throws up significant challenges, but not many as extreme as the two earthquakes of 6.4 and 7.1 on the Richter scale that occurred during one performance of the Apollo 11 immersive experience at the Rose Bowl in Pasadena CA! After the show, the OmniCal rig check was rerun, a few of the alignment points were adjusted and the calibration just snapped back into place despite the fact that everything had moved due to the earthquakes, including the projectors, cameras and projection surfaces.

Even the initial calibration for this event had been challenging. 3D projection was required onto the entire inner surface of the dome which stretched over the auditorium and onto two cylinders which formed the Saturn V rocket. With sixteen projectors used for the dome and another six on the cylinders, it was significantly faster to calibrate with OmniCal than any other method. Eight OmniCal cameras were used for calibration of both the dome and cylinders but with separate capture plans for each. In the original design concept on which the 3D modelling was based, the dome was completely smooth with no physical seams.

However, as the design process evolved, the dome construction changed to a collection of square panels that were suspended from pick-up points and were individually tensioned. This meant that there was more movement than expected in the dome due to tensioning adjustments as well as due to fluctuations in temperature and humidity. Nevertheless, the new modular structure of the dome made it easy to remodel using basic csv data taken from measurements into the corners of each panel using a laser measure on a tripod in the centre of the stage.

OmniCal was then be used to mesh deform the over simplified mesh made from this csv back to reality. Comparing mesh data from various parts of the project showed that some points of the dome had moved by over 30 cm at a time as the temperature changed. The regular recalibration required to adjust for these changes would not have been possible at such a scale without OmniCal.

A good working relationship

The imaging and logistics contributions from STEMMER IMAGING have been invaluable in ensuring that the OmniCal system has made a real difference to calibration for 3D projection mapping. By recognising and addressing the time constraints inherent in live experience production schedules, OmniCal has made it possible to incorporate even more complex mapped environments in an industry that rarely stands still.

Ed Plowman, Chief Technology Officer, disguise

Short company profile

The disguise technology platform enables creative and technical professionals to imagine, create and deliver spectacular live visual experiences at the highest level.

Combining real-time 3D visualisation-based software with high-performance hardware, disguise delivers challenging creative projects at scale and with confidence. Its new award-winning Extended Reality (xR) workflow is empowering users to bring to life immersive visual experiences that inspire and engage remote audiences everywhere.

disguise xR has already powered immersive real-time productions for music artists such as Katy Perry and Billie Eilish, enterprise businesses like SAP and Lenovo, educational institutions like the University of Michigan, broadcast TV shows like MTV Video Music Awards and America’s Got Talent, commercial brands like Nike and Under Armour and many other virtual experiences in more than 35 countries.

With an ever-increasing global partner network and working alongside the world’s most talented visual designers and technical teams in live events, TV broadcasts, films, concert touring, theatre, fixed installations and corporate and entertainment events, disguise is building the next generation of collaborative tools to help artists and technologists realise their vision.

Looking for advice?

If you have any questions regarding your future projects, you can always contact our experts at STEMMER IMAGING. Of course, we also offer a wide range of cameras, optics, illumination and software for your individual machine vision solution.

STEMMER IMAGING

MACHINE VISION APPLICATIONS

Successful examples from many industries