Computational imaging for cultural heritage
Machine vision has established itself as an effective method for quality assurance in production and the detection of errors and is used primarily for this purpose. ARTMYN proves that this technology is also suitable for applications in completely different areas: The Swiss company uses vision systems based on STEMMER IMAGING components to open up new insights into the world of art.
"Re-discover Art" is ARTMYN´s slogan, and what the Swiss company, born at the Swiss Federal Institute of Technology in Lausanne and based in Saint-Sulpice near Lausanne, has created is in fact a completely new, previously unimagined possibility to give insights into the world of art. With its 5D technology, ARTMYN allows art lovers to discover visual artworks in a completely new way and with unprecedented levels of detail.
With only the length, width and height of an artwork as input, ARTMYN automatically captures at high resolution the 3D surface topography as well as two additional components describing the material reflectance at each pixel for interactive re-illumination. As a result, ARTMYN technology creates interactive 5D images and movies that allow a special and emotional experience when viewing an artwork on screen. "Our scanning process captures something we see as the DNA of an art object by highlighting its unique qualities and presenting them in an innovative way," said Loïc Baboulaz, one of ARTMYN's founders and Chief Technical Officer.
Thousands of images combined
The characteristics of a painting or sculpture are captured by a special scanner developed by Baboulaz and his colleagues in 2016. Depending on the physical size of the artwork, a few hundred up to a hundred thousand images are taken and sent to the ARTMYN data processing centre. Different light sources and spectra are used during this process, including ultraviolet and infrared illumination, in order to be able to capture the individual properties exactly.
ARTMYN's scanners travel around the globe to take pictures of artworks and calculate their digital twins. Image source: ARTMYN
During the scanning process different light sources from the visible, the ultraviolet and the infrared spectrum are used. Image source: ARTMYN
In the subsequent data processing step, ARTMYN's proprietary algorithms bring together data that often spans more than 2 terabytes, combining the information from the large number of images taken to extract the relevant features of the artwork. "In this way, the system calculates, among other things, the 3D topography, the surface reflection and details of the colors used," explains Baboulaz. "The result is then formatted for online visualization and sent to our web server."
In one respect, the requirements for the vision system used in this application differ significantly from industrial applications: The captured images do not have to be processed in real time. "For us, above all, the reliable processing of the entire data set acquired for an artwork is important," emphasizes Baboulaz. As a demanding part of the task besides the actual image acquisition, he mentions the development of special algorithms that are able to extract the relevant information effectively from the very large amounts of data.
This requires a great deal of expertise – which is indeed available: The ARTMYN team can rely on decades of experience in computing and computer vision in science and industry. According to Baboulaz, the Swiss company also obtained further support for specific questions about image acquisition from relevant experts: "In 2016, we approached STEMMER IMAGING with a long list of stringent requirements to find the right camera setup for our scanners. Key elements were the ability to capture at high resolution and high-speed and also to offload the acquired data from RAM memory to disk. Since that time, STEMMER has provided invaluable support and advice to correctly adapt camera technology to our scanners. We always had a very constructive exchange on technical issues and could also rely on the commercial advice of our partner."
During the initial development phase of the system, ARTMYN worked with a consumer-grade SLR camera. "The quality of these images was excellent, but the life expectancy of such consumer cameras is only about 200,000 images," recalls Baboulaz. "As up to 100,000 images can be taken when shooting a single large artwork, we would have needed a new camera after every second artwork. In addition, all these required images for one object could take days to be fully captured."
The artificial change of the lighting direction makes the materiality of an artwork even more apparent and improves the recognition of image details. Image source: ARTMYN
Art lovers can look at each brush stroke individually and from every direction. Image source: ARTMYN
The solution that is now available from ARTMYN is suitable for many different applications. Thanks to the unique streaming technology, the digital twin of an artwork can be accessed and visualized online and from any device connected to the Internet. In particular, the interactive representation of the 5D images in real time allows a special kind of art enjoyment. The technology also simplifies the creation of interactive, audiovisual guides without incurring costs, risks or constraints, e.g. when a camera hovers over a fragile piece of art.
The option of generating e-catalogs is particularly interesting for museums, galleries and auction houses, Baboulaz emphasizes: "Both our interactive viewer and videos with guided tours can be elegantly embedded into such electronic catalogs, which can be created in an interactive way and designed very appealingly using our technology.“
The ARTMYN system also offers unique features for security and sales. Thanks to the high level of precision of the captured image and to its 5D content, an individual fingerprint of an artwork can be produced: "It is practically impossible to falsify a piece of art once it has been scanned by us," emphasizes Baboulaz.
This feature can also be used for automated damage detection, such as scanning an artwork once each time before and after lending it to a museum for example, Baboulaz points out: "Our algorithms objectively compare the first and second data set and provide a heatmap of potentially damaged areas, which can also serve as the basis for an insurance claim." In contrast to the human eye, the system reliably detects even the smallest damage.
A digital condition report is also easy to implement by allowing experts to log their remarks and appreciation of the state of an artwork directly in its 5D file. Such a status report can then be shared digitally, interactively and online with potential prospects, e.g. to discuss details of a sale.
As another option, the ARTMYN CTO mentions an art passport that can be created for each artwork as a secured, digital document for the owner and includes private access to the digital twin and its fingerprint. Such a passport also encapsulates all relevant documents such as a condition report, a certificate of authenticity or the history of the artwork, and thus prove both the quality and the authenticity of the associated artwork.
Thanks to streaming technology, the digital twin of a piece of art can be viewed and visualized online and from any device connected to the Internet. Image source: ARTMYN
A new look at art
Baboulaz is happy to tell that there is huge interest in this new way of looking at art: "Museums and foundations digitizing their collections also like to introduce their art treasures to a broader and younger audience. With our system, this is possible in a novel and attractive way: For visitors it is particularly interesting to see the actual artifact behind a protective glass under a given lighting, and at the same time view the digital twin, which can be manipulated as desired. In addition, interactive guides provide a fantastic way to visually expose both the artwork itself and the artist's technique. "Against this background, it is not surprising that the ARTMYN solution quickly became established in the art scene. "Without the competent support of STEMMER IMAGING during the development phase of our system, that would not have succeeded so quickly," emphasizes Baboulaz.
Born at the Swiss Federal Institute of Technology in Lausanne, ARTMYN offers a new generation of technological tools and services for the art eco-system. Combining thousands of photographs captured with different light sources and spectrums – including UV and infrared lights, ARTMYN generates interactive 5D images and movies providing an emotional experience on screen.