Finland’s University of Oulu is renowned as a leading centre for the development of advanced communications technologies, and now, in collaboration with Mid Sweden University, it has revealed the first fruits of the Immerse project – specifically, a new type of immersive augmented reality (AR) application that allows tasks requiring special expertise, such as repairing a device, to be completed without an expert present.
Funded by Interreg Aurora and beginning in August 2024, the Immerse project looks to promote remote work and operation with immersive technologies to address the unique challenges and skills shortages experienced in northern regions. The project’s likely beneficiaries are organisations in the mining, forestry and manufacturing industries, whose business efficiency and work safety the new technology improves.
For Immerse, the University of Oulu developed an AR-assisted task guidance application, which works by having an expert create instructions for performing a task using a digital twin produced by 3D scanning technology of the target device and adding graphical annotations such as arrows and text boxes. Users can download the instructions they need from a library, which they can then follow step-by-step on the mobile device’s screen. The instructions are projected onto the item being worked on using AR technology.
A demo version based on the technology has been built, although development work is still ongoing, and currently, the application is used mainly to demonstrate the technology with simple tasks. However, the intention is to release an open source implementation in the future, which will include all the essential tools.
The researchers believe that through their work, anyone, in principle, can create a digital twin of the target, create instructions for maintenance or repair tasks, and use the mobile application to repeat them in a real environment. For example, changing a bicycle tyre could be easily accomplished with the application when the instructions are created by the bicycle manufacturer specifically for that model.
“AR-assisted guidance has indeed been studied worldwide, and there are also commercial applications, but they typically require the presence of an expert and interaction with the user, whereas our proposed solution allows the task to be performed independently without personal guidance,” said Janne Heikkilä, professor of machine vision and signal analysis at the University of Oulu.
“The application could be compared to a much more precise version of popular tutorial videos that many use when they need to do something beyond their expertise,” added Heikkilä.
The future focus will be on using artificial intelligence in creating instructions, making the work process “effortless” and reducing the amount of expert work. “The mobile application also requires a lot of development to make it user-friendly,” added Heikkilä. “The aim is to enable the use of AR or XR glasses, eliminating the need to hold the mobile device in hand.”
Immerse’s project manager in Oulu, researcher Janne Mustaniemi, said: “The project combines our university’s years of experience in image-based 3D reconstruction and the recent developments in the field, which have made digital twins even more accurate.”