Research Report — Taikichiro Mori Memorial Research Fund
Academic Year 2017

Cross-Domain Information Retrieval for Augmented Reality Interactions

Iori Mizutani (Cyber Informatics)

Master's programme, 2nd year

Keio University, Graduate School of Media and Governance

Background

Augumented Reality (AR) technology has come into play on the shopfloor in recent years. Major industrial machine suppliers are seeking for a new integration with AR technology that improves their existing routine operations in manufacturing execution systems, especially after Microsoft HoloLens made its appearance on the market. Meanwhile, Semantic Web lead by W3C is a concept of database creation on top of the web object ontology. Web resources are described in RDF (Resource Description Framework) and each domain has its vocabulary (sometimes called "predicate") to represent a relationship between one entity and another. This relationship (subject-predicate-object) is called Triplestore, which also is a query format to retrieve the information from the Semantic Web ontology. The Semantic Web is suitable for providing a backend system for an AR interface because an AR intergration of industrial automation usually requires the access to cross-domain information to interact with physical onjects (e.g., robots or item boxes), which are not often managed in a single system. With the unified backend system based on the Semantic Web, there is no restriction in the extension of the domains as long as the vocabulary is defined for the new domain to be appended. This research proposes a Semantic Web based framework for AR interactions with physical objects.

Contribution

The objective of this research is to present an possible AR integration of particular industrial operations, such as item management or manufacture execution. In my other project and my master thesis, I have studied an RFID-based automatic identification system for item management and logistics. One of the next goal is to to provide accesses to the accumulated information captured by the automatic identification system. The accumulated information consists of IDs and properties of objects, that belong to different information system.

One approach was to combine various kinds of data to produce integrated information for physical objects and to create positive feedback loops to enhance operation processes in the industry. [Figure 1] A control interface of the manufacturing robot arm, for example, is a possible “property” of the physical object “robot arm” to be accessed in the AR interface to help the workers on site. One of the challenges on the shop floor is how to selectively choose an interested information (“a control interface” of “a robot arm”) in the information platform. The resolution and retrieval of the information can be queried by the ontology (semantic database) to maximize the benefit of digitalization for supporting human workers of different roles to execute the processes. [Figure 2]

With this platform, users can also examine the feasibility of installing a new asset in a machine that guarantees the asset ability to accomplish a task in the real-world setting. AR interface is utilized to provide access to the corresponding information management systems and to show role-relevant information. Virtual copies of any physical assets available in this system can be explored virtually even if they are currently not physically available in the inventory. [Figure 3]

This platform focuses on providing meaning to data by establishing a shared understanding of the world through ontologies of interlinked terms, concepts, relationships, and entities. A semantically enabled information source is based on models describing the entities that constitute a part of the domain, and the relationship that such entities hold among them. The hierarchical nature of semantic models allows obtaining information at different levels of abstraction. Since such models have logic built upon it, queries can be made to a specific type of asset (e.g., a gripper) and its attributes, but it is also possible, and straightforward to move a level up on the hierarchy to get information from the parent category that a particular asset belongs to (e.g., robot end effector for a gripper), or move horizontally to obtain information of the entities that a specific asset is related to (e.g., robot arm that a gripper is attached to). To demonstrate the benefit from having an AR interface, I also implemented a control interface of the robot arm. [Figure 4] In this example, the user can obtain the robotic controller end-point by querying the context: the location, the operator, and the task that the asset is capable of.

The proposed platform doesn't only help the workers to attain the interface of the asset but also the cross-domain information. In the stock administration operation in the inventory management, sometimes the workers have to consider the deterioration or the expiration date of products, and often they must be consumed or sold earlier than the other products that last longer. However, at a limited space in the warehouse, the boxes, for example, are not sorted in order of the expiration date. [Figure 5, Figure 6] Then the platform can help workers to provide those data combined with the automatic identifcation of objects. The demo enables users to provide the intuitive information through the fixed station that is constatnly tracking the object movement.

Figure 1. The AR platform to unify the access to the control of industrial assets	Figure 2. A unified access to integrated information from cross-domain systems
Figure 3. An example of virtual invenotry: compatible grippers for the robot arm interacting with	Figure 4. Dynamic control of the robot arm with the AR interface
Figure 5. Experimental setup for individual object setup with AR interface	Figure 6. Identification of the object metadata (expiration date) and providing it through the AR interface

Activity

25.5.2017 CJKT Auto-ID Workshop in Taipei, Taiwan

I have presented this research with a title "Cross-Domain Information Access through AR Interface" in the Auto-ID joint workshop. The workshop 38 presentations, and I have received the Best Presentation Prize.

23.10.2017 IoT 2017 (The 7th International Conference on the Internet of Things) in Linz, Austria

I have submitted a conference paper "Intuitive Interaction with Semantics using Augmented Reality - a Case Study of Workforce Management in an Industrial Setting" in collaboration with Siemens and it was presented in IoT 2017.

23.2.2018 Presentation at ETH Zürich Distributed Systems Group, in Zürich, Switzerland

I have presented this work to the DGS group in ETH Zürich to discuss possible extension of this project.