Building conceptual media and application systems

* This laboratory has been completed.

This project aims to build a concept media base that contains images and audio information as well as text information for building an application system that has flexible functions according to specific situations similar to the way in which humans work. We welcome to this consortium anyone who belongs to an organization that deals with languages, sensory data and multi-modal information. An associative concept dictionary that resulted from the first term is available (on a license basis).nn

Number of participating organizations2


The most difficult challenge at present in building advanced information processing systems similar to humans is how to handle “situations”. In other words, how can systems handle context and this requires systems that can flexibly handle situations. However, humans receive information in various modes using sensory organs and integrate all of the information to understand the context of a situation. Humans determine the meaning of not only words but also visual and auditory information based on these situations. In this project, a multi-modal concept media base will be built to seek the mechanism of understanding the situations and meanings of multi-modal information and how it can be applied to various systems.

Results of the first term of the Consortium

This laboratory built an associative concept dictionary of tens of thousands of words by conducting a large-scale association experiment using 800 basic nouns selected from textbooks of Japanese language for elementary schools. Recently another 200 words were added. Concepts include adjectives that express attributes or senses.

How to proceed during the second term Consortium project

The key for storing text, images and sound in a database is semantic information. Various bits of information are stored for each search centered on semantics. The basis of this is concepts. Therefore, vast amounts of conceptual information to be used by humans are first investigated and arranged as an associative concept dictionary. The basis of this is the associative concept dictionary made during the first-term Consortium project.
For the current second term project, information that expresses sensual or physical concepts using adjectives will be added to the associative concept dictionary. Distances between concepts are defined to allow easy handling and to express relationships among all concepts in terms of multi-dimensional distances. Then image and aural information is collected in a form, for example, of conversations between two or more people and bits of the conversation are tagged for associating with conceptual information. Once this tagging work is completed, easy searches for facial expressions, behaviors and emotional expressions contained in aural information will be possible.


  • Making an associative concept dictionary
  • Sensitivity database
  • Neural network system that handles sensitivity information
  • Status model on acquisition of command of language
  • Prosody analysis of voice of conversations or learning of Japanese
  • Recognition of music

Researchers, managers and professors

Shun IshizakiDirector Professor, Faculty of Environment and Information Studies
Jun Shigematsu Professor, Faculty of Policy Management
Shigenori Tanaka Professor, Faculty of Environment and Information Studies
Masahiro Fukaya Professor, Faculty of Policy Management
Mutsumi Imai Associate Professor, Faculty of Environment and Information Studies