<<Taikichiro Mori Memorial Research Fund>>

Graduate Student Researcher Development Grant Report

 

Research Project: An Imagination-based Image-Retrieval System and Its Application to Spatio-Temporal Analyzers with 5D World Environment Map

Project Researcher: Nguyen Thi Ngoc Diep

Affiliation:  1^st year Master’s student, Graduate School of Media and Governance, Keio University, Japan

 

Research Abstract

In this study, I develop a new imagination-based image-retrieval system with the integration of logical and correlation computing functions. The important aim is to realize image retrieval by reflecting imaginations appearing in a human brain. Furthermore, I will design and implement Spatio-temporal analyzers in 5D World Environment Map with this image-retrieval system. The 5D world environment map is a new cyber-physical environment for organizing, analyzing and retrieving context-dependently environmental information based on human imaginations with time-series multi-geographical views. This application provides users with 5-dimensional information overview: spatiotemporal 4 dimensions plus the multimedia data related to physical environment. The main features of the 5D world environment map system are: (1) to dynamically create various context-dependent patterns of multimedia stories of environment on spatiotemporal spaces according to a user’s imagination processes, and (2) to dynamically extract social-environmental rules by using feature selection and support vector machine methods. This application provides users visual tools to semantically analyze and discover new models of the ecological and global environment.

    The research theme is derived from the need of integrating information and communication technologies (ICT) and real world to solve multidisciplinary problems. Especially, since I am joining the Cyber Informatics program and Environmental Innovators course, I will have chances to reflect the environmental fieldwork activities into my ongoing research on imagination-based information retrieval methods and spatiotemporal 5D world map. As the basis of this study, I have presented 4 international conference papers（IASTED SEA 2010 USA, IADIS e-SOCIETY 2011 Spain, KCIC 2012 Indonesia, and EJC 2012 Czech Republic) from 2010 to 2012, SFC ORF 2010 and received many good assessments. I also received SFC Student Award for “Excellent Graduation Project” for the project named “5D World PicMap: Imagination-based Image Search System with Spatiotemporal Analyzers” (supervisor: Yasushi Kiyoki) on March 2011.

 

 

Research Activities and Results

 

Data Collection and System Implementation

I have implemented a new collaborative mashup system, named “WEE – Web-based Environmental Education, for global environmental monitoring research and education in which all participants could associatively organize multimedia data about physical environments and visually monitor various environmental topics based on time- series multi-geographical views. The main features of this system are: (1) collaborative multimedia database creation using “social sensors”, (2) information aggregation from Web resources, (3) content-based image and document retrieval, and (4) big-picture visualization with spatiotemporal view controllers and content-based image, document retrieval.

   Participative Database Creation Method using Social Networks

Primal social networks such as Twitter or Facebook were not designed for collaboration but their usability for conversational purposes has been varying from daily communication to crowd sourcing. Together with the development of information communication technology devices such as smart phones, nowadays SNS users can easily share information from anywhere and at anytime. Based on these technology backgrounds, I have designed a new usage of Twitter, one of the most popular SNS, as an educational tool for collecting environmental database while fostering students participating in classes’ activities.

 

Figure 1 shows how to create cross-topic shared databases with participation of students. In one class, a teacher divides students into some groups, then raises one topic for each group and gives one hash-tag (#topic_tag) to each group. Each student of each group will collect information relating to the given topic and upload data to Twitter (tweeting a tweet). The students can conduct these activities as group works. Besides, the students can go out for fieldwork activities with their smartphones to take photos and real time tweet some tweets. Or they can collect information from the Internet and tweet those by Twitter web interface. Every tweet is tweeted required to specify location, timestamp and include the given hash-tag, and other multimedia data such as photos or description text relating to the topic. One module in my system is implemented to get all tweets having a hash-tag (#topic_tag) and store data to one database table for one topic. These shared databases are used for spatiotemporal visualization onto the world map so that the users of this system can obtain the broader view of topics using others’ collected data by the time data are uploaded.

 

                      

Figure 1. Participative Database Creation using Twitter in WEE System

 

 

 

System Architecture

 

 

 

Experimental results via Educational Scenarios

       Shared-view interface of WEE system for environmental monitoring using collaborative database

 

           

Global Environmental Monitoring by Timeline by big-picture visualization with spatiotemporal view controllers of WEE System: (a) “desertification”, “flood”, “ocean pollution”, “deforestation”, and “wildfire” in July 5^th, 2012

 

 

      Fieldwork activities

Inspired by a report of the Centre for the Environment and Community Research (CECR), part of the Vietnam Union of Science and Technology Associations which stated approximately 80% of the lake surface area in Hanoi are disappeared during 50 years (1983-1996) and the health of lakes have been becoming worse. I have conducted a fieldwork at Hanoi, Vietnam.  This activity is also a required subject of Environmental Innovator course and the course provided me transportation financial support to conduct it. The vulnerability of the lakes is highly related to the urbanization and the lack of the public awareness. I mainly monitored the ecological conditions and socio-economic impacts of three important lakes in Hanoi: West lake, Hoan Kiem lake and Bay Mau lake. West Lake is the largest lake of the capital and a popular place for recreation with many surrounding gardens, hotels, villas and other entertainment centers. Hoan Kiem Lake is a lake in the historical center of Hanoi and also one of the major lake, which serves as a focal point for its public life. Bay Mau Lake is a lake inside the Thong Nhat Park, one of the largest parks in the capital. I also monitored some lakes, ponds at a rural area around Thuan Quang, Duong Xa village, Gia Lam district, Hanoi City, Vietnam. At the end of this fieldwork, I visited and studied about environmental policies at the Centre for the Environment and Community Research (CECR) and Vietnam Environment Administration, Ministry of Natural Resources and Environment. The following figure shows my routes and spots during my fieldwork.

 

 

Publications

 

         [Journal] D.T.N Nguyen and Y. Kiyoki, “An imagination-based query creation method for image retrieval,” Information Modelling and Knowledge Bases XXIV, p. 201, 2013.  

[Book] Kiyoki, Y., Sasaki, S., Nguyen, N. T., Nguyen, D.T.N., "Cross-Cultural Multimedia Computing with Impression-based Semantic Spaces", Conceptual Modelling and Its Theoretical Foundations, Lecture notes in Computer Science, Springer, pp. 316-328, 2012.

[Proceeding] Nguyen, D.T.N., Sasaki, S. and Kiyoki, Y.:  “WEE: A Mashup System Using Social Networks
For Collaborative Environmental Monitoring Education With Spatiotemporal Analysis And Visualization”, Proceedings of the IASTED International Conference Web-based Education (WBE 2013) February 13 - 15, 2013 Innsbruck, Austria.

[Submitting] Nguyen, D.T.N., Sasaki, S. and Kiyoki, Y.: “A Time-Constrained Image Retrieval System Towards Big Image Database Realization”,  IEEE International Conference on Image Processing (ICIP 2013),  September 15-18, 2013 Melbourne, Austrialia.

[Research Report] Nguyen, D.T.N., Sasaki, S. and Kiyoki, Y.: "5D World Environmental Map System with Spatiotemporal Analyzers for Collaborative Research and Education", Japanese Society for Information and Systems in Education (JSiSE) Research Report, Hokkaido July 2012.

[Poster] Nguyen, D.T.N., Sasaki, S. and Kiyoki, Y.: “Visualization and Analysis of Global Environmental Change with Multimedia Retrieval”, International Forum for Sustainable Asia and the Pacific (ISAP 2012) Student Poster Session.

Future Works

I have collected a lot of data after the fieldwork at Hanoi City, Vietnam. From now on, I would work to refine the data and improve WEE system to allow users to access to the data properly and helpfully.

 

Acknowledgements

I would like to express my gratitude to Taikichiro Mori Memorial Research Fund, which supports great financial policies to researchers like me to make progress on our ways.