Projects | ISLab, the University of Osaka

JST Top ASPIRE 'Deep Sensing: Jointly optimizing sensing and processing for CV/AI applications'

Fri, 03 Apr 2026 00:00:00 +0900

This collaborative research aims to generalize the “deep sensing” framework, which jointly optimizes both sensing and processing, and demonstrate its feasibility and efficiency across a wide range of computer vision and AI applications. Our team comprises leading research groups in Computational Photography and Computer Vision. We collaboratively develop a new computational camera system and apply it to various downstream computer vision tasks to achieve high accuracy, low data acquisition requirements, and reduced computational costs. Through our joint efforts, we aim to establish the “deep sensing” framework as a standard approach in the computer vision and AI communities. Additionally, by promoting collaborative and complementary research and encouraging researcher mobility, this project seeks to strengthen international research networks between the US/Canada and Japan.

Construction of an Integrated Analysis Platform for Vision and Omics

Thu, 02 Apr 2026 00:00:00 +0900

Construction of an Integrated Analysis Platform for Vision and Omics

In life science and medical research, both image data (Vision)—such as microscopic and pathological images—and omics data (Omics)—such as gene expression levels and protein abundance—are often used.

Microscopic and pathological images provide information about cellular and tissue morphology and spatial structure, whereas omics data comprehensively captures molecular-level states such as genes and proteins. These data types describe biological systems from different perspectives, and integrating them enables a deeper understanding of biological phenomena.

In recent years, advances in technologies such as spatial transcriptomics have made it possible to obtain gene expression data along with spatial information within tissues, leading to active research on the integrated analysis of image and molecular data. However, because image data and omics data differ significantly in format and dimensionality, effectively integrating them is not straightforward.

In this research, we develop methods for integrated analysis of image and omics data using image analysis techniques and machine learning. For example, by associating morphological features extracted from tissue images with gene expression data, we aim to deepen the understanding of cellular states and disease mechanisms. Through such integrated analysis of Vision and Omics, we seek to generate new insights that contribute to our understanding of diseases and to research on diagnosis and treatment.

Label-efficient Medical Image Analysis

Thu, 02 Apr 2026 00:00:00 +0900

Label-efficient Medical Image Analysis

We focus on medical images acquired for disease diagnosis and treatment—such as pathological images, microscopic images, fundus images, and endoscopic images—and conduct research on medical image analysis aimed at identifying lesion locations and classifying disease types using machine learning techniques.

With the advent of deep learning, recognition technologies have rapidly advanced, and high-accuracy analysis is becoming achievable when sufficient training data is available.

However, there is a major challenge in medical image analysis: it is difficult to prepare a large amount of high-quality labeled data (annotations). Labeling medical images must be performed by physicians or specialists, requiring substantial time and effort. For example, for pathological images, tumor regions must be precisely annotated at the pixel level, and cell-level annotation requires expert knowledge and long hours of work.

Against this background, our laboratory is working on label-efficient learning, which aims to achieve high-accuracy analysis with limited labeled data. Specifically, we develop methods such as semi-supervised learning, which leverages a small amount of labeled data together with a large amount of unlabeled data, and weakly supervised learning, which learns from coarse labels, and apply them to medical image analysis. Through these approaches, we aim to significantly reduce annotation costs while developing medical image analysis technologies that can be used in real clinical settings.

Optimization of Physical Encoders for Recognition Tasks and an Application to Pathological Diagnosis

Sat, 20 May 2023 00:00:00 +0900

Computational Optical Imaging for Endoscopic Surgery

Sat, 01 Oct 2022 00:00:00 +0900

Knowledge VQA

Wed, 01 Jul 2020 10:13:28 +0900

Visual question answering (VQA) with knowledge is a task that requires knowledge to answer questions on images/video. This additional requirement of knowledge poses an interesting challenge on top of the classic VQA tasks. Specifically, a system needs to explore external knowledge sources to answer the questions correctly, as well as understanding the visual content.

We created a dedicated dataset for our knowledge VQA task and made it open to the public so that everyone can enjoy our new task. The results are presented at AAAI 2020.

Representation of videos has been a major research topic for various deep learning applications including visual question answering. This is a challenging problem especially for tasks that involves vision and language and some researchers pointed out that deep neural network-based models mainly use natural language text but not the vision. We propose to use textual representation of videos, in which SOTA models for detection/recognition are used for generating text together with some rules. The results are presented at ECCV 2020.

We also work on question answering on art, which requires high-level understanding of paintings themselves as well as associated knowledge on them.

Publications

Noa Garcia, Chentao Ye, Zihua Liu, Qingtao Hu, Mayu Otani, Chenhui Chu, Yuta Nakashima, and Teruko Mitamura (2020). A Dataset and Baselines for Visual Question Answering on Art. Proc. European Computer Vision Conference Workshops.
Noa Garcia and Yuta Nakashima (2020). Knowledge-Based VideoQA with Unsupervised Scene Descriptions. Proc. European Conference on Computer Vision.
Noa Garcia, Mayu Otani, Chenhui Chu, and Yuta Nakashima (2020). KnowIT VQA: Answering knowledge-based questions about videos. Proc. AAAI Conference on Artificial Intelligence.
Zekun Yang, Noa Garcia, Chenhui Chu, Mayu Otani, Yuta Nakashima, and Haruo Takemura (2020). BERT representations for video question answering. Proc. IEEE Winter Conference on Applications of Computer Vision.
Noa Garcia, Chenhui Chu, Mayu Otani, and Yuta Nakashima (2019). Video meets knowledge in visual question answering. MIRU.
Zekun Yang, Noa Garcia, Chenhui Chu, Mayu Otani, Yuta Nakashima, and Haruo Takemura (2019). Video question answering with BERT. MIRU.

Australian History in Newspaper and AI

Wed, 01 Jul 2020 10:13:06 +0900

In collaboration with Prof. Fujikawa at Graduate School of Letters, the University of Osaka, we are working on exploring Australian history through public meetings, of which call for participation appears in newspapers from back then.

We explore ways to analyze such newspapers with state-of-the-art technologies in NLP to make OCR output better and to automatically detect/structure call for participation.

MLPhys: Foundation of Machine Learning Physics

Wed, 01 Jul 2020 10:12:37 +0900

Throughout its long history, physics has provided the most precise testing ground in the natural sciences, solving problems in various natural hierarchies in collaboration with the mathematical sciences.

On the other hand, the field of machine learning is a major research field, a mathematical system that forms the foundation of artificial intelligence and has seen explosive progress in recent years due to advances in computational science. We are launching the transformative research area “Machine Learning Physics” to integrate these two major fields.

For more detail, visit here.

CREST 3D Image Recognition AI for Cancer Diagnosis Support

Wed, 01 Jul 2020 10:08:48 +0900

Society 5.0 Projects

Wed, 01 Jul 2020 10:07:15 +0900

D3 Center, the University of Osaka is now working on Society 5.0 using information science and technology.

In the world of Society 5.0, innovations in IoT, Big Data, robotics, and AI will be part of everyday life, helping people lead active and high-quality lives, creating a super-smart society. This project encourages collaboration across projects and university organizations, thus promoting faster adoption of research results in real-world society.

Under this big project, we are working two sub-projects:

Sensing technologies that aggregate various types of information from publicly and ubiquitously available social probes including social networking services are essential for providing various services in the world of Society 5.0. We are working towards establishing a social sensing technology that can infer emotional states of people for timely services.

Future school technology for Society 5.0

Education everywhere is one of SDGs, and e-learning is one solution toward this. We collectively working on a broad range of technologies related to e-learning.

Esthetic Dentistry and Optical Analysis

Wed, 01 Jul 2020 10:06:58 +0900

There have been ever-increasing demands for esthetics in the oral cavity after tooth restorations. Especially in the anterior, functional restoration is not enough; restored teeth should have similar color tones and light transmissions to natural teeth because the appearance of restored teeth when exposed to light varies greatly depending on the material used for the crown restoration device and the abutment structure.

Therefore, by analyzing the optical properties of various dental restoration materials and dental tissues via optical simulation, the behavior of light in natural teeth and dental restorations is visualized and analyzed for esthetic tooth restorations.

Brain Pharmaceutics

Wed, 01 Jul 2020 10:06:11 +0900

Coming soon

AI Hospital

Wed, 01 Jul 2020 10:05:12 +0900

The University of Osaka Medical Hospital has launched Artificical Intelligence Center for Medical Research and Application (AIM), which supports physicians, nurses, and all the medical staff collaborating with medical information specialists and data scientists to boost the medical application of AI in daily practices of the hospital.

We are collaborating with AIM to provide cutting-edge technologies.

Ophthalmology and AI

In ophthalmology or any other departments, vessels in retinal fundus images provide rich information on the cardiovascular system of human bodies. We proposed a state-of-the-art method, coined IterNet for extracting vessels from retinal fundus images as in the (e) in the figure below.

On top of this technology, we also invented a new method for classifying vessels into artery/vein, which takes a two-step approach: We firstly segment vessels in input images using an IterNet-based method and then classify them into artery/vein with some post-processing.

Law and AI

Wed, 17 Jun 2020 23:02:32 +0900

In collaboration with Prof. Noriko Okubo at Graduate School of Law and Politics, the University of Osaka, we are studying to automatically evaluate how green laws are enforced in different countries.

Green laws’ participation principle consists of 1) the information access right, 2) participation in the policy decision process, 3) the judicial access; however, actual implementation varies country to country, and legal methodologies have been explored for evaluating their effectiveness. This work investigates legal evaluation criteria on the green laws’ participation principle, analyzes Japanese participation system’s pros and cons in a comparative perspective, and propose some recommendations to establish the environmental democracy.

The difficulty lies in how to automatically find out related legislations, cases, statutes, etc. in different languages. As the first attempt, we proposed a method for identifying the topic of such legal documents through analyzing citation networks in addition to classic topic modeling. The figure below shows citation networks among different types of legal documents (e.g., cases-prior cases).

Buddha Face and AI

Wed, 17 Jun 2020 22:52:41 +0900

In collaboration with Prof. Fujioka with Graduate School of Letters/School of Letters, the University of Osaka, we are attempting to create an AI for analyzing various aspects of Buddha faces in images.

Focusing on the face of the Buddha image, i.e., “Buddha face”, we analyze the characteristics of the style of each region, era, and author using statistical and machine learning approaches based on images and 3D geometric data, building a genealogy of Buddha faces. This is to realize style judgment based on the knowledge obtained from data, not based on the experience of art historians, which promotes the globalization of the Buddha statue research and also helps to identify the genealogy of Buddha faces propagated through the Silk Road, giving a new perspective on the spread of culture in Asia.

We have built several interfaces to browse through a large corpus of precious Buddha faces for facilitating annotations on the basic meta-data on the statues, which will then serve as a source to train more sophisticated models for analyzing them.

For example, we built a model that can embed various information on target entities (i.e., Buddha status), such as authors, eras, places, etc., into a vector representation of images and use them for other tasks like classification, through the model below.

Projects | ISLab, the University of Osaka

JST Top ASPIRE 'Deep Sensing: Jointly optimizing sensing and processing for CV/AI applications'

Construction of an Integrated Analysis Platform for Vision and Omics

Construction of an Integrated Analysis Platform for Vision and Omics

Label-efficient Medical Image Analysis

Label-efficient Medical Image Analysis

Optimization of Physical Encoders for Recognition Tasks and an Application to Pathological Diagnosis

Computational Optical Imaging for Endoscopic Surgery

Knowledge VQA

Publications

Australian History in Newspaper and AI

MLPhys: Foundation of Machine Learning Physics

CREST 3D Image Recognition AI for Cancer Diagnosis Support

Society 5.0 Projects

Social sensing for Society 5.0

Future school technology for Society 5.0

Esthetic Dentistry and Optical Analysis

Brain Pharmaceutics

AI Hospital

Ophthalmology and AI

Law and AI

Buddha Face and AI