The research group of Yusuke Matsui, a specially appointed researcher at the National Institute of Informatics, has collaborated with Dwango Co., Ltd. and the University of Tokyo to perform "clustering" on about 10 billion big data at high speed with a small memory capacity. Developed a high-quality method.This enables big data clustering processing even on a personal computer with general capacity.

 AI research processes huge and complex data (big data).Clustering is the basic work of data processing in which similar data are grouped together from a large amount of data, but when the data becomes huge, the conventional method slows down the processing speed and requires a large amount of memory.It was difficult to execute clustering with a single general personal computer, and distributed parallel processing using a large number of servers was required.

 This time, the data was compressed by a new technology (direct product quantization), and it was possible to express it with less memory (100 to 4000 times memory saving) than the conventional method.Next, the process of grouping similar data and averaging the groups is repeated for this compressed data, but in addition to the technology proposed in the past, high-speed clustering (10 to 1000) is performed by the newly devised efficient averaging technology. Double the speed) has become possible.

 As a result, the process of classifying 1 million images into 10 types of groups can be executed in about 1 hour with one computer (memory capacity 32GB, number of CPU cores 4) (about 300 computers are required with the conventional method). .. The process of classifying 10 billion image data into 10 types could be executed in about 12 hours.

 As a result, a huge amount of image data such as social media can be easily processed by a general personal computer.Since it will be easier for general engineers and researchers to handle big data, it is expected to be used in a wide range of fields such as the development of artificial intelligence (AI) that applies deep learning.

Tokyo University

Established in the 10th year of the Meiji era.A university with the longest history in Japan and at the forefront of Japanese knowledge

The University of Tokyo was established in 1877 (Meiji 10) by integrating the Tokyo Kaisei School and the Tokyo Medical School.Since its establishment, it has developed education and research in a unique way in the world as a leading university in Japan and an academic center for the fusion of East and West cultures.As a result, many human resources have been produced in a wide range of fields, and many research achievements […]

University Journal Online Editorial Department

This is the online editorial department of the university journal.
Articles are written by editorial staff who have a high level of knowledge and interest in universities and education.