In this study, a framework was constructed that modified Stable Diffusion, and local field potentials recorded from the rat's cerebral cortex were input as the source of the image instead of noise, to implement a system that outputs images in real time according to the state of the rat's brain. Local field potentials are measurements of electrical signals emitted by neurons, and the research group cut these out every 1/30th of a second, converted them into latent vectors, and input them into the latent diffusion model. Since successive latent vectors every 1/30th of a second are similar to each other, the images obtained through the latent diffusion model also show a smooth transition, and images that gradually change according to changes in brain waves are generated.

　There is no precedent for a drawing system that combines brain activity and artificial intelligence in this way, and this report is the world's first. In this study, images were generated by inputting only noise, but Stable Diffusion can also input instruction text that serves as a guide for image generation. Therefore, by giving instruction text that reflects the rat's internal state, it may be possible to generate images that correspond to the rat's "mood," such as a bright image when the rat is interested, and a quiet image when the rat is sleepy.

　The method developed this time can be applied not only to brain activity, but also to any time-series signals, such as other biosignals from the heart and intestinal peristalsis, and natural phenomena such as wind and waves. This result is expected to open up a new genre of artistic creation techniques.

Paper information:[PLOS ONE] Diffusion model-based image generation from rat brain activity