As the application of artificial neural networks continues to evolve, the complexity of machine vision algorithms has increased significantly, requiring a huge amount of computing power. However, the slowdown of Moore's Law has led to saturating performances of electronic processing units, making it difficult to meet the growing demands for high computing power and high energy efficiency. In the meantime, computing with photons presents an opportunity for revolutionary advancements in current vision computing. Unfortunately, existing intelligent optical computing architectures rely on electronic processors as computing relays, which significantly limit the computing advantages of high-speed and high-parallel optical computing, restricting ultra-fast machine vision applications.

Recently, a research group led by Associate Professor Lu FANG from the Department of Electronic Engineering at Tsinghua University developed an ultrafast intelligent optical computing architecture in the spatiotemporal domain. They introduced spatiotemporal cross-dimensional matching methods and a spatiotemporal optical matrix-vector computing model. This research realized the first 3D spatiotemporal intelligent optical computing system, which surpassed the constraints of digital I/O and improved machine vision processing speed by three orders of magnitude, reaching the nanosecond level.

Figure 1. Spatiotemporal Photonic Computing Model

Lu’s group developed a spatiotemporal photonic computing model (STPC) for continuous optical computation by characterizing a multi-dimensional optical field propagation model for the first time. To address the problem of intrinsic dimension mismatch in spatial and temporal optical fields, they proposed a spatial and spectral multiplexing computational model that matches the spatiotemporal optical computation dimension. Furthermore, they established a spatiotemporal matrix multiplication and addition computational model to expand the spatiotemporal optical computation dimension. All computational operations were conducted in the optical simulation domain, thereby eliminating constraints on the speed of spatiotemporal optical computation caused by the transfer and read-write of electronic memory. The paper also proposed multi-variable all-optical nonlinear functions, which were used to construct multi-layer nonlinear neural networks that supported high-speed inference for complex visual intelligence tasks. Experimental results on several video datasets indicated that spatiotemporal optical computation could increase intelligent video processing speed up to several hundred megahertz. This approach is expected to bring about new opportunities in high-performance intelligent computing during the post-Moore era and real-time analysis and control of transient scientific phenomena.

Figure 2: High-Speed Motion tracking

Figure 3. Nanosecond Ultrafast Computing

This work was published in Science Advances with the title "Ultrafast Dynamic Machine Vision with Spatiotemporal Photonic Computing", and was selected as the Journal’s feature article. Tiankuang Zhou and Wei Wu (Ph.D. students) are the co-first authors of the paper. The corresponding author is Prof. Lu Fang from the Department of Electronic Engineering.

Article link: https://www.science.org/doi/10.1126/sciadv.adg4391