NII S. Koyama's Lab Research Overview AY2026

Reseach overview of NII S. Koyama's Lab (https://www.ap.nii.ac.jp/)

Transcript

1. Audio Processing Research Group − Research Overview AY2026 − Shoichi Koyama, Ph. D. National Institute of Informatics / SOKENDAI
2. March 31, 2026 2 Core Technologies of Sound Field Analysis and Synthesis VR/AR audio Active noise control Local-field recording and reproduction Visualization/auralization Room acoustic analysis Our Research Topics Sound field analysis/synthesis and its applications
3. What is sound field analysis/synthesis? March 31, 2026 3 Estimating sound field in a target region using multiple microphones Synthesizing a desired sound field in a target region using multiple loudspeakers Signal processing and machine learning based on wave physics for spatial audio processing Analysis Synthesis Microphone Loudspeaker
4. Basic Technologies March 31, 2026 4 Analysis Synthesis Microphone Loudspeaker • Kernel interpolation with the Helmholtz-equation constraint • Sparse modeling for high-resolution sound field estimation • Physics-informed machine-learning-based estimation [Ueno+ IEEE SPL 2018, IEEE TSP 2021] [Murata+ IEEE TSP 2018, Koyama+ JASA 2018, IEEE JSTSP 2019] [Ribeiro+ IEEE/ACM TASLP 2024, Koyama+ IEEE SPM 2025] Physics-informed signal processing and machine learning
5. Analysis Synthesis Microphone Loudspeaker Basic Technologies March 31, 2026 5 • Weighted pressure and mode matching for sound field control • Optimization of source and sensor placement • Amplitude matching for multizone sound field control [Ueno+ IEEE/ACM TASLP 2019, Koyama+ JAES 2023] [Koyama+ IEEE/ACM TASLP 2020, Nishida+ IEEE TSP 2022] [Koyama+ IEEE ICASSP 2021, Abe+ IEEE/ACM TASLP 2023] Enhancing flexibility and scalability to broaden the range of applications Physics-informed signal processing and machine learning
6. SOUND FIELD ANALYSIS March 31, 2026 6
7. Kernel Interpolation of Sound Field ➢ Kernel interpolation with the Helmholtz-equation constraint – The estimated function should satisfy the governing equation of the acoustic field – Derived a kernel function to constraint solution of kernel ridge regression satisfying Helmholtz equation March 31, 2026 7 Estimate continuous sound field from discrete mics [Ueno+ IEEE SPL 2018, IEEE TSP 2021, Koyama+ IEEE ICASSP 2022 Tutorial] Kernel function: Helmholtz eq: Microphone Target region:
8. Kernel Interpolation of Sound Field ➢ Experimental results using real data from MeshRIR data set – Reconstructing pulse signal from single loudspeaker w/ 18 mic March 31, 2026 8 True Proposed Gaussian kernel (Black dots indicate mic positions) Impulse response measurement system [Koyama+ 2021]
9. Physics-Informed Machine Learning for Sound Field Estimation ➢ Kernel function is adapted to acoustic environment under the constraint of Helmholtz eq. March 31, 2026 9 Physics-constrained neural kernel adapted to acoustic environment Directed component Residual component Weighting function adapted to directed and residual components of sound field separately [Ribeiro+ IEEE/ACM TASLP 2024, Koyama+ IEEE SPM 2025] Kernel function based on plane wave decomposition
10. Physics-Informed Machine Learning for Sound Field Estimation March 31, 2026 10 Physics-constrained neural kernel adapted to acoustic environment Our adaptive kernel outperforms the state-of-the-art techniques [Ribeiro+ IEEE/ACM TASLP 2024, Koyama+ IEEE SPM 2025]
11. Application to Binaural Reproduction March 31, 2026 11 Conversion into binaural sounds ➢ Binaural reproduction in real world is difficult, compared to binaural synthesis in VR space ➢ Binaural reproduction from recordings of multiple small arrays ➢ Broad listening area by using flexible and scalable recording system Binaural reproduction from mic array recordings for VR audio Recording Reproduction [Iijima+ JASA 2021]
12. Application to Binaural Reproduction ➢ Recording system using multiple Ambisonic mics and 360-degree cameras March 31, 2026 12 Small mic arrays (Ambisonic mics) 360-degree cameras Demo Proposed Single array [Iijima+ IEEE WASPAA 2021 (demo)] Error distribution
13. SOUND FIELD SYNTHESIS March 31, 2026 13
14. Sound Field Synthesis ➢ Optimization problem to obtain loudspeaker driving signals March 31, 2026 14 Synthesizing desired pressure field w/ multiple loudspeakers Loudspeaker Target region: Synthesized sound field Desired sound field Conventional techniques rely on approximation by discretization of the region or truncation of series expansion Driving signal Transfer function
15. Weighted Pressure/Mode Matching ➢ Weighted pressure matching – Cost function is approximated by using kernel interpolation – Driving signal is obtained as weighted least squares solution ➢ Weighted mode matching – Cost function is approximated by using spherical wavefunction expansion – Driving signal is obtained as weighted least squares solution March 31, 2026 15 Pressure/mode matching for continuous target region [Koyama+ JAES 2023] [Ueno+ IEEE/ACM TASLP 2019] Generalization
16. Weighted Pressure/Mode Matching ➢ Comparison between Pressure Matching and Weighted Pressure Matching March 31, 2026 16 PM WPM Pressure Error [Koyama+ JAES 2023]
17. Amplitude Matching for Multizone Sound Field Control ➢ Amplitude matching: Synthesizing desired magnitude distribution, leaving phase distribution arbitrary March 31, 2026 17 Target region Generating multiple personal sound zones by using loudspeakers Desired amplitude No closed form solution, but iterative algorithms, e.g., alternating direction method of multipliers (ADMM), can be applied Element-wise absolute value [Abe+ IEEE/ACM TASLP 2023]
18. Amplitude Matching for Multizone Sound Field Control March 31, 2026 18 https://youtu.be/oYw7kmpZcY4 Full version:
19. Perceptual Quality Enhancement of Sound Field Synthesis ➢ Owing to discrete placement of secondary sources, spatial aliasing artifacts are unavoidable – E.g., Synthesizing sound field by 12 loudspeakers at 800 Hz March 31, 2026 19 Desired Pressure Matching Pressure ▪ Degradation of sound localization ▪ Coloration of source signals
20. Perceptual Quality Enhancement of Sound Field Synthesis ➢ Interaural level difference (ILD) is the dominant cue for horizontal sound localization above 1500 Hz, compared with interaural time difference (ITD) ➢ Amplitude response should be accurately synthesized as much as possible, rather than phase response, to alleviate coloration effects March 31, 2026 20 Synthesizing amplitude (or magnitude) distribution leaving phase distribution arbitrary at high frequencies Applying amplitude matching for high frequencies Pressure Magnitude [Kimura+ IEEE WASPAA 2023]
21. Perceptual Quality Enhancement of Sound Field Synthesis ➢ Average error of ILD at each listening position: ➢ Amplitude response at the center March 31, 2026 21 PM Proposed ILD is accurately reproduced Flat amplitude response is obtained [Kimura+ IEEE WASPAA 2023]
22. Application to Spatial Active Noise Control ➢ Environmental noise is still unsolved problem ➢ Active noise control (ANC) is aimed to cancel noise by loudspeaker signals, but its effect is limited to local region ➢ ANC in 3D space based on sound field analysis/synthesis March 31, 2026 22 Noise suppression by loudspeaker signals Quiet zone
23. Application to Spatial Active Noise Control ➢ Cost function of regional noise power is estimated by kernel interpolation of sound field ➢ Adaptive filtering algorithm based on kernel interpolation is also derived March 31, 2026 23 ANC in 3D space based on sound field interpolation ➢ Conventional cost function ➢ Proposed cost function : Power of error mics : Regional noise power [Ito+ IEEE ICASSP 2019 (Best Student Paper Award), Koyama+ IEEE/ACM TASLP 2021]
24. Application to Spatial Active Noise Control March 31, 2026 24 https://youtu.be/VhCPxi5BW34 Full version:
25. Summary ➢ Recent research topics – Physics-informed machine learning for sound field estimation, Spatial active noise control, DNN-based HRTF interpolation/personalization, Personal sound zone generation, Source and sensor placement for sound field control ➢ Keywords – Kernel methods, Gaussian process, Reproducing kernel Hilbert space, Sparse modeling, Deep neural network, Physics-informed neural network, Adaptive filter, Convex optimization, Physical acoustics, Partial differential equation March 31, 2026 25 From theory to applications in audio processing