Introduction of NII S. Koyama's Lab (AY2025)

Transcript

1. Introduction of Our Lab Audio Processing Research Group AY2025 Shoichi Koyama, Ph. D. Digital Content and Media Sciences Research Division, National Institute of Informatics
2. March 4, 2025 2 Basic Technologies of Sound Field Analysis and Synthesis VR/AR audio Active noise control Local-field recording and reproduction Signal enhancement Visualization/auralization Room acoustic analysis Summary Sound field analysis/synthesis and its applications
3. What is sound field analysis/synthesis? March 4, 2025 3 Estimating sound field inside target region using multiple mics Synthesizing desired sound field inside target region using multiple loudspeakers Wavefield-informed signal processing and machine learning for sound field analysis and synthesis Analysis Synthesis Microphone Loudspeaker
4. Basic Technologies March 4, 2025 4 Analysis Synthesis Microphone Loudspeaker • Kernel interpolation with constraint of Helmholtz eq • Sparse modeling for enhancing resolution • Physics-informed machine learning-based estimation [Ueno+ IEEE SPL 2018, IEEE TSP 2021] [Murata+ IEEE TSP 2018, Koyama+ JASA 2018, IEEE JSTSP 2019] [Ribeiro+ IEEE/ACM TASLP 2024, Koyama+ IEEE SPM 2025] Wavefield-informed signal processing and machine learning
5. Analysis Synthesis Microphone Loudspeaker Basic Technologies March 4, 2025 5 • Weighted pressure and mode matching for sound field control • Optimization of source and sensor placement • Amplitude matching for multizone control [Ueno+ IEEE/ACM TASLP 2019, Koyama+ JAES 2023] [Koyama+ IEEE/ACM TASLP 2020, Nishida+ IEEE TSP 2022] [Koyama+ IEEE ICASSP 2021, Abe+ IEEE/ACM TASLP 2023] Enhancing flexibility and scalability to make the range of applications broader Wavefield-informed signal processing and machine learning
6. SOUND FIELD ANALYSIS March 4, 2025 6
7. Kernel Interpolation of Sound Field Ø Kernel interpolation with constraint of Helmholtz eq – Estimated function should satisfy governing equation of acoustic field – Derived kernel function to constraint solution of kernel ridge regression satisfying Helmholtz eq March 4, 2025 7 Estimate continuous sound field from discrete mics Target region: Microphone <latexit sha1_base64="YqiyTflU53yE7Th8bfA2lgdHgBE=">AAACHXicbVDLSgNBEJyNrxhfUY9eBkMgQQm7IagXIagHjxHMA7IxzE46yZCZ3WVmVghLvsCP8Bu86tmbeBWP/omTx8EkFjQUVd10d3khZ0rb9reVWFldW99Ibqa2tnd299L7BzUVRJJClQY8kA2PKODMh6pmmkMjlECEx6HuDa7Hfv0RpGKBf6+HIbQE6fmsyyjRRmqnszns3gDXBJ/gwUMR53GUcz0Ry9GpGwjokTy+xHY7nbEL9gR4mTgzkkEzVNrpH7cT0EiAryknSjUdO9StmEjNKIdRyo0UhIQOSA+ahvpEgGrFk3dGOGuUDu4G0pSv8UT9OxETodRQeKZTEN1Xi95Y/M9rRrp70YqZH0YafDpd1I041gEeZ4M7TALVfGgIoZKZWzHtE0moNgnObfHEyGTiLCawTGrFgnNWKN2VMuWrWTpJdISOUQ456ByV0S2qoCqi6Am9oFf0Zj1b79aH9TltTVizmUM0B+vrF4oXn7I=</latexit> ( + k2)u(r, !) = 0 [Ueno+ IEEE SPL 2018, IEEE TSP 2021, Koyama+ IEEE ICASSP 2022 Tutorial] Kernel function: Helmholtz eq:
8. Kernel Interpolation of Sound Field Ø Experimental results using real data from MeshRIR data set – Reconstructing pulse signal from single loudspeaker w/ 18 mic March 4, 2025 8 True Proposed Gaussian kernel (Black dots indicate mic positions) Impulse response measurement system [Koyama+ 2021]
9. Physics-Informed Machine Learning for Sound Field Estimation Ø Kernel function is adapted to acoustic environment under the constraint of Helmholtz eq. March 4, 2025 9 Physics-constrained neural kernel adapted to acoustic environment Microphone Directed component Residual component Kernel function based on plane wave decomposition <latexit sha1_base64="BpUMj8y6fI2+LPQuRMWoXjnLaqw=">AAACvHicfZFNT9tAEIY3pi3gfhDgyMVqVKmqqsiuIuACQrQHLlWp1IRIcYTGm3GyzX5Yu2tEsPwfeqWn/q3+G9bBqRpoOtJKj955d2dmJ8k4MzYMfze8tSdPn61vbPrPX7x8tdXc3ukZlWuKXaq40v0EDHImsWuZ5djPNIJIOF4k049V/uIKtWFKfrOzDIcCxpKljIJ1Ui/WE3UUXjZbYTucR/AYohpapI7zy+3Gr3ikaC5QWsrBmEEUZnZYgLaMciz9ODeYAZ3CGAcOJQg0w2Lebhm8ccooSJV2R9pgrv59owBhzEwkzinATszDXCX+KzfIbXo4LJjMcouS3hdKcx5YFVSzByOmkVo+cwBUM9drQCeggVr3Q0tVErE0Q1HV0iY1pe/Hn9CNrPGzk75kqMEq/a6IQY8FXJfuC8bx+4r+Z2RyYXS0yugeYYLdYFn8oZVWJhfWBbmdRg83+Bh6H9rRfrvztdM6Oa23u0H2yGvylkTkgJyQM3JOuoSS7+QHuSU/vWNv5E09cW/1GvWdXbIU3tUdblbe9g==</latexit> ⇢ = 0 <latexit sha1_base64="3hf60yYYQ7HvahG+/44qvpc1Skk=">AAACvnicfZFLaxsxEMflTR+u+7LTYy5LTaGUYnaLSXopmKSHXEITiB/gNWFWnrWF9VgkbbGz7JfINTnka+XbROvYIU6aDgh+/OcvzYwmTjkzNghuKt7Wi5evXlff1N6+e//hY72x3TMq0xS7VHGlBzEY5Exi1zLLcZBqBBFz7MezgzLf/4vaMCVP7SLFkYCJZAmjYJ00iPRU/QpbwVm9GbSCZfhPIVxBk6zi+KxRuY7GimYCpaUcjBmGQWpHOWjLKMeiFmUGU6AzmODQoQSBZpQvGy78L04Z+4nS7kjrL9WHN3IQxixE7JwC7NQ8zpXiv3LDzCY/RzmTaWZR0rtCScZ9q/xyen/MNFLLFw6AauZ69ekUNFDr/mijSiw2ZsjLWtokpqjVot/oRtZ45KQ/KWqwSn/LI9ATAfPCfcEk+l7S/4xMro2OnjO6R5hg51jk9/Sslcm1dU1up+HjDT6F3o9WuNtqn7Sbnf3Vdqtkh3wmX0lI9kiHHJJj0iWUcHJBLsmV1/EST3jqzupVVnc+kY3w5rebAt9p</latexit> ⇢ = 1.0 <latexit sha1_base64="G3HeMTKiNSRWAGIadt4AyvfR3dw=">AAACvnicfZFLSyNBEMc74zu+9ehlMAgiEmZE1IsQdj3sRVQwGsgEqenUJI39GLp7xOwwX2Kvetivtd9me2IixldBw49//burqitOOTM2CP5VvKnpmdm5+YXq4tLyyura+saNUZmm2KSKK92KwSBnEpuWWY6tVCOImONtfP+zzN8+oDZMyWs7SLEjoCdZwihYJ7Ui3VenB/Xgbq0W1INh+B8hHEGNjOLybr3yN+oqmgmUlnIwph0Gqe3koC2jHItqlBlMgd5DD9sOJQg0nXzYcOHvOKXrJ0q7I60/VN/eyEEYMxCxcwqwffM+V4qf5dqZTU46OZNpZlHSl0JJxn2r/HJ6v8s0UssHDoBq5nr1aR80UOv+aKJKLCZmyMta2iSmqFajM3Qjazx30kWKGqzSe3kEuifgsXBf0Iv2S/rOyOTY6Ogro3uECfYbi/yVvrQyObaOye00fL/Bj3BzUA+P6odXh7XGj9F258kW2Sa7JCTHpEF+kUvSJJRw8oc8kWev4SWe8NSL1auM7mySifAe/wOdV99q</latexit> ⇢ = 2.0 <latexit sha1_base64="RgAQQ0q6zDhPdd8dXTMhg6LNaeU=">AAACwnicfZHBThsxEIadpRQItAV65LJqhIRQFe0iVDgi4NBLVSo1gJSN0Kwzmxhs72LPIoLZ1+i1vFbfpt6QVA2UjmTp0z+/PeOZtJDCUhT9agRzr+ZfLywuNZdX3rx9t7q2fmrz0nDs8Fzm5jwFi1Jo7JAgieeFQVCpxLP06qjOn92gsSLX32lUYE/BQItMcCAvJckQyCWpcqaqLlZbUTsaR/gc4gm02CROLtYaD0k/56VCTVyCtd04KqjnwJDgEqtmUlosgF/BALseNSi0PTduugo3vdIPs9z4oykcq3/fcKCsHanUOxXQ0D7N1eK/ct2Ssv2eE7ooCTV/LJSVMqQ8rCcQ9oVBTnLkAbgRvteQD8EAJz+nmSqpmvmDq2sZm9mq2UyO0X/Z4BcvfS3QAOVm2yVgBgpuKz+CQfKxpv8ZhZ4aPb1k9I8IJe6wcn/oRavQU+uU/E7jpxt8Dqc77fhTe/fbbuvgcLLdRbbBPrAtFrM9dsA+sxPWYZwV7Af7yR6C4+AyuA7sozVoTO68ZzMR3P8GjfHiRg==</latexit> ˆ r Weighting function adapted to directed and residual components of sound field separately [Ribeiro+ IEEE/ACM TASLP 2024, Koyama+ IEEE SPM 2025]
10. Physics-Informed Machine Learning for Sound Field Estimation March 4, 2025 10 Ground truth (600 Hz) NN PINN Adaptive kernel NMSE: -6.8 dB NMSE: -16.3 dB NMSE: -24.8dB Physics-constrained neural kernel adapted to acoustic environment Our adaptive kernel outperforms the state-of-the-art techniques [Ribeiro+ IEEE/ACM TASLP 2024, Koyama+ IEEE SPM 2025]
11. Application to Binaural Reproduction March 4, 2025 11 Conversion into binaural sounds Ø Binaural reproduction in real world is difficult, compared to binaural synthesis in VR space Ø Binaural reproduction from recordings of multiple small arrays Ø Broad listening area by using flexible and scalable recording system Binaural reproduction from mic array recordings for VR audio Recording Reproduction [Iijima+ JASA 2021]
12. Application to Binaural Reproduction Ø Recording system using multiple Ambisonic mics and 360-degree cameras March 4, 2025 12 Small mic arrays (Ambisonic mics) 360-degree cameras Demo Proposed Single array [Iijima+ IEEE WASPAA 2021 (demo)] Error distribution
13. SOUND FIELD SYNTHESIS March 4, 2025 13
14. Sound Field Synthesis Ø Optimization problem to obtain loudspeaker driving signals March 4, 2025 14 Synthesizing desired pressure field w/ multiple loudspeakers Loudspeaker Target region: Synthesized sound field Desired sound field Conventional techniques rely on approximation by discretization of the region or truncation of series expansion Driving signal Transfer function
15. Weighted Pressure/Mode Matching Ø Weighted pressure matching – Cost function is approximated by using kernel interpolation – Driving signal is obtained as weighted least squares solution Ø Weighted mode matching – Cost function is approximated by using spherical wavefunction expansion – Drigving signal is obtained as weighted least squares solution March 4, 2025 15 Pressure/mode matching for continuous target region <latexit sha1_base64="FaZMjJYiGcGtjVU4KwK7fZp2dL4=">AAADQXicfVHdbtMwGHXC3yh/HVxyY61C2lhXNRMaXE6AxJ/QhrRuk+quchwntWo7ke2gFc9PgcQL8RQ8AneIW7jASdOJbYxPinJyzvn85fOJC8606fe/BeGVq9eu31i62bp1+87de+3l+/s6LxWhA5LzXB3GWFPOJB0YZjg9LBTFIub0IJ6+qPSDj1Rplss9MyvoSOBMspQRbDw1bn95CxEuCpUfQ8SkGVu0I2iGHUScpuYEolhYNPUO7FYrrNzakUUCm4lO7Z5zsGbfuXXE/dAEV19vKs9G1GivErdRvUvXNCphE6qdW4NIsWxiTo424ang5kPG7U6/168LXgRRAzqgqd3xcvAZJTkpBZWGcKz1MOoXZmSxMoxw6lqo1LTAZIozOvRQYkH1yNYX6OAjzyQwzZV/pIE1+3eHxULrmYi9s179vFaR/9KGpUmfjSyTRWmoJPNBacmhyWGVBkyYosTwmQeYKOb/FZIJVpgYn1nrzJhYdOtL0qn227ykfktF33tmp6AKm1w9tgirTOBj57fOULdC/zMyuTB6dJnRH8IE+0SdPUWXWplcWBfIxxidD+0i2N/sRVu9rQ9POtvPm0CXwEOwAlZBBJ6CbfAa7IIBIOB3sBKsB93wa/g9/BH+nFvDoOl5AM5U+OsPMtITnw==</latexit> J → ! ω(r)T(K + ωI)→1(Gd ↑ udes) 2 dr <latexit sha1_base64="nTw1yy7DzN7N6vmgNA7GkhRSPtc=">AAADUHicfVFdb9MwFHVaPkbHRwePvFhUSB0aVTOhMY2XCZBASGhDWrdJdVc57k1qLXYi25lWPP8KfgN/iDf+CW/gpOnEOsaVont8zrFv7r1RnnJt+v2fQaN56/aduyv3Wqv3Hzx81F57fKizQjEYsCzN1HFENaRcwsBwk8JxroCKKIWj6PRdqR+dgdI8kwdmlsNI0ETymDNqPDVuf/+ECc1zlZ1jwqUZW7InIKEOkxRic4FJJCw5oyqfctctD8qtn1giqJnq2B642tjFFaW4TGzp+uBcmSYOv1xSCufq+8rroMsXFE+mZr3OFyeb+FJ385Ljdqff61eBr4OwBh1Ux/54LfhGJhkrBEjDUqr1MOznZmSpMpyl4Fqk0JBTdkoTGHooqQA9stU8HX7umQmOM+U/aXDF/n3DUqH1TETeWQ1iWSvJf2nDwsTbI8tlXhiQbF4oLlJsMlwuB0+4AmbSmQeUKe7/FbMpVZQZv8LWlTKR2KiGpGPtu3kPvksFnz2zl4OiJlMvLKEqEfTc+a4TslGi/xm5XBg9usnoH+GCfwVnL9GNVi4X1gXyawyXl3YdHG72wq3e1pdXnd239UJX0FP0DHVRiF6jXfQR7aMBYsFqEAY7wZvGj8avxu9mMLc26oyeoCvRbP0BL88YTg==</latexit> J → ! ω(r)T ˚ Gd ↑ ˚ udes 2 dr <latexit sha1_base64="Fp5GKbYefSIl0j/MnXJIJ4W13yA=">AAADP3icfZHNbhMxEMe9y1cJXykcuVhESG0p0S5ChQtSBUgth4ogkaZSNo283tnEqu1d2V5EsPwU8EI8Bk/ADXGFE95NgmibMpKlv37z94zHk5acaRNF34Lw0uUrV6+tXW/duHnr9p32+t1DXVSKQp8WvFBHKdHAmYS+YYbDUamAiJTDID15VecHH0BpVsj3ZlbCSJCJZDmjxHg0bn9JpsTYJBU2cw6/wK2NWu+5Y5sIYqY6t/vO1WjgxnOkhO0dzNmew49wwn23jOAavHGbx/Zx7PCKInhVlQZWS6OHGWjnxu1O1I2awOdFvBAdtIjeeD34nGQFrQRIQznRehhHpRlZogyjHFwrqTSUhJ6QCQy9lESAHtnm+xx+6EmG80L5Iw1u6L83LBFaz0Tqnc04Z3M1XJUbViZ/PrJMlpUBSeeN8opjU+B6FzhjCqjhMy8IVcy/FdMpUYQav7HWqTap2G5+SOfaT/Ma/JQKDjx5W4IiplBbNiFqIshH56eeJNu1+p+RyaXRq4uMvggT7BM4+1ddaGVyaV0qv8b47NLOi8Mn3Xinu/PuaWf35WKha+g+eoA2UIyeoV20j3qojyj6HeBgM9gKv4bfwx/hz7k1DBZ37qFTEf76A66OE5k=</latexit> ˆ d = (GHWPMG + ωI)→1GHWPMudes <latexit sha1_base64="iRLjXRHsy83SU+byoSU5r8DKDEw=">AAADa3icfZHtbtMwFIadho9RvrrtH/DDohoabFQNmgZ/kCZAYkhUDIm2k5qucpyT1FrsRLaDKJavghviNrgI7gEnbSfWfRzJ0qvnvOfYxycqMqZ0t/vHa/g3bt66vXaneffe/QcPW+sbA5WXkkKf5lkujyOiIGMC+prpDI4LCYRHGQyj0/dVfvgdpGK5+KZnBYw5SQVLGCXaoUnrdzgl2oQRN7G1+C1uboec6KlkIq3pR2tPTI1UYg6trdjQTuZIctPrObZagXdwmLk3xARX5JN9fmJeBhZf3xpf1nulpjyrcfkYlLWTVrvb6daBL4pgIdpoEUeTde9XGOe05CA0zYhSo6Bb6LEhUjOagW2GpYKC0FOSwshJQTiosam/2uItR2Kc5NIdoXFN/68whCs145Fz1pOt5ip4WW5U6uTN2DBRlBoEnV+UlBnWOa72hmMmgeps5gShkrm3YjolklDttts8d03Ed+sfUoly03wAN6WEniNfCpBE5/KFCYlMOflh3dRpuFup64xMLI1OXWV0TRhnP8GaM3WllYmldancGoPVpV0Ug1edYL+z/3WvffBusdA19Bg9RdsoQK/RATpER6iPqPfM++z1vUHjr7/pP/KfzK0Nb1Gzic6Fv/UPG8MjCA==</latexit> ˆ d = ( ˚ GHWMM ˚ G + ωI)→1 ˚ GHWMM ˚ udes [Koyama+ JAES 2023] [Ueno+ IEEE/ACM TASLP 2019] Generalization
16. Weighted Pressure/Mode Matching Ø Comparison between Pressure Matching and Weighted Pressure Matching March 4, 2025 16 PM WPM Pressure Error [Koyama+ JAES 2023]
17. Amplitude Matching for Multizone Sound Field Control Ø Amplitude matching: Synthesizing desired magnitude distribution, leaving phase distribution arbitrary March 4, 2025 17 Target region Generating multiple personal sound zones by using loudspeakers Desired amplitude No closed form solution, but iterative algorithms, e.g., alternating direction method of multipliers (ADMM), can be applied Element-wise absolute value [Abe+ IEEE/ACM TASLP 2023]
18. Amplitude Matching for Multizone Sound Field Control March 4, 2025 18 https://youtu.be/oYw7kmpZcY4 Full version:
19. Perceptual Quality Enhancement of Sound Field Synthesis Ø Owing to discrete placement of secondary sources, spatial aliasing artifacts are unavoidable – E.g., Synthesizing sound field by 12 loudspeakers at 800 Hz March 4, 2025 19 Desired Pressure Matching Pressure § Degradation of sound localization § Coloration of source signals
20. Perceptual Quality Enhancement of Sound Field Synthesis Ø Interaural level difference (ILD) is the dominant cue for horizontal sound localization above 1500 Hz, compared with interaural time difference (ITD) Ø Amplitude response should be accurately synthesized as much as possible, rather than phase response, to alleviate coloration effects March 4, 2025 20 Synthesizing amplitude (or magnitude) distribution leaving phase distribution arbitrary at high frequencies Applying amplitude matching for high frequencies Pressure Magnitude [Kimura+ IEEE WASPAA 2023]
21. Perceptual Quality Enhancement of Sound Field Synthesis Ø Average error of ILD at each listening position: Ø Amplitude response at the center March 4, 2025 21 PM Proposed ILD is accurately reproduced Flat amplitude response is obtained [Kimura+ IEEE WASPAA 2023]
22. Application to Spatial Active Noise Control Ø Environmental noise is still unsolved problem Ø Active noise control (ANC) is aimed to cancel noise by loudspeaker signals, but its effect is limited to local region Ø ANC in 3D space based on sound field analysis/synthesis March 4, 2025 22 Noise suppression by loudspeaker signals Quiet zone
23. Application to Spatial Active Noise Control Ø Cost function of regional noise power is estimated by kernel interpolation of sound field Ø Adaptive filtering algorithm based on kernel interpolation is also derived March 4, 2025 23 ANC in 3D space based on sound field interpolation Ø Conventional cost function Ø Proposed cost function <latexit sha1_base64="Pnlxe5THpW3gUCE+pMJvaUIbsmQ=">AAACFXicbZDLSsNAFIYnXmu9RV2KMFgUVyUpRd0IRTcuXFSwF2himUwn7dCZJMxMhBKz8iF8Bre6diduXbv0TZy0WdjWHwY+/nMO58zvRYxKZVnfxsLi0vLKamGtuL6xubVt7uw2ZRgLTBo4ZKFoe0gSRgPSUFQx0o4EQdxjpOUNr7J664EIScPgTo0i4nLUD6hPMVLa6poHDkdqgBFLblJ4fAGdR+h4PCGppvtK1yxZZWssOA92DiWQq941f5xeiGNOAoUZkrJjW5FyEyQUxYykRSeWJEJ4iPqkozFAnEg3GX8jhUfa6UE/FPoFCo7dvxMJ4lKOuKc7s6PlbC0z/6t1YuWfuwkNoliRAE8W+TGDKoRZJrBHBcGKjTQgLKi+FeIBEggrndzUFo+nOhN7NoF5aFbK9mm5elst1S7zdApgHxyCE2CDM1AD16AOGgCDJ/ACXsGb8Wy8Gx/G56R1wchn9sCUjK9f5U+ekQ==</latexit> L = kek2 : Power of error mics : Regional noise power <latexit sha1_base64="pm/hhsXKsYUZ/lWoZalEg8K5t2U=">AAACNHicbVDLSgMxFM34rPVVdekmWJS6KTOlqAuFohsXghXsAzq1ZNK0DU0yQ5IRynR+xo/wG9zqTnAj4tZvMNN2YVsPBA7nnMu9OV7AqNK2/W4tLC4tr6ym1tLrG5tb25md3aryQ4lJBfvMl3UPKcKoIBVNNSP1QBLEPUZqXv8q8WuPRCrqi3s9CEiTo66gHYqRNlIrc+5ypHsYsegmhkcX0KVCtyL3lpMuiuEwzLkej2R8PHwowFFU8qgdj0XYymTtvD0CnCfOhGTBBOVW5tNt+zjkRGjMkFINxw50M0JSU8xInHZDRQKE+6hLGoYKxIlqRqNfxvDQKG3Y8aV5QsOR+nciQlypAfdMMjlUzXqJ+J/XCHXnrBlREYSaCDxe1AkZ1D5MKoNtKgnWbGAIwpKaWyHuIYmwNsVObfF4bDpxZhuYJ9VC3jnJF++K2dLlpJ0U2AcHIAcccApK4BqUQQVg8ARewCt4s56tD+vL+h5HF6zJzB6YgvXzC9DNrBs=</latexit> L = Z ⌦ |u(r)|2dr [Ito+ IEEE ICASSP 2019 (Best Student Paper Award), Koyama+ IEEE/ACM TASLP 2021]
24. Application to Spatial Active Noise Control Ø Band-limited noise (500-800Hz), T60 : 240ms March 4, 2025 24 Proposed: -10.5 dB MPC: -6.2 dB (dB) Regional noise reduction is achieved by the proposed method [Ito+ IEEE ICASSP 2019 (Best Student Paper Award), Koyama+ IEEE/ACM TASLP 2021]
25. Summary Ø Recent research topics – Physics-informed machine learning for sound field estimation, Spatial active noise control, DNN-based HRTF interpolation/personalization, Personal sound zones generation, Source and sensor placement for sound field control Ø Keywords – Kernel methods, Gaussian process, Reproducing kernel Hilbert space, Sparse modeling, Deep neural network, Physics-informed neural network, Adaptive filter, Convex optimization, Physical acoustics, Partial differential equation March 4, 2025 25