MMSP 2025

Transcript

1. LEARNING 3D MESH SALIENCY FROM SPIRAL PATCH FEATURES MMSP 2025 - Beijing, China Olivier L ´ EZORAY1, Anass NOURI2 1Universit´ e Caen Normandie, ENSICAEN, Normandie Univ, GREYC UMR 6072, Caen, FRANCE 2Laboratoire des Syst` emes ´ Electroniques, Traitement de l’Information, M´ ecanique et ´ Energ´ etique, Ibn Tofail University, Kenitra, MOROCCO olivier.lezoray@unicaen.fr https://lezoray.users.greyc.fr
2. Outline 1. Introduction 2. Spiral patches 3. Roughness, geometric and spectral saliencies 4. Saliency learning from multi-scale spiral cues O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 2 / 29
3. Outline 1. Introduction 2. Spiral patches 3. Roughness, geometric and spectral saliencies 4. Saliency learning from multi-scale spiral cues O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 3 / 29
4. Introduction ▶ Recent technological advances have led to the generation of huge amounts of 3D data ▶ Even with cheap hardware and software, one can easily generate 3D data: ▶ 3D data acquisition with common smartphone (e.g., Photogrammetry) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 4 / 29
5. New applications Fields ▶ With this proliferation of such 3D Data, new application fields have appeared ▶ Digital Forensics, Cultural Heritage, Body Scanning One often needs to know which regions from a 3D mesh are important: adaptive simplification or decimation, viewpoint selection O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 5 / 29
6. 3D Data Saliency Saliency for images ? ▶ Salient regions of an image are visually more noticeable by their contrast with respect to surrounding regions Saliency for 3D meshes ? ▶ If a point from the 3D data stands out strongly from its surrounding, then, it could be considered as a salient 3D point. Our approach ▶ We analyze the geometry of 3D mesh normals using spiral patches. ▶ Roughness, geometric and spectral saliency cues are extracted from the patches’ surround analysis ▶ Saliency is predicted at each vertex by a regression on the cues extracted at multiple scales. O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 6 / 29
7. Outline 1. Introduction 2. Spiral patches 3. Roughness, geometric and spectral saliencies 4. Saliency learning from multi-scale spiral cues O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 7 / 29
8. Outline 1. Introduction 2. Spiral patches 3. Roughness, geometric and spectral saliencies 4. Saliency learning from multi-scale spiral cues O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 8 / 29
9. Spiral patches Notations ▶ A mesh M is represented by a graph G = (V,E) with V = {v1,...,vm } and E ⊂ V×V ▶ vi ∼ vj is used to denote two adjacent vertices ▶ N (vi ) = {vj,vj ∼ vi } gives the set of all the adjacent vertices to vi within a 1-hop (vertices that can be reached in one walk). ▶ Lim et al. have proposed local spiral hop descriptors: the surrounding vertices of one vertex can be enumerated by following a spiral O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 9 / 29
10. Spiral patches Definitions ▶ Rk(vi ) = k-ring(vi ): ordered set of vertices whose shortest path to vi is exactly k-hops long vi R1 7 R1 1 R1 2 R1 3 R1 4 R1 5 R1 6 R2 1 R2 2 R2 3 R2 4 R2 5 R2 6 R2 7 R2 8 R2 9 R2 10 R2 11 R2 12 R2 13 O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 10 / 29
11. Spiral patches Definitions ▶ Rk(vi ) = k-ring(vi ): ordered set of vertices whose shortest path to vi is exactly k-hops long ⇒ R0(vi ) = {vi } vi R1 7 R1 1 R1 2 R1 3 R1 4 R1 5 R1 6 R2 1 R2 2 R2 3 R2 4 R2 5 R2 6 R2 7 R2 8 R2 9 R2 10 R2 11 R2 12 R2 13 0-ring(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 10 / 29
12. Spiral patches Definitions ▶ Rk(vi ) = k-ring(vi ): ordered set of vertices whose shortest path to vi is exactly k-hops long ⇒ R0(vi ) = {vi } and R1(vi ) = N (vi ) vi R1 7 R1 1 R1 2 R1 3 R1 4 R1 5 R1 6 R2 1 R2 2 R2 3 R2 4 R2 5 R2 6 R2 7 R2 8 R2 9 R2 10 R2 11 R2 12 R2 13 1-ring(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 10 / 29
13. Spiral patches Definitions ▶ Rk(vi ) = k-ring(vi ): ordered set of vertices whose shortest path to vi is exactly k-hops long ⇒ R0(vi ) = {vi } and R1(vi ) = N (vi ) ▶ k-disk(vi ) = l=0,...,k Rl(vi ): set of vertices that can be reached from vi in 0 to k walks. vi R1 7 R1 1 R1 2 R1 3 R1 4 R1 5 R1 6 R2 1 R2 2 R2 3 R2 4 R2 5 R2 6 R2 7 R2 8 R2 9 R2 10 R2 11 R2 12 R2 13 1-disk(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 10 / 29
14. Spiral patches Definitions ▶ Rk(vi ) = k-ring(vi ): ordered set of vertices whose shortest path to vi is exactly k-hops long ⇒ R0(vi ) = {vi } and R1(vi ) = N (vi ) ▶ k-disk(vi ) = l=0,...,k Rl(vi ): set of vertices that can be reached from vi in 0 to k walks. ▶ k-hop(vi ) = k-disk(vi )\{vi } vi R1 7 R1 1 R1 2 R1 3 R1 4 R1 5 R1 6 R2 1 R2 2 R2 3 R2 4 R2 5 R2 6 R2 7 R2 8 R2 9 R2 10 R2 11 R2 12 R2 13 2-hop(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 10 / 29
15. Spiral patches Definitions ▶ Rk(vi ) = k-ring(vi ): ordered set of vertices whose shortest path to vi is exactly k-hops long ⇒ R0(vi ) = {vi } and R1(vi ) = N (vi ) ▶ k-disk(vi ) = l=0,...,k Rl(vi ): set of vertices that can be reached from vi in 0 to k walks. ▶ k-hop(vi ) = k-disk(vi )\{vi } ▶ R(k+1)(vi ) = N (R(k)(vi ))\k-disk(vi ) is the set of vertices that can be reached in 1 walk from Rk(vi ) without going through its k-disk (that contains vertices that can be reached from vi in 0 to k walks) vi R1 7 R1 1 R1 2 R1 3 R1 4 R1 5 R1 6 R2 1 R2 2 R2 3 R2 4 R2 5 R2 6 R2 7 R2 8 R2 9 R2 10 R2 11 R2 12 R2 13 2-ring(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 10 / 29
16. Spiral patches Definitions ▶ The Spiral patch Sp(vi,k) is an ordered sequence from the concatenation of the ordered rings Sp(vi,k) = (vi,1-ring(vi ),...,k-ring(vi )) = R0 1 (vi ),R1 1 (vi ),R1 2 (vi ),...,Rk |Rk| (vi ) ▶ It has 2 degrees of freedom: the direction (clockwise or counterclockwise) of the rings and the first chosen vertex R1 1 (vi ) ▶ We fix : ▶ the orientation clockwise ▶ the initial vertex R1 1 (vi ) is the one in the direction of the shortest geodesic path to vi: R1 1 (vi ) = arg min vj∈N (vi) dG(vi,vj ) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 11 / 29
17. Spiral patches Comparison ▶ The size of the operator Sp(vi,k) varies for the vertices ▶ Lim et al. either truncate or zero-pad each spiral depending on its size to compare two Spiral patches ▶ We propose a more accurate hierarchical comparison d(Sp(vi,k),Sp(vj,k)) = k ∑ l=0 d(Rl(vi),Rl(vj)) Two k-rings are compared by mapping the vertices of the largest ring to the smallest one: d(Rl(vi),Rl(vj)) = |Rl (vi )| ∑ n=0 d(Rl n (vi),Rl n′ (vj)) with n′ = n·|Rl (vj )| |Rl (vi )| and |Rl(vi)| > |Rl(vj)| First vertex spiral patch Second vertex spiral patch 0-ring 1-ring 2-ring The distance between two vertices is then the distance between their graph signal vectors: d(Rl n (vi),Rl m (vj)) = ∥F(Rl n (vi))−F(Rl m (vj))∥2 that are vertex normal vectors: F(vi) = n(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 12 / 29
18. Spiral patches Comparison ▶ The size of the operator Sp(vi,k) varies for the vertices ▶ Lim et al. either truncate or zero-pad each spiral depending on its size to compare two Spiral patches ▶ We propose a more accurate hierarchical comparison d(Sp(vi,k),Sp(vj,k)) = k ∑ l=0 d(Rl(vi),Rl(vj)) Two k-rings are compared by mapping the vertices of the largest ring to the smallest one: d(Rl(vi),Rl(vj)) = |Rl (vi )| ∑ n=0 d(Rl n (vi),Rl n′ (vj)) with n′ = n·|Rl (vj )| |Rl (vi )| and |Rl(vi)| > |Rl(vj)| First vertex spiral patch Second vertex spiral patch 0-ring 1-ring 2-ring The distance between two vertices is then the distance between their graph signal vectors: d(Rl n (vi),Rl m (vj)) = ∥F(Rl n (vi))−F(Rl m (vj))∥2 that are vertex normal vectors: F(vi) = n(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 12 / 29
19. Spiral patches Comparison ▶ The size of the operator Sp(vi,k) varies for the vertices ▶ Lim et al. either truncate or zero-pad each spiral depending on its size to compare two Spiral patches ▶ We propose a more accurate hierarchical comparison d(Sp(vi,k),Sp(vj,k)) = k ∑ l=0 d(Rl(vi),Rl(vj)) Two k-rings are compared by mapping the vertices of the largest ring to the smallest one: d(Rl(vi),Rl(vj)) = |Rl (vi )| ∑ n=0 d(Rl n (vi),Rl n′ (vj)) with n′ = n·|Rl (vj )| |Rl (vi )| and |Rl(vi)| > |Rl(vj)| First vertex spiral patch Second vertex spiral patch 0-ring 1-ring 2-ring The distance between two vertices is then the distance between their graph signal vectors: d(Rl n (vi),Rl m (vj)) = ∥F(Rl n (vi))−F(Rl m (vj))∥2 that are vertex normal vectors: F(vi) = n(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 12 / 29
20. Spiral patches Comparison ▶ The size of the operator Sp(vi,k) varies for the vertices ▶ Lim et al. either truncate or zero-pad each spiral depending on its size to compare two Spiral patches ▶ We propose a more accurate hierarchical comparison d(Sp(vi,k),Sp(vj,k)) = k ∑ l=0 d(Rl(vi),Rl(vj)) Two k-rings are compared by mapping the vertices of the largest ring to the smallest one: d(Rl(vi),Rl(vj)) = |Rl (vi )| ∑ n=0 d(Rl n (vi),Rl n′ (vj)) with n′ = n·|Rl (vj )| |Rl (vi )| and |Rl(vi)| > |Rl(vj)| First vertex spiral patch Second vertex spiral patch 0-ring 1-ring 2-ring The distance between two vertices is then the distance between their graph signal vectors: d(Rl n (vi),Rl m (vj)) = ∥F(Rl n (vi))−F(Rl m (vj))∥2 that are vertex normal vectors: F(vi) = n(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 12 / 29
21. Spiral patches Comparison ▶ The size of the operator Sp(vi,k) varies for the vertices ▶ Lim et al. either truncate or zero-pad each spiral depending on its size to compare two Spiral patches ▶ We propose a more accurate hierarchical comparison d(Sp(vi,k),Sp(vj,k)) = k ∑ l=0 d(Rl(vi),Rl(vj)) Two k-rings are compared by mapping the vertices of the largest ring to the smallest one: d(Rl(vi),Rl(vj)) = |Rl (vi )| ∑ n=0 d(Rl n (vi),Rl n′ (vj)) with n′ = n·|Rl (vj )| |Rl (vi )| and |Rl(vi)| > |Rl(vj)| First vertex spiral patch Second vertex spiral patch 0-ring 1-ring 2-ring The distance between two vertices is then the distance between their graph signal vectors: d(Rl n (vi),Rl m (vj)) = ∥F(Rl n (vi))−F(Rl m (vj))∥2 that are vertex normal vectors: F(vi) = n(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 12 / 29
22. Spiral patches Comparison ▶ The size of the operator Sp(vi,k) varies for the vertices ▶ Lim et al. either truncate or zero-pad each spiral depending on its size to compare two Spiral patches ▶ We propose a more accurate hierarchical comparison d(Sp(vi,k),Sp(vj,k)) = k ∑ l=0 d(Rl(vi),Rl(vj)) Two k-rings are compared by mapping the vertices of the largest ring to the smallest one: d(Rl(vi),Rl(vj)) = |Rl (vi )| ∑ n=0 d(Rl n (vi),Rl n′ (vj)) with n′ = n·|Rl (vj )| |Rl (vi )| and |Rl(vi)| > |Rl(vj)| First vertex spiral patch Second vertex spiral patch 0-ring 1-ring 2-ring The distance between two vertices is then the distance between their graph signal vectors: d(Rl n (vi),Rl m (vj)) = ∥F(Rl n (vi))−F(Rl m (vj))∥2 that are vertex normal vectors: F(vi) = n(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 12 / 29
23. Spiral patches Comparison ▶ The size of the operator Sp(vi,k) varies for the vertices ▶ Lim et al. either truncate or zero-pad each spiral depending on its size to compare two Spiral patches ▶ We propose a more accurate hierarchical comparison d(Sp(vi,k),Sp(vj,k)) = k ∑ l=0 d(Rl(vi),Rl(vj)) Two k-rings are compared by mapping the vertices of the largest ring to the smallest one: d(Rl(vi),Rl(vj)) = |Rl (vi )| ∑ n=0 d(Rl n (vi),Rl n′ (vj)) with n′ = n·|Rl (vj )| |Rl (vi )| and |Rl(vi)| > |Rl(vj)| First vertex spiral patch Second vertex spiral patch 0-ring 1-ring 2-ring The distance between two vertices is then the distance between their graph signal vectors: d(Rl n (vi),Rl m (vj)) = ∥F(Rl n (vi))−F(Rl m (vj))∥2 that are vertex normal vectors: F(vi) = n(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 12 / 29
24. Spiral patches Comparison ▶ The size of the operator Sp(vi,k) varies for the vertices ▶ Lim et al. either truncate or zero-pad each spiral depending on its size to compare two Spiral patches ▶ We propose a more accurate hierarchical comparison d(Sp(vi,k),Sp(vj,k)) = k ∑ l=0 d(Rl(vi),Rl(vj)) Two k-rings are compared by mapping the vertices of the largest ring to the smallest one: d(Rl(vi),Rl(vj)) = |Rl (vi )| ∑ n=0 d(Rl n (vi),Rl n′ (vj)) with n′ = n·|Rl (vj )| |Rl (vi )| and |Rl(vi)| > |Rl(vj)| First vertex spiral patch Second vertex spiral patch 0-ring 1-ring 2-ring The distance between two vertices is then the distance between their graph signal vectors: d(Rl n (vi),Rl m (vj)) = ∥F(Rl n (vi))−F(Rl m (vj))∥2 that are vertex normal vectors: F(vi) = n(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 12 / 29
25. Spiral patches Comparison ▶ The size of the operator Sp(vi,k) varies for the vertices ▶ Lim et al. either truncate or zero-pad each spiral depending on its size to compare two Spiral patches ▶ We propose a more accurate hierarchical comparison d(Sp(vi,k),Sp(vj,k)) = k ∑ l=0 d(Rl(vi),Rl(vj)) Two k-rings are compared by mapping the vertices of the largest ring to the smallest one: d(Rl(vi),Rl(vj)) = |Rl (vi )| ∑ n=0 d(Rl n (vi),Rl n′ (vj)) with n′ = n·|Rl (vj )| |Rl (vi )| and |Rl(vi)| > |Rl(vj)| First vertex spiral patch Second vertex spiral patch 0-ring 1-ring 2-ring The distance between two vertices is then the distance between their graph signal vectors: d(Rl n (vi),Rl m (vj)) = ∥F(Rl n (vi))−F(Rl m (vj))∥2 that are vertex normal vectors: F(vi) = n(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 12 / 29
26. Spiral patches Comparison ▶ The size of the operator Sp(vi,k) varies for the vertices ▶ Lim et al. either truncate or zero-pad each spiral depending on its size to compare two Spiral patches ▶ We propose a more accurate hierarchical comparison d(Sp(vi,k),Sp(vj,k)) = k ∑ l=0 d(Rl(vi),Rl(vj)) Two k-rings are compared by mapping the vertices of the largest ring to the smallest one: d(Rl(vi),Rl(vj)) = |Rl (vi )| ∑ n=0 d(Rl n (vi),Rl n′ (vj)) with n′ = n·|Rl (vj )| |Rl (vi )| and |Rl(vi)| > |Rl(vj)| First vertex spiral patch Second vertex spiral patch 0-ring 1-ring 2-ring The distance between two vertices is then the distance between their graph signal vectors: d(Rl n (vi),Rl m (vj)) = ∥F(Rl n (vi))−F(Rl m (vj))∥2 that are vertex normal vectors: F(vi) = n(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 12 / 29
27. Spiral patches Comparison ▶ The size of the operator Sp(vi,k) varies for the vertices ▶ Lim et al. either truncate or zero-pad each spiral depending on its size to compare two Spiral patches ▶ We propose a more accurate hierarchical comparison d(Sp(vi,k),Sp(vj,k)) = k ∑ l=0 d(Rl(vi),Rl(vj)) Two k-rings are compared by mapping the vertices of the largest ring to the smallest one: d(Rl(vi),Rl(vj)) = |Rl (vi )| ∑ n=0 d(Rl n (vi),Rl n′ (vj)) with n′ = n·|Rl (vj )| |Rl (vi )| and |Rl(vi)| > |Rl(vj)| First vertex spiral patch Second vertex spiral patch 0-ring 1-ring 2-ring The distance between two vertices is then the distance between their graph signal vectors: d(Rl n (vi),Rl m (vj)) = ∥F(Rl n (vi))−F(Rl m (vj))∥2 that are vertex normal vectors: F(vi) = n(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 12 / 29
28. Spiral patches Comparison ▶ The size of the operator Sp(vi,k) varies for the vertices ▶ Lim et al. either truncate or zero-pad each spiral depending on its size to compare two Spiral patches ▶ We propose a more accurate hierarchical comparison d(Sp(vi,k),Sp(vj,k)) = k ∑ l=0 d(Rl(vi),Rl(vj)) Two k-rings are compared by mapping the vertices of the largest ring to the smallest one: d(Rl(vi),Rl(vj)) = |Rl (vi )| ∑ n=0 d(Rl n (vi),Rl n′ (vj)) with n′ = n·|Rl (vj )| |Rl (vi )| and |Rl(vi)| > |Rl(vj)| First vertex spiral patch Second vertex spiral patch 0-ring 1-ring 2-ring The distance between two vertices is then the distance between their graph signal vectors: d(Rl n (vi),Rl m (vj)) = ∥F(Rl n (vi))−F(Rl m (vj))∥2 that are vertex normal vectors: F(vi) = n(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 12 / 29
29. Spiral patches Comparison ▶ The size of the operator Sp(vi,k) varies for the vertices ▶ Lim et al. either truncate or zero-pad each spiral depending on its size to compare two Spiral patches ▶ We propose a more accurate hierarchical comparison d(Sp(vi,k),Sp(vj,k)) = k ∑ l=0 d(Rl(vi),Rl(vj)) Two k-rings are compared by mapping the vertices of the largest ring to the smallest one: d(Rl(vi),Rl(vj)) = |Rl (vi )| ∑ n=0 d(Rl n (vi),Rl n′ (vj)) with n′ = n·|Rl (vj )| |Rl (vi )| and |Rl(vi)| > |Rl(vj)| First vertex spiral patch Second vertex spiral patch 0-ring 1-ring 2-ring The distance between two vertices is then the distance between their graph signal vectors: d(Rl n (vi),Rl m (vj)) = ∥F(Rl n (vi))−F(Rl m (vj))∥2 that are vertex normal vectors: F(vi) = n(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 12 / 29
30. Spiral patches Comparison ▶ The size of the operator Sp(vi,k) varies for the vertices ▶ Lim et al. either truncate or zero-pad each spiral depending on its size to compare two Spiral patches ▶ We propose a more accurate hierarchical comparison d(Sp(vi,k),Sp(vj,k)) = k ∑ l=0 d(Rl(vi),Rl(vj)) Two k-rings are compared by mapping the vertices of the largest ring to the smallest one: d(Rl(vi),Rl(vj)) = |Rl (vi )| ∑ n=0 d(Rl n (vi),Rl n′ (vj)) with n′ = n·|Rl (vj )| |Rl (vi )| and |Rl(vi)| > |Rl(vj)| First vertex spiral patch Second vertex spiral patch 0-ring 1-ring 2-ring The distance between two vertices is then the distance between their graph signal vectors: d(Rl n (vi),Rl m (vj)) = ∥F(Rl n (vi))−F(Rl m (vj))∥2 that are vertex normal vectors: F(vi) = n(vi) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 12 / 29
31. Spiral patches Comparison O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 13 / 29
32. Outline 1. Introduction 2. Spiral patches 3. Roughness, geometric and spectral saliencies 4. Saliency learning from multi-scale spiral cues O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 14 / 29
33. Outline 1. Introduction 2. Spiral patches 3. Roughness, geometric and spectral saliencies 4. Saliency learning from multi-scale spiral cues O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 15 / 29
34. Roughness ▶ Chen et al. have shown that curvature plays an important role for saliency detection ▶ Lee et al. have proposed the first mesh saliency detection based on differences between Gaussian-weighted mean curvatures ▶ We consider the more robust notion of roughness (from Wang et al.) that we extend to larger neighborhoods (by taking γ powers of the Laplacian) R(vi ) = κ(vi )− ∑ vj ∈γ-hop(vi ) Lγ i j ·κ(vj ) ∑ vj ∈γ-hop(vi ) Lγ i j with κ(vi ) the mean curvature. O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 16 / 29
35. Geometric and spectral saliencies Definitions Gradient operator ▶ We define the gradient at a vertex vi as the nonlocal vector of all the distances between the spiral patches of vi and its neighbors within its γ-hop(vi ): ∇f(vi ) = [d(Sp(vi,k),Sp(vj,k)),vj ∈ γ-hop(vi )]T with F(vi ) = N(vi ) ▶ One needs to have γ > k: the support of the gradient is larger than the spiral patch support. O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 17 / 29
36. Geometric and spectral saliencies Definitions Geometric saliency ▶ The geometric saliency is the normalized L1 norm of the gradient: GS(vi ) = 1 |γ-hop(vi )| ∥∇f(vi )∥1 = 1 |γ-hop(vi )| ∑ vj ∈γ-hop(vi ) d(Sp(vi,k),Sp(vj,k)) O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 18 / 29
37. Geometric and spectral saliencies Definitions Spectral saliency ▶ The structure tensor J is the outer product of the gradient: J(vi ) = ∇T f(vi )·∇f(vi ). ▶ Its eigenvalues provide a discriminative description of the local geometry by summarizing the distribution of the gradients ▶ The spectral saliency is defined as the Structure Tensor Total Variation (STTV): SS(vi ) = |γ-hop(vi )| ∑ j=1 λ2 j with λj the eigenvalues of J(vi ). O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 19 / 29
38. Roughness, Geometric and spectral saliencies O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 20 / 29
39. Outline 1. Introduction 2. Spiral patches 3. Roughness, geometric and spectral saliencies 4. Saliency learning from multi-scale spiral cues O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 21 / 29
40. Multi-scale Saliency cue computation ▶ We compute the saliency cues at three different resolutions obtained by mesh decimation with Qslim ▶ Given a mesh M1, we obtain meshes M2 and M3 with half and a quarter vertices ▶ We compute the saliency cues at each scale S1 M1 , S2 M2 , S3 M3 ▶ As the number of vertices decreases across scales, we cannot directly combine these computed multi-scale cues ▶ Given a saliency Sθ Mθ computed at scale θ, it is mapped back on the original mesh M1 by the projection Sθ M1 = proj(Sθ Mθ ,M1 ). ▶ This process yields three single-scale saliencies Sθ M1 for θ ∈ [1,3] O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 22 / 29
41. Multi-scale Saliency cue computation - Illustration O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 23 / 29
42. Learning saliency from the multi-scale saliency cues ▶ Saliency prediction is casted as a regression problem at the vertex level from our multi-scale spiral cues. ▶ A feature vector Φ(vi )T = [Rθ(vi ),GSθ(vi ),SSθ(vi ) : θ ∈ [1,3]] containing roughness, geometric and spectral saliency cues is computed at three different scales. ▶ This feature vector is fed to a MLP trained with MSE from the reference saliency values of the Schelling dataset (contains 400 meshes across 20 object categories). O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 24 / 29
43. Results on the Schelling dataset Schelling Dataset Year PLCC Multiscale Gaussian 2005 0,2230 Ranking patches 2015 0,3010 Spectral Processing 2014 0,3240 RPCA 2021 0,3360 CS2Point 2023 0,4010 Local to Global Saliency 2018 0,4070 Sparse metric-based 2020 0,4304 Cluster-based 2015 0,4321 Salient Regions 2016 0,4370 Cfs-CNN 2021 0,4550 MF-M5P 2012 0,4670 MIMO-GAN-CRF 2023 0,4760 Attention-embedding 2023 0,4910 Spiral Regressor (Ours) 2025 0,5567 Table: PLCC on the Schelling Dataset versus SOTA. O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 25 / 29
44. Comparison with two deep learning approaches Schelling Dataset PLCC Average results per category Spiral Regressor (Ours) 0.6062 MF-SAE (2019) 0,5639 Results on the decimated Schelling dataset Spiral Regressor (Ours) 0.5770 DS-Net (2023) 0,5732 Table: PLCC on the Schelling Dataset with the test protocols of MF-SAE and DS-NET. O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 26 / 29
45. Ablative study Roughness Geometric Spectral All PLCC 0.4045 0.4072 0.4157 0.5567 Table: Ablative study on the saliency cues. Scale PLCC 1 0.4880 2 0.4935 3 0.4667 1 & 2 0.5030 All 0.5567 Table: Ablative study on the scales. O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 27 / 29
46. Visual results Figure: Results on the Schelling dataset. Top: reference. Bottom : prediction. O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 28 / 29
47. The End Publications available at : https://lezoray.users.greyc.fr O. L´ ezoray & A. Nouri Learning 3D mesh saliency from spiral patch features 29 / 29