ISPA 2023

Transcript

1. 1 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 No Reference 3D mesh quality assessment using deep convolutional features Zaineb Ibork(1,2), Anass Nouri(1), Olivier Lezoray(2), Christophe Charrier(2), Raja Touahni(1) (1)SETIME Laboratory, Information Processing and A.I Team, Faculty of Sciences, Ibn Tofail University, Kenitra, Morocco (2)Normandie Univ, UNICAEN, ENSICAEN, CNRS, GREYC, 14000 Caen, France {zaineb.ibork,anass.nouri, touahni.raja}@uit.ac.ma {olivier.lezoray, christophe.charrier}@unicaen.fr le 18/09/2023
2. 2 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 2 Zaineb Ibork Plan Introduction and objectives 01 Methodology and Database construction 02 Experimentations and results 04 Proposed method 03 Conclusion 05
3. 3 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 Problematic ➢ 3D Meshes can be altered by pre-processing steps as: acquisition, compression or denoising ➢ In this context, visual quality assessment algorithms can be used to quantify the amount of distortions that affect a 3D mesh and hence degrade its visual rendering. ➢ Objective visual quality assessment methods can be categorized based on the availability of a reference object: Full-Reference(FR), No-Reference(NR) and Reduced-Reference (RR) methods. ➢ We introduce a no-reference mesh quality assessment index based on deep convolutional features named: DCFQI (Deep Convolutional Features Quality Index) Objective ➢ Assess the visual quality of a 3D mesh without referring to its reference version.
4. 4 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 4 Zaineb Ibork Plan Introduction and objectives 01 Methodology and Database construction 02 Experimentations and results 04 Proposed method 03 Conclusion 05
5. 5 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 Methodology ➢ We utilized a pre-trained CNN-based model with thirteen layers for feature extraction and an MLP with two layers for regression, considering 2D representations of a mesh as input and generating a subjective quality score as the output. ➢ After training, the network became capable of predicting the quality of a mesh that was not part of the training phase. ➢ We constructed a base model using 'leave-one-model-out cross-validation'. ➢ Subsequently, we built another model using the same method, but it was initialized with weights from a model trained cumulatively on the database.
6. 6 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 Databases construction: 3D mesh rendering Armadillo’s 11 rendered views: the views in the first row are obtained by fixing θe = 0 and graduating θa by 60 degrees at a time. We switch this process in the second row
7. 7 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 Database construction: Transformation Process from View to Patches
8. 8 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 8 Zaineb Ibork Plan Introduction and objectives 01 Methodology and Database construction 02 Experimentations and results 04 Proposed method 03 Conclusion 05
9. 9 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 The pipeline of the proposed quality assessment index that estimate the quality of a rendered image (2D view or 2D view patch)
10. 10 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 10 Zaineb Ibork Plan Introduction and objectives 01 Methodology and Database construction 02 Experimentations and results 04 Proposed method 03 Conclusion 05
11. 11 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 Description ➢ 88 models between 40K and 50K vertices generated from 4 reference objects. ➢ Two types of distortion: noise addition and smoothing ➢ Distortions were applied with different strengths and at four locations: on the whole model, on smooth areas, on rough areas and on intermediate areas. ➢ Subjective evaluations were made at normal viewing distance, using a SSIS (Single Stimulus Impairment Scale) method with 12 observers. ➢ Each model is associated to a Mean Opinion Score after normalization and outlier removal. Mesh database: LIRIS/EPFL 3D Model General-Purpose Armadillo Dyno Venus RockerArm
12. 12 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 Mesh database: LIRIS/EPFL 3D Model General-Purpose Smoothed mesh Taubin15 Noised mesh noise0015 Original mesh
13. 13 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 Resulting neural networks issued from the leave-one-mesh-out cross-validation (LOMO-CV)
14. 14 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 Base Model SROOC VALUES FOR THE BASE MODEL TRAINED FOR 20 EPOCHS ON Bview OR Bpatch DATABASES
15. 15 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 Base Model SROOC VALUES FOR THE BASE MODEL TRAINED WITH AN EARLY STOPPING ON Bview AND Bpatch DATABASES
16. 16 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 Cumulative Model Training
17. 17 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 Cumulative Model results SROOC AND PCC CUMULATIVE MODEL (CM) VS RE-TRAINED CUMULATIVE MODEL (RCM) RESULTS TRAINED ON DATABASES BVIEW OR BPATCH
18. 18 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 Results versus SOTA COMPARISON OF OUR PROPOSED DCFQI VIEW/PATCH BASED BASE AND CUMULATIVE MODELS (DCFQI-VBM & DCFQI-VCM / DCFQI-PBM & DCFQI-PCM RESPECTIVELY) WITH THE STATE OF THE ART NO-REFERENCE METRICS.
19. 19 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 19 Zaineb Ibork Plan Introduction and objectives 01 Methodology and Database construction 02 Experimentations and results 04 Proposed method 03 Conclusion 05
20. 20 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 Conclusion Future works In this paper, we presented a no-reference mesh quality assessment approach: ➢ It renders the mesh in 2D views that can be subsequently divided in patches. ➢ From these images, deep features are extracted by the pre-trained VGG16 CNN and fed into a MLP that performs quality prediction. ➢ This base model is competitive with the state-of-the-art, even at the view level. ➢ Finally a cumulative training has been proposed to obtain a single final model for prediction that goes beyond the state-of-the-art. Future works will consider ➢ Combining both view and patch predictions. ➢ The case of colored meshes.
21. 21 Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 Thank you! Any questions? Special thanks: This work received funding from PHC TOUBKAL TBK/22/142-CAMPUS N°47259YH.