Rethinking the kernel camera framework

Video4Linux (V4L2) has enabled hundreds of different multimedia devices, from TV tuners to the webcam that has accompanied you during the pandemic. Unfortunately, it has been neglected by vendors when they have implemented camera stacks for mobile devices.

In this talk we will explore the reasons why V4L2 has not proliferated with complex ISP systems and what can be done to support those devices from an Open Source perspective.

Ricardo Ribalda, Google

Transcript

1. Sergey Senozhatsky, Fei Shao, Yunke Cao, Tomasz Figa, Ricardo Ribalda Kernel CAM Rethinking the kernel camera framework Paris, 1st June 2022
2. Proprietary + Confidential Once upon a time
3. Operating system (for driver developers) Program My framework Driver Hardware open(/dev/my_device) read(fd) ioctl(fd,IOC_,... ) my_init() my_alloc() my_read_from() inb() outb()
4. Operating system (for driver developers) Joystick interface xpad gamecon
5. Cameras are special
6. Great diversity of output • There are more than 200 video formats ◦ grep -c "^#define V4L2_PIX_FMT" include/uapi/linux/videodev2.h -> 202! Userspace Kernel Hardware Userspace Kernel Hardware Userspace Kernel Hardware
7. Multiple ways to do the same thing • Do you want a 640x480 image from a 24Mpix (6000*400)?: ◦ Cropping ◦ Binning ◦ Interpolate • Image is too dark? ◦ Exposure time ◦ Analog Gain ◦ Digital Gain
8. Do it fast or don’t do it • Big data rates ◦ 1080p RGB32@60fps -> 474 MiB/sec! • Low latency • Most of the time handled by other hardware ◦ GPU ◦ TPU ◦ CODEC
9. Cameras drive the consumer market • Work from home/anywhere • E-learning • Deciding factor for purchase
10. Cameras in Linux today
11. Video4Linux 2 (non media controller) Program Video Buffer 2 Driver Hardware Control Framework Video Helpers Media Subsystem libv4l2 • Output can be converted via software to 4 standard formats. • The driver makes most of the decisions for us. • Hardware produces decent images by default.
12. Video4Linux 2 (media controller) Program Video Buffer 2 Driver Hardware Control Framework Video Helpers Media Subsystem Video Buffer 2 Driver Hardware Control Framework Video Helpers Video Buffer 2 Driver Hardware Control Framework Video Helpers Video Buffer 2 Driver Hardware Control Framework Video Helpers Video Buffer 2 Driver Hardware Control Framework Video Helpers
13. Video4Linux 2 (post media controller) Program Video Buffer 2 Driver Hardware Control Framework Video Helpers Media Subsystem Video Buffer 2 Driver Hardware Control Framework Video Helpers Video Buffer 2 Driver Hardware Control Framework Video Helpers Video Buffer 2 Driver Hardware Control Framework Video Helpers Video Buffer 2 Driver Hardware Control Framework Video Helpers • Software stack needs to know about the internals of the hardware. • All the decisions are taken by the software stack. • A lot of Image tuning before we can use the images.
14. Libcamera
15. Libcamera
16. Secret Sauce
17. Secret Sauce
18. Secret Sauce
19. Secret Sauce
20. Recap Aware Userspace Kernel Hardware Agnostic Userspace Kernel Hardware V4L2 Media Controller
21. Proprietary + Confidential Kernel CAM
22. What is CAM? Userspace Kernel Hardware • New kernel subsystem. • No media abstractions. • Fluid collaboration between userspace and other subsystems. • Two components: ◦ Entities ◦ Operations
23. KCAM components: Entities Root MIPI Codec Sensor ISP DMA VCM • Organized in a tree, based on the hardware architecture. • Single register-set. • Can throw events.
24. KCAM components: Entities Root MIPI Codec Sensor ISP DMA VCM • Organized in a tree, based on the hardware architecture. • Single register-set. • Can throw events.
25. KCAM components: Operations • Read and/or write to an entity ◦ regmap ◦ parameter buffers • Can depend on: ◦ Events ◦ Other operation ◦ Fence • Can create a fence http://www.paella10.com/calculadora/paellas/paella-valenciana/
26. KCAM components: Operations Operation List Configure Sensor Trigger image Init ISP Start DMA ioctl ready Circular buffer fence_out fence_in signal
27. Show me the code!! #define CAM_MAX_DEPENDENCIES 8 struct cam_operation_add { __u32 id; __u32 fence_out; __u32 flags; /* * Pre-execution dependencies list and dependency execution mode */ __u32 mode; struct cam_dependency deps[CAM_MAX_DEPENDENCIES]; /* * Execution context specific data (if any) */ __u64 delay_ns; __u64 rd_wr_list; __u32 entity; } __attribute__((packed)); enum cam_dependency_type { CAM_DEPENDENCY_NONE, CAM_DEPENDENCY_OP, CAM_DEPENDENCY_EVENT, CAM_DEPENDENCY_FENCE_IN, }; struct cam_dependency { __u32 type; __u32 id; } __attribute__((packed)); enum cam_dependency_mode { CAM_DEPENDENCY_WEAK_ORDER, CAM_DEPENDENCY_STRICT_ORDER, };
28. Testing (lots of) • Software testing: ◦ kunit ◦ libkc ◦ vcam ◦ error injection • Hardware testing ◦ ChromeOS test infra ◦ human test
29. Comparison Userspace Kernel Hardware libcamera • Limitless functionality • Fast upstream • Small drivers • Operations vs Streams Kernel CAM Userspace Kernel Hardware libcamera Media Controller
30. Governance • There must be an open source stack before a driver is merged. ◦ Similar to DRM model. • Compliance-test with minimum requirements.
31. Proprietary + Confidential Future
32. Benchmark the stack Camera Service Kernel Userspace Hardware Clients Android Camera Framework Sors Svers SoC ISP Drivers h/EEPROM Sensor/Actuator/Flash/EEP ROM SoC VMs So SoC ISP Chrome Video Capture Device VMs Generic V4L2 Frameworks Sensor/Actuator/Fl ash/ EEPROM Drivers Sensor/Actuator/Fl ash/ EEPROM Drivers Sensor/Actuator/Flash/ EEPROM Drivers Camera Service Clients Android Camera Framework Sors Svers CAM drivers h/EEPROM Sensor/Actuator/Flash/EEP ROM SoC VMs So SoC ISP Chrome Video Capture Device VMs CAM Sensor/Actuator/Fl ash/ EEPROM Drivers Sensor/Actuator/Fl ash/ EEPROM Drivers Sensor/Actuator/Flash/ EEPROM Drivers Libcamera - cam Secret Sauce Secret Sauce
33. Land upstream
34. Contributing
35. How to contribute • Show me the code! ◦ https://chromium.googlesource.com/chromiumos/third_party/kernel/+/ref s/heads/kcam • Mailing List ◦ kernel-cam@chromium.org • Bugs: ◦ WIP • Join the team ◦ https://crosjobs.page.link/event
36. Thank you! ribalda@chromium.org https://crosjobs.page.link/event
37. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 37
38. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 38
39. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 39
40. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 40
41. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 41
42. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 42
43. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 43
44. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 44
45. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 45
46. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 46
47. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 47
48. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 48
49. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 49
50. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 50
51. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 51
52. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 46
53. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 53
54. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 54
55. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 55
56. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 56
57. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 57
58. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 58
59. Thank you! ribalda@chromium.org https://crosjobs.page.link/event 0xffffffff