Embedded Linux Optimization Techniques: How Not To Be Slow ?

This talk is a retroactive overview of an embedded system's project issues and provides useful tips&tricks about what must be avoided in order not to introduce flaws by design. It provides tools and techniques for debugging your system and let you avoid falling into pitfalls.

Transcript

1. COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. 1 Benjamin Zores ELCE 2010 – 26th October 2011 – Prague, Czech Republic Embedded Linux Optimization Techniques: How Not To Be Slow ? This is a placeholder image only. Please select an image to reflect the content of your PPT presentation. Visit our approved corporate photography collection on the MarCom Store at: https://all.alcatel-lucent.com/marcomstore/
2. 2 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. Embedded Linux Optimizations Techniques: How Not To Be Slow ? About Me … ALCATEL LUCENT SOFTWARE ARCHITECT • Expert and Evangelist on Open Source Software. • 8y experience on various multimedia/network embedded devices design. • From low-level BSP integration to global applicative software architecture. OPEN SOURCE PROJECT FOUNDER, LEADER AND/OR CONTRIBUTOR FOR: • OpenBricks Embedded Linux cross-build framework. • GeeXboX Embedded multimedia HTPC distribution. • Enna EFL Media Center. • uShare UPnP A/V and DLNA Media Server. • MPlayer Linux media player application. EMBEDDED LINUX CONFERENCE FORMER EDITIONS SPEAKER • ELC 2010 GeeXboX Enna: Embedded Media Center. • ELC-E 2010 State of Multimedia in 2010 Embedded Linux Devices.
3. 3 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. From our “IP Touch” IP phone ... - MIPS32 @ 275 MHz. - 8/16 MB RAM, 4/8/16 MB NOR. - Physical keys input. - Basic 2D framebuffer display. - Powered by VxWorks OS. Embedded Linux Optimizations Techniques: How Not To Be Slow ? About My Job … … to next-generation enterprise IP phones. - Brainstorming exercise from our R&D Labs. - Introduced as a proof-of-concept feasibility study, allowing us to explore modern Linux technologies. - Early Requirements: - Powered by GNU/Linux OS, not Android. - Open to HTML/JS-based WebApps. - Remaining parts are open to imagination.
4. 4 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. • Return of experience from feasibility study: - You may want to see this presentation as one big exercise. - It won’t help you boost your system (sorry folks ). - But hopefully it’ll prevent you from facing some common troubles. • Share a few tips and tricks for: - Correctly choosing your hardware. - Wisely selecting your software architecture and components. - Measuring and profiling your system. - Isolating the performances bottlenecks. - Optimizing your Linux embedded system. • Ultimately, avoid your software to be slow by design. Embedded Linux Optimizations Techniques: How Not To Be Slow ? What You May Expect …
5. 5 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. In 20 years (from my i286 Desktop to my Core i5 laptop): • My CPU got 10000x faster. • My RAM got 12800x bigger (and faster). • My HDD got 8192x times bigger (and faster). And yet my PC takes ages to boot and I need more time to open up my text editor ... Seriously, What Went Wrong ??? Embedded Linux Optimizations Techniques: How Not To Be Slow ? Preamble
6. 6 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. Rule #1: Know Your Hardware ! Embedded Linux Optimizations Techniques: How Not To Be Slow ?
7. 7 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. • CPU SIMD Optimizations and Execution Modes: - Thumb-1/2: Tradeoff between code size and efficiency … - Jazelle: Don’t do JAVA on ARM without it ! - VFP / NEON: Impressive performance boost on all FPU operations; Use integer-based routines otherwise. => Tradeoff between performances and portability (generic builds are meant for portability). • Audio Management: - Choice #1: Legacy hardware DSP audio decoding (with complex shmem architecture) ? - Choice #2: Software Cortex-A9 audio decoding (within 50 MHz or so) ? • Display / Input Optimizations: - GPU Capabilities: 2D blitting, 3D, post-processing ? Ensure you’ll never fail into software fallback ! Don’t bother rendering more frames than your LCD can display. - TouchScreen: Calibrate your driver not to read more often than your max display FPS rate. Reading on I2C consumes resources that you may never be able to interpret. Embedded Linux Optimizations Techniques: How Not To Be Slow ? Common Considerations …
8. 8 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. Embedded Linux Optimizations Techniques: How Not To Be Slow ? Embedded SoC Comparison … Our Test Case SoC Apple iPhone 3GS Apple iPhone 4 Samsung Galaxy S2 Today PC Introduction Date 2009 2009 2010 2011 2011 CPU ARM1176 ARM Cortex-A8 ARM Cortex-A8 ARM Cortex-A9 MP Intel Core-i5 2500T Frequency (MHZ) 500 600 1000 2 x 1200 4 x 2300 Memory Size (MB) 256 256 512 1024 Unlimited L2 Cache Size (kB) None 256 640 1024 6144 FPU No Yes Yes Yes Yes Specialized Instructions Thumb-1, Jazelle Thumb-2, Jazelle, VFPv3, NEON Thumb-2, Jazelle, VFPv3, NEON Thumb-2, Jazelle, VFPv3, NEON MMX, SSEx Hardware GFX Limited 2D Blitter Full 3D Full 3D Full 3D Full 3D Hardware Video Engine Limited SD Limited SD Limited HD Full HD Full HD Memory Bandwidth (GB/s) 1.33 1.6 3.2 6.4 21.3 Performances (DMIPS) 625 (1.25 DMIPS/MHz) 1200 (2.00 DMIPS/MHz) 2000 (2.00 DMIPS/MHz) 6000 (2.5 DMIPS/MHz/Core) 59800 (6.5 DMIPS/MHz/Core) CPU PC Equivalency Pentium Pro @ 233 MHz (1996) Pentium II @ 400 MHZ (1998) Pentium III @ 600 MHz (2000) 2x ATOM @ 1.3 GHz (2008) N.A.
9. 9 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. Rule #2: Embedded is NOT Desktop ! Embedded Linux Optimizations Techniques: How Not To Be Slow ?
10. 10 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. • Brutal Facts: - Embedded devices get more and more powerful each year. - But not everybody uses high-end ARM SoCs. - Still resources limited: CPU, memory bandwidth, run on batteries, slow I/Os ... So why would you use the same kind of software than on a PC ? Android somehow came out and diverged from GNU/Linux for some reason ... • Good Hints on some desktop-oriented performances eating software/technologies: - Abstraction Framework, - Messaging Bus, - Garbage Collector, - Virtual Machine, Use these with care ! Badly used, they are sources of terrible difficulties. Embedded Linux Optimizations Techniques: How Not To Be Slow ? Embedded is NOT Desktop … - Interpreted Language, - XML, - Data Parsing and Serialization.
11. 11 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. Rule #3: Isolate Your System’s Bottlenecks ! Embedded Linux Optimizations Techniques: How Not To Be Slow ?
12. 12 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. • Optimization requires accurate measurement. • Measure must: - Be deterministic and repeatable. - Not impact system’s behavior. - Be the less intrusive as possible. • Try to cover as much usability scenarios as possible; don’t limit yourself to average Joe use cases. Embedded Linux Optimizations Techniques: How Not To Be Slow ? Measurement and Benchmarking …
13. 13 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. • Need for global feature/solution benchmark (requires end-to-end implementation) - At Input Level: • Record scenario: At tslib level, we retrieve X/Y coordinates, pressure level and timestamp. • Replay scenario: We inject raw data to /dev/input/eventX and let the software handle events. - => Least intrusive input (mimics final human behavior). • Can also be fully automated through simple client/server approach. Embedded Linux Optimizations Techniques: How Not To Be Slow ? Benchmarking: An External Approach (1/2) …
14. 14 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. • At Output Level: • External video camera recording. • Need to define scenario start and end conditions (e.g. some widgets appearance / disappearance). • On a remote PC, play back the recorded video to measure delta between start/stop conditions using OpenCV libraries. - Measure is the least intrusive (no impact on target). - Can be used for non-regression tests on a given global feature. - But you still have no clue which exact part of your code is slow. - Accuracy depends on camera's capability (usually 30fps, so 33ms minimum threshold). Embedded Linux Optimizations Techniques: How Not To Be Slow ? Benchmarking: An External Approach (2/2) …
15. 15 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. • Modern Linux kernel introduced support for hardware counters - Introduced as Performance Counters ( see http://goo.gl/LldPv ) in 2.6.31. - Renamed as Performance Events ( see http://goo.gl/KWIfo ) in 2.6.32+ - Successor of Oprofile. - See tools/perf/ directory in kernel. • Example of usage (on OMAP 4430 Pandaboard): - Requirements: You need debugging symbols to accurately trace your system. - User-space Profiling: perf top –U - Kernel–space Profiling: perf top -K Embedded Linux Optimizations Techniques: How Not To Be Slow ? Benchmarking: An Internal Approach …
16. 16 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. Perftools also can be used for global system profiling by generating a time chart: • On target: perf timechart record (will generate your perf.data samples). • On host: perf timechart –i perf.data –o output.svg Embedded Linux Optimizations Techniques: How Not To Be Slow ? Determining Workflows … D-Bus events messaging can be generated using dbus-monitor, or better, bustle. - Though very intrusive (impacts on performances). - Can be extended to include tcpdump network messages into workflow. - See http://willthompson.co.uk/bustle/ for more details.
17. 17 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. Rule #4: Kill the Message Bus ! “Don’t Shoot The Messenger”, Shakespeare, 1598 Embedded Linux Optimizations Techniques: How Not To Be Slow ?
18. 18 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. • Study about different RPC architectures: - Basic RPC function call between client and server. - Measure consists of 10000 calls on an AMD Athlon XP 2800+, 1 GB RAM. • Interesting results, CORBA is known to be slow but: - DCOP is 3x slower. - DBUS is 18x slower. • Full analysis details are available at: - http://eleceng.dit.ie/frank/rpc/CORBAGnomeDBUSPerformanceAnalysis.pdf Embedded Linux Optimizations Techniques: How Not To Be Slow ? RPC Frameworks Comparison … CORBA (ms) DCOP (ms) D-Bus (ms) VOID Call 626 1769 9783 IN Integer Call 629 1859 10469 OUT Integer Call 660 1824 10399 IN/OUT Integer Call 686 1903 11162 IN String Call 650 1902 10510 OUT String Call 730 1870 10455 IN/OUT String Call 682 1952 11239
19. 19 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. • Some IPC benchmark figures: - Performed on TI Pandaboard (TI OMAP 4430 @ 2x 1GHz). - Reading rows from a SQLite database (75k rows chunks). - Different use cases: - Native SQLite direct library function call. - Client/Server approach with UNIX sockets messaging channel. - Client/Server approach with D-Bus messaging channel. - Client/Server approach with D-Bus messaging channel with file descriptor support. • See “IPC Performance” utility (http://goo.gl/5ygSU). Embedded Linux Optimizations Techniques: How Not To Be Slow ? Messaging Benchmarks … 0 5000 10000 15000 20000 25000 30000 35000 1000 75 000 150 000 225 000 300 000 Direct UNIX Socket D-Bus D-Bus FD (rows) (ms)
20. 20 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. • D-Bus really is meant only for eventing/broadcasting; avoid passing data on it. • There are more efficient and straightforward alternatives between 2 applications. • Avoid passing large data: use D-Bus with UNIX file descriptor support instead. • Remove paranoid message header/body checks/assertions: d i f f - N a u r d b u s - 1 . 5 . 0 . o r i g / d b u s / d b u s - m e s s a g e . c d b u s - 1 . 5 . 0 / d b u s / d b u s - m e s s a g e . c - - - d b u s - 1 . 5 . 0 . o r i g / d b u s / d b u s - m e s s a g e . c 2 0 1 1 - 0 8 - 0 6 1 2 : 3 1 : 5 0 . 6 2 4 2 4 8 0 7 1 + 0 2 0 0 + + + d b u s - 1 . 5 . 0 / d b u s / d b u s - m e s s a g e . c 2 0 1 1 - 0 8 - 0 6 1 2 : 3 2 : 4 9 . 2 6 4 2 4 8 1 0 3 + 0 2 0 0 @ @ - 3 9 5 5 , 7 + 3 9 5 5 , 7 @ @ D B u s V a l i d a t i o n M o d e m o d e ; d b u s _ u i n t 3 2 _ t n _ u n i x _ f d s = 0 ; - m o d e = D B U S _ V A L I D A T I O N _ M O D E _ D A T A _ I S _ U N T R U S T E D ; + m o d e = D B U S _ V A L I D A T I O N _ M O D E _ W E _ T R U S T _ T H I S _ D A T A _ A B S O L U T E L Y ; o o m = F A L S E ; Embedded Linux Optimizations Techniques: How Not To Be Slow ? D-Bus Messaging: Be Careful … 25% D-Bus Messaging Speedup
21. 21 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. Rule #5: Go Native !!! Embedded Linux Optimizations Techniques: How Not To Be Slow ?
22. 22 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. • Desktop Legacy Applicative Architecture Sample: - C/C++ code. - Graphical applications using native function calls to libraries. - Eventing through signals. - IPC through SysV IPC or UNIX /TCPIP Sockets. - Mastered memory usage. - Easily debuggable (using gdb or valgrind). - Easily profilable (using gcov, Oprofile, or Linux PerfTools). Application’s portability, skin-ability and easiness of deployment really depends on how you write your code  Embedded Linux Optimizations Techniques: How Not To Be Slow ? Desktop Software Architecture Comparison …
23. 23 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. • Desktop Web Applicative Architecture Sample: - JS/HTML/CSS code. - Graphical user application using interpreted JavaScript functions with bindings to native middleware apps/libs. - WebServices usage and JSON data (de)serialization to exchange with middleware apps. - JavaScript-based Apps: - Easy and fast to write. - Even easier to skin, customize and deploy. - But interpreted and compiled in time, making them really hard to impossible to properly debug and/or profile. - Slower than any native equivalent. Tradeoff needs to be made. Embedded Linux Optimizations Techniques: How Not To Be Slow ? Desktop Software Architecture Comparison …
24. 24 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. • Browser Architecture: - Makes JS portable to your legacy OS. - Specific bindings for OS and architectures. - Specifically designed modules to access the hardware beneath (audio, video, graphics, WebGL ...). • OS Concepts: - Scheduler and Memory Allocator. - Applications Security / Sandboxing ... • Bindings for OS native services: - HTML5 Local Storage - HTML5 Audio/Video tags … Modern browsers are to JavaScript what POSIX used to be for C. Embedded Linux Optimizations Techniques: How Not To Be Slow ? Browser Architecture Perspective: A Virtualized OS…
25. 25 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. Conclusion Embedded Linux Optimizations Techniques: How Not To Be Slow ?
26. 26 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. Know Your hardware. Embedded is NOT Desktop. Isolate Your System’s Bottlenecks. Kill the Message Bus. Go Native ! Embedded Linux Optimizations Techniques: How Not To Be Slow ? The Embedded Linux Rules Set …
27. 27 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. • Your SoC never has been that powerful ... - ... ain't a reason for wasting it though. • Don’t mimic software development trend ! - Embedded Systems aren’t desktop PCs. - They can’t be programmed the same way. - Guess why Google’s Android differs from GNU/Linux ? • Back to the Basics ! - It's not that more difficult to code in C/C++ than in JS or other "high-level language". - It's been proven to work; guess how's been coded your high-level language. - Go straight to the point: avoid as many indirection layers as you can. Embedded Linux Optimizations Techniques: How Not To Be Slow ? Conclusion …
28. 28 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. Backup Slides: Miscellaneous Tips & Tricks Embedded Linux Optimizations Techniques: How Not To Be Slow ?
29. 29 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. • Data (De)Serialization: - Consumes a lot of CPU time: avoid at ALL cost whenever possible. - Only serialize elements you really need to, not the whole class content. - When possible, use shared memory instead. - Check serializer routines from the FOSS you include: - e.g. qjson adds extra white spaces that make it nice on Wireshark. - Our serialized 'contact' object (40 kB) contained 4 kB of white spaces. • Logs (seen in so many programs …): - Check log macro level THEN compute log string, and not the opposite: # d e f i n e L O G ( l v l , f o r m a t , a r g . . . ) d o { \ # d e f i n e L O G ( l v l , f o r m a t , a r g . . . ) d o { \ s n p r i n t f ( f m t , s i z e o f ( f m t ) , " % s : % s \ n " , f o r m a t ) ; \ i f ( l v l < D E B U G _ L E V E L ) \ v a _ s t a r t ( v a , f o r m a t ) ; \ s n p r i n t f ( f m t , s i z e o f ( f m t ) , " % s : % s \ n " , f o r m a t ) ; \ i f ( l v l < D E B U G _ L E V E L ) \ v a _ s t a r t ( v a , f o r m a t ) ; \ v f p r i n t f ( s t d e r r , f m t , v a ) ; \ v f p r i n t f ( s t d e r r , f m t , v a ) ; \ v a _ e n d ( v a ) ; \ v a _ e n d ( v a ) ; \ } w h i l e ( 0 ) ; } w h i l e ( 0 ) ; Embedded Linux Optimizations Techniques: How Not To Be Slow ? Miscellaneous Tips & Tricks (1/2) …
30. 30 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. • Memory Allocation: - Avoid Memory Fragmentation: you’d better keep some objects in memory than continuously (de)allocating them. - Real-Time Memory Management: for performance critical apps, you’d better use a pre-allocated memory pool, that will never go in page fault (slooooooow). • Compiler Optimizations: - GCC can do wonders by adding various optimizations flags (usually –march=…, -Ox, and –mfpu=neon when using floating point on ARM), but it’s a tradeoff with portability. - Isolate your critical sections code into dedicated C file and use Acovea (see http://goo.gl/KdLqK ) for determining the best compiler options through evolutionary algorithms. - Rewrite your critical sections code using GCC inline ASM (very useful on codec routines). - See some FPU calculation on Pandaboard: Go http://goo.gl/hT9Q7 for benchmark sources. Embedded Linux Optimizations Techniques: How Not To Be Slow ? Miscellaneous Tips & Tricks (2/2) … Measured Execution Time (usec) C 2730 (reference time) C with GCC Optimizations -O3 -fomit-frame-pointer -mcpu=cortex-a9 -ftree-vectorize -ffast-math 2594 (1.05x faster) C with GCC Optimizations and NEON SIMD -mfloat-abi=softfp -mfpu=neon 366 (7.45x faster) Inline NEON ASM -mfloat-abi=softfp -mfpu=neon 275 (9.9x faster)
31. 31 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. • Web Rendering Engine Optimizations: - Tune your rendering engine to use JIT. - Tune your rendering engine not to render invisible widgets (off-screen or hidden layers). - Tune your rendering engine to have a limited object cache (otherwise you'll quickly get low on free memory, which will induce more page faults and slow down your whole system until OOM gets its job done). • Simplify your CSS: - Use regular images instead of slow CSS transformations. - Use solid pattern instead of gradients. - Use correct images size instead of software rescaling them each time. - E.g: Scroll lists with CSS gradient pattern took 90% CPU while using CSS solid pattern only took 3% in tests. Embedded Linux Optimizations Techniques: How Not To Be Slow ? Web Technologies Optimizations (if you _really_ wanna go this way) …
32. 32 COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED. • Parsing HTML is a CPU hog: Remove complexity by lowering DOM's depth as much as possible. • When designing WebServices, you’d better return a lot of information in one call than to proceed with multiple WS calls (anyway, you're asynchronous, right ?) • Don't refresh your MMI as much as possible, this is a terribly slow operation: You’d better wait for all of your data to be ready. • If you're lucky enough to have a recent engine, try delegating some graphics to GPU through OpenGL/WebGL to provide hardware acceleration. • Additional JavaScript tips were provided at Oreilly’s conference “How to Make JavaScript Fast” (see http://goo.gl/K7VYd ). Process as much logic code as possible in C/C++ (i.e. go Native !!) => See Google’s Chrome NativeClient approach ( http://code.google.com/p/nativeclient/ ). Embedded Linux Optimizations Techniques: How Not To Be Slow ? Web Technologies Optimizations (if you _really_ wanna go this way) …