ApacheCon Europe Big Data 2016 – Parquet Format in Practice & Detail

Apache Parquet is among the most commonly used column-oriented data formats in the big data processing space. It leverages various techniques to store data in a CPU- and I/O-efficient way. Furthermore, it has the capabilities to push-down analytical queries on the data to the I/O layer to avoid the loading of nonrelevant data chunks. With various Java and a C++ implementation, Parquet is also the perfect choice to exchange data between different technology stacks.

As part of this talk, a general introduction to the format and its techniques will be given. Their benefits and some of the inner workings will be explained to give a better understanding how Parquet achieves its performance. At the end, benchmarks comparing the new C++ & Python implementation with other formats will be shown.

Transcript

1. What is Parquet? How is it so efficient? Why should I actually use it? Parquet in Practice & Detail
2. About me • Data Scientist at Blue Yonder (@BlueYonderTech) • Committer to Apache {Arrow, Parquet} • Work in Python, Cython, C++11 and SQL xhochy uwe@apache.org
4. Agenda Origin and Use Case Parquet under the bonnet Python & C++ The Community and its neighbours
5. About Parquet 1. Columnar on-disk storage format 2. Started in fall 2012 by Cloudera & Twitter 3. July 2013: 1.0 release 4. top-level Apache project 5. Fall 2016: Python & C++ support 6. State of the art format in the Hadoop ecosystem • often used as the default I/O option
6. Why use Parquet? 1. Columnar format
 —> vectorized operations 2. Efficient encodings and compressions
 —> small size without the need for a fat CPU 3. Query push-down
 —> bring computation to the I/O layer 4. Language independent format
 —> libs in Java / Scala / C++ / Python /…
7. Who uses Parquet? • Query Engines • Hive • Impala • Drill • Presto • … • Frameworks • Spark • MapReduce • … • Pandas
8. • More than a flat table! • Structure borrowed from Dremel paper • https://blog.twitter.com/2013/dremel-made-simple-with-parquet Nested data Document DocId Links Name Backward Forward Language Url Code Country Columns: docid links.backward links.forward name.language.code name.language.country name.url
9. Why columnar? 2D Table row layout columnar layout
10. File Structure File RowGroup Column Chunks Page Statistics
11. Encodings • Know the data • Exploit the knowledge • Cheaper than universal compression • Example dataset: • NYC TLC Trip Record data for January 2016 • 1629 MiB as CSV • columns: bool(1), datetime(2), float(12), int(4) • Source: http://www.nyc.gov/html/tlc/html/about/ trip_record_data.shtml
12. Encodings — PLAIN • Simply write the binary representation to disk • Simple to read & write • Performance limited by I/O throughput • —> 1499 MiB
13. Encodings — RLE & Bit Packing • bit-packing: only use the necessary bit • RunLengthEncoding: 378 times „12“ • hybrid: dynamically choose the best • Used for Definition & Repetition levels
14. Encodings — Dictionary • PLAIN_DICTIONARY / RLE_DICTIONARY • every value is assigned a code • Dictionary: store a map of code —> value • Data: store only codes, use RLE on that • —> 329 MiB (22%)
15. Compression 1. Shrink data size independent of its content 2. More CPU intensive than encoding 3. encoding+compression performs better than compression alone with less CPU cost 4. LZO, Snappy, GZIP, Brotli
 —> If in doubt: use Snappy 5. GZIP: 174 MiB (11%)
 Snappy: 216 MiB (14 %)
16. https://github.com/apache/parquet-mr/pull/384
17. Query pushdown 1. Only load used data 1. skip columns that are not needed 2. skip (chunks of) rows that not relevant 2. saves I/O load as the data is not transferred 3. saves CPU as the data is not decoded
18. Competitors (Python) • HDF5 • binary (with schema) • fast, just not with strings • not a first-class citizen in the Hadoop ecosystem • msgpack • fast but unstable • CSV • The universal standard. • row-based • schema-less
19. C++ 1. General purpose read & write of Parquet • data structure independent • pluggable interfaces (allocator, I/O, …) 2. Routines to read into specific data structures • Apache Arrow • …
20. Use Parquet in Python https://pyarrow.readthedocs.io/en/latest/install.html#building-from-source
21. Get involved! 1. Mailinglist: dev@parquet.apache.org 2. Website: https://parquet.apache.org/ 3. Or directly start contributing by grabbing an issue on https://issues.apache.org/jira/browse/PARQUET 4. Slack: https://parquet-slack-invite.herokuapp.com/
22. We’re hiring! Questions?!