Revolute (Lightning Talk @ Big Data Scala meetup)

Project Revolute (Lightning Talk @ Big Data Scala meetup)

Transcript

3. * Revolute
4. Alex Boisvert
5. Work @
6. SQL-like query language (for Big Data)
7. ( Scala ) embedded DSL
8. * Apache Hive * Scala Query * Cascalog Inspired by
9. Familiar + Type-safe + Expressive + Interactive
10. “Thin Layer” on top of Cascading.
11. A Taste of Revolute →
12. object Persons extends Table[(String, Int, String)] { def name = column[String]("name") def age = column[Int]("age") def gender = column[String]("gender") def * = name ~ age ~ gender }
13. -- SQL select p.* from Persons ↓ /* Revolute */ for { p Persons } yield p.* ←
14. /* output multiple fields */ for { p Persons } ← yield p.name ~ p.age
15. /* filtering */ for { p Persons if (p.age > 21) ← } yield p.name ~ p.age
16. /* combinators for complex expressions */ for { p Persons ← if (p.age > 21) && (gender === “m”) } yield p.name ~ p.age
17. /* How much time are people spending per location? */ for { Join(p, l) (Persons innerJoin Locations) ← on (_.name is _.name) _ Query.groupBy (p.name ~ l.city) ← time TimeSpent(l.timestamp) ← } yield p.name ~ l.city ~ time
18. // context provides table bindings for taps/sinks // // e.g. Traffic table HDFS or S3 → // Parter table MySQL → // Summary table Google Spreadsheet → flow(context) { val myQuery = for { … } yield … insert { for { (partner, segment, views) <- myQuery if segment in Set("a", "b", "c") } yield partner ~ views } into Summary }
19. * Query - one or more table joined together (“join”) - field selection and function application (“select”) - one or more filters (“where”) - grouping and sorting (“group by”, “sort by”) - aggregation based on groupings (“count()”, …) * Nest & chain queries * 1-1, 1-0/1, 1-N mappings * Null, Option & PartialFunction filtering … and (eventually) more awesome.
20. aboisvert / revolute @ github