Share
Embed
Aggregation Framework | Mikhail Burtylev
Mikhail Burtylev
Meetup #6
Minsk MongoDB User Group
September 18, 2012
Transcript
37 slides
-
Map/Reduce • Big hammer - simple tasks should be easier.
• JavaScript Problems to solve -
Declarative framework No JavaScript required Describe a chain of operations
to apply Expression evaluation Return computed values C++ implementation Higher performance than JavaScript Solu'on (v2.1+) -
Components Pipelines A pipeline is a series of operations.
The members of a collection are passed through a pipeline to produce a result. Expressions Computed values (fields & operators). Document format using prefixes. -
Invoca'on Command: db.runCommand{ aggregate: “users", pipeline: [{$op1, $op2, ...}]});
Helper: db.users.aggregate( {$pipeline_operation1}, {$pipeline_operation2}, ...); -
Result { result: [ {…}, … ], ok: 1
} Document size limit (16MB) -
$match Filter documents. Similar to find({...}). $where and geospatial
operations are not allowed. db.users.aggregate([ {$match: {age: {$gt: 16}}} ]); -
$project Reshape documents. Similar to find()’s optional argument. Include/exclude/rename
fields. Computed fields. Nested documents. -
Projec'ons {$project: { title: 1, /* include field if
it exists */ _id: 0, /* exclude */ /* calculated field */ age_month: {$multiply: [“$age”, 12]}, user: “$name”, /* rename */ city: “$home.city”, /* move to top */ add: {age: “$age”} /* sub-document */ } } -
$unwind Hands out array elements one at a time
in the context of their surrounding documents. • Multiple documents can be generated from a single source document. • Nothing for absent or empty array. • Error for non-array fields. -
$group Aggregates items into buckets defined by a key
(_id): • single field • document • constant value Other fields are computed: $sum, $avg, $min, $max, $first, $last, $push, $addToSet (no sub-documents allowed) -
Unwinding, Grouping db.authors.aggregate( {$project: {name: 1, tags: 1, _id:
0}}, {$unwind: "$tags"}, {$group: { _id: “$tags”, /* single field */ author_count: {$sum: 1}, /*count*/ /* array construction */ authors: {$addToSet: "$name"}}}); -
$limit Only allow the specified number of documents to
pass. Similar to find(…).limit(…). db.aggregate.users([ {$sort: {age: -1}}, {$limit: 10}]); -
$skip Skips over the specified number of documents. Similar
to find(…).skip(…). db.aggregate.users([ {$sort: {age: -1}}, {$skip: 10}]); -
New Opera'ons Aggregation Framework: New operations can be added
in future versions. $out Saves the document stream to a collection. Functions like a tee, so that intermediate results can be saved. -
Expressions Return computed values. Used with $project and $group.
Can be nested. $multiply: [“$age”, 12] $multiply: [ {$ifNull: [“$age”, 10]}, 12] Boolean Comparison Arithmetic String Date Conditional -
Boolean operators $not $and, $or (input array of one
or more values, short-circuit logic) BSON conversion standards: • null, undefined, 0 - false • Non-zero values, dates, strings, objects - true -
Comparison operators Compare numbers, strings and dates. Two operands
in array. $cmp, $eq, $ne $gt, $gte, $lt, $lte {$gt: [“$field1”, “$field2”]} -
Arithme'c operators $add, $multiply Multiple numbers in array. $subtract,
$divide, $mod Two numbers in array. -
String operators $strcasecmp Case-insensitive $toLower, $toUpper $substr Not encoding
aware! (Latin alphabet) -
Date operators Extract values from date objects. $dayOfYear, $dayOfMonth,
$dayOfWeek $year, $month, $week $hour, $minute, $second -
Condi'onal operators Null value substitution {$ifNull: [“$f1”, “$f2”]} Ternary
conditional {$cond: [ <bool expression>, <true-case>, <false-case>]} -
Early Filtering: $match (appropriate index can be used to avoid
scanning the entire collection). Sorting: $sort operator can take advantage of an index when placed at the beginning of the pipeline or placed before the following aggregation operators: $project, $unwind, $group. Op'miza'on -
Sharding Mongos • analyzes pipeline, and forwards operations up
to $group or $sort to shards; • merges $sort and $group results; • processes remaining operations. Early $match can exclude shards. Required more CPU resources. -
[ {$match: {…}}, /* filter data */ {$sort: {…}}, /*
sort documents */ {$limit: {…}}, /* limit document stream */ {group: {…}}, /* group data */ {$sort: {…}}, /* sort by calc. values */ {$project: {…}} /* reshape data */ ] Order of opera'ons -
SQL & MongoDB db.users.aggregate([ {$match: { age: {$gte: 16}}},
{$group: { _id: “$city”, count: {$sum: 1}}}, {$match: { count: {$lt: 10000}}}, {$sort: { adult_count: -1}}, {$project: { _id: 0, city: “$_id”, adult_count: “$count”}}]); select - $project where/having - $match group by - $group order by - $sort select city, count(id) from users where age >= 16 group by city having count(id) < 10000 order by 2 desc -
Limita'ons The pipeline cannot operate on values of the
following types: • Binary • Symbol • MinKey • MaxKey • DBRef • Code • CodeWScope -
Limita'ons If any single aggregation operation consumes more than
10% of system RAM the operation will produce an error (5% - warning). -
Limita'ons $where or geospatial operations can’t be used in
$match queries as part of the aggregation pipeline. -
Used materials http://docs.mongodb.org/manual/aggregation/ http://www.slideshare.net/cwestin63/mongodbs-new-aggregation-framework http://www.slideshare.net/mongodb/introduction-to-the-new-aggregation- framework http://www.mongodb.org/display/DOCS/Aggregation http://stackoverflow.com/questions/12337319/mongodb-group-group-and- mapreduce
http://blog.mongodb.org/post/16015854270/operations-in-the-new- aggregation-framework https://jira.mongodb.org/browse/SERVER-3253 http://habrahabr.ru/post/139643/