MongoDB Transaction Pattern

1. MongoDB Transaction Pattern Who? Ian Yang From? Intridea Inc. When? 2013-05-15 Wed
2. Outline 1 Overview 2 Write Operation Properties 3 Two Phrase Commit 4 Recovering 5 Rollback 6 Multiple Applications 7 References
3. Transaction (ACID) Atomicity all or nothing. Consistency does not violate any integrity constraints Isolation allow concurrent execution of transactions Durability Transaction is persistent after commit
4. MongoDB Features Atomicity Only on the level of a single document. Write Concern Return write operation result
5. MongoDB Transaction Atomicity done or rollback Consistency rollback may leave system in inconsistent state. Isolation other transactions can see intermediate result. Durability write concern, replica set, journal.
6. Requirements of Write Operations in Transaction Revertible for recover and rollback Idempotent for resume, recover and rollback
7. Not Revertible db.users.save(user); db.users.update ({ username: "ian"}, user); db.users.update ({ username: "ian"}, {$set: {balance: 120}});
8. Revertible db.users.update ({ username: "ian"}, {$inc: {balance: 20}}); db.users.update ({ username: "ian"}, {$inc: {balance: -20}}); db.users.update ({ username: "ian"}, {$push: {tx: 20}}); db.users.update ({ username: "ian"}, {$pull: {tx: 20}});
9. Not Idempotent db.users.update ({ username: "ian"}, {$inc: {balance: 20}}); db.users.update ({ username: "ian"}, {$push: {tx: 20}}); db.users.update( {username: "ian", balance: 100}, {$inc: {balance: 20}} );
10. Idempotent db.users.update( {username: "ian", tx: {$ne: tx._id}}, {$inc: {balance: 20}, $push: {tx: tx._id}} ); db.users.update( {username: "ian", tx: tx._id}, {$inc: {balance: -20}, $pull: {tx: tx._id}} );
11. Sample Transfer 20 dollars from Ian to Daniel
12. Create Transaction Transaction provides a unique identifier, enough information to recover and rollback, a document for atomic write operation.
13. var tx = { _id: ObjectId (), source: ian._id , dest: daniel._id , amount: 20 }; db.transactions.create(tx);
14. Transaction State Machine State represents transaction stages Different actions should be taken in different stages to recover or rollback
15. Step 1: Create Transaction var tx = { _id: ObjectId (), source: ian._id , dest: daniel._id , amount: 20, state: 'initial ' }; db.transactions.create(tx);
16. Step 2: initial → pending db.transactions.update( {_id: tx._id , state: 'initial '}, {$set: {state: 'pending '}} );
17. Step 3: Apply Transaction db.users.update( {_id: tx.source , tx: {$ne: tx._id}}, {$push: {tx: tx._id}, $inc: { balance: - tx.amount }} ); db.users.update( {_id: tx.dest , tx: {$ne: tx._id}}, {$push: {tx: tx._id}, $inc: { balance: tx.amount }} );
18. Step 4: pending → committed db.transactions.update( {_id: tx._id , state: 'pending '}, {$set: {state: 'committed '}} );
19. Step 5: Remove Pending Transaction db.users.update( {_id: tx.source , tx: tx._id}, {$pull: {tx: tx._id}} ); db.users.update( {_id: tx.dest , tx: tx._id}, {$pull: {tx: tx._id}} );
20. Step 6: committed → done db.transactions.update( {_id: tx._id , state: 'committed '}, {$set: {state: 'done'}} );
21. Recover Initial Transactions Recover from Step 2: initial → pending
22. Recover Pending Transactions Recover from Step 3: Apply Transactions
23. Recover Committed Transactions Recover from Step 5: Remove Pending Transaction
24. General Rule Recover from the Step just following the step transaction just entered current state. State transition creates a check point
25. Rollback Operations Better to create a new transaction in reverse direction for Committed Transactions and Done Transactions Use following steps to rollback initial and pending transactions
26. Rollback 1: state → canceling db.transactions.update( {_id: t._id , state: {$in: ['initial ', ' pending ']}}, {$set: {state: 'canceling '}} );
27. Rollback 2: Undo the Transaction db.users.update( {_id: tx.dest , tx: {$ne: tx._id}}, {$pull: {tx: tx._id}, $inc: { balance: - tx.amount }} ); db.users.update( {_id: tx.source , tx: tx._id}, {$pull: {tx: tx._id}, $inc: { balance: tx.amount }} );
28. Rollback 3: canceling → canceled db.transactions.update( {_id: t._id , state: {$in: ['canceling ']}}, {$set: {state: 'canceled '}} );
29. Recover Rollback Restart from Rollback 2 for canceling transactions
30. Conclusion Create a document for any transaction Store transaction info in the document Apply only Revertible and Idempotent operations
31. Conclusion (Cont.) Transit state to save current work Transit state if several operation branches exists
32. Enough? Only 1 application can handle a give transaction at any point in time.
33. Background A picks a pending transaction to recover B picks also a pending transaction to recover
34. Scenario 1 A is so slow, when it starts apply transaction, B already has done the transaction. A must rollback. If A exists abnormally, the transaction is in inconsistent state.
35. Scenario 2 B just finished Step 5: Removing Pending Transaction A applied transactions and exited abnormally B change state to done without error
36. Solution findAndModify lock db.transactions.findAndModify ({ query: {state: "initial", application: {$exists: 0}}, update: {$set: {state: "pending", application: "A1"}}, new: true });
37. Lock Release db.transactions.findAndModify ({ query: {application: "A1"}, update: {$unset: {application: 1}}, new: true });
38. Lock Forever? Application exists abnormally before release the lock Solution: timeout
39. Timeout Lock now = new Date(); timeout = new Date(now.valueOf () + 60 * 10 * 1000); db.transactions.findAndModify ({ query: {state: "initial", lockUntil: {$not: {$gte: now}}}, update: {$set: {state: "pending", lockUntil: timeout}}, new: true });
40. Client Requirements Client timeout to ensure the operation is finished in expected time. Check time after return from block IO Defer transactions cleanup
41. References http://docs.mongodb.org/ manual/tutorial/ perform-two-phase-commits/ http://docs.mongodb.org/ manual/tutorial/ isolate-sequence-of-operations/
42. Q & A