Upgrade to Pro — share decks privately, control downloads, hide ads and more …

From Node.js to Design Patterns - BuildPiper

From Node.js to Design Patterns - BuildPiper

A design pattern provides a general reusable solution for the common problems that occur in software design. The pattern typically shows relationships and interactions between classes or objects. The idea is to speed up the development process by providing well-tested, proven development/design paradigms. Design patterns come in three different categories. Creational patterns include the generation of object instances. Structure refers to how an object is made and how things behave and interact In this Webinar(Live Meetup) we will be covering - What is node js - When to use node js - Async I/O operations in node js - Advantages of Async/Await - Some interesting - async patterns - Performance comparison

Luciano Mammino

December 02, 2022
Tweet

More Decks by Luciano Mammino

Other Decks in Technology

Transcript

  1. 1

  2. $ ~ whoami 👋 I'm Luciano ( 🍕🍝) Senior Architect

    @ fourTheorem (Dublin ) nodejsdp.link 📔 Co-Author of Node.js Design Patterns 👉 Let's connect! (blog) (twitter) (twitch) (github) loige.co @loige loige lmammino 2
  3. Always re-imagining We are a pioneering technology consultancy focused on

    AWS and serverless | | Accelerated Serverless AI as a Service Platform Modernisation loige ✉ Reach out to us at 😇 We are always looking for talent: [email protected] fth.link/careers 3
  4. Fact: Async JavaScript is tricky! callbacks promises Async/Await async generators

    streams event emitters util.promisify() Promise.all() Promise.allSettled() 😱 loige 5
  5. Agenda Async WUT?! Callbacks Promises Async / Await async Patterns

    Mixed style async A performance trick! loige 6
  6. What does async even mean? In JavaScript and in Node.js,

    input/output operations are non- blocking. Classic examples: reading the content of a file, making an HTTP request, loading data from a database, etc. loige 7
  7. Blocking style vs JavaScript Blocking style JavaScript 1. Assign a

    variable 2. Read data from a file 3. Print to stdout 1. Assign a variable 2. Read data from a file 3. Print to stdout loige 8
  8. Blocking style vs JavaScript Blocking style JavaScript 1. Assign a

    variable 2. Read data from a file 3. Print to stdout 1. Assign a variable 2. Read data from a file 3. Print to stdout loige 9
  9. Blocking style vs JavaScript Blocking style JavaScript 1. Assign a

    variable 2. Read data from a file 3. Print to stdout 1. Assign a variable 2. Read data from a file 3. Print to stdout loige 10
  10. Blocking style vs JavaScript Blocking style JavaScript 1. Assign a

    variable 2. Read data from a file 3. Print to stdout 1. Assign a variable 2. Read data from a file 3. Print to stdout loige 11
  11. Blocking style vs JavaScript Blocking style JavaScript 1. Assign a

    variable 2. Read data from a file 3. Print to stdout 1. Assign a variable 2. Read data from a file 3. Print to stdout (done) loige 12
  12. Blocking style vs JavaScript Blocking style JavaScript 1. Assign a

    variable 2. Read data from a file 3. Print to stdout 1. Assign a variable 2. Read data from a file 3. Print to stdout (done) (done) loige 13
  13. Non-blocking I/O is convenient: you can do work while waiting

    for I/O! But, what if we need to do something when the I/O operation completes? loige 14
  14. doSomethingAsync(arg1, arg2, (err, data) => { // ... do something

    with data }) You are defining what happens when the I/O operations completes (or fails) with a function. doSomethingAsync will call that function for you! loige Anatomy of callback-based non-blocking code 17
  15. doSomethingAsync(arg1, arg2, (err, data) => { if (err) { //

    ... handle error return } // ... do something with data }) Always handle errors first! loige Anatomy of callback-based non-blocking code 18
  16. getLatestBooking(userId, (err, booking) => { if (err) { console.error(err) return

    } if (booking) { console.log(`Found booking for user ${userId}`, booking) } else { console.log(`No booking found for user ${userId}`) } }) 1 2 3 4 5 6 7 8 9 10 11 12 An example loige 20
  17. A more realistic example Fetch the latest booking for a

    given user If it exists, cancel it If it was already paid for, refund the user loige 21
  18. getLatestBooking(userId, (err, booking) => { if (err) { console.error(err) return

    } if (booking) { console.log(`Found booking for user ${userId}`, booking) cancelBooking(booking.id, (err) => { if (err) { console.error(err) return } if (booking.paid) { console.log('Booking was paid, refunding the user') refundUser(userId, booking.paidAmount, (err) => { if (err) { console.error(err) return } console.log('User refunded') }) } }) } else { console.log(`No booking found for user ${userId}`) } }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 cancelBooking(booking.id, (err) => { if (err) { console.error(err) return } if (booking.paid) { console.log('Booking was paid, refunding the user') refundUser(userId, booking.paidAmount, (err) => { if (err) { console.error(err) return } console.log('User refunded') }) } }) getLatestBooking(userId, (err, booking) => { 1 if (err) { 2 console.error(err) 3 return 4 } 5 6 if (booking) { 7 console.log(`Found booking for user ${userId}`, booking) 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 } else { 27 console.log(`No booking found for user ${userId}`) 28 } 29 }) 30 if (booking.paid) { console.log('Booking was paid, refunding the user') refundUser(userId, booking.paidAmount, (err) => { if (err) { console.error(err) return } console.log('User refunded') }) } getLatestBooking(userId, (err, booking) => { 1 if (err) { 2 console.error(err) 3 return 4 } 5 6 if (booking) { 7 console.log(`Found booking for user ${userId}`, booking) 8 cancelBooking(booking.id, (err) => { 9 if (err) { 10 console.error(err) 11 return 12 } 13 14 15 16 17 18 19 20 21 22 23 24 25 }) 26 } else { 27 console.log(`No booking found for user ${userId}`) 28 } 29 }) 30 refundUser(userId, booking.paidAmount, (err) => { if (err) { console.error(err) return } console.log('User refunded') }) getLatestBooking(userId, (err, booking) => { 1 if (err) { 2 console.error(err) 3 return 4 } 5 6 if (booking) { 7 console.log(`Found booking for user ${userId}`, booking) 8 cancelBooking(booking.id, (err) => { 9 if (err) { 10 console.error(err) 11 return 12 } 13 14 if (booking.paid) { 15 console.log('Booking was paid, refunding the user') 16 17 18 19 20 21 22 23 24 } 25 }) 26 } else { 27 console.log(`No booking found for user ${userId}`) 28 } 29 }) 30 loige 22
  19. getLatestBooking(userId, (err, booking) => { if (err) { console.error(err) return

    } if (booking) { console.log(`Found booking for user ${userId}`, booking) cancelBooking(booking.id, (err) => { if (err) { console.error(err) return } if (booking.paid) { console.log('Booking was paid, refunding the user') refundUser(userId, booking.paidAmount, (err) => { if (err) { console.error(err) return } console.log('User refunded') }) } }) } else { console.log(`No booking found for user ${userId}`) } }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 loige 23
  20. getLatestBooking(userId, (err, booking) => { if (err) { console.error(err) return

    } if (booking) { console.log(`Found booking for user ${userId}`, booking) cancelBooking(booking.id, (err) => { if (err) { console.error(err) return } if (booking.paid) { console.log('Booking was paid, refunding the user') refundUser(userId, booking.paidAmount, (err) => { if (err) { console.error(err) return } console.log('User refunded') }) } }) } else { console.log(`No booking found for user ${userId}`) } }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 loige THE PIRAMID OF DOOM (or callback hell 🔥) 24
  21. function cancelAndRefundBooking(booking, cb) { cancelBooking(booking.id, (err) => { if (err)

    { return cb(err) } if (!booking.paid) { return cb(null, {refundedAmount: 0}) } refundUser(booking.userId, booking.paidAmount, (err) => { if (err) { return cb(err) } return cb(null, {refundedAmount: booking.paidAmount}) }) }) } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 loige 26
  22. getLatestBooking(userId, (err, booking) => { if (err) { console.error(err) return

    } if (booking) { cancelAndRefundBooking(booking, (err, result) => { if (err) { console.error(err) return } console.log(`Booking cancelled (${result.refundedAmount} refunded)`) }) } }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 loige 27
  23. With callbacks we are not in charge! We need to

    trust that the async function will call our callbacks when the async work is completed! loige 30
  24. Promise help us to be more in control! const promiseObj

    = doSomethingAsync(arg1, arg2) An object that represents the status of the async operation loige 31
  25. const promiseObj = doSomethingAsync(arg1, arg2) A promise object is a

    tiny state machine with 2 possible states pending (still performing the async operation) settled (completed) ✅ fullfilled (witha value) 🔥 rejected (with an error) loige Promise help us to be more in control! 32
  26. const promiseObj = doSomethingAsync(arg1, arg2) promiseObj.then((data) => { // ...

    do something with data }) loige Promise help us to be more in control! 33
  27. const promiseObj = doSomethingAsync(arg1, arg2) promiseObj.then((data) => { // ...

    do something with data }) promiseObj.catch((err) => { // ... handle errors } loige Promise help us to be more in control! 34
  28. Promises can be chained ⛓ This solves the pyramid of

    doom problem! doSomethingAsync(arg1, arg2) .then((result) => doSomethingElseAsync(result)) .then((result) => doEvenMoreAsync(result) .then((result) => keepDoingStuffAsync(result)) .catch((err) => { /* ... */ }) 35 loige
  29. Promises can be chained ⛓ This solves the pyramid of

    doom problem! doSomethingAsync(arg1, arg2) .then((result) => doSomethingElseAsync(result)) // ... .catch((err) => { /* ... */ }) .finally(() => { /* ... */ }) loige 36
  30. How to create a promise new Promise ((resolve, reject) =>

    { // ... do something async // reject(someError) // resolve(someValue) }) loige 38
  31. How to create a promise (example) function queryDB(client, query) {

    return new Promise((resolve, reject) => { client.executeQuery(query, (err, data) => { if (err) { return reject(err) } resolve(data) }) }) } 1 2 3 4 5 6 7 8 9 10 11 loige 40
  32. How to create a promise (example) queryDB(dbClient, 'SELECT * FROM

    bookings') .then((data) => { // ... do something with data }) .catch((err) => { console.error('Failed to run query', err) }) .finally(() => { dbClient.disconnect() }) 1 2 3 4 5 6 7 8 9 10 queryDB(dbClient, 'SELECT * FROM bookings') 1 .then((data) => { 2 // ... do something with data 3 }) 4 .catch((err) => { 5 console.error('Failed to run query', err) 6 }) 7 .finally(() => { 8 dbClient.disconnect() 9 }) 10 .then((data) => { // ... do something with data }) queryDB(dbClient, 'SELECT * FROM bookings') 1 2 3 4 .catch((err) => { 5 console.error('Failed to run query', err) 6 }) 7 .finally(() => { 8 dbClient.disconnect() 9 }) 10 .catch((err) => { console.error('Failed to run query', err) }) queryDB(dbClient, 'SELECT * FROM bookings') 1 .then((data) => { 2 // ... do something with data 3 }) 4 5 6 7 .finally(() => { 8 dbClient.disconnect() 9 }) 10 .finally(() => { dbClient.disconnect() }) queryDB(dbClient, 'SELECT * FROM bookings') 1 .then((data) => { 2 // ... do something with data 3 }) 4 .catch((err) => { 5 console.error('Failed to run query', err) 6 }) 7 8 9 10 queryDB(dbClient, 'SELECT * FROM bookings') .then((data) => { // ... do something with data }) .catch((err) => { console.error('Failed to run query', err) }) .finally(() => { dbClient.disconnect() }) 1 2 3 4 5 6 7 8 9 10 loige 41
  33. Let's re-write our example with Promise Fetch the latest booking

    for a given user If it exists, cancel it If it was already paid for, refund the user loige 42
  34. getLatestBooking(userId) .then((booking) => { if (booking) { console.log(`Found booking for

    user ${userId}`, booking) return cancelBooking(booking.id) } console.log(`No booking found for user ${userId}`) }) .then((cancelledBooking) => { if (cancelledBooking && cancelledBooking.paid) { console.log('Booking was paid, refunding the user') return refundUser(userId, cancelledBooking.paidAmount) } }) .then((refund) => { if (refund) { console.log('User refunded') } }) .catch((err) => { console.error(err) }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 getLatestBooking(userId) 1 .then((booking) => { 2 if (booking) { 3 console.log(`Found booking for user ${userId}`, booking) 4 return cancelBooking(booking.id) 5 } 6 console.log(`No booking found for user ${userId}`) 7 }) 8 .then((cancelledBooking) => { 9 if (cancelledBooking && cancelledBooking.paid) { 10 console.log('Booking was paid, refunding the user') 11 return refundUser(userId, cancelledBooking.paidAmount) 12 } 13 }) 14 .then((refund) => { 15 if (refund) { 16 console.log('User refunded') 17 } 18 }) 19 .catch((err) => { 20 console.error(err) 21 }) 22 .then((booking) => { if (booking) { console.log(`Found booking for user ${userId}`, booking) return cancelBooking(booking.id) } console.log(`No booking found for user ${userId}`) }) getLatestBooking(userId) 1 2 3 4 5 6 7 8 .then((cancelledBooking) => { 9 if (cancelledBooking && cancelledBooking.paid) { 10 console.log('Booking was paid, refunding the user') 11 return refundUser(userId, cancelledBooking.paidAmount) 12 } 13 }) 14 .then((refund) => { 15 if (refund) { 16 console.log('User refunded') 17 } 18 }) 19 .catch((err) => { 20 console.error(err) 21 }) 22 .then((cancelledBooking) => { if (cancelledBooking && cancelledBooking.paid) { console.log('Booking was paid, refunding the user') return refundUser(userId, cancelledBooking.paidAmount) } }) getLatestBooking(userId) 1 .then((booking) => { 2 if (booking) { 3 console.log(`Found booking for user ${userId}`, booking) 4 return cancelBooking(booking.id) 5 } 6 console.log(`No booking found for user ${userId}`) 7 }) 8 9 10 11 12 13 14 .then((refund) => { 15 if (refund) { 16 console.log('User refunded') 17 } 18 }) 19 .catch((err) => { 20 console.error(err) 21 }) 22 .then((refund) => { if (refund) { console.log('User refunded') } }) getLatestBooking(userId) 1 .then((booking) => { 2 if (booking) { 3 console.log(`Found booking for user ${userId}`, booking) 4 return cancelBooking(booking.id) 5 } 6 console.log(`No booking found for user ${userId}`) 7 }) 8 .then((cancelledBooking) => { 9 if (cancelledBooking && cancelledBooking.paid) { 10 console.log('Booking was paid, refunding the user') 11 return refundUser(userId, cancelledBooking.paidAmount) 12 } 13 }) 14 15 16 17 18 19 .catch((err) => { 20 console.error(err) 21 }) 22 .catch((err) => { console.error(err) }) getLatestBooking(userId) 1 .then((booking) => { 2 if (booking) { 3 console.log(`Found booking for user ${userId}`, booking) 4 return cancelBooking(booking.id) 5 } 6 console.log(`No booking found for user ${userId}`) 7 }) 8 .then((cancelledBooking) => { 9 if (cancelledBooking && cancelledBooking.paid) { 10 console.log('Booking was paid, refunding the user') 11 return refundUser(userId, cancelledBooking.paidAmount) 12 } 13 }) 14 .then((refund) => { 15 if (refund) { 16 console.log('User refunded') 17 } 18 }) 19 20 21 22 getLatestBooking(userId) .then((booking) => { if (booking) { console.log(`Found booking for user ${userId}`, booking) return cancelBooking(booking.id) } console.log(`No booking found for user ${userId}`) }) .then((cancelledBooking) => { if (cancelledBooking && cancelledBooking.paid) { console.log('Booking was paid, refunding the user') return refundUser(userId, cancelledBooking.paidAmount) } }) .then((refund) => { if (refund) { console.log('User refunded') } }) .catch((err) => { console.error(err) }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 loige 43
  35. Sometimes, we just want to wait for a promise to

    resolve before executing the next line... const promiseObj = doSomethingAsync(arg1, arg2) const data = await promiseObj // ... process the data await allows us to do exactly that loige 45
  36. const data = await doSomethingAsync(arg1, arg2) // ... process the

    data We don't have to assign the promise to a variable to use await Sometimes, we just want to wait for a promise to resolve before executing the next line... loige 46
  37. try { const data = await doSomethingAsync(arg1, arg2) // ...

    process the data } catch (err) { // ... handle error } Unified error handling If we await a promise that eventually rejects we can capture the error with a regular try/catch block loige 47
  38. Async functions async function doSomethingAsync(arg1, arg2) { // ... }

    special keyword that marks a function as async loige 48
  39. Async functions async function doSomethingAsync(arg1, arg2) { return 'SomeValue' }

    function doSomethingAsync(arg1, arg2) { return Promise.resolve('SomeValue') } loige 49
  40. Async functions async function doSomethingAsync(arg1, arg2) { throw new Error('SomeError')

    } function doSomethingAsync(arg1, arg2) { return Promise.reject(new Error('SomeError')) } loige 50
  41. Async functions async function doSomethingAsync(arg1, arg2) { const res1 =

    await doSomethingElseAsync() const res2 = await doEvenMoreAsync(res1) const res3 = await keepDoingStuffAsync(res2) // ... } inside an async function you can use await to suspend the execution until the awaited promise resolves loige 51
  42. Async functions async function doSomethingAsync(arg1, arg2) { const res =

    await doSomethingElseAsync() if (res) { for (const record of res1.records) { await updateRecord(record) } } } Async functions make it very easy to write code that manages asynchronous control flow loige 52
  43. Let's re-write our example with async/await Fetch the latest booking

    for a given user If it exists, cancel it If it was already paid for, refund the user loige 53
  44. async function cancelLatestBooking(userId) { const booking = await getLatestBooking(userId) if

    (!booking) { console.log(`No booking found for user ${userId}`) return } console.log(`Found booking for user ${userId}`, booking) await cancelBooking(booking.id) if (booking.paid) { console.log('Booking was paid, refunding the user') await refundUser(userId, booking.paidAmount) console.log('User refunded') } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 async function cancelLatestBooking(userId) { } 1 const booking = await getLatestBooking(userId) 2 3 if (!booking) { 4 console.log(`No booking found for user ${userId}`) 5 return 6 } 7 8 console.log(`Found booking for user ${userId}`, booking) 9 10 await cancelBooking(booking.id) 11 12 if (booking.paid) { 13 console.log('Booking was paid, refunding the user') 14 await refundUser(userId, booking.paidAmount) 15 console.log('User refunded') 16 } 17 18 const booking = await getLatestBooking(userId) async function cancelLatestBooking(userId) { 1 2 3 if (!booking) { 4 console.log(`No booking found for user ${userId}`) 5 return 6 } 7 8 console.log(`Found booking for user ${userId}`, booking) 9 10 await cancelBooking(booking.id) 11 12 if (booking.paid) { 13 console.log('Booking was paid, refunding the user') 14 await refundUser(userId, booking.paidAmount) 15 console.log('User refunded') 16 } 17 } 18 if (!booking) { console.log(`No booking found for user ${userId}`) return } async function cancelLatestBooking(userId) { 1 const booking = await getLatestBooking(userId) 2 3 4 5 6 7 8 console.log(`Found booking for user ${userId}`, booking) 9 10 await cancelBooking(booking.id) 11 12 if (booking.paid) { 13 console.log('Booking was paid, refunding the user') 14 await refundUser(userId, booking.paidAmount) 15 console.log('User refunded') 16 } 17 } 18 console.log(`Found booking for user ${userId}`, booking) async function cancelLatestBooking(userId) { 1 const booking = await getLatestBooking(userId) 2 3 if (!booking) { 4 console.log(`No booking found for user ${userId}`) 5 return 6 } 7 8 9 10 await cancelBooking(booking.id) 11 12 if (booking.paid) { 13 console.log('Booking was paid, refunding the user') 14 await refundUser(userId, booking.paidAmount) 15 console.log('User refunded') 16 } 17 } 18 await cancelBooking(booking.id) async function cancelLatestBooking(userId) { 1 const booking = await getLatestBooking(userId) 2 3 if (!booking) { 4 console.log(`No booking found for user ${userId}`) 5 return 6 } 7 8 console.log(`Found booking for user ${userId}`, booking) 9 10 11 12 if (booking.paid) { 13 console.log('Booking was paid, refunding the user') 14 await refundUser(userId, booking.paidAmount) 15 console.log('User refunded') 16 } 17 } 18 if (booking.paid) { console.log('Booking was paid, refunding the user') await refundUser(userId, booking.paidAmount) console.log('User refunded') } async function cancelLatestBooking(userId) { 1 const booking = await getLatestBooking(userId) 2 3 if (!booking) { 4 console.log(`No booking found for user ${userId}`) 5 return 6 } 7 8 console.log(`Found booking for user ${userId}`, booking) 9 10 await cancelBooking(booking.id) 11 12 13 14 15 16 17 } 18 async function cancelLatestBooking(userId) { const booking = await getLatestBooking(userId) if (!booking) { console.log(`No booking found for user ${userId}`) return } console.log(`Found booking for user ${userId}`, booking) await cancelBooking(booking.id) if (booking.paid) { console.log('Booking was paid, refunding the user') await refundUser(userId, booking.paidAmount) console.log('User refunded') } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 loige 54
  45. Mini summary Async/Await generally helps to keep the code simple

    & readable To use Async/Await you need to understand Promise To use Promise you need to understand callbacks callbacks → Promise → async/await Don't skip any step of the async journey! loige 55
  46. Sequential execution const users = ['Peach', 'Toad', 'Mario', 'Luigi'] for

    (const userId of users) { await cancelLatestBooking(userId) } 1 2 3 4 5 const users = ['Peach', 'Toad', 'Mario', 'Luigi'] 1 2 for (const userId of users) { 3 await cancelLatestBooking(userId) 4 } 5 for (const userId of users) { } const users = ['Peach', 'Toad', 'Mario', 'Luigi'] 1 2 3 await cancelLatestBooking(userId) 4 5 await cancelLatestBooking(userId) const users = ['Peach', 'Toad', 'Mario', 'Luigi'] 1 2 for (const userId of users) { 3 4 } 5 const users = ['Peach', 'Toad', 'Mario', 'Luigi'] for (const userId of users) { await cancelLatestBooking(userId) } 1 2 3 4 5 loige 57
  47. Sequential execution (gotcha!) const users = ['Peach', 'Toad', 'Mario', 'Luigi']

    users.forEach(async (userId) => { await cancelLatestBooking(userId) }) 1 2 3 4 5 loige ⚠ Don't do this with Array.map() or Array.forEach() Array.forEach() will run the provided function without awaiting for the returned promise, so all the invocation will actually happen concurrently! 58
  48. Concurrent execution (Promise.all) const users = ['Peach', 'Toad', 'Mario', 'Luigi']

    await Promise.all( users.map( userId => cancelLatestBooking(userId) ) ) 1 2 3 4 5 6 7 loige Promise.all() receives a list of promises and it returns a new Promise. This promise will resolve once all the original promises resolve, but it will reject as soon as ONE promise rejects 59
  49. Concurrent execution (Promise.allSettled) const users = ['Peach', 'Toad', 'Mario', 'Luigi']

    const results = await Promise.allSettled( users.map( userId => cancelLatestBooking(userId) ) ) 1 2 3 4 5 6 7 loige [ { status: 'fulfilled', value: true }, { status: 'fulfilled', value: true }, { status: 'rejected', reason: Error }, { status: 'fulfilled', value: true } ] 60
  50. Node.js offers promise-based alternative APIs Callback-based Promise-based setTimeout, setImmediate, setInterval

    import timers from 'timers/promises' import fs from 'fs' import fs from 'fs/promises' import stream from 'stream' import stream from 'stream/promises' import dns from 'dns' import dns from 'dns/promises' loige 63
  51. util.promisify() import { gzip } from 'zlib' // zlib.gzip(buffer[, options],

    callback) import { promisify } from 'util' const gzipPromise = promisify(gzip) const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 1 2 3 4 5 6 7 import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 import { promisify } from 'util' 2 3 const gzipPromise = promisify(gzip) 4 5 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 6 console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 7 import { promisify } from 'util' import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 3 const gzipPromise = promisify(gzip) 4 5 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 6 console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 7 const gzipPromise = promisify(gzip) import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 import { promisify } from 'util' 2 3 4 5 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 6 console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 7 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 import { promisify } from 'util' 2 3 const gzipPromise = promisify(gzip) 4 5 6 7 import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) import { promisify } from 'util' const gzipPromise = promisify(gzip) const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 1 2 3 4 5 6 7 loige 64
  52. Promisify by hand 🖐 import { gzip } from 'zlib'

    // zlib.gzip(buffer[, options], callback) function gzipPromise (buffer, options) { return new Promise((resolve, reject) => { gzip(buffer, options, (err, gzippedData) => { if (err) { return reject(err) } resolve(gzippedData) }) }) } const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 function gzipPromise (buffer, options) { } import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 3 return new Promise((resolve, reject) => { 4 gzip(buffer, options, (err, gzippedData) => { 5 if (err) { 6 return reject(err) 7 } 8 9 resolve(gzippedData) 10 }) 11 }) 12 13 14 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 15 console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 16 return new Promise((resolve, reject) => { }) import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 function gzipPromise (buffer, options) { 3 4 gzip(buffer, options, (err, gzippedData) => { 5 if (err) { 6 return reject(err) 7 } 8 9 resolve(gzippedData) 10 }) 11 12 } 13 14 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 15 console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 16 gzip(buffer, options, (err, gzippedData) => { }) import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 function gzipPromise (buffer, options) { 3 return new Promise((resolve, reject) => { 4 5 if (err) { 6 return reject(err) 7 } 8 9 resolve(gzippedData) 10 11 }) 12 } 13 14 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 15 console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 16 if (err) { return reject(err) } import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 function gzipPromise (buffer, options) { 3 return new Promise((resolve, reject) => { 4 gzip(buffer, options, (err, gzippedData) => { 5 6 7 8 9 resolve(gzippedData) 10 }) 11 }) 12 } 13 14 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 15 console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 16 resolve(gzippedData) import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 function gzipPromise (buffer, options) { 3 return new Promise((resolve, reject) => { 4 gzip(buffer, options, (err, gzippedData) => { 5 if (err) { 6 return reject(err) 7 } 8 9 10 }) 11 }) 12 } 13 14 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 15 console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 16 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 function gzipPromise (buffer, options) { 3 return new Promise((resolve, reject) => { 4 gzip(buffer, options, (err, gzippedData) => { 5 if (err) { 6 return reject(err) 7 } 8 9 resolve(gzippedData) 10 }) 11 }) 12 } 13 14 15 16 import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) function gzipPromise (buffer, options) { return new Promise((resolve, reject) => { gzip(buffer, options, (err, gzippedData) => { if (err) { return reject(err) } resolve(gzippedData) }) }) } const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 loige 65
  53. What if we we want to do the opposite? 🤷

    Convert a promise-based function to a callback-based one loige 66
  54. var env = nunjucks.configure('views') env.addFilter('videoTitle', function(videoId, cb) { // ...

    fetch the title through youtube APIs // ... extract the video title // ... and call the callback with the title }, true) 1 2 3 4 5 6 7 OK, this is not a common use case, so let me give you a real example! Nunjucks async filters {{ data | myCustomFilter }} We are forced to pass a callback-based function here! Ex: {{ youtubeId | videoTitle }} loige 67
  55. util.callbackify() import { callbackify } from 'util' import Innertube from

    'youtubei.js' // from npm async function videoTitleFilter (videoId) { const youtube = await new Innertube({ gl: 'US' }) const details = await youtube.getDetails(videoId) return details.title } const videoTitleFilterCb = callbackify(videoTitleFilter) videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { if (err) { console.error(err) return } console.log(videoTitle) }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 import { callbackify } from 'util' 1 import Innertube from 'youtubei.js' // from npm 2 3 async function videoTitleFilter (videoId) { 4 const youtube = await new Innertube({ gl: 'US' }) 5 const details = await youtube.getDetails(videoId) 6 return details.title 7 } 8 9 const videoTitleFilterCb = callbackify(videoTitleFilter) 10 11 videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { 12 if (err) { 13 console.error(err) 14 return 15 } 16 17 console.log(videoTitle) 18 }) 19 import Innertube from 'youtubei.js' // from npm import { callbackify } from 'util' 1 2 3 async function videoTitleFilter (videoId) { 4 const youtube = await new Innertube({ gl: 'US' }) 5 const details = await youtube.getDetails(videoId) 6 return details.title 7 } 8 9 const videoTitleFilterCb = callbackify(videoTitleFilter) 10 11 videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { 12 if (err) { 13 console.error(err) 14 return 15 } 16 17 console.log(videoTitle) 18 }) 19 async function videoTitleFilter (videoId) { const youtube = await new Innertube({ gl: 'US' }) const details = await youtube.getDetails(videoId) return details.title } import { callbackify } from 'util' 1 import Innertube from 'youtubei.js' // from npm 2 3 4 5 6 7 8 9 const videoTitleFilterCb = callbackify(videoTitleFilter) 10 11 videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { 12 if (err) { 13 console.error(err) 14 return 15 } 16 17 console.log(videoTitle) 18 }) 19 const videoTitleFilterCb = callbackify(videoTitleFilter) import { callbackify } from 'util' 1 import Innertube from 'youtubei.js' // from npm 2 3 async function videoTitleFilter (videoId) { 4 const youtube = await new Innertube({ gl: 'US' }) 5 const details = await youtube.getDetails(videoId) 6 return details.title 7 } 8 9 10 11 videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { 12 if (err) { 13 console.error(err) 14 return 15 } 16 17 console.log(videoTitle) 18 }) 19 videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { if (err) { console.error(err) return } console.log(videoTitle) }) import { callbackify } from 'util' 1 import Innertube from 'youtubei.js' // from npm 2 3 async function videoTitleFilter (videoId) { 4 const youtube = await new Innertube({ gl: 'US' }) 5 const details = await youtube.getDetails(videoId) 6 return details.title 7 } 8 9 const videoTitleFilterCb = callbackify(videoTitleFilter) 10 11 12 13 14 15 16 17 18 19 import { callbackify } from 'util' import Innertube from 'youtubei.js' // from npm async function videoTitleFilter (videoId) { const youtube = await new Innertube({ gl: 'US' }) const details = await youtube.getDetails(videoId) return details.title } const videoTitleFilterCb = callbackify(videoTitleFilter) videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { if (err) { console.error(err) return } console.log(videoTitle) }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 loige 68
  56. Callbackify by hand ✋ import Innertube from 'youtubei.js' // from

    npm async function videoTitleFilter (videoId) { // ... } function videoTitleFilterCb (videoId, cb) { videoTitleFilter(videoId) .then((videoTitle) => cb(null, videoTitle)) .catch(cb) } videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { // ... }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 function videoTitleFilterCb (videoId, cb) { videoTitleFilter(videoId) .then((videoTitle) => cb(null, videoTitle)) .catch(cb) } import Innertube from 'youtubei.js' // from npm 1 2 async function videoTitleFilter (videoId) { 3 // ... 4 } 5 6 7 8 9 10 11 12 videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { 13 // ... 14 }) 15 loige 69
  57. The request batching pattern multiple users (no batching) Web server

    DB /api/hotels/rome /api/hotels/rome /api/hotels/rome 73 loige
  58. The request batching pattern multiple users (with batching!) Web server

    DB /api/hotels/rome /api/hotels/rome /api/hotels/rome 74 📘 Requests in-flight /api/hotels/rome ✅ loige
  59. The web server import { createServer } from 'http' const

    urlRegex = /^\/api\/hotels\/([\w-]+)$/ createServer(async (req, res) => { const url = new URL(req.url, 'http://localhost') const matches = urlRegex.exec(url.pathname) if (!matches) { res.writeHead(404, 'Not found') return res.end() } const [_, city] = matches const hotels = await getHotelsForCity(city) res.writeHead(200) res.end(JSON.stringify({ hotels })) }).listen(8000) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 createServer(async (req, res) => { 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 const [_, city] = matches 14 const hotels = await getHotelsForCity(city) 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 }).listen(8000) 19 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ import { createServer } from 'http' 1 2 3 4 createServer(async (req, res) => { 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 const [_, city] = matches 14 const hotels = await getHotelsForCity(city) 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 }).listen(8000) 19 createServer(async (req, res) => { }).listen(8000) import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 const [_, city] = matches 14 const hotels = await getHotelsForCity(city) 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 19 const url = new URL(req.url, 'http://localhost') const matches = urlRegex.exec(url.pathname) import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 createServer(async (req, res) => { 5 6 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 const [_, city] = matches 14 const hotels = await getHotelsForCity(city) 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 }).listen(8000) 19 if (!matches) { res.writeHead(404, 'Not found') return res.end() } import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 createServer(async (req, res) => { 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 9 10 11 12 13 const [_, city] = matches 14 const hotels = await getHotelsForCity(city) 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 }).listen(8000) 19 const [_, city] = matches import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 createServer(async (req, res) => { 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 14 const hotels = await getHotelsForCity(city) 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 }).listen(8000) 19 const hotels = await getHotelsForCity(city) import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 createServer(async (req, res) => { 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 const [_, city] = matches 14 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 }).listen(8000) 19 res.writeHead(200) res.end(JSON.stringify({ hotels })) import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 createServer(async (req, res) => { 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 const [_, city] = matches 14 const hotels = await getHotelsForCity(city) 15 16 17 18 }).listen(8000) 19 import { createServer } from 'http' const urlRegex = /^\/api\/hotels\/([\w-]+)$/ createServer(async (req, res) => { const url = new URL(req.url, 'http://localhost') const matches = urlRegex.exec(url.pathname) if (!matches) { res.writeHead(404, 'Not found') return res.end() } const [_, city] = matches const hotels = await getHotelsForCity(city) res.writeHead(200) res.end(JSON.stringify({ hotels })) }).listen(8000) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 loige 75
  60. The data fetching function (with batching) let pendingRequests = new

    Map() function getHotelsForCity (cityId) { if (pendingRequests.has(cityId)) { return pendingRequests.get(cityId) } const asyncOperation = db.query({ text: 'SELECT * FROM hotels WHERE cityid = $1', values: [cityId], }) pendingRequests.set(cityId, asyncOperation) asyncOperation.finally(() => { pendingRequests.delete(cityId) }) return asyncOperation } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 loige 76
  61. Benchmarks loige.link/req-batch-bench Without request batching With request batching (+90% avg

    req/sec)* * This is an artificial benchmark and results might vary significantly in real-life scenarios. Always run your own benchmarks before deciding whether this optimization can have a positive effect for you. loige 77
  62. Closing Notes JavaScript can be a very powerful and convenient

    language when we have to deal with a lot of I/O (e.g. web servers) The async story has evolved a lot in the last 10-15 years: new patterns and language constructs have emerged Async/Await is probably the best way to write async code today To use Async/Await correctly you need to understand Promise and callbacks Take your time and invest in learning the fundamentals loige 78
  63. Cover Picture by on Marc-Olivier Jodoin Unsplash THANKS! 🙌 ❤

    nodejsdp.link loige Grab these slides! 😍 Grab the book 79