Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Finding a lost song with Node.js and async iter...

Finding a lost song with Node.js and async iterators

Did you ever get that feeling when a random song pops into your brain and you can’t get rid of it? Well, that happened to me recently and I couldn’t even remember the title of the damn song! In this talk, I want to share with you the story of how I was able to recover the details of the song by navigating some music-related APIs using JavaScript, Node.js and the magic of async iterators!

Luciano Mammino

April 29, 2021
Tweet

More Decks by Luciano Mammino

Other Decks in Technology

Transcript

  1. Finding a lost song with Node.js & async iterators Luciano

    Mammino ( ) @loige Dublin, Node.js Meetup 2021-04-29 loige.link/async-it 1
  2. Photo by on Darius Bashar Unsplash A random song you

    haven't listened to in years pops into your head... 3
  3. It doesn't matter what you do all day... It keeps

    coming back to you! Photo by on Attentie Attentie Unsplash 4
  4. And now you want to listen to it! Photo by

    on Volodymyr Hryshchenko Unsplash 5
  5. But, what if you can't remember the title or the

    author?! Photo by on Tachina Lee Unsplash 6
  6. Today, I'll tell you how I solved this problem using

    - Last.fm API - Node.js - Async Iterators Photo by on Quinton Coetzee Unsplash 8
  7. Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋

    Senior Architect @ fourTheorem (Dublin ) 9
  8. Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋

    Senior Architect @ fourTheorem (Dublin ) nodejsdp.link Co-Author of Node.js Design Patterns 👉 9
  9. Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋

    Senior Architect @ fourTheorem (Dublin ) nodejsdp.link Co-Author of Node.js Design Patterns 👉 20% eBook discount on Packt 20NODEDUBLIN 9
  10. Let me introduce myself first... I'm Luciano ( 🍕🍝) 👋

    Senior Architect @ fourTheorem (Dublin ) nodejsdp.link Co-Author of Node.js Design Patterns 👉 Connect with me: (blog) (twitter) (github) loige.co @loige lmammino 20% eBook discount on Packt 20NODEDUBLIN 9
  11. We are business focused technologists that deliver. | | Accelerated

    Serverless AI as a Service Platform Modernisation Do you want to ? work with us loige 10
  12. I could only remember some random parts and the word

    "dark" (probably in the title) loige 12
  13. 13

  14. 14

  15. import querystring from 'querystring' import axios from 'axios' const query

    = querystring.stringify({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json' }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const response = await axios.get(url) console.log(response.data) loige 22
  16. loige We are getting a "paginated" response with 50 tracks

    per page but there are 51 here! 🤔 23
  17. loige We are getting a "paginated" response with 50 tracks

    per page but there are 51 here! 🤔 (let's ignore this for now...) 23
  18. loige We are getting a "paginated" response with 50 tracks

    per page but there are 51 here! 🤔 How do we fetch the next pages? (let's ignore this for now...) 23
  19. let page = 1 while (true) { const query =

    querystring.stringify({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', page }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const response = await axios.get(url) console.log(response.data) if (page === Number(response.data.recenttracks['@attr'].totalPages)) { break // it's the last page! } loige 25
  20. let page = 1 while (true) { const query =

    querystring.stringify({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', page }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const response = await axios.get(url) console.log(response.data) if (page === Number(response.data.recenttracks['@attr'].totalPages)) { break // it's the last page! } loige 25
  21. let page = 1 while (true) { const query =

    querystring.stringify({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', page }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const response = await axios.get(url) console.log(response.data) if (page === Number(response.data.recenttracks['@attr'].totalPages)) { break // it's the last page! } loige 25
  22. let page = 1 while (true) { const query =

    querystring.stringify({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', page }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const response = await axios.get(url) console.log(response.data) if (page === Number(response.data.recenttracks['@attr'].totalPages)) { break // it's the last page! } loige 25
  23. let page = 1 while (true) { const query =

    querystring.stringify({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', page }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const response = await axios.get(url) console.log(response.data) if (page === Number(response.data.recenttracks['@attr'].totalPages)) { break // it's the last page! } loige 25
  24. let page = 1 while (true) { const query =

    querystring.stringify({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', page }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const response = await axios.get(url) console.log(response.data) if (page === Number(response.data.recenttracks['@attr'].totalPages)) { break // it's the last page! } loige 25
  25. // ... for (const track of response.data.recenttracks.track) { console.log( track.date?.['#text'],

    `${track.artist['#text']} - ${track.name}` ) } console.log('--- end page ---') // ... loige 28
  26. loige * Note that page size here is 10 tracks

    per page Every page has a song with undefined time... This is the song I am currently listening to! It appears at the top of every page. 29
  27. loige * Note that page size here is 10 tracks

    per page Sometimes there are duplicated tracks between pages... 😨 29
  28. loige ... tracks (newest to oldest) 31 Page1 Page2 ...

    Page1 Page2 new track moved from page 1 to page 2
  29. loige ...* tracks (newest to oldest) 34 * we are

    done when we get an empty page (or num pages is 1)
  30. loige ...* tracks (newest to oldest) 34 Page1 * we

    are done when we get an empty page (or num pages is 1)
  31. loige ...* tracks (newest to oldest) 34 Page1 t1 *

    we are done when we get an empty page (or num pages is 1)
  32. loige ...* tracks (newest to oldest) 34 Page1 before t1

    (page 1 "to" t1) t1 * we are done when we get an empty page (or num pages is 1)
  33. loige ...* tracks (newest to oldest) 34 Page1 before t1

    (page 1 "to" t1) t1 t2 * we are done when we get an empty page (or num pages is 1)
  34. loige ...* tracks (newest to oldest) 34 Page1 before t1

    (page 1 "to" t1) t1 t2 before t2 (page 1 "to" t2) * we are done when we get an empty page (or num pages is 1)
  35. let to while (true) { const query = querystring.stringify({ method:

    'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', limit: '10', to }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const response = await axios.get(url) const tracks = response.data.recenttracks.track console.log( `--- ↓ page to ${to}`, `remaining pages: ${response.data.recenttracks['@attr'].totalPages} ---` ) for (const track of tracks) { console.log(track.date?.uts, `${track.artist['#text']} - ${track.name}`) } if (response.data.recenttracks['@attr'].totalPages <= 1) { break // it's the last page! } const lastTrackInPage = tracks[tracks.length - 1] to = lastTrackInPage.date.uts } loige 35
  36. let to while (true) { const query = querystring.stringify({ method:

    'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', limit: '10', to }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const response = await axios.get(url) const tracks = response.data.recenttracks.track console.log( `--- ↓ page to ${to}`, `remaining pages: ${response.data.recenttracks['@attr'].totalPages} ---` ) for (const track of tracks) { console.log(track.date?.uts, `${track.artist['#text']} - ${track.name}`) } if (response.data.recenttracks['@attr'].totalPages <= 1) { break // it's the last page! } const lastTrackInPage = tracks[tracks.length - 1] to = lastTrackInPage.date.uts } loige 35
  37. let to while (true) { const query = querystring.stringify({ method:

    'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', limit: '10', to }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const response = await axios.get(url) const tracks = response.data.recenttracks.track console.log( `--- ↓ page to ${to}`, `remaining pages: ${response.data.recenttracks['@attr'].totalPages} ---` ) for (const track of tracks) { console.log(track.date?.uts, `${track.artist['#text']} - ${track.name}`) } if (response.data.recenttracks['@attr'].totalPages <= 1) { break // it's the last page! } const lastTrackInPage = tracks[tracks.length - 1] to = lastTrackInPage.date.uts } loige 35
  38. let to while (true) { const query = querystring.stringify({ method:

    'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', limit: '10', to }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const response = await axios.get(url) const tracks = response.data.recenttracks.track console.log( `--- ↓ page to ${to}`, `remaining pages: ${response.data.recenttracks['@attr'].totalPages} ---` ) for (const track of tracks) { console.log(track.date?.uts, `${track.artist['#text']} - ${track.name}`) } if (response.data.recenttracks['@attr'].totalPages <= 1) { break // it's the last page! } const lastTrackInPage = tracks[tracks.length - 1] to = lastTrackInPage.date.uts } loige 35
  39. let to while (true) { const query = querystring.stringify({ method:

    'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', limit: '10', to }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const response = await axios.get(url) const tracks = response.data.recenttracks.track console.log( `--- ↓ page to ${to}`, `remaining pages: ${response.data.recenttracks['@attr'].totalPages} ---` ) for (const track of tracks) { console.log(track.date?.uts, `${track.artist['#text']} - ${track.name}`) } if (response.data.recenttracks['@attr'].totalPages <= 1) { break // it's the last page! } const lastTrackInPage = tracks[tracks.length - 1] to = lastTrackInPage.date.uts } loige 35
  40. We know how to iterate over every page/track. How do

    we expose this information? loige 38
  41. const reader = LastFmRecentTracks({ apikey: process.env.API_KEY, user: 'loige' }) //

    callbacks reader.readPages( (page) => { /* ... */ }, // on page (err) => { /* ... */} // on completion (or error) ) loige 39
  42. const reader = LastFmRecentTracks({ apikey: process.env.API_KEY, user: 'loige' }) //

    event emitter reader.read() reader.on('page', (page) => { /* ... */ }) reader.on('completed', (err) => { /* ... */ }) loige 40
  43. const reader = LastFmRecentTracks({ apikey: process.env.API_KEY, user: 'loige' }) //

    streams <3 reader.pipe(/* transform or writable stream here */) reader.on('end', () => { /* ... */ }) reader.on('error', () => { /* ... */ }) loige 41
  44. import { pipeline } from 'stream' const reader = LastFmRecentTracks({

    apikey: process.env.API_KEY, user: 'loige' }) // streams pipeline <3 <3 pipeline( reader, yourProcessingStream, (err) => { // handle completion or err } ) loige 42
  45. const reader = LastFmRecentTracks({ apikey: process.env.API_KEY, user: 'loige' }) //

    ASYNC ITERATORS! for await (const page of reader) { /* ... */ } // ... do more stuff when all the data is consumed loige 43
  46. const reader = LastFmRecentTracks({ apikey: process.env.API_KEY, user: 'loige' }) //

    ASYNC ITERATORS WITH ERROR HANDLING! try { for await (const page of reader) { /* ... */ } } catch (err) { // handle errors } // ... do more stuff when all the data is consumed loige 44
  47. The iterator protocol An object is an iterator if it

    has a next() method. Every time you call it, it returns an object with the keys done (boolean) and value. loige 47
  48. function createCountdown (from) { let nextVal = from return {

    next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 48
  49. function createCountdown (from) { let nextVal = from return {

    next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 48
  50. function createCountdown (from) { let nextVal = from return {

    next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 48
  51. function createCountdown (from) { let nextVal = from return {

    next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 48
  52. function createCountdown (from) { let nextVal = from return {

    next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 48
  53. const countdown = createCountdown(3) console.log(countdown.next()) // { done: false, value:

    3 } console.log(countdown.next()) // { done: false, value: 2 } console.log(countdown.next()) // { done: false, value: 1 } console.log(countdown.next()) // { done: false, value: 0 } console.log(countdown.next()) // { done: true } loige 49
  54. function * createCountdown (from) { for (let i = from;

    i >= 0; i--) { yield i } } loige 51
  55. function * createCountdown (from) { for (let i = from;

    i >= 0; i--) { yield i } } loige 51
  56. function * createCountdown (from) { for (let i = from;

    i >= 0; i--) { yield i } } loige 51
  57. const countdown = createCountdown(3) console.log(countdown.next()) // { done: false, value:

    3 } console.log(countdown.next()) // { done: false, value: 2 } console.log(countdown.next()) // { done: false, value: 1 } console.log(countdown.next()) // { done: false, value: 0 } console.log(countdown.next()) // { done: true, value: undefined } loige 52
  58. The iterable protocol An object is iterable if it implements

    the @@iterator* method, a zero-argument function that returns an iterator. loige *Symbol.iterator 53
  59. function createCountdown (from) { let nextVal = from return {

    [Symbol.iterator]: () => ({ next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } }) } } loige 54
  60. function createCountdown (from) { let nextVal = from return {

    [Symbol.iterator]: () => ({ next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } }) } } loige 54
  61. function createCountdown (from) { let nextVal = from return {

    [Symbol.iterator]: () => ({ next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } }) } } loige 54
  62. function createCountdown (from) { return { [Symbol.iterator]: function * ()

    { for (let i = from; i >= 0; i--) { yield i } } } } loige 55
  63. const countdown = createCountdown(3) for (const value of countdown) {

    console.log(value) } // 3 // 2 // 1 // 0 loige 56
  64. The async iterator protocol An object is an async iterator

    if it has a next() method. Every time you call it, it returns a promise that resolves to an object with the keys done (boolean) and value. loige 58
  65. import { setTimeout } from 'timers/promises' function createAsyncCountdown (from, delay

    = 1000) { let nextVal = from return { async next () { await setTimeout(delay) if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 59
  66. import { setTimeout } from 'timers/promises' function createAsyncCountdown (from, delay

    = 1000) { let nextVal = from return { async next () { await setTimeout(delay) if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 59
  67. import { setTimeout } from 'timers/promises' function createAsyncCountdown (from, delay

    = 1000) { let nextVal = from return { async next () { await setTimeout(delay) if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 59
  68. const countdown = createAsyncCountdown(3) console.log(await countdown.next()) // { done: false,

    value: 3 } console.log(await countdown.next()) // { done: false, value: 2 } console.log(await countdown.next()) // { done: false, value: 1 } console.log(await countdown.next()) // { done: false, value: 0 } console.log(await countdown.next()) // { done: true } loige 60
  69. const countdown = createAsyncCountdown(3) console.log(await countdown.next()) // { done: false,

    value: 3 } console.log(await countdown.next()) // { done: false, value: 2 } console.log(await countdown.next()) // { done: false, value: 1 } console.log(await countdown.next()) // { done: false, value: 0 } console.log(await countdown.next()) // { done: true } loige 60
  70. import { setTimeout } from 'timers/promises' // async generators "produce"

    async iterators! async function * createAsyncCountdown (from, delay = 1000) { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } loige 62
  71. import { setTimeout } from 'timers/promises' // async generators "produce"

    async iterators! async function * createAsyncCountdown (from, delay = 1000) { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } loige 62
  72. import { setTimeout } from 'timers/promises' // async generators "produce"

    async iterators! async function * createAsyncCountdown (from, delay = 1000) { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } loige 62
  73. The async iterable protocol An object is an async iterable

    if it implements the @@asyncIterator* method, a zero-argument function that returns an async iterator. loige *Symbol.asyncIterator 63
  74. import { setTimeout } from 'timers/promises' function createAsyncCountdown (from, delay

    = 1000) { return { [Symbol.asyncIterator]: async function * () { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } } } loige 64
  75. import { setTimeout } from 'timers/promises' function createAsyncCountdown (from, delay

    = 1000) { return { [Symbol.asyncIterator]: async function * () { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } } } loige 64
  76. function createLastFmRecentTracks (apiKey, user) { return { [Symbol.asyncIterator]: async function

    * () { let to while (true) { const query = querystring.stringify({ method: 'user.getrecenttracks', user, api_key: apiKey, format: 'json', to }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const response = await axios.get(url) const tracks = response.data.recenttracks.track yield tracks if (response.data.recenttracks['@attr'].totalPages <= 1) { break // it's the last page! } const lastTrackInPage = tracks[tracks.length - 1] to = lastTrackInPage.date.uts } } } } loige 67
  77. function createLastFmRecentTracks (apiKey, user) { return { [Symbol.asyncIterator]: async function

    * () { let to while (true) { const query = querystring.stringify({ method: 'user.getrecenttracks', user, api_key: apiKey, format: 'json', to }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const response = await axios.get(url) const tracks = response.data.recenttracks.track yield tracks if (response.data.recenttracks['@attr'].totalPages <= 1) { break // it's the last page! } const lastTrackInPage = tracks[tracks.length - 1] to = lastTrackInPage.date.uts } } } } loige 67
  78. function createLastFmRecentTracks (apiKey, user) { return { [Symbol.asyncIterator]: async function

    * () { let to while (true) { const query = querystring.stringify({ method: 'user.getrecenttracks', user, api_key: apiKey, format: 'json', to }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const response = await axios.get(url) const tracks = response.data.recenttracks.track yield tracks if (response.data.recenttracks['@attr'].totalPages <= 1) { break // it's the last page! } const lastTrackInPage = tracks[tracks.length - 1] to = lastTrackInPage.date.uts } } } } loige 67
  79. Let's search for all the songs that contain the word

    "dark" in their title! 🧐 loige 69
  80. async function main () { const recentTracks = createLastFmRecentTracks( process.env.API_KEY,

    'loige' ) for await (const page of recentTracks) { for (const track of page) { if (track.name.toLowerCase().includes('dark')) { console.log(`${track.artist['#text']} - ${track.name}`) } } } } loige 70
  81. For a more serious package that allows you to fetch

    data from Last.fm: loige npm install scrobbles 72
  82. Cover picture by on Thanks to Jacek Spera, , ,

    , for reviews and suggestions. Rod Long Unsplash @eoins @pelger @gbinside @ManuEomm - loige.link/async-it loige.link/async-it-code for await (const _ of createAsyncCountdown(1_000_000)) { console.log("THANK YOU! 😍 ") } loige nodejsdp.link 20% eBook discount on Packt 20NODEDUBLIN 73