Promise.all the things?

Promise.all the things?

If you google for asynchronous mapping or parallelisation for javascript you'll most likely end up using Promise.all for those tasks, which is perfectly fine.

But what about processing a lot of items at the same time? For example if you need data from an additional API it can quickly end up in a home grown DoS of your own or someone else infrastructure.

To prevent this you can process this list of items in batches. For this I came across p-map by Sindre Sorhus (if you are using modern javascript the chance is very likely that you are already using one of his modules).

With p-map it's easy to turn something like this

const wait = ms => new Promise(resolve => setTimeout(() => resolve(ms), ms))

async function process() {
  const items = [1,2,3,4,5]
  const mappingFunction = item => wait(item * 100)
  return Promise.all(items.map(mappingFunction))
}

process().then(console.log).catch(console.error)

Into this

const pMap = require("p-map")
const wait = ms => new Promise(resolve => setTimeout(() => resolve(ms), ms))

async function process() {
  const items = [1,2,3,4,5]
  const mappingFunction = item => wait(item * 100)
  return pMap(items, mappingFunction, { concurrency: 3 })
}

process().then(console.log).catch(console.error)

Does not look very different but the last one processes only three items in parallel and the other ones as soon as if there is capacity left.

If everything is processed, the promises resolves and returns and array with the resulting data in the same order as it was pushed into, just like Promise.all would return.

Show Comments