Async Iterator

Problems with Synchronous Iterators

As mentioned in the chapter on Iterators, the Iterator interface is a protocol for data traversal. By calling the next method of an iterator object, you get an object representing the information at the current position of the traversal pointer. The object returned by the next method has the structure {value, done}, where value represents the current data value, and done is a boolean indicating whether the traversal has ended.

function idMaker() {
  let index = 0;

  return {
    next: function() {
      return { value: index++, done: false };
    }
  };
}

const it = idMaker();

it.next().value // 0
it.next().value // 1
it.next().value // 2
// ...

In the code above, the variable it is an iterator. Each time it.next() is called, it returns an object representing the current traversal position.

There is an implicit rule here: the it.next() method must be synchronous, meaning it must return a value immediately when called. In other words, once it.next() is executed, value and done must be obtained synchronously. If the traversal pointer happens to point to a synchronous operation, there is no problem, but for asynchronous operations, this is not ideal.

function idMaker() {
  let index = 0;

  return {
    next: function() {
      return new Promise(function (resolve, reject) {
        setTimeout(() => {
          resolve({ value: index++, done: false });
        }, 1000);
      });
    }
  };
}

In the code above, the next() method returns a Promise object, which is not allowed and does not conform to the Iterator protocol. Any code containing asynchronous operations is not allowed. In other words, the next() method in the Iterator protocol can only contain synchronous operations.

The current workaround is to wrap asynchronous operations as Thunk functions or Promise objects, where the value property of the return value of next() is a Thunk function or Promise object that will eventually return the real value, while the done property is still produced synchronously.

function idMaker() {
  let index = 0;

  return {
    next: function() {
      return {
        value: new Promise(resolve => setTimeout(() => resolve(index++), 1000)),
        done: false
      };
    }
  };
}

const it = idMaker();

it.next().value.then(o => console.log(o)) // 1
it.next().value.then(o => console.log(o)) // 2
it.next().value.then(o => console.log(o)) // 3
// ...

In the code above, the return value of the value property is a Promise object, used to hold asynchronous operations. However, writing code this way is cumbersome, unintuitive, and semantically confusing.

ES2018 introduced (opens new window) the "Async Iterator", providing a native iterator interface for asynchronous operations, where both value and done properties are produced asynchronously.

Async Iteration Interface

The most notable syntax feature of async iterators is that calling the iterator's next method returns a Promise object.

asyncIterator
  .next()
  .then(
    ({ value, done }) => /* ... */
  );

In the code above, asyncIterator is an async iterator. After calling the next method, it returns a Promise object. Therefore, you can use the then method to specify the callback function that runs after the Promise object's state becomes resolve. The parameter of the callback function is an object with value and done properties, which is the same as with synchronous iterators.

As we know, the synchronous iterator interface of an object is deployed on the Symbol.iterator property. Similarly, the async iterator interface of an object is deployed on the Symbol.asyncIterator property. Regardless of what kind of object it is, if its Symbol.asyncIterator property has a value, it indicates that the object should be iterated asynchronously.

Here is an example of an async iterator.

const asyncIterable = createAsyncIterable(['a', 'b']);
const asyncIterator = asyncIterable[Symbol.asyncIterator]();

asyncIterator
.next()
.then(iterResult1 => {
  console.log(iterResult1); // { value: 'a', done: false }
  return asyncIterator.next();
})
.then(iterResult2 => {
  console.log(iterResult2); // { value: 'b', done: false }
  return asyncIterator.next();
})
.then(iterResult3 => {
  console.log(iterResult3); // { value: undefined, done: true }
});

In the code above, the async iterator actually returns values twice. On the first call, it returns a Promise object; when the Promise object resolves, it then returns an object representing the current data member's information. This means that the async iterator's ultimate behavior is consistent with the synchronous iterator, except that it first returns a Promise object as an intermediary.

Since the next method of an async iterator returns a Promise object, it can be placed after the await command.

async function f() {
  const asyncIterable = createAsyncIterable(['a', 'b']);
  const asyncIterator = asyncIterable[Symbol.asyncIterator]();
  console.log(await asyncIterator.next());
  // { value: 'a', done: false }
  console.log(await asyncIterator.next());
  // { value: 'b', done: false }
  console.log(await asyncIterator.next());
  // { value: undefined, done: true }
}

In the code above, after processing the next method with await, there is no need to use the then method. The entire flow is already very close to synchronous processing.

Note that the next method of an async iterator can be called consecutively without waiting for the Promise object produced by the previous step to resolve. In this case, next method calls will accumulate and automatically run in sequence. Here is an example that puts all next method calls inside Promise.all.

const asyncIterable = createAsyncIterable(['a', 'b']);
const asyncIterator = asyncIterable[Symbol.asyncIterator]();
const [{value: v1}, {value: v2}] = await Promise.all([
  asyncIterator.next(), asyncIterator.next()
]);

console.log(v1, v2); // a b

Another approach is to call all next methods at once, then await the last operation.

async function runner() {
  const writer = openFile('someFile.txt');
  writer.next('hello');
  writer.next('world');
  await writer.return();
}

runner();

for await...of

As previously introduced, the for...of loop is used to iterate over synchronous Iterator interfaces. The newly introduced for await...of loop is used to iterate over asynchronous Iterator interfaces.

async function f() {
  for await (const x of createAsyncIterable(['a', 'b'])) {
    console.log(x);
  }
}
// a
// b

In the code above, createAsyncIterable() returns an object with an async iterator interface. The for...of loop automatically calls the next method of the object's async iterator, which returns a Promise object. await is used to handle this Promise object, and once it resolves, the obtained value (x) is passed into the body of the for...of loop.

One use of the for await...of loop is that async interfaces with deployed asyncIterable operations can be directly placed into this loop.

let body = '';

async function f() {
  for await(const data of req) body += data;
  const parsed = JSON.parse(body);
  console.log('got', parsed);
}

In the code above, req is an asyncIterable object used for asynchronously reading data. As you can see, using the for await...of loop makes the code very concise.

If the Promise object returned by the next method is rejected, for await...of will throw an error, which should be caught using try...catch.

async function () {
  try {
    for await (const x of createRejectingIterable()) {
      console.log(x);
    }
  } catch (e) {
    console.error(e);
  }
}

Note that the for await...of loop can also be used with synchronous iterators.

(async function () {
  for await (const x of ['a', 'b']) {
    console.log(x);
  }
})();
// a
// b

Node v10 supports async iterators, and Stream has implemented this interface. Below is the difference between the traditional approach and the async iterator approach for reading files.

// Traditional approach
function main(inputFilePath) {
  const readStream = fs.createReadStream(
    inputFilePath,
    { encoding: 'utf8', highWaterMark: 1024 }
  );
  readStream.on('data', (chunk) => {
    console.log('>>> '+chunk);
  });
  readStream.on('end', () => {
    console.log('### DONE ###');
  });
}

// Async iterator approach
async function main(inputFilePath) {
  const readStream = fs.createReadStream(
    inputFilePath,
    { encoding: 'utf8', highWaterMark: 1024 }
  );

  for await (const chunk of readStream) {
    console.log('>>> '+chunk);
  }
  console.log('### DONE ###');
}

Async Generator Functions

Just as Generator functions return a synchronous iterator object, the purpose of async Generator functions is to return an async iterator object.

Syntactically, an async Generator function is a combination of an async function and a Generator function.

async function* gen() {
  yield 'hello';
}
const genObj = gen();
genObj.next().then(x => console.log(x));
// { value: 'hello', done: false }

In the code above, gen is an async Generator function. After execution, it returns an async Iterator object. Calling the next method on this object returns a Promise object.

One of the design goals of async iterators is to enable Generator functions to use the same interface when handling both synchronous and asynchronous operations.

// Synchronous Generator function
function* map(iterable, func) {
  const iter = iterable[Symbol.iterator]();
  while (true) {
    const {value, done} = iter.next();
    if (done) break;
    yield func(value);
  }
}

// Async Generator function
async function* map(iterable, func) {
  const iter = iterable[Symbol.asyncIterator]();
  while (true) {
    const {value, done} = await iter.next();
    if (done) break;
    yield func(value);
  }
}

In the code above, map is a Generator function. The first parameter is an iterable object iterable, and the second parameter is a callback function func. The purpose of map is to process each value returned by iterable using func. There are two versions of map above: the first handles synchronous iterators, and the second handles async iterators. As you can see, the two versions are written in essentially the same way.

Here is another example of an async Generator function.

async function* readLines(path) {
  let file = await fileOpen(path);

  try {
    while (!file.EOF) {
      yield await file.readLine();
    }
  } finally {
    await file.close();
  }
}

In the code above, the await keyword is used before asynchronous operations, meaning the operation after await should return a Promise object. Wherever the yield keyword is used, that is where the next method pauses. The value of the expression after it (i.e., the value of await file.readLine()) becomes the value property of the object returned by next(), which is consistent with synchronous Generator functions.

Inside an async Generator function, both await and yield commands can be used simultaneously. You can think of it this way: the await command is used to input values produced by external operations into the function, while the yield command is used to output values from inside the function.

The usage of the async Generator function defined in the code above is as follows.

(async function () {
  for await (const line of readLines(filePath)) {
    console.log(line);
  }
})()

Async Generator functions can be combined with the for await...of loop.

async function* prefixLines(asyncIterable) {
  for await (const line of asyncIterable) {
    yield '> ' + line;
  }
}

The return value of an async Generator function is an async Iterator, meaning that each time its next method is called, it returns a Promise object. This means what follows the yield command should be a Promise object. If, like in the example above, a string follows the yield command, it will be automatically wrapped into a Promise object.

function fetchRandom() {
  const url = 'https://www.random.org/decimal-fractions/'
    + '?num=1&dec=10&col=1&format=plain&rnd=new';
  return fetch(url);
}

async function* asyncGenerator() {
  console.log('Start');
  const result = await fetchRandom(); // (A)
  yield 'Result: ' + await result.text(); // (B)
  console.log('Done');
}

const ag = asyncGenerator();
ag.next().then(({value, done}) => {
  console.log(value);
})

In the code above, ag is the async iterator object returned by the asyncGenerator function. After calling ag.next(), the execution order of the code above is as follows.

1. ag.next() immediately returns a Promise object.
2. The asyncGenerator function begins execution, printing Start.
3. The await command returns a Promise object, and the asyncGenerator function pauses here.
4. When line A becomes fulfilled, the produced value is placed into the result variable, and the asyncGenerator function continues execution.
5. The function pauses at the yield on line B. Once the yield command obtains its value, the Promise object returned by ag.next() becomes fulfilled.
6. The callback function specified by the then method after ag.next() begins execution. The parameter of this callback function is an object {value, done}, where value is the value of the expression after the yield command, and done is false.

Lines A and B function similarly to the following code.

return new Promise((resolve, reject) => {
  fetchRandom()
  .then(result => result.text())
  .then(result => {
     resolve({
       value: 'Result: ' + result,
       done: false,
     });
  });
});

If an async Generator function throws an error, it causes the Promise object's state to become reject, and the thrown error is caught by the catch method.

async function* asyncGenerator() {
  throw new Error('Problem!');
}

asyncGenerator()
.next()
.catch(err => console.log(err)); // Error: Problem!

Note that a regular async function returns a Promise object, while an async Generator function returns an async Iterator object. You can think of it this way: async functions and async Generator functions are two methods of encapsulating asynchronous operations, both used to achieve the same purpose. The difference is that the former comes with its own executor, while the latter is executed via for await...of, or by writing your own executor. Below is an executor for async Generator functions.

async function takeAsync(asyncIterable, count = Infinity) {
  const result = [];
  const iterator = asyncIterable[Symbol.asyncIterator]();
  while (result.length < count) {
    const {value, done} = await iterator.next();
    if (done) break;
    result.push(value);
  }
  return result;
}

In the code above, the async iterator produced by the async Generator function will automatically execute through a while loop. Each time await iterator.next() completes, it enters the next iteration. Once the done property becomes true, it breaks out of the loop and the async iterator execution ends.

Here is a usage example of this auto-executor.

async function f() {
  async function* gen() {
    yield 'a';
    yield 'b';
    yield 'c';
  }

  return await takeAsync(gen());
}

f().then(function (result) {
  console.log(result); // ['a', 'b', 'c']
})

With the introduction of async Generator functions, JavaScript now has four forms of functions: regular functions, async functions, Generator functions, and async Generator functions. Please note the differences between each type. Basically, if you have a series of asynchronous operations that execute sequentially (such as reading a file, then writing new content, then saving to disk), you can use an async function; if you have a series of asynchronous operations that produce the same data structure (such as reading a file line by line), you can use an async Generator function.

Async Generator functions can also receive externally passed data through the parameters of the next method.

const writer = openFile('someFile.txt');
writer.next('hello'); // executes immediately
writer.next('world'); // executes immediately
await writer.return(); // waits for writing to finish

In the code above, openFile is an async Generator function. The parameter of the next method passes data into the function's internal operations. Each next method call is executed synchronously, and the final await command waits for the entire write operation to finish.

Finally, synchronous data structures can also use async Generator functions.

async function* createAsyncIterable(syncIterable) {
  for (const elem of syncIterable) {
    yield elem;
  }
}

In the code above, since there are no asynchronous operations, the await keyword is not used.

yield* Statement

The yield* statement can also be followed by an async iterator.

async function* gen1() {
  yield 'a';
  yield 'b';
  return 2;
}

async function* gen2() {
  // result will eventually equal 2
  const result = yield* gen1();
}

In the code above, the result variable inside the gen2 function has a final value of 2.

Like synchronous Generator functions, the for await...of loop will expand yield*.

(async function () {
  for await (const x of gen2()) {
    console.log(x);
  }
})();
// a
// b