Parsing individual data items from huge JSON streams in Node.js
Let's say you have a huge amount of JSON data
and you want to parse values from it in Node.js.
Perhaps it's stored in a file on disk
or, more trickily,
it's on a remote machine
and you don't want to download the entire thing
just to get some data from it.
And even if it is on the local file system,
the thing is so huge that reading it in to memory
and calling JSON.parse
will crash the process
with an out-of-memory exception.
Today I implemented a new method
for my async JSON-parsing lib, BFJ,
which has exactly this type of scenario in mind.