Introducing Node Streams

[00:00:03]
>> Will Sentance: We learned about the, on this very note, event paradigm. The event system, nature of Node. What if Node used the event, message-broadcasting, pattern, where it flags out a message depending on stuff that happens in Node, to auto trigger a function to send out a message event each time a sufficient batch of the JSON data had being loaded in?

[00:00:29]
And then at each point, have that auto run a function. And that function can start cleaning that batch of data, store it, store it. While we're running that function, cleaning that batch of data, Node is pulling in the next batch, triggering the function again to clean that. And we're effectively gonna end up doing these in parallel.

[00:00:57]
While we're pulling in batches, we're gonna simultaneously be running the clean tweet function batch by batch in here. I can see this reducing our time down to, I'm gonna have a feeling, 15 seconds, basically. Because we're gonna spend all the time we're pulling in batches in the background simultaneously cleaning tweets in the foreground in JavaScript, which is only gonna take ten seconds.

[00:01:20]
So we can do all of this while this is happening in the background in chunks as it comes in.
>> Will Sentance: Should be pretty cool. I'm gonna feel there's a bit of an issue, though, because the data is gonna come in in chunks at a rate. I think I'm gonna switch this up, haha haha.

[00:01:43]
Actually, it's gonna take 20 seconds, there we go, to clean that much data. The data is gonna come in, auto trigger a function to run on the next batch. But the running of the cleaning of this batch will yet have finished. So is that auto triggered function allowed to jump in and run at the same time as the previous ones running?

[00:02:02]
No, but it can’t just be it’s not allowed to. There must be some really, really strict rules for when it’s allowed to throw the next running of the function in. And that’s gonna introduce to us some queuing stuff. [BLANK AUDIO]