Startup vs. Runtime Performance

[00:00:00]
>> I think as an industry we have become little immune to this. Where everybody keeps claiming how amazingly performant they are, but I think they're missing the wrong thing. If you look at these benchmarks, and I'm sure you've seen this, right? They do things like, how long does it take to render a thousand rows, right?

[00:00:21]
Sure, it's a use case. But that's not the problem for 99% of the websites. None of the print sites are not slow because they're trying to render 1000 rows, right? And this benchmarks lie for several reasons. First of all, well, not that they lie. They don't lie, I mean, they show the truth here.

[00:00:40]
They're just measuring the thing that doesn't really necessarily matter. So the first problem I have is that all of these benchmarks assume that JavaScript is already downloaded, right? Not the case for most sites. The second thing they assume is that the jitter has already been warmed up. So what happens is, they run this benchmark over and over and over again, and the first few runs get thrown away because they're too unpredictable.

[00:01:06]
The reason that they're unpredictable is because the jitter is not ready, right? The jitter hasn't collected sufficient amount of data about the runtime of the application to do this. Now, when you run a code in tight loop, Jitter does all kinds of crazy things that are not a representative of what actually happens in production.

[00:01:23]
Everything from the fact that a tight loop will have really good caching in terms of L2 cache inside the CPU. It will do really good caching in the property lookup in the VM which is not true in the actual application code, etc. So there's a whole bunch of reasons why warmed up Jit is actually incorrect of the actual performance, and specifically, it's incorrect because when you navigate to a page on a fresh page, no jitter.

[00:01:55]
You are running in interpretive mode and everything is being running interpretively.
>> Sorry, when you say jitter, that has different baggage for me based on my background. When I hear jitter, I think of fluctuations in execution or latency time, but I think what you're really referring to is the just-in-time compiler.

[00:02:12]
>> Sort of yes, here jitter means just-in-time compiler, right? So,
>> I just wanted to clarify.
>> Yes, it's good clarification. So, JavaScript is a interpreted language, right? That can be compiled down to assembly. And that's the job of the Jitter, to compile it down. But it turns out it's actually extremely difficult to compile it down.

[00:02:35]
And so, what happens in practice is that the Jitter collects information about what it thinks your JavaScript is actually doing. And based on that, it takes a whole bunch of shortcuts. And then if you break one of these assumptions that the Jitter makes, then the Jitter does this thing called deopt.

[00:02:51]
In other words, it says, yeah, I generated all this code, I thought it was correct, but it turns out it's not, I'm gonna throw it away. And so, all of these benchmarks that you have here are my point is that they're kind of unfair. None of them are unfair, they're not measuring the thing that you want.

[00:03:07]
The problem of our industry is not that the application is slow once it's up and running, although there are some applications that have this problem. The problem with the industry is just getting the application to run is the problem. Yeah, we talked about the Jitter, we talked about this totally focuses on the dumb update and it ignores the fact that there is this thing called time to interactive and total blocking time.

[00:03:30]
And so, my point with these benchmarks is that while they're important, they're not actually going after, an improvement in this benchmark here will probably have no measurable improvement in your startup performance of the application. So, people will oftentimes be like, I should use framework x cuz it's twice as fast as the other thing, on my application is gonna be twice as fast.

[00:03:53]
You run it, you rebuild your app in the new technology and you realize the application is just as slow as it was before. There's no difference in the thing that we care about. So, instead we did a different benchmark here. So first, we have this tool called Mitosis.

[00:04:08]
Mitosis allows you to write kind of pseudo code of an application and write it once, and Mitosis will generate that code in idiomatic format, in angular, in react, in whatever technology you want, right? And this allows you to really have apples to apples to comparisons, right? Because a lot of the people that wrote the movie app complained, well, there's actually two different versions of the movie app.

[00:04:31]
Ours does log in, yours doesn't do log in, or whatever. It's all fair points and so with Mitosis you can actually literally build the same exact app across the technologies. And the thing to kind of compare here is time to interactive, right? This is the blue line and the total amount of JavaScript.

[00:04:52]
And you can see is that as the total amount of JavaScript goes up, the blue line goes up as well, right? And again, it totally makes sense because what we're talking about here is, the way hydration works is it eagerly executes your application. The more application you have, the longer it takes, right?

[00:05:09]
So, the more JavaScript you ship, the longer the startup is gonna be. It's as simple as that. Figure out a way not to ship JavaScript, your startup will be fine.