Vitest Code Coverage Configuration

[00:00:00]
>> So let's talk about strategies around that. And we'll also talk about, I'm trying to mostly create a fictitious situation to talk about, go back to the GitHub actions build process and how would you generate one of these and be able to download it later and kinda track stuff over time.

[00:00:13]
So that's also what we're doing, so spoiler alert. But if I kinda go, and I have the kind of pre-baked version cuz watching me write this is not super useful or interesting to anyone involved. In this test directory, in your vitest.config, they're both using the same libraries under the hood.

[00:00:34]
There are three popular libraries for code coverage, NYC, C8, and Istanbul. In the same way, vitest and Jest are API compliant with each other, they are mostly all compliant with each other. So which one you choose, I don't care, right? I use C8, with the C8, it's named after V8, annotates the code.

[00:01:01]
Why do I use C8? Because that is the one vitest would prompt you to choose either Istanbul or C8. That was first on the list in the drop-down, right? I'd love to tell you there's a better reasoning about it. But there are some configuration features,right? You can have all.

[00:01:18]
all is I think a super interesting one if you're trying to get that heuristic. Because if you think about this, normally what would happen is you do npm run test, neat. It goes to your test files. It's basically walking that dependency tree with your import statements, which means files that never get imported never get counted towards code coverage, right?

[00:01:39]
Which is either exactly what you want or you don't want. If you're just glancing, hey, I'm working on test coverage for this file cuz I think it's important, then you really only wanna point to a given file. And if you do run any of the filters, your coverage report is gonna be really bogus anyway, because you might not have the coverage cuz that test runner didn't do it.

[00:01:56]
all will at least try its best to truly walk the tree and find everything and start working through all the files. It's also slower. But it's good if you're trying to get a sense. Then again, my test utilities, my tests themselves, I don't want them counting. Again, to the joke that we've had all day, who tests the tests?

[00:02:16]
Not me. TypeScript files, right, configuration files, it'll depend on your application. Again, big JSON files that are maybe fixtures or something. Sometimes when I find weird situations, I will grab the network request JSONs and I'll throw them in a directory in case I ever wanna do something with them later, put the UI in weird states.

[00:02:35]
You can choose to ignore a bunch of stuff. You can choose to include a bunch of stuff. I think that this is arguably more important in practicality. Because, again, let's say you run this thing and it's like a sea of red, right? You're like, cool, let me start pushing back the ocean with a broom, right?

[00:02:58]
You might know, hey, the files in this directory, I know that this particular file is critically important to my application and it has terrible code coverage, right? You might start with one piece of your application, right? We are in this workshop, in this course, we're not going for the platonic ideal of anything.

[00:03:16]
We're going from, I have a ten-year-old code base. I inherited something. No, my god, I can't sleep at night. What do I do? You might choose the two or three most critical parts of your app and choose to maybe enforce the code coverage at that point because you know that they're critical, right?

[00:03:33]
We're doing things because they make us feel better and make us more confident, not cuz we have read a blog post on the Internet that told us we should, right? The other one that I think is interesting, which you saw statements, features, branches, choose one of what you think is the most important.

[00:03:51]
You do that by feel, by glancing the code. A lot of this is by feel. I hate to break it to you. And this will actually fail if you use dash coverage, if the threshold drops below that, to which you might be asking, hey Steve, how did you settle on 59.79% code coverage?

[00:04:14]
I would like to draw your attention to line 23, which is thresholdAutoUpdate. Which is I put in the number 50 because I glanced at it and I saw at the time it was 54. But we've been solving some issues during the workshop and I think I added some stuff last night.

[00:04:31]
And I have thresholdAutoUpdate set to true, which just basically says, it can't go down, right? If for some reason we wrote some tests and we got this thing up to 60, guess what it would auto update my file to be? 60, right? Especially if you're being very like nuanced with the input, as much as I'm against, cuz it's just like one of those things.

[00:04:59]
If you had it at 100 or something really high, so to your point before, why was line 15 red? I'm sure I could spend an hour and figure it out. I'm sure nobody wants to watch that right now. So I'm gonna say I don't know, move on with my life, right?

[00:05:14]
You will turn this off if it gets to the point where it's breaking for arbitrary things, but there might be things, like I'm trying to fix this, I'm trying to, we have a problem. I'm solving a problem, and then you might choose to turn that off. You know what I mean?

[00:05:26]
Once you've gotten to the point where you're no longer worried about it, and that is totally fine as well. But, Dustin has a question.
>> What is the all: true?
>> Yeah, we talked about a little bit, if you don't have that on, it will only check the coverage of the files hit by your test runner, right?

[00:05:49]
Which means by definition there will be some files that have 0% test covered, cuz no test file ever imports them, cuz vitest is looking for star. It's looking in this case, I have some weird things on this one, there is a default exclude that you can see, but for the regular ones, I'm avoiding this svelte test, weird stuff in this repo.

[00:06:09]
But vitest will load anything usually with *.test.ts, whatever. And it will look at those imports and annotate everything, files that are never imported by a test aren't ever counted if you have all set to false, which is the default. It's all true, we'll also do a filesystem scan, try to collect them all that would have matched and then analyze them as well.

[00:06:34]
It's tricky cuz you will then end up with things you never aspired to have test coverage, again like vitest.config.ts. That is a file that I will never test, [LAUGH] right? I don't want to test, I don't care. And I have it in the ignore one. So if you do all, this grows, right?

[00:06:55]
If you don't have all on, you can probably get rid of this, right? And so it's like, how much pain do you want and what direction? A lot of these things depend on the unique, all good code bases have a lot of things in common. All terrible code bases are unique and special in their own way.

[00:07:12]
This is why we're kinda taking a very wide philosophical stance here, because the terrible code base that you are dealing with, if it was that easy to solve by applying a few things, it would have been fixed by now, right? You could have probably just even run something that, Prettier can handle all the formatting issues.

[00:07:30]
You're left with that six deep nested conditional. You know what I mean? With three, four loops in there, right? You have to deal with that one. Prettier will definitely move that semicolon back for you though. And yeah, use the tools appropriately. And so, yeah, we're kinda casting a wide net in that sense.

[00:07:50]
But like I said, I think what could be interesting, and there are services where you could, I think CodeCov is one, Coveralls which I like as a name, will also integrate with your build process and chart it out over time. And again, to the point we made earlier, the number, [SOUND] that's dangerous.

[00:08:10]
A graph that shows it going down is possibly useful. You know what I mean? Saying like we have to have 80. And I think Amazon does this, right, it's like a Bezos thing, where it can't drop below 90 or something like that.
>> So philosophical question about that, I think you already answered it, but I wanna just clarify.

[00:08:29]
>> Yeah.
>> When you say the 80-20 rule for example-
>> Yeah.
>> You're not saying, just look at it. If it says 80%, we're good, you gotta actually look at what 80% is being covered.
>> Yeah, it's like, are you covering the right 80%, right?
>> The 20% that's not covered could be the most important.

[00:08:41]
>> Yeah, could be the 80% of what your code does. You know what I mean? It's like we joked before about that agony for performance, right? If the most hit code path in real world production usage is not covered, that's probably more important than if you had 80% of files that don't really get used all that often, right?

[00:09:01]
And also, any number that you read on the Internet should be treated with suspicion. Because they're written by a bunch of backend engineers who have microservices where everything is really condensed down and we have to deal with wild bunch of state and Safari hacks. Used to be IE hacks, right?

[00:09:21]
And things are tougher over here-
>> Viewports and?
>> What's that?
>> And viewports and devices.
>> And viewports and weird, Firefox does this a little bit differently, and Mac OS wants to put a gloat, that kinda stuff. So, and like I said, this is only doing what vitest does.

[00:09:42]
So you could have an amazing set of, is that a question we got from after lunch, you could have an amazing set of browser-based tests. And maybe someone's be like, yo, there's a tool that merges these things, I don't know about it, because it's usually working as the built assets at that point.

[00:09:57]
We don't have access to this anymore. But you could have something that is covering it effectively, as well. I worked at a company a few years ago, they hired three full time people whose job it was to clone down the repo and spin it up and do things, right?

[00:10:21]
[LAUGH] And so 10 to 100% code coverage, whose job it is? But yeah, and so it's useful tool. But the coverage for it I think is super interesting. And if you wanna do the programmatic stuff with it, there are different reporters that you can use. Yeah, reporter, and I think I can do it as an array.

[00:10:40]
Yeah, you see that autocomplete happening in there? The modern era is really great, [LAUGH] as I said before. This is when people who wrote compiled language would be like, the compiler tells me everything and finds all the problems. I'm like, must seem nice, and I feel like I'm starting to live in that world.

[00:10:58]
These are some of the ones built in. Do I know what they all are? No. TeamCity seems like a thing that I should know what it is. I don't. JSON, if you were gonna build your own or wanna track it over time, right, cuz, yeah, do I believe that you should write your own code coverage monitoring tool?

[00:11:16]
I do not. Have I worked at companies which won't let you install third party anything ever and understand that you might have to give them where you work? I do understand that. So JSON might be what you choose to do. That I read, as I was talking, as Cobra Tuna.

[00:11:30]
That is not what that says. But yeah, and so there are different things you can do. You can generate this, so on and so forth. It would be interesting if you wanna know what happens, how it annotates the code, that's the data structure. You are welcome to format that with Prettier and look at it in your free time.

[00:11:52]
I will not be doing that. And this is also partially a reason why maybe you don't wanna check this in the version control because, you know what we don't need in our lives? Everyone checking, having merge conflicts with that. So, yeah, but you might have your build process do it.