Testing Goals & Prebuilt Rules

[00:00:00]
>> Marcy Sutton: So the biggest thing, no matter what kind of test you're writing is that you should really focus on the outcome and not the implementation. And there's lots of great things that people have written about testing, methodologies, and I know Kent C Dodds is a testing master. So definitely, if you're interested in this area go learn from him.

[00:00:21]
But yeah, this is the idea that you could swap out your entire implementation and your test would still pass which is amazing, that's such a cool feeling to have that work. And if you're testing things for accessibility, I mean, focus, there's slight variation just depending on your design and depending on what you're building.

[00:00:40]
But, I mean, if you're just restyling something and your end user's not really supposed to know that it changed, that's a good case for having these tests be kind of swappable or the code swappable because the user contract for focus management and for accessibility stuff should be unchanged theoretically.

[00:00:58]
So that really applies for accessibility. So in this space, I guess this applies to unit tests as well but especially in integration tests. If you use an accessibility testing API, you don't have to write all of those rules. Trying to maintain color contrast rules, I've worked on those, it's a lot of work.

[00:01:18]
And there's a lot of changes to specifications like HTML and Aria that people who are working on test frameworks all the time are keeping abreast of. So that way you don't have to stay on top of like, all right, what did the Aria working group decide for accessible name calculation or something like that?

[00:01:36]
That can change. I know for a fact the axe-core team is keeping on top of that, so I don't have to any more, which is pretty cool. So things like axe-core, that's a JavaScript library that's open source that you could use in your unit tests. There is a cypress integration that I think we already have installed, so we'll play with that.

[00:01:56]
If you're using web driver, slimline web driver, you can use axe-webdriverjs. The WAVE API is another API that you could try. accessibilityjs is an API from GitHub. And then there's Lighthouse, which uses axe. So it's kind of a similar approach. But if you were doing a bigger suite of tests, you could theoretically use Lighthouse to test for accessibility too.

[00:02:22]
So as far as automation and accessibility, we can't catch everything, I mentioned that a little earlier. But we can catch estimated about 30 to 50% of issues by volume for accessibility issues. It really depends on the rule set. So if you're using something like pally which uses HTML code sniffer by default, I believe that rule set will be different than what axe would return or WAVE would return.

[00:02:50]
And it's really a way to just catch some of those low hanging fruit issues in an automated test fashion. So things that we were finding in browser extensions having that baked in your automated tests can be pretty cool. One caveat that I really wish would evolve, maybe it will in the future but screen readers can't actually be automated yet.

[00:03:12]
Research project for someone, hint, hint? Maybe Python or NVDA is written in Python? I don't know. It would be really cool if we could automate those and somehow assert that screen reader output is what we expect. There is some cool stuff happening with Puppeteer and the accessibility tree.

[00:03:32]
But ultimately what I'm concerned with is what the screen reader is doing, not what the accessibility tree is doing, because they can differ. So yeah, just a note about screen readers not being automated. But we really wanna write a combination of tests to catch a variety of bugs, like having some unit tests for your components, having some integration tests.

[00:03:52]
They just do different things well, and it's nice to have a variety. I guess if you could only choose one, if you're gonna start somewhere, I'd probably start with integration test just because it doesn't really matter what framework you're using or anything. You could use cypress with whatever, basically.

[00:04:10]
It's just looking at pages as rendered. So I'd probably start there, it's also just a really great API.