Shai Reznik: How to Write Tests the Most Test Effective Way
If you need to cover a distance of 30 kilometers, you can walk for 6 hours. Or you can learn how to drive a car and get there in 30.
That’s what Shai Reznik, Test Effective Coach, learned the hard way. He worked so hard trying to juggle a 9-to-5 job with growing a startup that he got a panic attack. In the end, he decided automated what could be automated, including testing, is the only way to stay sane.
In this podcast episode, Darko and Shai talk about Shai’s career path from being a full-time developer to growing his startup to consultancy, why writing tests can save developers’ time, and how to write tests the test effective way.
Darko Fabijan (00:02):
Hello, and welcome to Semaphore Uncut, a podcast for developers about building great products. Today, I’m excited to welcome Shai Reznik. Shai, thank you so much for joining us. Please go ahead and introduce yourself.
Shai Reznik (00:16):
Okay. I’m Shai Reznik, I help business companies save time and money by integrating cost effective testing into their workflow without stopping development. For that, I use training programs, courses, coaching and consulting.
How Shai started with business consulting
Darko Fabijan (00:42):
Great. And that’s super important. From another aspect, we are also trying to help companies do the same by supporting them with tools and CI/CD practices.
I’m curious to find out, how did you figure out that it’s such an important thing that you want to focus on? How did your career lead you to doing what you’re doing right now?
Shai Reznik (01:07):
Okay, awesome. So I have a website called HiRez.io, which is going to get a newer version with all the details inside. Currently it’s a learning page, and you can reach me there if you’re interested.
So my backstory, how did I get into this topic?
I started as a normal developer working at a nine to five job, and I learned about this thing called testing. But back in 2006, I thought to myself, “Okay, yeah, it looks interesting, but I don’t have time now, I need to develop stuff and I still need to learn the frameworks and stuff. So I will get to that at one point.”
And a few years later, I was working in a company and they had crazy deadlines there and they kept giving me more and more tests because they counted on me to deliver fast.
Shai Reznik (02:08):
And at the same time, me and a couple of friends started to work on a side project that we want to become a startup to raise fund for. And so I was working, I didn’t have kids back in the day and I wasn’t married.
So I said to my girlfriend (now my wife), ‘“Hey, let me invest all this time now. So we’ll be rich and we’ll succeed. And then we’ll go out, and have fun to dinner and stuff like that.”
So I spent all of my time, investing in development and trying to improve myself as a developer and trying to meet all the deadlines both of the day to day job and the after hours startup work. So I was working like crazy.
Shai Reznik (02:56):
And what I realized is that I was working in the nine to five job. I was working on a big feature, with workflows and trees and all that, complexity on the UI.
And I got to a point where every time I introduced a change, I created many bugs. And every time I try to fix a bug, I create four more bugs. And they kept asking me for estimations. “How much time can you finish this feature?” And I kept saying, “Oh, two weeks or something like that.”
And this sprint and the sprint ended. And I was like, “Did you finish it?” “No, I was stuck fixing bugs.” And then we got investors wanting to invest in our startup, but they wanted to see the product, the first version. I had bugs with that version.
How Shai started with writing tests
Shai Reznik (03:43):
And I was trying to juggle all the bugs, trying to figure out what the hell I do. And it kept postponing, two sprints, three sprints. And I ended up exhausted and frustrated and was out of answers, basically.
It reached a point because of all of this pressure. I started eating bad and not exercising. And I got to a place where I actually walked up the stairs and felt dizzy and sat down and almost passed out.
And it was like, I didn’t know what’s wrong with me. So I went to the hospital and they did a bunch of checks and didn’t find anything wrong. And the doctor said, “Hey, you might have a panic attack or something that.”
And I said, “Panic attack?” I didn’t know what’s a panic. So I looked and it was what I had. I had trouble breathing and all that stuff because of all of this pressure. So I was worn out, very nervous and stressed.
Shai Reznik (04:44):
And I remember saying after one of the springs, “I don’t know what to do. I don’t have any solution.” So I was a flex guy back in the day of flex Flash development and such.
And one of the Java developers looked at me and asked me after the retrospect, “Why aren’t you writing tests? It’ll help you scale.” And I was first of all, I don’t know how to do that. Second of all, it looks like a waste of time. If I don’t have time to finish the features, how do I have time to write tests?
And he said, “Yeah, but…” It gave me a good metaphor.
Shai Reznik (05:26):
It’s like you’re saying, “Okay, you need to walk six hours to get to your destination. Right? It would take you six hours or you can learn how to drive and then get there in 30 minutes.” But you don’t want to learn how to drive. You rather walk all the way, and maybe it’ll get longer and longer and such.
You need to walk six hours to get to your destination. Or you can learn how to drive and then get there in 30 minutes. But you don’t want to learn how to drive.
So I said, “Okay, that makes sense. How much time do you think it’ll take me to write a simple test?” He said, “Okay, if you just started out maybe two days just to write a simple unit test and just to check a pure function.”
One thing, I went to my manager and asked them, “I think that this is the way out. I don’t see another way out. I need to stop and spend two days trying to figure this out.” And he said, “Okay,” fortunately.
Shai Reznik (06:16):
So I went through the computer like crazy, trying to understand. So I installed the flex unit plugin to the flex builder, the editor. And I wrote my first test.
And I remember in flex, you used to wait for two minutes on every compilation, just to check out that it’s compiled, and then you went to the browser and checked all the things. So I ran the test and it took one millisecond. I don’t know. Something like one second and gave me the green bar of “Hey, this code works.”
And I was “Aah!” It was an ah-hah moment, an epiphany, eureka moment. I ran to the other room screaming, “Hey, it worked, it took one second,” and I was so happy and delighted. I had lots of motivation to go deeper and to study it and to learn.
Shai Reznik (07:10):
And the more I learned, the more I saw, and I remember the first time it actually caught a bug. Not in the code that I’m currently working on, but on another code because I changed some shared code somewhere and all of a sudden another test broke.
And I was like, “What?” I didn’t realize the impact of “Hey, it just stopped me from pushing more bugs and having complaints from the QA or the users without me having to go through this workflow of wasting time and stopping what I’m doing afterward to try and fix the emergency.’ It just saved me all this time in one click of a button and I got hooked immediately.
The deeper I got, the faster I could go. So that was how I got introduced to testing and why I started to fall in love with that.
The most common mistake when writing code
Shai Reznik (08:04):
So after that I finished working on the feature and I started implementing and learning about techniques like TDD or test driven development. Fortunately, we raised money with the startup. I quit my job and went to work as the technical lead and co-founder on the startup.
But I made the same mistake now as a manager. I said, “Let’s move fast and we’ll write test afterwards. Right?” We’ll write tests when we get to the first milestone or the second milestone and then we’ll worry about the test.
And we reached the first milestone and such. But again, we went to the same quicksand. Because you come to a point where you want to move faster, but the faster you move, the more bugs you create and the lower you sink in the quicksand. Right?
The faster you move, the more bugs you create and the lower you sink in the quicksand.
Shai Reznik (08:59):
So we went to this place and I was so into testing, but I didn’t implement it on my own startup. So I told my co-founder, “Hey, I’ve been here before, we need to add tests, okay. It will help us.”
He argued with me because he didn’t understand why I needed two weeks to work on something invisible that doesn’t bring value immediately. But fortunately, I managed to convince him and we started having tests and we could grow faster.
And actually technically wise, it was very successful. We had a great product, but we learned the hard way, the importance of a lean startup, because we didn’t do a lot of market research or user interviews and stuff like that. So we spent a lot of money and time building the wrong product.
Shai’s transition into consultancy
Shai Reznik (09:59):
So then we had to close down the startup. I joined a normal day to day workforce as a developer first. Then I started being a consultant, trying to take all of the lessons learned by my entrepreneurship, managing and developer experience and try to merge it all to help companies migrate.
Standup and improvisation in developer lectures
Shai Reznik (11:01):
But at first, I didn’t know anyone. I just wanted to ask questions, but there wasn’t a friendly group to do that. So I created it and then started doing talks myself because I didn’t know anyone. So I started just sharing my knowledge and my journey and such.
And over time, I got invitations to speak abroad and I started submitting also to conferences. And I started speaking at the world’s largest Angular conferences, for example, NG-Conf and Angular Connect and such.
My hobbies, regardless, were always doing comedy, like improvisation and stand up. So I started thinking maybe I can integrate the two things like teaching and improv and comedy to make the stuff that you will learn more interesting and more entertaining. So I started doing that in the lectures, in the big conferences.
Shai Reznik (11:55):
And sometimes it was a row. Sometimes it was a stage play. Sometimes it’s a rap song. Every time it was something different. So people started knowing me as the crazy Angular guy who teaches in a weird way, but a way that makes you remember the stuff better.
So I took all of this and that’s where I decided to start doing online courses about Angular. And when I did that, I started seeing that people kept asking the same thing. They wanted material about testing because they kept struggling with testing. I said, “Oh testing, that’s a subject close to my heart.”
How to find the balance between confidence and efficiency
Shai Reznik (12:51):
And when I did that, I realized that the way that we are learning how to test and actually implementing the strategies that are 20 years old, by now even longer, are not producing the wanted results. Especially when the company grows. At first, I felt it was only about confidence, right? We want the confidence to ship our code.
But then when I started implementing different strategies, I saw that it’s not only about confidence, it’s also about efficiency. I can have the largest end to end test suite in the world, but it will take five hours to run. So I will get all the confidence that they need, but I won’t be able to move faster because I will need to wait five hours every time. And it costs a lot and such.
So I started thinking about how I can find the balance between confidence and efficiency?
Shai Reznik (13:45):
How can I find the metric there that represents the optimum between them?
And the only word that I came up with was cost effective. How cost effective is using this testing strategy compared to the other testing strategy or using this tool compared to this tool. And actually doing experiments and measuring it.
For example, in Angular, the common wisdom was to write desk for the dome, compiling the components and actually rendering them and testing stuff on the dome. But from experience and also from experiments, we saw that it’s not effective. When you reach a certain amount of tests, your test suite becomes six minutes, 10 minutes, 20 minutes, and such.
Using class testing
So a better strategy was to use class testing, which just instantiates the class of the component. Because in Angular, it is separate from the template, not in React. And then you could just write the test for the method. You lose a bit of confidence, but you gain more efficiency and that way you get the optimum, that’s just one very specific example about Angular.
Shai Reznik (14:54):
But I started applying that logic to other testing strategies, which are not Angular specific. For example, end to end testing, okay? Now there’s a shift left in the QA and development communities where developers are being expected to write more tests, which is good.
But if you just throw at the developers “Hey, write a bunch of end-to-end tests for your feature”, you end up with a test of five hours. Instead of spending their time in a more cost effective way, on building or developing features, developers are wasting time on running around after tests and trying to fix them. Trying to figure out the root cause of this failure and wasting time on tasks that are not contributing to the business.
Writing tests the most test effective way
Shai Reznik (15:47):
So I started trying to figure out what is a better testing strategy for developers in terms of end-to-end. Not just saying, “Oh, you need to write end-to-end test,” because it’s vague in general.
But there are better strategies, for example, smoke testing, which is just writing the critical path test and not just for every feature and that way you get the most confidence and the most efficiency, the most test effective way.
That’s the name that I came up with for the cost effective metric. Cost effective is too general. So I would call it test effective.
And then I started putting everything that I’m doing under this term, test effective development because I’m focused on developers. Although a lot of QA automation people and developers are coming to my courses as well and wanting to learn how to write the test. So it’s another market that I didn’t think I’m addressing, but apparently I am.
How to optimize the rotting test suite
Darko Fabijan (17:09):
And if you have a client or a company approaching you and they already have what you described as a very inverted testing permit. over the years they have gathered so many, very expensive end-to-end tests, acceptance tests, integration tests. However people call them and whatever they do there actually. But in your terminology, they are not just test effective, they’re not cost effective.
Shai Reznik (17:43):
Darko Fabijan (17:44):
From that point on, do you advise that they rework a part of their test suit or just embrace a new practice going forward, and then eventually over time tackle that rotting test suit?
Shai Reznik (18:02):
So every consultant will say, “It depends,” right? Depends on the company and the size and such, but it starts with an analysis first.
For example, how much time does it take? How much money does it cost the company to run this test every time? Do the developers need to run them or does it run only in the CI? Are the developers writing that or the QA automation is in charge on that?
The answer to these questions will affect the conclusion of what the company should do. But the ideal case, I think the most test effective way in a sense, is to have the developers focus on writing a combination of what I call multi-action test and single action test, but in the most test effective strategies. And I will explain, okay? What are single election, multi-action? Why I’m not saying integration tests, unit tests and such.
The ideal case is to have the developers focus on writing a combination of multi-action test and single action test, but in the most test effective strategies.
Shai Reznik (19:01):
I have a problem with the terms integration and unit tests. Although this is the common way and again, the common wisdom, but I feel that are overloaded.
For example, if you read the goose book, the growing object or in the guided by test book, you’ll see that integration tests are referred to as external integration tests, stuff that you test your code against code that you don’t own.
But if you look up integration tests in Wikipedia, it says that it’s writing something in integration, like several things together.
Shai Reznik (19:34):
And unit test, for example, even if you go to the blogs or read the Martin Fowler definition on that, you’ll see solitary or mockists and classist approaches to unit test, which basically one means isolated test and the other means integrated tests. Like integrating several classes together as one unit.
Shai Reznik (19:58):
Again, this is an attitude type of thing and an approach, but it complicates the term. So I start looking at it and trying to use a different angle to think about these terms from the test effective lenses and trying to think in first principles, taking inspiration from Elon Musk, not that I’m as close to this alien, but I take inspiration from a lot of things.
And I took inspiration from the first principle thinking and trying to divide the unit test. What is a unit test and what are all the dimensions of testing in order to deconstruct them and rebuild them differently in order to find the most test effective way of approaching a certain situation.
Several dimensions of testing
So for example, I realized that there are several dimensions to testing. One of them being boundaries, which is what I just said, isolated versus integrated.
Shai Reznik (20:55):
Another one is action scope. For example, do I test just one action in a test? For example, calling a method or click in a button, or do I test multi-actions? Filling up the form, clicking a button and going to this page…
So this is a multi action test. And sometimes people are mixing those up without even knowing if this is the right approach or not. And what is more cost effective? Is it single action, multi action to which situation? These are the questions that I dealt with and compared to the scale of the program, what’s the better approach.
And another dimension for example, is the subject in under test. Like I mentioned, the dome or the class, should I instantiate this or that? Should I write this for the method? What’s the subject, what’s that dimension?
Shai Reznik (21:41):
So for example, in terms of unit integration, I started seeing that you can have a single action test, should test only one thing, but it could be either isolated or integrated and what are the benefits or downsides to each approach?
And I realized that it has a lot to do with the size of the app that we are testing. The app under test. For example, are we dealing with the library or a shared component library that we’re testing? This will be a whole different set of tests compared to the monolith of glue code, I call it, that ties up all their reusable components or reusable modules together, which will be a different type of tests. And what is more test effective in this way?
The small room metaphor
Shai Reznik (22:30):
So I came up with better terms. For example, a small app or a microservice or something that you can imagine as a small room, a room in an apartment store or something that, just a room.
And if you have all of your modules there, classes or files, as boxes, you can see all the boxes. And if you want to test them, you want to test them in a single action, integrated way. And that way you get the most test effectiveness because you get the most efficiency and the most confidence from this type of test, because you’re not isolating.
So you’re getting the real values and all that stuff. Cool. So that’s the small room metaphor with the boxes. But what if you have a stadium, and you have lots of boxes far away from you? That’s the monolith, okay, example, right? Like a stadium. So I thought about how I can find a metaphor for the test.
Flashlight tests and laser tests
Shai Reznik (23:27):
And then I thought about light. If I need to shed a light on these boxes in the small room, I could use a flashlight and hit all the boxes at once. And this would be the integrated way.
And if I’m in a stadium, a flashlight wouldn’t help me and couldn’t reach everything together. So I had to switch strategy and point a laser pointer to one of the far boxes. So I now thought about the name, okay, flashlight tests and laser tests. So depending on the size of the app, I can use different strategies and would know what they mean.
So for example, laser tests are always isolated, always, always isolated. They’re not partially integrated, meaning you mock or you fake some of the dependencies and you leave the others as is. You always fake everything that you can. With the exception of, if it’s a super pure external, dependency or something that.
Shai Reznik (24:22):
And the integrated you integrate all the way to the, EdgeX layer or communication layer or remote layer. And on the end to end side, I started looking at strategies smoke testing, where you test only the critical path.
So then you need all this context to understand, because it’s a complicated subject. It’s not, “Hey, right, I saw blog posts or tweets or something that. That’s saying, in one sentence you should write mostly integration tests.”
Great. Okay. Now when my test suite takes five hours, watch what I do with this sentence. Okay. So it’s a more complicated topic, but for some reason I’m super passionate about this topic. So I really love doing research and such.
So to answer your question from a while ago, the ideal situation is the developers writing either smoke and laser tests or smoke and flashlight tests, depending on the situation. Sometimes it can change between monorepo apps and libraries, right?
How to save developers’ time and spend it the most cost effective way
Shai Reznik (25:21):
But the developers should write a smoke test. If you’re suffering with a long test suite, which the developers need to maintain, I would say, try to identify the critical path of each of the entire app or product and each of the root features and minimize…
For example, if the developers are maintaining them, minimize those and just extract the smoke test out of your test suite and let the developers maintain debt. If you have a QA team, let them worry about the end-to-end test, the full test suite, because this is their job… No, not 24/7, hopefully, but the nine to five job, right?
The developers need quick feedback on only the most important path to test that everything is configured correctly or assembled correctly in their end to end. They want to focus on the more flashlight or laser test or single action test because they drive it. They’re designed for better code design as well. And that’s the best case scenario in terms of the development point of view.
Shai Reznik (26:26):
In terms of the QA, I read QA books and stuff, but I have never been a QA developer or QA automation or manual QA. And that’s probably maybe a topic for the future, but you have other methodologies on how to maintain that.
But if we were talking from a development point of view, the developers usually cost the highest in a company. They are the most expensive employees in a company. So we want to make sure that their time is the most cost effective time that can be. And maybe separate the need to have the full end to end suite, running only after the smoke test finished running.
That way, you save five hours of time. This is a practice from QA. It’s not something I invented, but I would at least run the smoke test before the full end to end suite. So that, I would’ve done in a general use case, but I said, it depends on the specific conditions.
How to do testing in regulated industries
Darko Fabijan (27:50):
As you said, it’s not an easy topic. There are so many variables. Size of team, nature of the application, maybe even the industry. Is it a regulated industry where everything has to be compliant and so on? Or it’s more freedom is just fine?
Shai Reznik (28:17):
Yeah. in insurance and tax and banking and all management stuff you don’t want any slips, between the cracks.
Darko Fabijan (28:25):
Shai Reznik (28:25):
You want to make sure you are fully covered, but sometimes you have these settings or other places you want to identify the most critical stuff and to make sure you cover them as much as you can.
But the other stuff which is less critical, you want to use the most cost effective or the test effective strategy that you can do and not waste unnecessary time on that.
How to start working with clients
Darko Fabijan (28:49):
And can you give us an overview about your services and when you come in, usually? When you’re talking with the companies and what are maybe some of the typical relationships that you have with the clients?
Shai Reznik (29:04):
Sure. So when I started, I started with focusing on individual developers and trying to train and give the knowledge necessary in order to get from zero knowledge to a place where I can feel comfortable writing my test, even in a TDD way, in a test driven way, where I write the test before the code and driving my co-design that way. So that’s where I started.
So I started with courses and trying to record my knowledge and to scale it in a video form. And when I started doing that, I got approaches from companies that wanted me to help them train the entire team because they realize that, stronger is the weakest link in the chain. And if you won’t train the entire team, you’ll get some of the people writing tests and some of the people not writing tests.
And the first group is running after the second group trying to fix their test for them. And so it’s not efficient as well.
Shai Reznik (29:56):
So you want everyone on board and that requires a type of thinking, because now you get to the world of first of all, the business psychology.
Because now we need to convince the managers who are not technical to invest time in the entire team transformation and it’s not easy. And also how to help the company or the team not stopping what they’re doing for a couple of months. And just adding tests to everything, but actually integrate it into their day to day.
So I started doing that. I started working with team leads in smaller companies like C-levels and such. Helping them and coaching them on how to start, how to measure what they currently have, how to set goals to what they want, what type of improvement and percentages?
Shai Reznik (30:45):
And that part was super important because that way they could project the time save or the money save for the company.
For example, if we improve the bugs to feature ratio on average in each sprint by 20%, then you can show how much money does it save the company. And that way you bridge the gap between the technical nerds like us who talk in tests and code to the business people who talks in numbers and money and cost effective, which is in terms of the business, not of the code and time save.
So I started doing that and we figured that it’s a longer timeframe, but it will help them not stop development.
Shai Reznik (31:30):
So currently, what I’m offering is a training program, which you can take as either an individual. So we have developers buying their own and you can take as a team.
So have I said, team leads or CTOs and stuff, buying for the entire team transformation. And you can have that, the training plus the coaching and consulting because I kept doing consulting to stay true to the code, stay with the code, not just be the educator, but actually be still a developer. But mainly I work on infrastructure and moving to better architectures and tests that.
Shai Reznik (32:11):
So I’m doing consultancy. Now I’m scaling up because I don’t have a lot of time doing the consultancy work anymore because I have a business to manage and all that stuff, but I still do the coaching and now I started training other coaches and other people to help me with that.
So those are the three ways of getting my services currently. In the future, I have future plans of certification, all that cool stuff, but yeah, it’s in the works. So yeah, that’s the services part.
Darko Fabijan (32:42):
In the beginning, you mentioned your site, but what are some ways of getting in touch, but also, you have a number of conference talk. So I guess for those, the YouTube and just, Googling your name or searching your name is the best way, but what are ways to get in touch?
Shai Reznik (33:01):
The current version of the web will change soon because first of all, I started with Angular testing specifically. So we have an Angular master class for people who are angular developers that want to learn how to be more test effective.
But we started seeing that most of what they teach is not Angular specific. For example, I have one team who I help with their unity code, in C#. And a lot of the same principles apply there as well. And we have the end to end or the smoke testing courses and Cypress courses, which are not relevant to a specific framework this and that, or they could be applied to any company.
Shai Reznik (33:41):
So currently we’ll see the master class for Angular testing, but yeah, you can see on YouTube, all of my other testing lectures and all of my crazy NG-Conf talks, I mentioned before.
And once the new website is up, you can reach through there. But you can also always hit me on Twitter or on my email firstname.lastname@example.org. So that’s the best way currently to reach me.
And if you live in Israel, you can follow me on the street and scare the hell out of me if you meet me in the street. So follow me on Twitter, not in the street.
Darko Fabijan (34:21):
Great, great, great. Well, thank you Shai. Many hard lessons that you learned through your career and yeah, I’m really glad that it got you hooked. The biggest pain point that you have detected and you kept on solving it and helping other people solve it. So, yeah, that’s really great.
Shai Reznik (34:38):
Yeah. Yeah. Thank you. You remind me that story and I always feel that I don’t want other people to get into the panic mode that I got into. The panic attack and going to the hospital and all that stuff. I really, really, really want to save that from other people.
So that’s what keeps me up and trying to find better ways. I didn’t mention the open source tools that I created and all that stuff, but that’s for another episode maybe…
Darko Fabijan (35:03):
Yeah, for sure. Yeah. I can relate to that completely. Especially as a young developer. That level of peace of mind that you have when you have your test suits, being green or just a couple of things, being red. Completely different to that when you don’t know where you are going, like you are working on a mind field and you don’t know what’s going to blow up.
Shai Reznik (35:25):
Darko Fabijan (35:27):
Again, thank you so much. It was very interesting and I think very useful for lots of our listeners. Good luck. Thank you.
Shai Reznik (35:34):
Thank you very much for having me.
Have a comment? Join the discussion on the forum
Originally published at https://semaphoreci.com.