In this episode, our host Anna Petrovicheva, CTO of OpenCV.AI, is talking to Simon Kozlov. Simon was the first machine learning engineer at Instrumental.ai, an artificial Intelligence California-based company that is helping consumer electronics companies find and fix problems better in their manufacturing lines. Before that he used to work in game development, working on 3D graphics and the infrastructure side of the graphics stack.
The video and audio versions you can find on Youtube, Spotify, SoundCloud, Google Podcasts, Apple Podcasts, and Stitcher.
The interview was recorded in March 2021.
How AI Makes Manufacturing Smart
Anna Petrovicheva: Hi, my name is Anna Petrovicheva, and today I’ll be talking with Simon Kozlov. Simon was the first machine learning engineer at Instrumental.ai, an artificial intelligence California-based company that is helping consumer electronics companies find and fix problems better in their manufacturing lines. Before getting immersed into the computer vision world, he used to work on the reverse problem of making real-time 3D graphics look awesome. In his spare time, he was teaching a deep learning course in his alma mater Novosibirsk State University.
Hi Simon, and thank you so much for joining OpenCV AI for Entrepreneurs podcast. I’m excited to have you here, and my first question to you would be: could you please tell us about the company you're working for? What are you doing there? And what is the goal of the company, what is the mission?
Simon Kozlov (SK): Thanks for having me, I am glad that such a thing exists, and really honored to be here. So, I work for a company called Instrumental. I'm one of the early engineers, I guess I’m the first machine learning person that joined the company, and we're in the manufacturing space – we're trying to use all of these great advances in AI and machine learning to help build products better. So, this is like a very high level, if we zoom in on the details, one of the things that I as an outsider learned about manufacturing is it's such a huge industry, it's a significant percentage of the whole look of the world economy, there are so many things going on. What we are doing specifically, and there are a lot of companies doing interesting things in the space, is we are focusing right now mostly on the consumer electronics segment, and what we help companies do is we help them make their existing lines run better. And usually, this follows this iterative process: you find that there is some kind of problem with your manufacturing process, you take steps to fix it, and you make sure that the problem is fixed, and then you repeat this loop as fast as you can in the time that you have allowed. And how quickly and how efficiently you go through this loop, determines a lot of the success metrics of manufacturing. Our product captures vital data during the manufacturing process with really rich data like raw images, very high-resolution, and AI tools that let you do this whole cycle from finding that the problem exists, that you might not be aware of, to figure out what was the problem, and taking steps to address it.
AP: Got it! So, do you see how artificial intelligence when applied to the manufacturing industry changes the industry? Do you see that the new lines that are being created are created with AI in mind? What is the effect there?
SK: This is a great question! The domain is so huge that it's really hard for one person to see it all, and my perspective is just that part of the manufacturing process that we work with, it's consumer electronics and it's geared towards the beginning of the process where you have a new product, and you get it to mass production, and you want to have a sustainable mass production. Manufacturing is very interesting in that: it's one of the industries that is very tech-heavy. For example, they've been one of the primary adopters of computer vision over the years. If you look at the last 20-30 years, manufacturing is one of the places where now really embraced machine vision and all of the approaches. At the same time, it's an industry that has certain ways of doing things, and some things have been built in the same way – like placeholders of high-tech technologies – over the last 20-30 years. So, it's an interesting mix: part of it really embraced technology for a specific reason but on the process side it's fairly conservative. This means that there are very different ways you can think of applying AI in the area. One way is to say okay, here's an existing process, and I’m going to make some particular part of it better; an example of that would be an optical inspection, animated inspection, AOI in the traditional computer vision which solves the problem the following way: I know the problem, I know where it happens, and it's more expensive for me to fix my process to stop making this particular defect, it's cheaper for me to catch it and remove that part from the line. It's kind of well-defined and people think about such issues in these terms. And you can apply AI and machine learning towards more complicated cases: that's sort of filling the same spot but better, and they're certainly a lot of opportunities to do that. They're also all kinds of more high-level things: you have a process, and you have a machine that does some industrial part, for example, injection molding, and it has a particular temperature mode and a bunch of parameters which really affect the outcome of the process. And currently, some people are like artisans in knowing what you are building, what the parameters should be, how you tweak it, and things like that.
You find that there is some kind of problem with your manufacturing process, you take steps to fix it, and you make sure that the problem is fixed, and then you repeat this loop as fast as you can in the time that you have allowed. And how quickly and how efficiently you go through this loop, determines a lot of the success metrics of manufacturing.
So, there are companies who attempt to automate that part and maybe AI could come up with a better combination or control things to influence the outcome. What we're trying to do is some of that; but we also try to impact the process of how things get made, meaning that changing the way that hardware engineers even find out about the problem, changing the steps that they take when they fix those problems. Imagine that you have a factory in China, it's overseas, and a company is a brand, the engineers who designed this product are in a different country, and you have very limited visibility in what's going on at the factory level. Usually, this means that the records that you might have been CSV – if you're really lucky, they will upload the CSV with thousands of rows and columns you don't understand. If you're unlucky, you have to get there, you have to fly to this factory and see exactly what's going on, and if you want to see images or something of what's actually happening, you really have to go there, which made challenging by the Covid and the current reality that we live in. People are adjusting to it. I think this is an event that moved the needle on the process in the last 10 or 20 years; this is the biggest event that has made the companies much more open, and made them rethink their processes in a Covid world – I can't call our world post-Covid yet.
AP: So, it seems that Covid had a positive impact on the company, right?
SK: It impacts in a lot of ways but it does seem that it would follow. What we're doing is we capture a lot of the vital data about manufacturing in electronic form and let people solve problems by interacting with this data as opposed to being physically on the line and then looking at things. It would follow that this would be something very useful in these times, and we do see that it proves to be useful for a lot of our customers.
AP: So, more and more companies understand that there is a huge need to automate things that are not automated, don’t they?
With AI technologies you can solve the same problem better, or you can solve new things that you previously have just been doing manually or struggling for. But now there's something that can automate a lot of it, so you actually end up doing creative work as opposed to manual work.
SK: Yes! Coming back to your previous question, you can apply AI to existing problems – a problem that's already well-defined, and with AI technologies you can solve the same problem better, or you can solve new things that you previously have just been doing manually or struggling for. But now there's something that can automate a lot of it, so you actually end up doing creative work as opposed to manual work, and it's much harder to explain and much harder to build the business around it. In that respect what happened with Covid in the manufacturing space, it made people much more open to new ideas and made them think from the first principles about what they are actually doing as opposed to relying on the processes that sat for tens of years.
Exciting Intersection of Theory and Practice: Applying AI to Real-Life Products
AP: I also wanted to ask you about how hard it is to apply artificial intelligence to real-life products in the manufacturing space. There are probably a lot of difficulties that you face when you apply this to noisy or dirty manufacturing facilities, etc.
SK: This is the most exciting stuff! My background is in engineering, and I got into machine learning and deep learning because of all of the amazing things happening in the area, but for me, this intersection of theory and practice is something most exciting. This is what makes me tick off: you have a particular problem, you come up with a solution, you try it in the lab and then see how well it works, and it seems to work, and then you get it in the real world. And reality has a way of surprising you; no matter how many times you did it, there's still something new to learn, something challenging, and so on. In more specific terms, what makes it fun in the manufacturing context: for example, one thing that makes it fun is that first you have this rich data in images, and there are problems that you're trying to find there, and this is an example of a severe imbalance if you think about it. We're looking for a particular defect, let's say, you want to have a model that separates defect from non-defect based on the image. And the general setting that we have is maybe you have thousands of images, the challenge is you can't get more because this is how many units were built at that point, you cannot go outside in the world and capture more data of the next phone that's being built – there's been only a thousand of them built. And your realistic defect count is something like one percent if you're really lucky; one percent is a huge defect, super important to fix, it's like everyone knows about it. The defects that matter at this point are also 0.1 or 0.5 percent because if you have a 0.5 defect and you don't fix it, and you create a manufacturing line, 0.5 at the scale of the billions of phones is overwhelmingly bad. So, this means your model receives say a thousand units, thousand images, and if you're lucky, ten of them are defects – and this is when you're lucky. In practice, you don't get lucky, and the value is five, or four, or three… So, how do you do that? This is something that they don't teach you at school. There's a way to deal with the balanced data but what if it's that imbalanced, and there's no way to get more? That's one interesting challenge.
Another challenge is as follows: as an example, usually the images we work with in deep learning space, the inputs to a lot of the classification networks, are like 224 by 224 or around that number. The realistic image that you can get is something like 25 megapixels. And it raises a question: okay, I have this image, I know some units failed or not failed, how do you even start with that? The size of the actual defect is maybe like 50 by 50 pixels, and if you resize the whole image then you won't even see it. This happens everywhere! It’s the general theme and the same challenges that people talk about like in academia, or struggle with or investigate with deep learning. However, the real world has a way of pushing it to an extreme on some axis that you don't expect, and then it becomes an interesting engineering and research problem.
AP: Yeah, that's exciting! I also love that: when you read some state-of-the-art papers, they typically operate on large data that is pretty much balanced, and when you start applying to some real-life problems, this is what you actually see. Thanks for sharing that!
Three Pillars of Teaching a Course in AI and ML
AP: I wanted to ask you about your teaching experience. As far as I know, you were teaching a course in artificial intelligence and machine learning, and you did it online remotely, even before it was a mainstream way of teaching artificial intelligence, and also other courses in universities. So, could you please tell us about this experience and what would you maybe recommend to people who study online right now?
SK: The story is as follows: I come from Novosibirsk State University, depending on the year it's one of the top three or top five engineering schools in Russia, and this is my alma mater. Since then I moved to the US and worked in companies, large and small. But in the back of my mind there was a background thread that was running: dude, you should give back somehow! I got the degree for free, which is amazing – looking at what happens in the world, specifically the US, the fact that I got this kind of education for free just because I had good grades in school, and passed the exam, is just beyond fantastic! And I always thought about how to give back to all of this. And the opportunity presented itself: I think it was my professor that visited me at some point, and we were talking about it, and at that point, I’ve been working and deploring for a couple of years, and I was really excited about it. Maybe some of your viewers resonate with this: some people can't shut up when they are excited about something, and they talk about that to anyone they meet. So, we were talking and a question arose: why doesn't anybody teach it, why isn't there a course. He told me they had been trying to find somebody who would teach this for some years already, and nobody wanted to. And I asked if he wanted me to do something like this. I think this situation is quite typical in my career. They were desperate enough to accept me – I’m not a professor, I don’t do research in the space, but I work in the space. However, they didn't have a better option, so I ended up teaching the course. I was in Silicon Valley, they were in the middle of Siberia, and there was a stream on YouTube, which was fun. Imagine that you have the first lecture, there's an auditorium where people coming in, there's maybe 100 people who signed up for this course; you sit down in front of a big projector and your teaching assistant opens a YouTube link and hits play – and this is a live stream with the audience, which is kind of surreal – at least it felt surreal in 2019.
AP: What is the time difference between you and the students?
SK: It was late and night for me, and it was morning for them. And halfway through the course, the light savings time kicked in; in Russia, they just don't have the daylight savings, and California still does, and my lecture started at 11 PM and then it suddenly had to start at midnight because of the daylight savings. However, their time didn't change, so they were good.
AP: Would you like to share any advice with people who are learning remotely right now? Maybe some tips and tricks that you learned from this experience?
What I’m looking for in a university course or an online course is a full set of three things: first, there have to be lectures, and not just some text but videos. Second, there has to be homework; you need to get value from the course, you have to do the homework, and if you don't, then 50 percent or more of the value disappears. Third, you have to have a community where you ask questions.
SK: Yes, a couple of things come to my mind. A preface: everybody's learning style is very different. Some people learn from practice only, they need to feel it with their fingertips to get a feel for it. I’m the kind that likes some structure. I think a university course is a perfect structure, there's some theory, there's some practice, and then you're ready to take on new things. What I’m looking for in a university course or an online course is a full set of three things: first, there have to be lectures, and not just some text but videos, as for me a video of somebody explaining something works really well. Second, there has to be homework; you need to get value from the course, you have to do the homework, and if you don't, then 50 percent or more of the value disappears. Third, you have to have a community where you ask questions: you ask them if you're stuck, you help other people, you get involved. The nice thing about online courses is that those three things don't have to be from the same place. Well, lectures and homework may come from the same place, but as for the community, there're a lot of options. You can organize such a community by yourself by just asking friends or buddies who also go to this course, you can join one of the existing online communities and ask questions. There's a bunch of great communities, both in Russian and in English. So, for me those three things are essential. If you have a buddy that you can go through this course with, this is amazing! I passed some of the courses with my wife and that was an awesome experience!
AP: I think this advice on the community is very important because this is something that makes it personal for you, that makes you really finish the course that you started on some online platform, right.
SK: Absolutely. Also, there's a lot of social feedback, and if it's a vibrant community, you start learning tangential things – what other cool things are happening in the space, you make great relationships. Again, I’m a nerd, and I’m not the loudest person at a party, let me put it this way; and if I look back, I can see circles of friends that I made because of the changing interests, so it’s a very natural and pleasant way of meeting people and keeping in touch. It turns out a couple of years down the road always pay off – I’m on this podcast, for example!
The Path from Game Development to Deep Learning
AP: Tell me about your career path into artificial intelligence and deep learning. As far as I know, you worked as a game development engineer, and so what was the transition path to machine learning and artificial intelligence? Could you please share your experience with that?
SK: I guess they're two parts of it: “why the hell would you do it?” And the second is “how would you do it?”, so I’ll start with the first part. I made games, it started with 3D graphics, and I really liked all of these shaders, GPU, all of the cool stuff to make things run fast, and I grew into thinking about the platform and the product overall. If any of you have kids of a certain age, if you know the word Roblox – blame it all on me, I am the problem, even though obviously there are a lot more people working on it. And at some point – I call this middle middle-aged crisis for engineers which is when you have worked at some space for like 10 years or a little more – and it does not feel as exciting anymore for some reason. There's a combination of reasons: you learn fewer new things, and you start looking at the larger picture. When you work in the game industry, even if you develop graphics or cool technology, in the end, your product is entertainment, you're in the entertainment industry, so it means your final product is this fleeting sensation of feeling awesome. And this is great, this is a very valid product that you can sell, and it will make people's lives better, but I wanted to see my product being something more tangible, something that stays in the world, unlike this fleeting feeling that just disappears after it happened. Moreover, I was tired of graphics and games as a space, and I was looking at what else I could do. I was looking for this excitement of learning something new and working on something that at the point felt more meaningful. I look at a bunch of things: I was interested in AR, I was interested in robotics, and even took a gander at blockchain, the intellectual part of the blockchain I mean. This coincided with the rise of deep learning, and with machine learning, it felt right, it was like love at first sight. So, this is the why-question. The how-question wasn't rosy. At that point, I had around ten years of experience, and it turns out that switching careers as a senior engineer is not easier but might be even harder than as a junior engineer. Say you are a senior engineer and you want to work with machine learning, and you apply to work on it, and people expect you to be a senior machine learning person with years of experience and knowing everything about the space, and you're clearly not that. Even if you are okay with reduced payment and you want to be like a junior person learning about the stuff, it doesn't work – it just doesn't click in people's minds, and they just don't talk to you. That was surprising, downshifting is not the thing that often happens. So, the question is what to do? The solution for me was similar to what I alluded to before – to find a company that is so desperate to hire someone then they would allow you to work on things that you really like. For me this was Dropbox. Dropbox is an amazing company, and what makes Dropbox different from any other company that I’ve ever worked for is a unique combination of how qualified and friendly people are. The bar for hiring engineers is extremely high (I’m sure for other people it is very high as well), and somehow, they managed to hire great engineers and have a friendly company culture, which is an amazing thing to experience. Naturally, they always have a problem hiring people because if your bar is super high, then it will be a challenge. So, I applied as an engineer to the ML team, they told me it was so hard for them to hire engineers, and if they find someone who they feel could be an engineer here, so I used all of my negotiation leverage not towards the better salary, or a position, or anything. I assured them I wanted to be an engineer on this team, I wanted to absorb what was happening, and this is how I got to the ML team. Most of the stuff that I’ve worked on was far from ML, there's always enough engineering things to do, but it was an opportunity to absorb things that were happening around, and I spent a little more than a year there. The next step, if you're interested: I joined Instrumental as a first-mission learning person. They were trying to find a machine learning person to join a startup and they gave me a chance. And it has been working out so far.
I wanted to see my product being something more tangible, something that stays in the world, unlike this fleeting feeling that just disappears after it happened.
AP: You sound like you just got lucky but I believe that you put a lot of hard work into that: to learn a lot, to get new skills.
SK: Absolutely! This is luck! The other part of it is that you keep trying, keep both learning new skills and applying to a lot of companies until you hit this magic desperation combination… I’m kidding! What I’m trying to say is that there is the part about growing your skills and being able to perform the work; a lot of the skills can be grown only by doing things, by having an opportunity to work on something real. And for a lot of positions in the outside world, they might not know that you have those skills because it's hard for them to measure that; they look at your background. Once this solution worked for me, I’m not sure if it works for everyone: find an opportunity that is maybe a loose combination of those things, and you have to keep trying a lot to find somebody who would give you a chance – and it's a privilege to receive that opportunity and obviously, there's a promise on your part to do your best. By chaining a couple of these things, you could smooth out the curve from transitioning from what you've been doing before to a very different domain to work afterward. Am I making sense?
AP: Totally! And then, when you get this opportunity, you definitely have to work hard, learn new things, and don't give up! When you don't have this chance with one company, you will have it with some other company.
SK: Or with some other team, or under different circumstances.
AP: I enjoyed talking to you! Thank you for being on our podcast, that was my last question for today, and I wish you a very good day!
SK: You too, thanks for having me!