In Conversation With Vicki Cheung: SRE, MLOps, and Lessons Learned from Lyft, OpenAI, and Duolingo

Join HashiCorp's Nic Jackson and Anubhav Mishra as they sit down with Vicki Cheung for a fireside chat.

Speakers: Vicki Cheung, Nic Jackson, Anubhav Mishra

_Unedited Transcript_

Nic Jackson:

All right. So welcome to another episode of In Conversation With, and this time we've got our good friend Vicki Cheung, who is the CTO of Stealth, a stealthy startup that we might get a little bit of insights into later on. But Vicky's also had a really illustrious career. She's worked with Lyft, OpenAI, started up things at Duolingo. And we're going to dig into all of these things because it's really impressive the amount that she's achieved and this is the purpose of In Conversation With. It's been able to bring the human, demystify the way that they kind of got through their career and to hopefully help you all out with your careers too. So Vicky, welcome to the show and thank you so much for coming.

Vicki Cheung:

Yeah, thanks for having me.

Nic Jackson:

All right. Well, so the first question, what we generally try to dig into is people's background. Now, what was it that first got you interested in Computer Science? Why did you decide that would be a thing to study?

Vicki Cheung:

So I think it started when I was in elementary school, probably around 10. I grew up in Hong Kong and right around the time when I was 10, they started actually having computer classes in public schools. So the first year curriculum was something like, "Have you guys heard of Logo?" I just associate it with drawing the thing. So we learned that and we had to make HTML websites and because I went to a public school, we didn't really have any fancy software so we had to actually write the HTML by hand. And I guess I thought that was actually really fun and empowering in one way because I was like, "Oh, I'm just this 10 year old kid, but I can sign up for GeoCity or whatever, and then put on this website." And it was just like, "People don't know that I'm just 10."

Vicki Cheung:

I think snowballed from there where I started making. I guess the next thing was probably PHP sites or something, and then started going further down the rabbit hole. And I didn't really think that hard about what I wanted to do because it just seemed so natural that like, "Oh, this is the thing that I enjoy spending my free time on and it also happens to be a thing I can study and be a profession." And really there wasn't really a lot of sort of wondering about it, I guess in my mind.

Nic Jackson:

You started off looking at HTML and I think HTML was fascinating too, because it, as you said, it gives you the ability to create this thing on this internet thing that's public and that's pretty amazing. Where did that kind of a depth come from? Because the thing is when you started looking at your career and you're looking at things you're doing with Duolingo, you're not so much working with the PHB and the front end or was it? You kind of starting to dig deeper, you're going deeper into all system's programming and things like that. Was that something that made you turn direction, you kind of think, "Oh, this is actually really interesting, this depth of complexity."

Vicki Cheung:

Yeah, well, I actually really see myself as a generalist because I find all things engineering very fascinating. I really like making websites and making things beautiful. I think that's such a joy, but I also really liked digging into systems and understanding the intricacies and everything. I guess the first time I really thought about systems was probably when I was in high school. I was really a big nerd, I guess I kept going down this engineering rabbit hole. So I would buy college Computer Science textbooks and I would just read them and do the exercises. So I bought this systems book that I actually eventually used in college. It just went into like, "Oh, how all those system pieces work." And yeah, I thought that was super fascinating too.

Nic Jackson:

It's definitely, I think, when I was researching through some of the things you've done and there's definitely some twists and turns and I'm looking forward to getting into those, but let's start at the start, Duolingo. Now, Duolingo is a great platform. You started off there when it was literally just finding out. How did you get that role? Tell us about how you got the role, what you did there.

Vicki Cheung:

Yeah, I would say everything that happened was very much a combination of being at the right place at the right time and also, I guess, proactively seeking out the things that I like to do. So I guess to go back to the very beginning, Duolingo started out of CMU, Carnegie Mellon University, which is where I went for college. I would say it was a single decision that really changed the course of my life and my career, which is when I was in high school. I went to a summer camp at CMU where I was taking some college classes and they told me to just stay. So I dropped out of high school and just start college because I was like, "Oh, I think this is what I like to do."

Vicki Cheung:

And then when I was in college, I would seek out these professors I really enjoyed working with and I was like, "Oh, can I work with you on research or whatnot?" So one of them was Louis von Ahn, who was one of the professors in my freshman courses. And he was like, "Oh yes, actually I'm working on this thing called Duolingo. It's between me and my PhD students sovereign. And we got a couple other people, like interns and other students working on it." And I was like, "Sure, it sounds good." And so I started doing that as an internship and the first task, I still remember the first day I showed up at, I guess it was still a campus office then, there's this giant NLP book on my desk and they're like, "Okay, can you just read about speech recognition and work on our speech recognition engine."

Vicki Cheung:

And I was in college at the time. And I was like, "Sure, that sounds fun." So, that's where I spent my summer on. And then we just really enjoyed working together as a team. And I was like, "Oh, this seems a mission for a company that really resonates with me." And so I was like, "Oh, I just stick around." Yeah, so I kept working, I guess, part-time throughout my senior year. And then shortly after that, we raised funding and spun out as a company. So long story short, that's sort of how I got there. And I guess the other thing about what I worked on there is a little bit of everything because when you're one of the first people in a company, you just kind of do whatever it needs to be done to ship the thing.

Vicki Cheung:

So after speech recognition, I actually went on to work on mostly full stack stuff. I built a lot of the web front end and then wrote the Android up and did a lot of the API layer stuff also. And I guess over the course of a few years, drifted lower and lower into the stack, but I would say very much all over the place.

Nic Jackson:

You're an incredible Engineer. And that goes without so saying, but do you attribute a lot of the understanding, the knowledge that you've picked up over the years because of that attitude of wanting to know how everything works, being able to wanting to dig in and do a little bit of everything, understand how the API side works, but also have an understanding how the front end works. How do you feel kind of, does it give you additional sympathy, say when you're working on a backend that you have mechanical sympathy for how the front end works or for front end devs? Is there a beneficial thing, do you think?

Vicki Cheung:

Yeah, for sure. I think a lot of what different areas of the stack is optimizing for is different. And if you don't know what the person you're working with is optimizing for, it's very hard to work effectively as a team, I think. So even if you've never personally experienced doing front end and stuff, I think, it does help to just ask them, "What are you optimizing for?" And build empathy that way. But certainly having been in their shoes, I'm like, "Oh yeah, I know this is really a hard thing to do for you. So I'm going to try to build it as much as I can to ease your pain."

Nic Jackson:

What sort of advice would you give companies when they're trying to retrain their staff? You've touched on things like education and growth and stuff. Is there anything specific as an employee, specifically when you're at that junior marginal, moving over to the mid-tier and upwards that you could give to organizations to help retain great people?

Vicki Cheung:

Yeah, I think everybody is a little different, everybody has different priorities in their career. Now I would say the first thing you need to do is understand those priorities. I think, when I was managing, I would have fairly regular career check-ins with people. Just like, what are you thinking about your career? What do you want to work on in the next quarter that would make you feel you're really satisfying your career goals and try to help them find those opportunities in the company. And I do think that sometimes the right answer is for people to leave a company and experience something else. But I think a lot of times, companies don't even explicitly ask that question for people like, what do you want in your career? And so it's very easy to lose people that way.

Nic Jackson:

So for me, whenever I've moved on to a different company, I've just had this feeling that it was time to go. I've never really taken a measured approach to it. And that's probably because I'm not so smart. But do you think there is a measured way that as a growing developer, how do you weigh up the decision of stay or go? Do you think is there a formula or some device that you'd be able to give folks who are maybe stuck in that?

Vicki Cheung:

I mean, I think a lot of times it is a feeling. Your intuition can tell you a lot, before you maybe even know about it and be able to articulate the reasons. But I also think that there are a few things that can help, as I've moved through the companies, I always start with... I guess when I'm deciding where to go next, I have a list of goals in mind. And when I started the company, I just very explicitly told my manager, "These are my goals. These are the reasons I started at this company. I want to be able to meet these goals in whatever time frame."

Anubhav Mishra:

Yeah. But I think one of the interesting things I saw in New York area, which was to move from, let's say, but from the company we were talking about, TrueVault, to OpenAI and I think you did move girls, you went from, I guess, software engineering into infrastructure. And I know we've talked about this in the past, you really indulge into the infrastructure side of things. And so what prompted that change in moving from TrueVault to OpenAI?

Vicki Cheung:

Yeah. So, I got connected with the people who eventually started OpenAI while I was at TrueVault and the mission really resonated with me and I felt I could just learn a lot there. It was very different from working at a startup where you have your microservices, you have your website, you have just... I'm not trying to trivialize or generalize, people's stuck but there's certainly a certain way that you build startups or products that I think is very different from the way you build infrastructure and whatnot for a more R and D focused team. And so I feel that was going to be an interesting change for me.

Vicki Cheung:

And I just wanted to see what's up. So, yeah. As I started talking with the team there, I guess, again, it was another one of those where the company wasn't even formed yet. So it was just me and Greg Brockman working in his apartment. And yeah, I didn't really actively decide I was going to spend full time on infrastructure. It was more just again, there's only two of us, one knows that we'll have to do it.

Anubhav Mishra:

You sacrificed.

Vicki Cheung:

So we ended up actually both doing it. Another funny story about that was when I interviewed at OpenAI, they didn't really have an interview process or anything. Greg was like, "So I tried setting up this Kubernetes Cluster and it's not working, can you fix it?"

Vicki Cheung:

And that was the interview. And I had never even heard of Kubernetes before because it was pretty young, it was version one or something, and definitely hadn't used Terraform before that. So it was like, "Okay, a lot of new stuff. This looks fun." So yeah, just got in, got my hands dirty, started TCP dumping all over the place. Because I was like, "Ah, they're not talking." And yeah, at the end of the interview, I guess, got something working and then I was hired and that ended up being a major theme in that role.

Anubhav Mishra:

That is super cool, yeah. I mean, I think that's the most organic way of starting a company is to just stack and talk about problems that you have today and then hopefully find people that can fill in those roles. But thinking about this, right? You're a software engineer now doing infrastructure stuff. Yeah, of course, you're learning about new technologies like Kubernetes, [inaudible 00:15:58] and networking and how does that all work. Do you think there's a need for software engineers to learn more and more about the underlying infrastructure. Do you think, is it better to have just software engineers just work focused on code and leave the infrastructure things to experts who are doing networking and what SRE's hired into those roles. Do you think there is a benefit of being well-rounded in that way?

Vicki Cheung:

Yeah, I think it's kind of a spectrum. It's hard to say where software ends and infrastructure starts these days. Certainly if you work on Kubernetes, is that software engineering? I think it is. It's software engineering in a very infrastructure, heavy area. So yeah, I guess that's maybe one observation, but yeah, I do think that at least the hope is that more and more people can get started with software engineering without knowing too much about the stack. And I mean, already, with the cloud, you're shielded from a lot of hardware stuff. And so I feel that layers keeps getting pushed higher and higher, but I do think it is beneficial. It might be my personal bias from working in smaller teams. But I think it does help, after a certain point, maybe not when you're prototyping or just shipping your first product or something, but after a certain point, you start to run into maybe bottlenecks or different constraints in your designs that you might not be able to really pinpoint or debug without knowing the infrastructure.

Vicki Cheung:

So yeah, I think certainly you can get away with doing a lot without knowing, but after a certain point, I think it's useful.

Anubhav Mishra:

Do you have any tips for people that are thinking to do this transition moving from software, I guess more traditional software engineering role to a more SRE or operation engineering roles?

Vicki Cheung:

Yeah, that's a good question. I think the first thing is to try doing it and see if you like it. I think it is a little different. Yeah, there are times when you are writing code or building pieces on top of whatever Kubernetes or something, that's certainly, I think, to people who are more used to building products, maybe that's the kind of a workflow that looks very similar, but there are also other times when it feels you're spending a lot of time building up a pretty massive mental model of what's going on in the system and trying to put the pieces together in just the right way. And I think that not everyone enjoys that kind of Lego building. So I think playing around with it will help. I think another thing is, do you like thinking about failure cases? Do you thinking about reliability and security and all that stuff? Are those the kinds of things that you to seek out and fix proactively? If so, then maybe there's a future for you and operations or SRE.

Anubhav Mishra:

That was interesting. You moved to Lyft, right? And I think you were leading as, I guess, a software engineering team at Lyft. Again, you've moved away from infrastructure looks like, maybe not. Could you tell us more about it?

Vicki Cheung:

Yeah, I think it was another one of those moments where I took a step back and was like, "What kind of impact do I want to have in the next few years in my career?" And decided that I really cared about helping people grow and I find a lot of satisfaction from that. And further, I think, creating a good work environment where people enjoy going to work, and I feel as a manager that's really your responsibility to make a good environment for your team. I also feel I wanted to spend some time working on diversity and inclusion issues and tech. And I think as a manager, you get to spend a lot of time doing that because you're responsible for the culture. And so all those things led me to, "Oh, let me try doing engineering management for a few years."

Vicki Cheung:

And yeah, I think I was still fairly deep in infrastructure. I was leading the Compute Team at Lyft where we were building out the Kubernetes platform for Lyft. So yeah, there was certainly a lot of talking together as a team on what architecture should look like and how do we get the whole company onto Kubernetes, what's the scale and so on. So yeah, there's definitely, I guess, in the great spectrum of engineering management, it was one that was maybe closer to the technical side then than not, but still, yeah, I really enjoyed it.

Anubhav Mishra:

All right. So I really enjoyed talking to you Vicki, but I think I want to ask this last question, the present of what you're doing. I know you left Lyft shortly after the IPO to start your own company assessed out. Of course, you can't share too many details, but we would love for you to share some of the high level details of what you're working on today.

Vicki Cheung:

Yeah. One of the interesting things I learned at OpenAI was how much the infrastructure experience sucks for things that are not core software engineering. So what I would say is if you look at building infrastructure for machine learning or something, there are a lot of hurdles to getting it right because it's such a intersectional area where you have to know about the constraints of machine learning and you have to be able to build large scale systems and it's, yeah, I think it's just like when I look at how we built our infrastructure at OpenAI, it feels we were always going against the current because whatever was being built like Kubernetes at the time was heavily optimized for stateless microservices, which was not at all what we were doing.

Vicki Cheung:

And so it was always like trying to make the infrastructure do the thing that's not built to do. And so that kind of got me further down into this intersectional area of machine learning and infrastructure, and thinking about all the difficulties people have for building nice platforms for doing these workloads. And then I guess, looking at the problem generally of what are the problems that machine learning engineers are actually running into to see where infrastructure can help. And so really studying a lot about what the users are doing. And I guess long story short, my co-founder and I, we met at OpenAI and he's from the ML research site. So we partner up and have the totally different backgrounds, I think, makes a very good team for understanding both sides of the spectrum and realizing that the number one problem people have is that machine learning engineers have very basic questions about their machine learning models and production that they can't really answer.

Vicki Cheung:

So for example, how healthy is my model? Maybe if you think about the world in software engineering before, there's great observability tools, for example, people are scared to deploy because "Oh, I don't know if this will break something." Even today people are like, "Oh, don't deploy on Friday because I don't want to deal with that." So if you think about that but 10 times worse or a hundred times worse for machine learning, then you get to a place where machine learning engineers aren't deploying models for maybe a whole year, because they're like, "I don't want to touch that. If I upgraded this library, I don't know if the performance will change." And they have no great way to tell.

Vicki Cheung:

So, that was really I guess the Genesis of a lot of what we're thinking about building is giving people that feedback loop of what is happening in production. And I think more generally, I like to think about it as bringing the disciplines and the lessons that we learned from software engineering into machine learning engineering, what have we built up in terms of tooling and software engineering that doesn't quite work yet for machine learning but that we could make it work if we change our mental model a little bit on what software is. So, that's sort of the larger mission.

Nic Jackson:

That's fascinating. And I'm wishing you the best on that. And I'm looking forward to you inviting Mishra and I over to San Francisco to have a little drink for your IPO party. We'd love a free sandwich and a glass of... Seriously-

Vicki Cheung:

I can't wait to-

Nic Jackson:

I genuinely-

Vicki Cheung:

...see you in person.

Nic Jackson:

Yeah, gosh! Oh, it's been so long, isn't it? I mean, I'm hoping [inaudible 00:26:33] going to go ahead in and the full [inaudible 00:26:37] we'll all be able to at least hang out for a little bit and just meet in person again, which has been quite, personally speaking, quite draining, not being able to do that. But last question, all good things have to come to an end and I'm super interested in knowing whether Vicki has a long-term plan. Where do you see yourself in 20 years?

Vicki Cheung:

Yeah. Well, I don't really have a plan per se because I feel 20 years ago I couldn't have predicted that I would be here. And so I just don't feel I can make very good predictions about that, but I do have general wishes. Yeah, so one is, I just know that in my core being, I like engineering and I like building and stuff. And I don't know, I think, I hope that in 20 years I'll have found meaningful ways to apply that skill and knowledge to some problem in the world that I really, really care about.

Vicki Cheung:

And yeah, I think another thing is 20 years is a long time. I think the sad thing is there are times when that becomes really hard to be a woman in tech. And I feel a lot of women in tech leave for different reasons. So 20 years is a long time. And unfortunately, we do see a lot of women leave in that timescale. So I guess what I hope is that I'll still be in tech and that I'll be part of making it a better place for other women to come.

Nic Jackson:

Well, I'm hoping that one morning I'm going to pick up my newspaper, I'm going to be sitting with my coffee and I'm going to be reading the headline which features you. And I can turn around and say, "Oh my gosh, it's Vicky. I knew it. She'd do it. She's done it. I know this lady. She's incredible." That'll make me super happy as well. So I'm keeping my fingers crossed for you, Vicky. You are going to do it. It's not a matter of if, it's a matter of when.

Vicki Cheung:

Well, thank you. Thank you.

Nic Jackson:

And on that, unfortunately we've got to go. And I've really enjoyed talking with Vicki always. And I really want to thank you so much for taking the time out, sharing your insights. I really think it's beneficial for folks to learn from others when they're trying to forge their own career paths because it's really not an easy thing to do. It's not really written down and very, very valuable insights you've prided on. Thank you again so much for that.

Vicki Cheung:

Yeah, thanks for having me again.