Wix.com’s Hila Fish – Doing Something With Your Data

Liran Haimovitch: Welcome to The Production First Mindset, a podcast where we discuss the world of building code from the lab, all the way to production. We explore the tactics, methodologies, and metrics used to drive real customer value by the engineering leaders actually doing it. I’m your host, Liran Haimovitch, CTO and Co-founder of Rookout.

Today, we have Hila Fish joining us. Hila is a Senior DevOps Engineer at Wix and believes that DevOps culture is what drives a company to perform at its best. When not working, Hila is a lead singer of a cover band and gives back to the community by co-organizing DevOps-related events. Thank you for joining us, and welcome to the show.

Hila Fish: Thanks. Thanks for having me.

Liran Haimovitch: How would you describe yourself?

Hila Fish: I usually say that in a professional frame, I would say that I’m a Senior DevOps Engineer for the last 15 years in the high-tech industry. I love tech. When I was a little girl, I carried floppy disks around and installed the stuff on my friends’ computers. I really love tech. I love computer games. I love everything related to tech. A friend asked me, “How can you sit 10 hours in front of a computer?” I said, “Because I like it.” 

Liran Haimovitch: Floppy disks are slow. You have to wait for them to copy stuff. 

Hila Fish: No, then – back then, yeah. Back then, I didn’t just sit around and do nothing. I did other stuff but tech evolved since then. But I do like just figuring things out. I really enjoy the troubleshooting aspect of my work. 

Liran Haimovitch: That’s why you’re a fan of open source. 

Hila Fish: Yeah, actually, I do. I think that open source is something that is very crucial for our tech industry. Because even if the big companies don’t drift in that way – but I think that they usually do because they understand that impacting the open-source world means impacting the tech world. Even if they don’t embrace it or don’t do proactive things to embrace it, I think that doing stuff on your own, if it’s contributing to open source, or when you have a task at work, don’t just go the obvious option and do it, but hey, maybe there’s an open-source project that you can integrate into your environment. And then it could be even better because it suits your use case better. 

But also on that note, I think that it also connects to the production-first mindset because I wouldn’t select a project to integrate into my environment if it’s not really beneficial for my environment. If there are only a few comments on the issues, people are not really engaged, and the ecosystem is not great, then it means that if the project will stop maintaining itself, then I will have issues in my environment. It’s not something that I would do, just because, “Hey, I want to do cool stuff and play with new, shiny things.”

Liran Haimovitch: What more can you share with us about yourself?

Hila Fish: Basically, my lead value that I take with me wherever I go is collaboration. I really like to collaborate with anyone. I remember the other day, a CTO from a company approached me on LinkedIn. She wanted to pitch me with her position at her company. But I said, “Sorry I can’t, but hey, let’s talk anyway.” Then we talked and I just gave her my two cents on which DevOps she should hire at the specific point in time that her company is in. I like to help, I like to give back whatever I can. If it’s by consulting someone, or giving a pep talk, whatever I can do, I like to help. About the cover band, I have like a gig next week. If someone wants to come, then just ping me on Facebook or LinkedIn or whatever. That’s mostly it.

Liran Haimovitch: You came onto the podcast because the term “the production-first mindset” resonated with you. Why is that?

Hila Fish: When I think about production-first mindset, I think about my way of contributing to a company. When I come to a company, I say, “Okay, I have my skillset but I don’t do it for nothing, I do it for the success of the company.” Production first mindset means for me how can I drive the company to a successful state in which she could reach more audience and eventually gain more success, more everything. When I think of production-first mindset, I think of what is the best thing to do at any point in time that will help the company succeed. If it conflicts with what I want to do, then I need to think what is most crucial for the company. 

I can give an example. When I just started my career in high tech, I was in 888, and I was in charge of a system that eventually I discovered only had like a few years or so. But then, I made a mistake and the system went down. I didn’t understand what it means because it was my first gig in the high tech. After that fact, after we fixed everything, my team leader took me outside, set me on a porch, and then he said, “Okay, I need to explain to you what production is.” I don’t recall the specifics of this talk, I just remember that there was a talk. I remember the feeling that I had like, “Okay, it’s important. I’m not here to get money, and that’s it, and then go home – no. Stuff matter.” Then this point of view made me just realize how important what I do for the company and then it stick with me for whatever years to come. 

Ten years later, if we go back, so ten years after that here at Wix, we had like monthly strategic planning. Then someone said, “Hey, we have the Presto, which is an open source for a – like data lake to search for another matter. We have Presto and then it runs on servers. Then they said, “Hey, we want maybe deployed on Kubernetes because…”

Liran Haimovitch: Kubernetes is cool. Everybody loves Kubernetes. 

Hila Fish: Exactly, right? They had like, I think one idea of why to deploy on Kubernetes. Then I stopped and said, “Okay, what does Hila wants? Hila wants to work on Kubernetes because it’s cool, it’s fun, it’s complicated, it’s complex. A lot of companies work in Kubernetes, and if I work and gain more confidence and more knowledge in Kubernetes, it will be best for me, for my career, because then I can work whatever I want to work.” 

But again, production-first mindset. Hila doesn’t only think about what Hila wants to do. Sorry for speaking like somebody or whatever. But I need to think about what is best for the product for the application. Then I told them, “Okay, guys. Before I know, Kubernetes is cool and it’s a buzzword, but we need to think it through. We need to see if it’s the best practice for the application. If Presto themselves say that we are production-ready declaration for Kubernetes. If it’s something that is suggested to do, maybe it’s not. Maybe for our skill, it’s not good. Maybe for other skills, it is good. We need to think it through and not just rush things and move, start even doing a POC just because, “Hey, it’s cool and we want to do it and we want to be there.” Production-first mindset means for me, we need to think what is best for the company, what is best for the application, and therefore, it will be best for everyone in the long term.

Liran Haimovitch: Because the kind of production is the real world, and you want to know that whatever you’re doing is making the impact you want in the real world rather than just shifting bits around or looking good on your resume.

Hila Fish: Yeah. Because eventually, every product that I’m in charge of means real users that have either satisfaction or…

Liran Haimovitch: Dissatisfaction. 

Hila Fish: Yeah. And I am a user as well. On other products, I am that user. I want to be able to help other users like me have the best experience possible.

Liran Haimovitch: What kind of values do you find most helpful to focus on when you’re trying to think production first?

Hila Fish: Okay. First of all, I think it’s a value not only for production-first mindset but also for a DevOps Engineer and its collaboration. Because if you don’t collaborate, then you wouldn’t reach the end gain. You wouldn’t gain the best outcome possible because you don’t know everything. Neither of us knows everything. If you just reach out to someone, consult about something, have more ideas, more points of view about something about how to do a task, about how to do an architecture or design architecture for an application, then a lot of great minds have more impact and more options to bring to the table rather than one. Collaboration, I think is the most important thing. I’m not sure professionalism is a value. 

Liran Haimovitch: Yeah, it’s a value. 

Hila Fish: Okay. Yeah, professionalism also matters and is most important for that kind of business. Because if you’re not professional, then you wouldn’t do things the best way possible. Like from end-to-end, if you want to make sure that everything is covered, then you would create a plan, you would do all sorts of things. There are a lot of things that are considered to be professional things to do. And if you wouldn’t do it, then you wouldn’t gain the best possible outcome.

Liran Haimovitch: Now, speaking of production first, I know one of the biggest challenges in software engineering in general, and I think one of the biggest challenges we’re hearing of today is kind of migrations. Whenever you have one technology in place, and then you’re trying to move to a different one for whatever reason, things often get messy. Things get complicated. No, it’s never easy to migrate between technologies.

Hila Fish: I agree. Because you want to prepare, you want to think everything through. But sometimes you have unexpected things that you need to deal with. For any plans that you can do, I think that you should do it. When you think things through, you cover everything that you could cover. When we have the unexpected, I think that’s where your professionalism and your experience come to play. Because the idea of not putting something on the spot, then don’t be nervous about it, and knowing who you should contact and who you should speak to in order to do this or that, I think this is something that comes from experience and dealing with projects along the way.

Liran Haimovitch: Now, before you joined Wix, you kind of did a pretty big migration project in your last role. Can you share with us a bit about that?

Hila Fish: Sure. There was an idea, and ultimately, implementation of a migration from Bitbucket to GitLab as the source control management tool. It was a very challenging project, to say the least. I did it on my own, so I was the only one that was basically assigned for this. I was able to get some help, but I took a decision not to include anyone, not because I didn’t want to but because I had a very tight schedule. I knew that if I needed to bring someone to speed, it will take more time to do so rather than to just do everything on my own. 

But even though I just – after I finished the project, I left a whole bunch of documentation that everyone could just read, divided into sections and all. Whatever I knew about GitLab after I finished the migration was there in the documentation. I didn’t leave anything behind because I wanted everyone to know what I know.

Liran Haimovitch: GitLab is a pretty big project, pretty huge platform. What features did you adopt out of it, at least initially?

Hila Fish: We decided at first to do only its first phase, to do the source control management. So, only import the repositories from Bitbucket to GitLab, and the mailed requests, permissions, login, and everything that’s related to their source control management. Only later on, they decided to also adapt the CI, but at first, it was only the source control management.

Liran Haimovitch: What are the key phases kind of moving your SCM? I’m wondering where would I begin if I were to do something like that.

Hila Fish: I think that the key phases for the Bitbucket and GitLab integration that I did was first to understand the nature of the migration. You need to understand if it is full-on migration, is it just the POC, what are the deadlines. If there aren’t any deadlines, let’s define ones. Because if you don’t know what you should do or what are the timelines to do something, then you can go a bit lost and don’t know how to continue from one point to another. 

After I understood the full extent of the migration, I built a plan for the migration. Everything that is related to architecture, networking, monitoring, permissions, access, also covers the training. Everything that I was responsible for the full extent of the migration, I covered in the plan, along with deadlines and everything. I would just – whenever I started implementation, I just went one by one and had a full vision of what I’m going to do. 

After that, I had a lot of research, a lot of reading material on GitLab documentation to know what is available for me, to know how do I build the architecture. Because we chose the self-hosted approach, so we deployed the GitLab on GKE. I had to check architecture from their point of view and see, “Okay, we should do this or that.” They had a limitation on which database to use. I had to put it in the documentation because I need to explain to other people why I chose this and not that. 

I had a lot of things to consider, and a lot of things to document along the way because I wanted people that after I left, we will get to it. After I left the company, I wanted people to be able to maintain it. I didn’t want anyone to feel like they are, “Oh my God, Hila left. I don’t know how to deal with it.” No, I wanted them to know every decision that I made along the way for the implementation, why I did it, and what they need to check in order to continue from this point on.

Liran Haimovitch: Just kind of wondering what are the challenges as you’re making this shift, as you’re deploying GitLab on a GKE. As you’re going through the phases, what was holding you back?

Hila Fish: I think there were two major challenges in the implementation. One is, let’s say other people or other teams that I had to collaborate with, and they have their own agenda and their own schedule to follow, and their own priority for the project. This was my first priority, but for them, it wasn’t. I needed to communicate with them in a way that will allow them to help me in my timeline, which is a very challenging thing to do because no one owes you anything and you should do whatever you want to do from their side. But I had my own schedule and I had the need to get their help. This was the first challenge. I had like two or three separate teams to coordinate with along the way. 

Another challenge was the actual pain points that I ran with doing the implementation. I think I opened 7 or 10 tickets for GitLab support, or something like that. We didn’t buy the support package, which means that I had to implement anything on my own, and they don’t really have to give me any priority for the support tickets that I open. Then I’m like, “Hmm, okay.” There was a lot of project management stuff that I needed to do in order to make sure that I’m progressing, even though I’m waiting for someone else to respond, either the GitLab support or the other teams in my company. I think that these were the major bottlenecks that I encountered.

Liran Haimovitch: Now, you mentioned that this was actually the last thing you did in your previous role. Then you left the company just after the project was launched and everything was migrated. What was it like living so shortly after finishing such a big project?

Hila Fish: First of all, I gave my early notice on the early stages of the project. Because I wanted them to decide if I was the one to do it or not because I didn’t want to decide for them. I wanted to make sure that I’m leaving on the best terms, and if they didn’t want me to do it, also no problem. But they decided that I should do it because I was a very organized person. They knew that something like that, like source control management, should be very organized. 

They said, “Sure, you should do it.” But I wanted to do it in the best way possible so that’s why I left a really monstrous documentation divided into sections, so you don’t really have to read everything. You can just dive into, “Oh, I have a certificate issue. Okay, here’s the certificate issue.” Documentation was the main name key for me. Also, I gave training to Dev, QA, my DevOps team members in order to just do the handoff the best way possible. I think, yeah, that’s mostly it. I just wanted to make sure that no one is dependent on me after I live, and that they know everything that I know after doing the migration.

Liran Haimovitch: Now, you’re working for Wix, and you’ve actually become a DevOps or SRE for part of the data engineering group. What does it mean to do data engineering?

Hila Fish: My take on it is that I usually – throughout my career, I was in the production teams. I was responsible for the infrastructure, for the main product of the company. Now, the focus shifted. Now, the main focus is data. Data, I think this is what means that engineering. It means that the main focus is not the product of the company, it’s data – data all over the place. How you deal with it and how you make the most out of it. I also had a talk with Josef Goldstein, who’s an R&D Manager at Wix, and I think you know him.

Liran Haimovitch: Yeah, a friend from the past. 

Hila Fish: He asked me to say hi. Josef also was able to explain it in a very clear manner, in a way that I also adopt this approach. He said that mostly, the development was divided into three pillars: database administration, system administration, and development. There are more but let’s simplify things. Then there was a need to – everything just tried to get shifted together, everything was just… 

Liran Haimovitch: What you’re saying is that data engineering is the full-stack engineering of databases. 

Hila Fish: Yeah, something like that. Because you need to make sure that the data is flowing as it should flow, that you derive conclusions from it, and you know what to do with it. It’s not just raw data coming in, you need to do something with it. This something is data engineering, which means that how you make the most value from the raw data.

Liran Haimovitch: What does it necessarily do for the data engineering team?

Hila Fish: Anything related to infrastructure for data. There are still Kubernetes, there are still Terraform, there are still a lot of things that you need to do for the infrastructure. But instead of the infrastructure being available or being accessible for the main product, it’s available and accessible for the data tools. So, Presto and Aerospike and a lot of things that they need to deal with the actual application. The Aerospike, Hadoop, Presto, and the configuration and everything, but I need to deal with the infrastructure and the server/Kubernetes side of things in order to make them work without thinking of anything infrastructure related.

Liran Haimovitch: Now, you’re an SRE, you’ve been one for quite a long while. Now, as you mentioned, quite early in your career, you had this production-first epiphany. Today, as you’re joining a new company, as you’re joining a new group, a new role, what do you focus on when you’re onboarding?

Hila Fish: Okay. I think that the most important thing when you’re onboarding, it’s being proactive. If you have a training plan, it’s great, but I had a training plan also and some things weren’t updated. I didn’t just say, “Okay, it’s not updated and then left it. No. I talked with whoever I needed to talk to, then I asked what are the differences, and then I updated the documentation for people after me to come and read it. 

Being productive is something that is very crucial in the onboarding process because you know what you need in order to succeed. You know which kind of data you need, information you need. If you’re a people person, then you need to get to know everyone, or if you need information, then you need to read everything. You know what is best for you in order to thrive in your role. Because of that, because you know yourself best, you should be proactive and navigate yourself in the best way that you know that will help you in your role.

Liran Haimovitch: Makes sense. Now, there is one question I pretty much ask all of my guests, one of my favorites. I know, we’ve all been software engineers for a long while and we’ve all seen our share of bugs. I also find interesting stories when I asked people to share about the most memorable bugs from their history.

Hila Fish: I don’t know if to portray it as a funny story, but it wasn’t funny. I had like a task to do some maintenance on the Jira server. I thought to myself, “Okay, yeah, it’s Jira, tickets.” No one will…

Liran Haimovitch: No one will care.

Hila Fish: Yeah, tickets. You can update your tickets afterward, everything is okay. It’s not asking for a call. I did my maintenance and I didn’t really – I think I updated only a few people and not everyone that I should have updated. Then eventually, I discovered that a lot of DevOps process is dependent on that Jira server, and then it created a bit of a mess. I was – people said, “Not good, not good.” 

I think my takeaway point from this is, if you’re not sure about something, then ask about it. Don’t just assume that, “Okay, it’s Jira, then it’s not really important. Maybe we are DevOps SREs, IT, whatever. We do sometimes stuff that is not as straightforward as it should be, like putting a very crucial process on a site server. Because “Hey, nobody touches the server so the utilization of that server is not high, so I can use it for other stuff.” No. No, don’t do it. Don’t just assume things. Check and update everyone, because it’s better to update a lot than nothing. And be careful, I think. 

Liran Haimovitch: Be careful. 

Hila Fish: Yeah. 

Liran Haimovitch: Hila, thank you very much for being here. 

Hila Fish: Thanks for having me.

Liran Haimovitch: That’s a wrap on another episode of The Production First Mindset. Please remember to like, subscribe, and share this podcast. Let us know what you think of the show, and reach out to me on LinkedIn or Twitter, @productionfirst. Thanks again for joining us.