Netsertive’s Herb Brittner – Going Beyond The Cookie Cutter

Liran Haimovitch: Welcome to The Production First Mindset, a podcast where we discuss the world of building code from the lab, all the way to production. We explore the tactics, methodologies, and metrics used to drive real customer value by the engineering leaders actually doing it. I’m your host, Liran Haimovitch, CTO and Co-founder of Rookout.

Today, we’re going to be discussing running Java and PHP side by side in modern cloud environments. With us is Herb Brittner, Senior Director of Technology at Netsertive. Thank you for joining us, and welcome to the show.

Herb Brittner: Thank you for having me.

Liran Haimovitch: So Herb, what can you tell us about yourself?

Herb Brittner: I am the Senior Director of Technology at Netsertive. I lead all software development, product, and IT teams. I have a background as a software engineer, and I’m very passionate about solving problems for customers.

Liran Haimovitch: Speaking of solving problems for customers, what kind of problems does Netsertive solve for its customers?

Herb Brittner: Netsertive is the digital marketing solution trusted by multi-location businesses across the country. We were founded in 2009, and have over 1500 retailers, franchises, auto dealers, and media companies that use our award-winning technology and team to run localized digital advertising, search engine optimization, and web solutions at scale.

Liran Haimovitch: What kind of challenges do multi-location businesses face when it comes to digital advertising?

Herb Brittner: There’s actually a lot. They face consistency; they face budgeting problems. They want to make sure that everybody has the same message across the network. And then when you’re creating the campaign, you want to be able to put your budget in certain places; some markets require more than others, and then at the end of the campaign, you want to be able to compare them. How did everybody do? Who are my high performers, my low performers? What can I do to help get everybody on the same path?

Liran Haimovitch: In a way, you’re kind of trying to run lots of mini-campaigns because some things are in common, but some things are differentiated between locations.

Herb Brittner: Yeah. The biggest challenge we have is bringing all that together, keeping the localized feel, and making sure everything’s not cookie-cutter along the way. 

Liran Haimovitch: How long has Netsertive been around? 

Herb Brittner: We started in 2009. I actually joined as a software engineer in 2012, so not quite at the beginning but I’m going on 10 years in October.

Liran Haimovitch: Cool. The company’s been around for a while, and I’m sure you have many generations of technology and many pieces of code written over the years. What does the service look like on the inside? 

Herb Brittner: Yeah. We are using a combination of mostly Java and PHP applications. Our newer websites are written in React. Everything’s running in Kubernetes in the cloud. We have a lot of database optimizations and database ETLs as well that run every day. We pull a lot of data into our data warehouse for reporting and analysis on a daily basis. There’s a lot of different stuff in there. It is from MySQL to PostgreSQL and some Snowflake as well.

Liran Haimovitch: What kind of data are you pulling into those databases, and where is the data coming from?

Herb Brittner: There are kind of two sides to the data in our databases. The first would be the configuration side, so setting up all the campaigns, all the ad groups, and all the ads, and keywords that go along with them. On the other side, there’s the performance data. That’s the reporting we get from people like Google, Microsoft, Facebook, and The Trade Desk. We pull that all together so that we can report on everything at different granularities to our customers. 

Liran Haimovitch: Then when you’re pulling all this data, and you’re customizing all of those reports, how much of the operation of the services of the customization of the ads are done by Netsertive of employees, versus how much of it is being done by the end customers themselves?

Herb Brittner: We have some of both. We have some self-serve customers in certain areas that can log in and make some changes to their campaigns themselves. Probably a majority of our customers are more of a white glove, where they work with our account managers to determine these media strategies, and our team will actually use the tool.

Liran Haimovitch: How do you differentiate the product offering and engineering processes between those two kinds of portals?

Herb Brittner: That kind of goes into a broader, how do you determine what you work on in general. At the end of the day, you have to align your product roadmap with your company’s goals. Those goals can change year to year, quarter to quarter maybe. Just keeping a pulse on what’s important to you and planning accordingly.

Liran Haimovitch: Speaking of which, what’s important to you, over the past few years, you’ve migrated most of your infrastructure into Kubernetes. In fact, you’re running quite a lot of PHP servers on Kubernetes. That’s fairly unique in the industry.

Herb Brittner: I suppose it is. PHP is what we started using back in the day. It was PHP for websites and APIs, and Java for automation applications that go around it. As we’ve grown, we’ve started moving more of those APIs into Java, different frameworks in Java, like Spring Boot. We still have some of the PHP APIs around. We aren’t really writing any new applications in PHP, but until we get all of them migrated, we’ll have them around. If there are small changes we need to make here and there, we’ll do that.

Liran Haimovitch: Over the past few years, what kind of changes and what kind of efforts have you made to make your PHP servers easier to maintain?

Herb Brittner: I think you hit the nail on the head earlier. We use Docker and Kubernetes to spin up and down test environments, deploy different versions of containers and applications, and APIs, essentially, using the microservice framework. Everything is kind of separated into their different duties. When you’re talking HTTP, API, CRUD operations, it doesn’t really make much difference if you’re using PHP or Java from the API layer. That definitely helped us out. Then deploying to production is the same, it’s just deploying that container to the production cluster.

Liran Haimovitch: Yeah, containers are great to kind of isolate away how the application is being built and packaged, and how technologies are being used. Today, you just have an image that runs an HTTP server, and it’s fairly opaque what’s happening on the inside. 

Herb Brittner: Yeah. That’s a great way for things to kind of work as they work. Yeah. We don’t have to upgrade everything in one fell swoop. We can do things as we get the capacity to do them and plan accordingly. It also lets developers kind of have a way to learn new things, try new technologies, and try new patterns over time as well. As long as the contract doesn’t change, yeah, it’s pretty opaque. That doesn’t make much difference. 

Liran Haimovitch: Have you faced any unique challenges trying to modernize those PHP environments? I mean, most Kubernetes users are using Java, Golang, Node, or ROS, and all of a sudden, you’re running PHP. Are there any sharp ends to that approach?

Herb Brittner: It’s been a little while since we actually did that work. But we invested in that with our DevOps team to streamline as much as possible around containerizing everything and getting things into Kubernetes. It’s been a few years and there are definitely some road bumps along the way and different things that you run into with PHP specifically. But once you get them all figured out and automated, today, it’s not even a question. Nobody asks that. It just works.

Liran Haimovitch: And yet, you’ve kind of ended up saying, “We’re going to move as much as possible to Java.” Kind of why not stick with PHP?

Herb Brittner: It’s actually a trend I’ve seen with developers in general. It’s hard to find anyone that wants to work with PHP these days. Those people exist, obviously, but more people want to do things like React, Node, and Java. I think people are just getting more vocal about their preferences. The market is in such a place where they can find whatever they want to work on, and whatever language they want to work in. Trying to just keep modernizing our stack is a good way for us to stay competitive in the market.

Liran Haimovitch: That’s a refreshingly honest answer. 

Herb Brittner: Yeah. 

Liran Haimovitch: Not saying that PHP is superior in any way in technologically. It’s not that it is or isn’t, but sometimes different factors come into the decision saying, “My employees are happier working for Java and my candidates are easier to hire.” It is a very honest answer. It’s refreshing. I guess, to a certain degree, it’s also interesting because – not necessarily because it’s the only funnel to developing PHP or Java, but people perceive it as more modern, more fun, more interesting, and the truth doesn’t matter that much.

Herb Brittner: Right. I mean, there are a few things. Like PHP has its quirks. Every language has its quirks. You’ve learned them and you get through it, and then you know what you need to do going forward. But yeah, there are definitely enhancements coming to PHP but it doesn’t seem as quick to evolve as other languages. When people learn languages, they generally start with Java or C#, maybe.

Liran Haimovitch: Why not go with C#? Now, why did you decide to go with Java and Spring for your new Web APIs?

Herb Brittner: We started with Java because we already did Java. There is a lot of familiarity with Java and object-oriented programming and the libraries did Spring Boot specifically. It simplifies and you can kind of create a template of an API right off the bat. I mean, when you’re creating a CRUD API, it’s pretty standard. You can Have your creates, you’re going to have your updates, you’re going to have your reads. It kind of gives you that out of the box, you just have to fill in the unique parts of what that API needs to handle and not worry about the boilerplate. Yeah, faster development and happier developers because they don’t have to focus on the mundane stuff. They’re focused on the business logic and the uniqueness of what they’re building. 

Liran Haimovitch: What kind of challenges did you find around spinning up those Java servers, running them in Kubernetes, and running them in production?

Herb Brittner: It was a lot of work just ahead of time to get things to a point where they could run successfully in Kubernetes. Historically, you have things on the Cron and certain apps run at different times, you try to think of different ways to do that. Kubernetes has its own Cron built-in. There are also other tools we use, like Argo for workflow management. We can say, “We’re going to run this application to pull data,” and then there are different stages that happen in the workflow after that. Figuring out all of those things and how to modernize things is the most time-consuming part of that.

Liran Haimovitch: I have to admit, I had some bad experiences with the Kubernetes Cron APIs. They’re fairly cumbersome, and we found that we didn’t have much observability into what was going on in there. How did you fare with the Cron API?

Herb Brittner: Exactly. That’s why we moved to that Argo workflow manager instead of the Kubernetes Cron’s. Because time zones actually were a problem. Whenever the time zones shifted and daylight savings took effect, now everything’s an hour off. Unless we updated all the different Helm Charts to fix that. You come up with your solution to fix it right away, and then you build this longer-term solution that hopefully resolves some of those problems.

Liran Haimovitch: How are you using Argo for workflow management, for ETL, for Cron’s? What’s your experience around that?

Herb Brittner: It’s actually really positive. I really liked that tool specifically. Because it gives you like the tree view. You can see the red or green status, you can see what stage of the workflow it’s on. It’s really similar to Airflow. I know that’s another common one that we investigated when we made that decision. We went with Argo at the end of the day because it was Kubernetes native. It’ll spin up and down containers and jobs and everything as it goes, and all of the observability was there. You could also see the logs and you can see all the different runs that happened today, scheduling things for every day, every hour, and every five minutes if you wanted to. It is very flexible.

Liran Haimovitch: What is it like troubleshooting? I mean, you’re running all those ETL jobs, you’re ingesting tons of data, I’m guessing you’re doing various aggregations and fancy calculations around the data. How do you monitor the data flows? How do you troubleshoot when things go wrong?

Herb Brittner: Argo is actually really helpful with that because if something fails, it’ll be read, and it puts some messages into our team chat. We’ll know immediately if something happened. And then we can dig into the logs after that to help resolve it. But knock on wood, because the Argos handling the workflow, and it has different stages of “I can go to the next step only if this succeeds,” or “If this fails, I can stop,” it actually makes things pretty easy. And it’s very reliable, in my opinion so far.

Liran Haimovitch: That’s great to hear. Now, I’m wondering, you’re running all these fancy new Java technologies, you’re using Kubernetes, you’re using Argo. You have Java Spring with APIs, you have Java ETL jobs. And then you have those PHP servers side by side, the old stuff nobody wants to work on, the legacy technology. What are the challenges? What are the unique elements of operating both of those generations side by side?

Herb Brittner: I think day to day, there are actually very few challenges. Because all of these technologies built around them make it more seamless. We’re not necessarily handling the PHP servers ourselves day to day, they spin up and down as needed. There’s all load balanced in Kubernetes, automatically. If we don’t need to make any changes to them, they just keep doing what they’re doing. It’s actually pretty minimal. It’s nice that the technology these days allows you to do that without as much overhead as you would have had in the past.

Liran Haimovitch: Yeah. I mean, sure. Getting the concept of pets versus cattle, right? As long as you have – your service is cattle, as long as they can be managed as a group, you can relegate most of that day-to-day to machines, the ship running, the day-to-day heavy lifting, and just kind of sail on focus on the business logic and the value on the future of the business.

Herb Brittner: Exactly. That’s the part that’s exciting, right? I mentioned earlier, I’m really passionate about solving problems for customers. I’m not really passionate about troubleshooting why a PHP server doesn’t want to turn on, right? Or it doesn’t want to work correctly. I mean, there are people that want to focus on that. But yeah, in the business framework, it just needs to work. And we need to be focusing on solving those problems, the right problems.

Liran Haimovitch: We’re definitely seeing that the rise of Platform as a Service, the rise of Kubernetes, of managed databases, are taking away much of the grind of day-to-day operations, of setting the IT infrastructure. So you can get Containers as a Service, you can get Database as a Service, you can get all the building blocks for application, so you can just focus on the code itself and what it’s going to do.

Herb Brittner: Yeah. I think that’s the biggest improvement we’ve made, even in the last couple of years, right? You don’t have to worry about how to get everything set up and running locally anymore. There are basically services to do almost everything. There’s a learning curve, that you have to get over to figure out how to do that, how to troubleshoot things, but it’s definitely nicer, and at least a faster development at the end of the day.

Liran Haimovitch: Yeah. Now, you’ve been at Netsertive for a decade now or you’re going to be at the end of the year. I have to ask, what’s the most memorable bug from your time with Netsertive?

Herb Brittner: My most memorable bug, probably likely didn’t make it to production. But when I first joined the team, right… 

Liran Haimovitch: That’s not fun. 

Herb Brittner: Well, I’m trying not to have any super memorable production bugs, but things go wrong and you have to fix them. But the one I remember the most actually didn’t make it to production. But it was a good learning experience for me and it’s something that I kind of kept with me forever. It’s working with databases and creating queries. You can really get into the mindset, “Oh, it’s easy.” You’re just selecting a star from a table, right” And then you get to more complex joins, and you get to more filtering, and all of those things, and you end up with a result set that’s got this Cartesian product in there where everything is multiplied by over many joins you pulled in, and all the numbers are wrong. 

Sometimes it can be hard to spot because some things look correct. But maybe for this customer, they had a couple of different records that ended up compiling on top of each other. Again, luckily, that wasn’t one that made it into a production environment, but it was good learning for me early on in my career. So, I’m always very careful writing queries after that. Because even simple queries can be deceivingly complex.

Liran Haimovitch: Yeah, queries or code, you can get them wrong. Especially if you don’t know what data is going to be run against, especially if it’s hard to predict data that’s going to end up in production that you’re going to be running the queries against.

Herb Brittner: Yeah, 100%. And then, of course, memorable bugs, maybe things that we weren’t in control of or things when your cloud provider has an outage, what do you do? How can you debug? How can you get back up and running as fast as you can? Or in some cases, maybe data from Google is delayed, and they have a bug where they can’t send reports out. Now, we have to figure out how to backfill things whenever they do get things back up and running, how do we notify all of our customers that there’s a delay there. And then even all of our automation after that, right? We’ll automate all of our bidding and budgeting based on that reporting, and if it doesn’t exist, we’ve got to wait.

Liran Haimovitch: Yeah, exactly. Herb, any last word for our audience?

Herb Brittner: I think the important thing about your technology stack is that you’re solving the right problem. The technology you use doesn’t necessarily matter as much as the outcomes. If you’re using a legacy technology, and you want to modernize it, you’ve got to come up with a plan and a migration strategy to get there, and ideally get there without trying to do everything at once. Trying to do things in stages is incredibly important. It’ll help you succeed and it’ll put a lot less stress on your team.

Liran Haimovitch: Makes perfect sense. It was great having you on the show. Thanks, everyone, for listening in. If you’re looking to get some PHP timing or Java time, you probably want to check out Netsertive.

Herb Brittner: Thank you. Happy to have you.

Liran Haimovitch: That’s a wrap on another episode of The Production First Mindset. Please remember to like, subscribe, and share this podcast. Let us know what you think of the show, and reach out to me on LinkedIn or Twitter, @productionfirst. Thanks again for joining us.