What Developers Should Know About DevOps With Ben Curtis | Rubber Duck Dev Show 115

[00:00:00] Speaker A: Hello. Welcome to the rubber duck Dev show. I'm Kristen. [00:00:04] Speaker B: I'm Ben. [00:00:06] Speaker A: And today we're going to talk about what developers should know about ops or operations or DevOps, if you like. But before we get into that, we usually do a weekend review. So I'll go ahead and start off the weekend review this week. So I actually just released my, my 300th episode of Scaling Postgres. So that was quite an accomplishment, doing six years of. Basically, it curates postgres content on a weekly basis. So it's a weekly show, and I've been doing it for six years. So I just had my 300th episode, so that was a big milestone. The other thing that's happening, there's a little bit of an API rant. So I have my own business, and Ben is a co founder as well. But it's a small company. I haven't delegated bookkeeping yet, so I still do the bookkeeping for the business in terms of accounting for things. And I kind of get a little. I'm not immediately up on current with everything, so when the new year comes, I have to do a backlog of an improvement bookkeeping to catch up. So I was a little bit behind. I haven't set up, like, a bank to Quickbooks transfer. I use quickbooks. Some people may say, well, that's your first problem. But I haven't set up a transfer to do that yet. And I didn't want to start it because it said something about 90 days behind. So I'm always at a question at this point, well, I'm a little bit further than 90 days behind. Will this work? I already know how to account for things now, so I've kind of still been doing things, mostly manually, but I said, this thing's got to have an API. So I looked into it, and I'm like, I'm looking for something that's just an API. You give me a security key and I can get my access to my data. I can create new expenses and all sorts of stuff. And then it goes through this whole process. I have to create a program, and then I have to create an app to use the API. And then they have, I think a store like Zoom has stores, and I'm like, I don't want to do that. But finally I was, like, looking, and it's like, what's the easiest thing to? And then I realized, oh, they do csv imports. I'm like, okay, sweet, I'll use that. But the problem is they only accept, like, three columns of data. And who you paid is not one of the required fields or it's not available. What insanity is this? So it works, but it's not great. And then there was a whole issue of trying to get my bank's PDF statements, get the raw data from that. I literally had to use a command line program to convert the pdfs into text because I couldn't use the pdfs from the bank, convert it into text, and then literally copy and paste the relevant data. It was insane. I eventually did it, but I would love just a very simple API to do. But that's. That's kind of the stuff that's happened for me this week. So, yeah, a little bit of a rant. How's your week this, Ben? [00:03:49] Speaker B: Yeah, it sounds like my week was a little less frustrating than. So I've been working in the past few weeks on a big change at Honey Badger, where we are moving our search index. So currently we're using elasticsearch to service all of our search requests that happen inside the app, and we're moving that over to Clickhouse. So we've been playing with clicklass lately, and it's fantastic. It's not a relational database, it's an analytical database. So it's great for doing queries against large sets of data. This past week, I've been wrapping that up and I just this morning polished off a pr and we've got it deployed to our staging environment. So now it's ready for review and hopefully within the next week or so that'll be deployed. So that ties off about a month of part time work. And I'm pretty excited to get that. [00:04:40] Speaker A: Out there, I bet. Yeah, it's that when it's almost at the finish line and there's not another 95% of the finish line, when you're literally. That's the great feeling. It's like, oh, we're super close now. That's cool. All right, so for those of you who don't know, this is Ben Curtis. He's the co founder of Honey Badger. And how would you describe Honey Badger? What's the elevator pitch for that? Yeah. [00:05:07] Speaker B: So honey Badger is a monitoring service for web developers. So we help you monitor the health of your application so you can provide an excellent customer support to your customers. So whether that be exception monitoring, uptime monitoring, cron monitoring, and pretty soon structured logging, any way you want to monitor application in production, we're here to help you for that. [00:05:26] Speaker A: That's great. And full disclosure, I have been a customer for years and I very much appreciate honey badgers. The service they provide. But before we kind of get into the main topic, which is what developers should know about DevOps also wanted to talk a little bit about your history, kind of like how you got started in tech or in programming. [00:05:51] Speaker B: Well, I think many people in our community, I'm primarily self taught when it comes to the tech stuff. I got my hands on a computer, the first one was a TRS 80 back in the day and started doing some basic programming where you copy the stuff out of the little, the magazine that came with. [00:06:08] Speaker A: Magazine you type in. [00:06:09] Speaker B: Yeah, totally. So that's when I got my start. I guess I was about, I don't know, seven or eight years old and then eventually graduated to the intel area and got an 80 88 on my desk and did some. [00:06:22] Speaker A: Wow, you're lucky. Holy cow. [00:06:25] Speaker B: Yeah, well I mean it was a family computer, right. [00:06:30] Speaker A: Still. [00:06:32] Speaker B: Taught myself some pascal and that was a lot of fun. I was in BBS scene and doing that and then eventually progressed and got introduced to Unix and Linux and Perl and PHP and decided I love doing web development. So this was back in the late ninety s and just gone from there and I've really enjoyed being able to just play with stuff. And then one day decided, you know what, I guess I can get a developer job. Graduated, got a job, moved out to the Seattle area and been working at web startups ever since. [00:07:08] Speaker A: So did you actually get a degree in computer science or something or was it unrelated? [00:07:16] Speaker B: No, actually I was kind of afraid actually going that when I was in college I didn't really actually want to be a developer for my job because I thought I really enjoy doing it. It's a great hobby and if I have that as a job then I think I might not like it anymore. Right. I think it might crush my spirit, as it were. And so I actually started off in English because I wanted to go into technical writing. And then I was like, no, I think I want to switch over to computer science. And turns out I don't like math. And when I got to the differential equations and stuff I was like, I'm out. And so I bailed and I went to the business program, I went to management information systems, so I went to Louisiana State University and at the time the IS program was in the business college and that's where all of the computer science dropouts went. Basically. They went over to is and that was a good choice for me. I mean I missed the algorithms and that sort of thing, all the stuff you learn in computer science program. But I really enjoyed the business side. And then right at the end of my college career, I'm like, you know what? I really enjoy development. They say you should do what you love. So I decided to throw all my plans out the window and get a developer job. So that worked out pretty well, actually. I think that was a good choice and being self taught technically, but having the college background on the business side I think was just a great combination for me. I think it's really worked out well. [00:08:43] Speaker A: Yeah, my degree, undergraduate was in biology. In retrospect, it would have been a better path given the path I've been on to do the business information systems or whatever degree it was in my school as well. Yeah. And I didn't even have it on my radar that that was a thing I knew. I always loved computers. I mean, I like biology too, but the computers after I had graduated kind of pulled me back over into doing it and things like my sister said, that makes more sense. After I made the transition, I'm like. [00:09:27] Speaker B: Okay, I think there's a lot of overlap there between the hard sciences and the technical stuff. You don't have to go all the way to computer science to really enjoy using that analytical part of your brain on tech stuff. [00:09:42] Speaker A: Yeah, I always loved how understanding how things worked. Like I was getting pc magazine and all these other words and looking like a book about memory on the, as you say, and how that works. And it was no different than understanding how a biological system work and worked and getting down to the nitty gritty, okay, this is DNA, this is the cell, this is how things. So yeah, it's just the same, but in terms of DevOps, because we're going to be talking about DevOps. How did you kind of get into that or start doing more DevOps related work? [00:10:28] Speaker B: Yeah, well, a large part of it was because I was broke as a kid. I didn't have a lot of cash and so I got into Linux really early. So I went to the Alabama School of Mathematics and Science, which was a brand new boarding school at the time and brand new at the time. And what we did there know it was like a college format, but it was the last couple years of high school and we got to experiment with a bunch of stuff that you didn't get in a typical high school in Alabama at the time. So we had a computer lab that was pretty well decked out. We had a blazing fast for the day, 9600 bits per second modem that was connected to a university that was hundreds of miles away. Right. Which eventually got upgraded to a t one line. So we were blown away with that, right. [00:11:19] Speaker A: Wow. [00:11:22] Speaker B: Yeah. At that time I was really interested in muds and talkers, which were a mud without the gaming component, right. And I really wanted to hack on that kind of stuff. And so the only really thing option at the time was you had to learn C and you had to do that on Unix and you had to learn sockets programming. And so I just dove in, right. So I got a Linux box set up and as I went through that process I started using Linux as my daily driver. And you have to learn at the time, you had to learn a whole bunch of system admin stuff just to be able to use your computer, right? You got to be able to connect up x windows and all this kind of stuff and learn how these things work. And so I really got into the whole what is a network and how does all this stuff work? As part of just my passion for I want to write a talker. So I did end up building that talker. It was a lot of fun. And I just kind of stuck with the system admin stuff. So everywhere I went since then I spent all my career building on open source, building on Linux, using things like Perl and Ruby and those open source supported kind of stuff. And so just like along with the development work, a lot of the system admin work just came along with it. And I think it's been really helpful to me to know what's the platform that I'm building on. How does the network work? I follow that all the way through. One time I had a business where I was doing consulting for people who were installing networks. So this is back in the Novell network days where you would go in and build this land out of coax and stuff. And so I got my Novell certifications and I was doing all that. So I just really love all things tech. So not just development, but all the hardware and networking stuff. I think that's how I got me here, where I really enjoy the blend of the systems and the development on top of the systems. [00:13:07] Speaker A: Yeah, I'm very similar in that, except I didn't do Linux related stuff. For me it was wanting to play pc games really early on. And in order to do that you had to really understand and precisely configure auto exec bat and config sys in Microsoft DOS. So I got to be a champ at that and really understanding it. So it was like in order for me to play games I had to really figure things out. And I've worked with Novell networks as well in certain cases. So, yeah, similar experience. That's interesting. What to you is DevOps? I think probably in history was probably called systems administration. But what do you consider? What is DevOps? [00:13:58] Speaker B: Yeah, I think there are definitely different opinions out there about what DevOps means, and I think there are a variety of valid opinions. For me, it really is the fusion of the operating, the system. Like you have the servers that are involved, the network, all the things that go along with that, load balancers and firewalls and all that. And you also have the code, the actual application that you want to run, that you want to deploy, that you want to put in front of customers. Right. Building an application is one thing, but it's not terribly useful if no one is actually using it. Right. So you have to get it in front of people and it has to be up. And so I think to me, DevOps is a combination of being able to do the development work, being able to build the things, but also being able to operate the things, being able to get them out there into the world, and being able to connect that all the way from start to finish. We might talk about this a bit more later, but I think one of the things that's come up with DHH and rails lately, the phrase the one person framework, right? And that's focused on that one person can build the app. Well, that's cool, and that's great. But also one person should be able to deploy the app, right. And maintain it. And so I think a lot of developers should be interested in the op side, like how does my thing run? And so to me, that's what DevOps is, just building the app, running the app. [00:15:26] Speaker A: Okay. And you had also mentioned something to me in the time we were speaking beforehand, something about site reliability. So what is that to you? And how is that different from DevOps? [00:15:38] Speaker B: Yeah. To me, I think that site reliability engineering, the SRE is a fancy label that Google came up with for basically the same idea. I mean, I think if you dive into it, you'll see there are differences and there are things that are specific to each name, but I think in general it's referencing the same idea. I think the SRE stuff goes more towards the larger organizations. We're going to monitor these particular metrics and we're going to have these particular things in place. And here's our checklist of things we need to have for our system to be considered a good production level kind of system. And that site reliability engineer is there to make sure those checkboxes are ticked, right. And that developers are meeting the spec that their team might put forth so that it will be well supported in production. Right. So I think the SRE comes a little bit more from the systems admin side and DevOps comes a little more from the dev side, but they're both, I think, playing in the same playground. [00:16:39] Speaker A: Okay, so what kind of, I guess, DevOps tools are you using? Like for example, if you could share what you're using at Honey Badger or what tool sets have you used even beforehand? [00:16:55] Speaker B: Yeah, I think I mentioned Linux before I started using that long, long time ago. But that's the foundation, right? I think for all of us who are probably, who are building rails, typically 99% of us are going to be building that on top of a Linux server, right? We might be using Mac for our desktop, but even that's got Unix under unless you're using. [00:17:14] Speaker A: Net. [00:17:18] Speaker B: Yeah, well again, that goes back to what I said. I was broke, right? So I never had the money to buy licenses and stuff from Microsoft. That's why I've been all 100% open source all my career. But I think regardless of what platform you're on, like you said, there's a bunch of tools that you're going to encounter. The ones that we use most often at honey Badger are terraform and ansible. So terraform is a great tool for working with all kinds of cloud and other providers on deploying infrastructure. So you can write a configuration file that defines what kind of server you want to have running, what kind of firewall you want to have in place, what kind of load balancer needs to be there and connects all these pieces together into a configuration that you can put under version control, you can collaborate on, you can make sure as it changes over time you're actually tracking those changes, which is much better than the alternative that we like to call click ops, where you go into the hosting console and you click the button to boot up that digitalocean droplet. Right? Or you click the button to add that firewall config. That's fine, and that's good for playing around and stuff. But when you have a system that you want to have repeatable, that you want to have reliable, that you want to be working with other people on, it's much much better to have all these decisions and these infrastructure pieces encoded into a configuration which we call infrastructure as code. Just like you have your app under git revision control, you have your system changes under revision control. And another tool that we use at honey Badger very often is ansible which there's a bit of overlap there. Both ansible and terraform can work with deploying instances and creating firewalls and stuff. But at honey Badger we use ansible more for the system level configuration. Like once you have an EC two instance running for example, what do you want to deploy on that thing? Like, well you need to have Ruby installed, do you need to have some logging software installed monitoring things, whatever you want to configure that particular instance to do. That's where Ansible comes in the picture. It installs those packages for you might install the database server, et cetera. So these two tools I think are great hand in hand to really make your deployment process repeatable and well documented. Like someone wonders how does it work? Well go look at the config, you can read through that and you can. [00:19:42] Speaker A: See it's literally, as you say, code. It's the code of your infrastructure. Yeah. Well just to let you know, I actually created a course on Ansible a number of years ago called Discover Ansible that I sold because for longest time I was basically using bash scripts to do the configuration. And then of course I was like I finally got to get my button gear and learn either chef Ansible or was the other one puppet. So I was like after looking at all of them, I really like the push ability of Ansible because I think chef was more a pull it pulled from a central repository whereas Ansible for me was like more simplistic. And hey, I just have the local configuration and I just push what the configuration to be when I want to as opposed to it randomly pull. I don't know, in my head I was like is it going to randomly pull things and I'm not going to be ready or something like that. I don't know. [00:20:47] Speaker B: And it's great. Ansible is great because if you only have one server, right, if you're not, oh exactly, your own thing, right? It's just like Capistrano was back in the day where just you push out the commands to the server, it does the things and then it's. [00:21:01] Speaker A: So I'm very familiar with Ansible and I know terraform is basically has come over the, I guess the DevOps scene and I don't know, taken over but it seems like Ansible is pushed down a little bit from that configuration perspective. But I guess call me the guy with a hammer banging the same type of nail. I use Ansible for my provisioning because I separate, I guess I'll call it configuration of the servers themselves and then provisioning actually bringing up a server, configuring the network. As you said, ansible can do all of that. I've kind of stuck with that. So I haven't dedicated time to learn terraform yet, although I'll probably have to at some point. Yeah. But it's hard because I am the guy running. Well, not quite a solo SaaS, but basically I have to be able to do almost everything from front end to back end to database to the systems administration. That full stack engineer, that's kind of what my head is. [00:22:13] Speaker B: Exactly. [00:22:14] Speaker A: I have to selectively choose what I choose to work on and know that's why I'm late to certain parties, like the move to terraform that's happened. So what was it like for you? Have you always been using ansible and did terraform? I guess. When did you start using terraform? And what convinced you to say, like, hey, we should start using this for provisioning versus something? [00:22:41] Speaker B: Yeah, yeah. I started with chef back in the day. That was awesome. I think Ezra introduced me to chef. I think many of us in the ruby and rails Communities got introduced to chef that way. And that was when I really got into the whole notion of being able to do this repeatable configuration is a good thing. And then over time, chef just got felt to me just too big. And like you said, you had to have this server that was running and pulling things down. It just felt like too much. And then Ansible was a great, I think, response to, I really, I used Ansible exclusively for quite a while. Really enjoyed that. But I think what really got me moved over to Terraform was two, one, this is kind of a minor thing, but it is a friction point. And that's with Ansible written in Python. You do have to have your Python environment stable, right? You do have times. I mean, we all know what it's like. You install the new version of Python and all of a sudden dependencies are broken. Then you got to do pip and you got same thing in the ruby world. That happens there too. So that's one little minor friction point. [00:23:44] Speaker A: I say a little, I say a little prayer every time. [00:23:48] Speaker B: Exactly. [00:23:49] Speaker A: Sometimes I'm very familiar with, yeah, I'm very familiar with Ruby, but stepping into the python world, I'm like, oh, please, everything, right, exactly. [00:23:59] Speaker B: So one benefit that terraform has is that it's written in go and so it compiles and it's going to work. Right. You don't have to worry about the dependency things. And then the other thing is when it really came down to when I was working with other people on the team, when we were multiple people working in this scenario, terraform is great because it keeps the local state of what's deployed inside local state file. Ansible doesn't do this, right, and some other systems do and some don't. It's just a matter of what's the preference. Ansible always runs the commands to make sure that whatever you want ends up happening, right. And then if it has checks in place. So let's say you want to install a new package and you tell Ansible, okay, make sure that the MySQl client package is installed so it'll go onto the server, it'll do the ssh bit, it'll check to see if that package is already installed and if it's the right version. And if not, if it is, okay, just skips that step, right? If it's not, then it'll install the package. But the way terraform works is actually when it does a thing, it keeps that state. Now it saves that. It's like, oh, I deployed this firewall and now I've got a record locally that I have deployed that firewall so I don't have to deploy it again. So the next time I run, I'll check and see, is that firewall still deployed? Does it still have the same configuration as what I expected when I saved it last time? Then I won't do anything, right, so terraform can save that state remotely, right? So they can put it up on s three. You can use Hashicorp's cloud thing and ansible has this too. Now with tower, you can save configuration. [00:25:32] Speaker A: And I haven't invested in that. I haven't invested time, not in terms of money, but time in order to understand exactly where that would work. [00:25:41] Speaker B: Yeah. So having that shared state is the other benefit of terraform. I can do a run, I can make sure some stuff is ready, and then I can go away and one of my teammates can come in, they can change something, they can do a run and they can see, oh, okay, here's the changes that are going to go into the environment. One of the benefits of having that state is terraform can tell you ahead of time, oh, well, I think I need to do this and that I'm going to destroy this thing or create that thing. Do you really want me to do that? Right. So to me that's like the main benefit of using terraform over ansible is that having that shared state and being able to see, okay, what's going to change, because, you know, that state? [00:26:16] Speaker A: Yeah. All right, that's interesting. And I guess we should say when you start using some of these tools, I mean they're a great benefit from keeping your systems in a repeatable and known state. So you want to be ultra cautious about trying to log into the server remotely and fiddling with anything. So I guess we should say be cautious of that because you may. Do you know what terraform does in that case? If it thinks the local state, excuse me, it stores the local state of what it believes exists on the cloud provider. But you blow away a database or you change the version of some package, do you know how it rectifies it or what it. Yeah, if there's manual intervention, what does it do? [00:27:11] Speaker B: Yeah, for the infrastructure stuff it will try and create it again because it disappeared. Right. You can refresh the state, you can tell terraform, go back and see what's out there and refresh the state so I can get it. I did some things you should know about to get it more. In reality, I don't ever use terraform with configuring stuff on a server I always use. [00:27:31] Speaker A: Right, right. Okay. Gotcha, gotcha, gotcha. [00:27:35] Speaker B: Yeah, it's definitely for that person who is used to be going onto the server and editing things. Yeah, it's definitely a bit of discipline to always do it in ansible, don't do it on the server. I mean ansible does have that diff command so you can actually see what it's going to do before it does it. But yeah, it's much easier if you just always get in the habit of oh, I'm going to do this change on ansible and then have it push it out for me. Otherwise we'll call. [00:28:04] Speaker A: So I'm assuming that honey badger runs on ruby on rails. [00:28:10] Speaker B: Oh yeah. [00:28:11] Speaker A: Is that correct? Okay, so how do you deploy new rails versions? I know you mentioned Capistrano, are you still using Capistrano or how are you doing your new version? Excuse me, you have a new pr you want to push up and deploy. How does that work now? [00:28:29] Speaker B: Yeah, well in the early days of honey Badger we did use Capistrano. We deployed, we had a server farm and we did that static ips and all that jazz. Nothing really changed a whole lot. Then we moved over to Amazon and when we did that, when you're hosting Ec two instances on Amazon, you have to be ready for those instances to disappear because Amazon is pretty ruthless about, hey, if something gets degraded we're going to shut it off and they try to give you a warning and be nice about it, but sometimes they don't. It's actually gotten better over the years. These instances tend to hang around longer than they used to, but still you have to have an automation in place to handle a server disappearing, which is not as much of a problem, let's say, in a hosted environment where you have your server that's actually in a know and you're leasing from. So what we did there was we would have what you call an Ami, a machine image that you create, and that machine image will get deployed to any new instance that boot. And so we would deploy via Capistrano like normal, and then we would have some code on the machine that would do the Capistrano things when it booted. So we grab the latest code and do all the things that Capistrano did. We had a script that did that for us. So periodically we would have a fresh image that we would create with the latest code. But over time codes would happen on master and you wouldn't have those on that image. And so it would just at boot pull that stuff down and do the things that capacitano did. These days we've moved away from EC two. Now we're doing ECS, which is AWS's way of running Docker containers. So now we've built all this infrastructure with terraform, we've got the load balancers, we've got the ECS cluster, and blah blah blah. And then we have GitHub actions, which when we do a push to master, it runs the tests, it compiles the assets, it puts the assets on s three for us. It builds a new docker container, a new image pushes that image to Amazon's repository and then tells code, deploy, go deploy that service again. And it pulls the latest image and runs the UCS tasks and does all that. So that's how we do it today. Basically we're building a new docker image every time we have a push to master or emerge to master. [00:30:48] Speaker A: And how is that orchestrated or what tool orchestrates that process to provision and deploy a new docker container. [00:30:59] Speaker B: So the daily day to day deployments, that's all done by code deploy and ecs. So once GitHub actions has run the tests, we have a workflow that does just the tests. We have a workflow that does standard and breakmen to check for vulnerabilities and things like that. Once all those tests pass, then that triggers another workflow that then does all the deployment steps. So it builds the Docker image, it runs the asset compilation, it then puts the image up on ECR. So it's all coordinated by GitHub actions and then Amazon's own deployment tool to do that. [00:31:41] Speaker A: All right. Yep. So back what I was saying, I kind of keep doing the same thing longer than most. I'm still using Capistrano for the majority of my deploys. Basically ansible prepares everything and then I just use ansible to deploy it. I have dipped my toes into containers. I haven't actually started using them for my infrastructure yet. I've used some. Was it LXC containers from Ubuntu? Are you familiar with those? So those have been super easy to deal with. And then I said, all right, I got to learn Docker because that's pretty much what everybody's else is using. I've used the LXC containers for years and it's pretty simple to use and I haven't had a problem with it. And then I started trying to use Docker and the level of complexity, I don't know what it is, at least for Ubuntu, the amount of packages and services I have to install, I'm like, why do I have to do it? [00:32:58] Speaker B: Yeah. For the longest time I was the same way. Not a fan of Docker, and I love Capistrano and just pushing stuff up and ansible and that sort of thing. But what really sold me on finally getting over the hump and converting to Docker was I mentioned we would build these machine images periodically that we were running. And the thing that got frustrating was new versions of Ruby. So my process was I'd have to boot up an instance, upgrade Ruby on that instance, deploy our app to it and make sure it all bundled properly. Right. And then burn a new image and then put that image into our config so that image would get deployed the next time we had auto scaling or whatever. And that whole rigmarole just got really old. I had it down, I had my checklist, but it was still manual process, something I had to do periodically. And Ruby comes out a new version every year, so you're guaranteed at least once a year doing this thing, and. [00:33:57] Speaker A: That'S if you don't do the version upgrade. [00:33:59] Speaker B: Yeah, right, exactly. And so I just got tired of it. Right. And so with the Docker version it's easier, right, because you do that locally, you bump your ruby version and your docker image, boom, push that to master and the pipeline takes care of it for you. [00:34:15] Speaker A: Yeah, I'm not denying that that is the way I should be going. It's just when I started attempting, it was like, why is this so complicated? Well, I think the problem is I had early exposure to the LXC containers and they were so easy to use and work with. And then they want me install all these. And then the cherry on top was after I installed Docker, all my LXC containers stopped working. And there was actually a networking change that was in there and someone had posted it's been present for years where I can't remember what it was, but there's some network change that Docker specifies that breaks the LXC containers on Ubuntu. So I'm like, wasn't that just great? Well, that's great to get rid of your competition, but okay, anyway, totally. [00:35:10] Speaker B: Yeah. [00:35:13] Speaker A: So I'm definitely going to eventually be moving to that path. And I'm really interested in the stuff that 37 signals is doing with what used to be called Mersk and is now Kamal. Are you familiar if you kept up with some of that? [00:35:31] Speaker B: A little bit, yeah. [00:35:33] Speaker A: So it looks like a simpler way to do working with containers in terms of deployment. So I'm definitely going to be looking into that. Presumably you don't need or is ECS essentially kubernetes? [00:35:57] Speaker B: Yeah, that's a good way of looking at it. Ecs, there's actually two flavors. I'm talking about the Fargate flavor of ECS. There's another one where ECs, where you actually run your own EC. Two instances. And then ECs is just the orchestrator for running your tasks on your instances. Typically when people are talking about ecs, they're not talking about that one, they're talking about the Fargate one, which is basically like they provide the compute resources, you just give them the container and they run it. Right. You don't have to worry about instances at much. It predates Kubernetes. And now Amazon has EKS, which is their hosted Kubernetes service for you. But yeah, you can think of ECS a lot like kubernetes being just the coordinator, the orchestrator for your. [00:36:44] Speaker A: So it's not, it's like Kubernetes, but it's not Kubernetes, it's a Kubernetes services. Eks, okay. Haven't even invested time into. [00:36:55] Speaker B: Mean there's, there's no shortage of services to buy from Amazon. [00:37:02] Speaker A: With regard to deployment, what do you think developers should be thinking about? Like if someone has a new app they're going to deploy, what is the path you would recommend it at this point, with the state of today, the. [00:37:20] Speaker B: Simplest possible path still, even ten years later, is still heroku, right, where you get push and your code is now magically on the Internet. And I think for someone who is just starting out doesn't know anything about anything, that's a great way to go, right. The problem comes when you actually want to scale that because it gets pretty pricey pretty quickly. And so what you see a lot of times is people get some traction, their app is great, it's got some scaling, and now they're like, wait, now I'm spending too much money. What can I look at, right? And I think for people who want to mimic that experience of just doing a git push, I think one great thing to look into is daku dokku. And it's basically, it's your own platform as a service, basically running on your own instances. So you poise to your Ubuntu instance and now you can get push. And that's fun. That's cool. I know some people using that. If you want to go to the next step where it's like, I'm actually running all my stuff and I know everything that's going on. I'm not just depending on over moving over to AWS with ECS. Now all I have to do is worry about having a docker container and it runs it for me. Now there's a whole world of services you have to be familiar with now with load balancers and all that jazz. So maybe a better intermediate step is, well, I want to spin up a droplet at Ubuntu, at Digitalocean, and I want to get my thing running there. I think Kamal is a great option for that. It gets you introduced to the world of Docker. You start building docker containers and dealing with all the learning that goes on there. But it's kind of like Capistrana was back in the day where you just ssh into your instance wherever it's running and it does the magic for you, right? It pulls the container and builds the things and it connects the web server for you. I think that's a great way to go these days for developers who are new to the idea of running their own app. [00:39:11] Speaker A: Okay, I guess let's shift for a little bit here and talk about the things the 37 signals was doing in terms of leaving the cloud, because we're talking about different cloud services and they're talking about leaving the cloud. And even I saw the hashtag that DHH David Hannemeyer Hansen started using was hashtag cloud exit. What was your thoughts on I'm assuming you've seen it and seen some of the discussions. So what are your thoughts on kind of that topic? [00:39:48] Speaker B: Well, I got to say back to my sysadmin days, I'm a big fan of having servers that I can put my hands on and I can see the blinking lights that harkens back to the. For me, I love that. And so I can understand that just the emotional appeal, the visceral appeal of that approach. But also I totally agree with David on when you're in Amazon, when you're in Google Cloud, you're going to be spending more. You're paying for the convenience of having elasticity of having. You can spin up this huge stack, right. And then you can tear it all down five minutes later. That comes with a price, right. We use that well at honey Badger because we have big spikes of traffic. Right. All of a sudden we might have 20 x the inbound volume that we had five minutes ago and so we need that elasticity. But if you don't, yes, I totally agree. You can save a heck of a lot of money if you just have a server running. Know, when we started honey Badger we were not at Amazon. We were at a provider that had colo facilities. And so it was kind of a mix between cloud and your own because it wasn't our own servers but we leased actual servers, actual hardware from them and it was in their rack. So it was cheap. It was like $75 a month for a nice fat server. And that got us a long way until we added a second one and a third one and proceeded that way. So I think if you really look at your use case, if you're in Amazon and you're like, you know what, we never scale. We always have just these two or three instances running and it's always consistent. Yeah, I think it's worth a look to see. Hey, maybe we can just get a couple of servers at a colo facility and save half the money. But at the other hand, if you're really not interested in playing with servers if you don't have the Ops team like 37 signals does have, then maybe it's good to just leave that in the hands of the AWS team or the Heroku team or just let someone handle that for you. I mean we have that same question to a lesser extent at Honey Badger with for example database like postgres. Like we used to run our own postgres because again I like running things and I know how to do it and so it's no big problem even with auto failover and stuff, I built the things that did that and it was hands off. But not everyone on the team at honey Badger knows how to do those things. And so if someone else is on call, they're kind of freaking out if the database goes away. Right. So over time we're like, you know what? It makes sense just to outsource that. Yes, we'll pay more to have a hosted postgres provider, which we love, by the way. Crunchy data shout out to them. Yes, it costs more, but I know that crunchy data is paying attention to making sure that database is okay. And nobody on my team has to worry about what happens if there's a failover. Right? Is that going to work? So that's the trade off. Like, it's fun to run your own servers, but that does come with some responsibilities. There are cost savings to have, but there's also time savings that you might lose. Right. So I think it's a very individual decision. [00:42:51] Speaker A: Yeah. As I was looking at it, we were thinking, yeah, when you have variable, because we talked about this in previous episodes of Throwbird Dev show, and when you have variable workloads, like you're saying suddenly you have huge traffic spikes and you can seamless, well, relatively seamlessly deploy additional servers to help that load. That makes a ton of sense to say in the cloud or at least go with a hybrid model. The other thing that in terms of making spend more efficiently is the more that least that I've observed, the more that you use their, I guess, value added services, I'll call it. So if you just look for commodity stuff, like just give me an EC two instance, that's going to be super cheap. If you want to say, hey, I want you to run the whole database for me and do everything related to that. That's going to be more expensive. Setting up a server and setting up, oh gosh, what is the load balancer? The name suddenly escaped me. Anyway. [00:44:04] Speaker B: Application load balancer. [00:44:05] Speaker A: Yeah, well, no, I'm saying an open source load balancer, you get an EC two instance, you install your own load balancer. It could be NgINX. But there's another. It's skipping my head for a moment. Anyway, that would be much cheaper than their elastic load balancer. [00:44:23] Speaker B: Right? [00:44:24] Speaker A: So I'm just saying the more services that you purchase, the more costs are going to go up. So if you can just stick with things that are more commodity that you could easily say, oh, well, I don't want to buy this from Amazon, I want to buy it from Google or I want to buy it from Microsoft or something that can help with costs as so. [00:44:48] Speaker B: But yeah, it's no joke that you get more performance out of your own hardware for a cheaper price than you would out of Amazon. [00:44:56] Speaker A: Yeah. The thing that I keep thinking of is that DHH said is that basically they're having I think 40% margins for the business. So clearly they're making a profit by the rates that they're charging. Exactly. For providing the services. But again, I totally agree with what you're saying. Like when you have dynamic workloads or if you just don't have enough bodies to provide redundancy to make sure that your system stay up. Like you're saying, you're only a person who knows how this database is set up. Well that's a big risk and being able to outsource it to someone else is very beneficial. So I totally agree with that. So also wanted to kind of shift and talk a little bit about here monitoring and observability. So you've got your thing deployed. How do you monitor the state of. This is an interesting question because honey badger does in a certain sense monitor states of applications and reporting and logging of things, exceptions. So how do you monitor honey Badger? [00:46:15] Speaker B: We have a whole lot of monitoring in place. We use internally a number of things. One, we use a lot of Cloudwatch. So since we have all of our stuff hosted at Amazon, it's really the default really to have logs show up in Cloudwatch and metrics show up in Cloudwatch. So we use a lot of Cloudwatch alarms and so we have terraform used to set those up. So for example, let's say we're running karafka. That's like sidekick, but it's using Kafka as the transport rather than redis. So if you have a Kafka task running an ECS, it's going to send out some logs and it's going to take up some memory, it's going to take up some cpu on that ECS task. And so we have a terraform config that tells this task like go and use this image and put these environment variables in the environment and oh by the way, create a cloud watch alarm that watches the cpu and watches the memory and then sends alerts over to this SNS topic which then forwards over to our slack channel. So that's a lot of what we have at honey Badger. Since we are building right now a structured logging tool. We're also working on pulling that stuff into our own internal instance of that structured logging tool so that we can try our own dog food. But predominantly, I think Cloudwatch logs and metrics and alarms is what we use to monitor. [00:47:36] Speaker A: Okay, so like honey badger. So I'm using Cloudwatch logs as well for monitoring disk space and cpu usage and a number of other things. And going back to what I was saying is that when you buy the commodity parts, like just get an EC two instance is cheap, when you ask them to do extra things, it starts getting more expensive. That's actually one of the higher cost things, the monitoring that I'm paying for in my AWS bill. So I've been kind of know I should really look into maybe setting up prometheus grafana to monitor. Know, I'm just been thinking different ways I could potentially monitor that. It would take an investment of my time, but it could save some money by doing that. But I'm kind of asking this because I was hearing some of the things that you were mentioning about honey Badger and the insights product. So could you explain kind of what that can offer and not saying it's a replaced for cloud metrics or whatever, but could you explain in your own words kind of what the insights. Do you call it a product? Do you call it a module? What do you call it for honeybee? [00:49:02] Speaker B: Yeah, we're kind of unsettled on what exactly we're calling it if it's a product or a module. But yeah, you can think of it as a module because it's going to be inside of honey Badger. Right. It's not something that's completely separate. And that's one of the selling points of it is we do want to bring that cloud watch insights because in Amazon there's Cloudwatch as a product and then that product has a bunch of features like Cloudwatch metrics and Cloudwatch logs and there's even cloud watch logs insights. So we're not 100% original in this name. We do want to take some of that functionality and bring it to our honey badger customers in the form of insights as the module or product, however you want to call it. And that's really a way to send in your structured logs. So let's say you have a rails app that's spitting out logs for each request, like the duration and the path and the parameters that were used. [00:49:53] Speaker A: Just basically your standard rails like production log. [00:49:56] Speaker B: Exactly, exactly. Now if you choose to put that into a JSON format using like the log rage gym or the semantic logger gym, if you put that out in a structured way and then send that over to honey badger insights. Now you can analyze, query that data. So now you can see, oh, how many requests am I doing? Or what is the average duration, or give me the duration over time, right. And you can chart out what's the time series data for that. So that's a way to, you don't have to set up that additional monitoring, right, because it's already there in your logs and maybe you have user sign ups, right. And so just take account of the number of times this particular route gets hit in my application based on the logs that I'm sending over. And now you've got activity stream happening that you can see a chart of your activity over time for your application. So our goal with insight is really to kind of bring that richness experience that you'd be paying for quite a bit in Cloudwatch or even datadog or splunk or these other big guys that charge a heck of a lot of money for doing these kinds of things and bringing that kind of experience to developers who are using honey badger at a price point that's a bit more affordable and with an experience that's not quite as complicated as sitting in the cockpit of AWS console. Right. But is more straightforward, we're going to give you a query language that is easier to use. So that's our goal. [00:51:19] Speaker A: I still am not satisfied with my configuration options I've set up for the metrics in cloud watch. I'm just like, why is it reporting like this? I have no idea, but I'm too busy to contact support and say, help me figure this out. So insights is designed specifically to consume logs. [00:51:50] Speaker B: Okay. [00:51:53] Speaker A: Sorry, go ahead. [00:51:55] Speaker B: I said, we have a good story today for monitoring exceptions and for monitoring the uptime like we do an external ping to make sure your site is still up and watching your. So one gap that we have in our product today is that we don't know operationally what's happening with your application during normal load, right, when you're not having errors. And that's what insights is for, is like how's the performance? How long is the view rendering taking, how long is the entire duration of the request taking and how many requests am I getting that sort of thing to help monitor that. [00:52:28] Speaker A: So I've used different logging products where you send them your logs and then you can peruse the logs. Is there any vision like in the future of applying certain metrics like extracting, I'll say pre extracting knowledge from the logs to give indications of something like requests per minute or some type of metric like that, yeah. [00:53:03] Speaker B: So our goal was to be able to provide you the flexibility to set that up so you could say, what kind of metrics am I interested in? So maybe there's a minute so I. [00:53:12] Speaker A: Could make a dashboard or something of that nature. [00:53:15] Speaker B: Exactly. You write a query that says, okay, ask my log, give me all the requests, and now give me account of the requests and group that by minute. Okay, now I've got a time series. And now, okay, push the button that says chart that for me. [00:53:28] Speaker A: And. [00:53:28] Speaker B: Okay, it looks good. Push the button says add that to a dashboard. And so now I always have my request per minute chart right there. Yeah, that's exactly what we built with insights. [00:53:37] Speaker A: Okay. [00:53:38] Speaker B: And then hopefully not here today, but hopefully soon. That's a pre made dashboard ready for you. We say, oh, we see you're sending in rails application data. We know you're going to want requests in a minute. Here's a chart that does it. There you go. [00:53:55] Speaker A: What about system based metrics? And I'm thinking about like we were using the example of cpu usage. Is there any way of tracking that within insights? [00:54:11] Speaker B: So whatever you can put into a log, we can do metrics based on that. Eventually we want to have an agent that you can install that will then report. [00:54:23] Speaker A: Right, okay. [00:54:24] Speaker B: But today we're using vector a lot. So vector is a tool that can do a lot of taking in data from one place and shoveling it to another place. And so you can, for example, if you're running collect d on your servers and it's collecting all your cpu stats and memory stats and stuff like that, you can actually ingest that into vector running locally as an agent. And then vector can package that up as a log and send that to insights. And then we can say, okay, here's the cpu stat for that minute. Here's the memory stat for that minute. And then we can chart some metrics based on those metrics as logs basically. [00:54:58] Speaker A: Okay. So it's just basically at this point it's up to the person using the insights account to say, all right, what do I want to measure in my server? I want cpu, I want disk space. Whatever metric you want to use, you put that in a log format and then send it up to insights to get information. Okay. [00:55:23] Speaker B: Yeah. And really, I think our philosophy really has been with honey Badger is we want to give developers every tool they need to make sure their application is healthy in production, that their customers are having a good experience. And insights is just the latest step in that. It's like, okay, I should pay attention to what my server is doing. I should pay attention to what requests are coming in on a normal basis. And just having that awareness as a developer, I think brings you into this realm of DevOps, this grand rainbow land of knowing how your application is performing in production is really the whole goal. [00:56:00] Speaker A: All right, now I should mention as I'm sitting here thinking about the discussion we're having. This is not sponsored at no, I just thought it would be great to get Ben on the show because I've been hearing him talking on his podcast about insights. I was like, oh. And I was thinking different ways I could use it. So this is really just an interest for me. And I invited him to come on. There's no sponsorship at all, even though it may think that way based on the discussions is going. But I'm generally interested in seeing what the product can do. So before we wrap up, kind of what would you say in terms of closing statements, why is it a good idea for developers to know more about operations? [00:56:46] Speaker B: Yeah, back a long time ago when I was early in the web Dev days, I thought it's good to know the basics of programming, like what happens down a few layers. It's good to know the networking stack. It's good to know why c is the way it is. I think my opinion on that has matured over time. I think at the time I was like, it's good to have the fundamentals right today. I think that's probably less important that you get into those details. But I think the idea of having a notion of what the fundamentals are, why is running n plus one queries a bad idea? Why is taking multiple API requests inside of a request without putting that in the background? Why is that a bad idea? Just these basics of understanding the environment that you're in and so we can understand those kinds of things as developers. And I think understanding how is my application performing is helpful for making sure you deliver a great experience to your customers. We feel strongly like we want everyone who uses our product to be happy using it. Right? We don't want them to experience bugs, we don't want them to experience slowdowns. And the only way we can know that they're not encountering bugs and not encountering slowdowns is if we have some monitoring there to tell us what's going on. Otherwise we're just flying blind. And we don't want to fly blind. We want to provide a great experience and so we want to know what that experience is. And I think that's really our whole driving force at Honey Badger is we want to be the tool that allows you, as a web developer to say, I'm providing a good experience to my customers because I know they're not encountering bugs left, right, and center. I know they're not encountering long page load times because I'm seeing that in the charts and in the alerts that I'm not getting right from honey Badger. So that's really our goal is we believe very much in serving our customers at the best we can and helping our customers serve their customers in the same way. [00:58:45] Speaker A: Yeah. And I can attest sometimes I get something coming through on honey Badger and it's any other log alerting exception tool, and I can respond immediately. I can see, okay, what was the user that this happened to? I can reach out to the user and say, hey, we just got notified that you encountered an error. We were looking at it, and it's because of XYZ reason. And some people are super amazed. It's like, holy cow, how did he know that, right? That this happened? But they're very appreciative, even though some of them may think it's a little bit creepy. [00:59:21] Speaker B: But anyway, yeah, I don't really think it matters what tool you use. I mean, there are so many different options out there, and we have plenty of competitors in the space and there are plenty of options. But I think really, the message that I would like to share is, like, use something, right? Find a way to make sure that you're monitoring your stuff so that you're providing a great experience as a developer. It sucks if no one's using your product. Right? That is the worst feeling in the world. Right? You build this thing and nobody uses it. That's terrible. But I think also what sucks is that when you have this thing and people are using it, but it's broken for them. And I think that's the message that I like to share, is like, hey, you should care that the people using your product, you should care what kind of experience they're having with it. [01:00:11] Speaker A: All right, so thanks, Ben. This has been great. So where can people find out more about the work that you're doing? [01:00:19] Speaker B: So I'm on various social networks. I'm on Twitter as stimpy. S-T-Y-M-P-Y. I'm on Mastodon. I'm Ben Curtis at hackyderm. But if you go to Honeybadger IO IO and you hit the about us page, you will see all my contact can if you forget any of the things I just said. [01:00:37] Speaker A: All right, sounds great. Well, I hope you guys enjoyed that. Please be sure to like and subscribe if you like this content. And be sure to visit rubberdevshow.com where you can find the links for all the content that we discussed here today, as well as go ahead and sign up for the email list there, because we send out every week the notification of the new shows as well as other content we're producing. Because we're taking the long form shows and putting them into shorter clips of about ten or 15 minutes videos so you can enjoy those as well. I hope everyone is going to have a great week. And until next time, happy coding. [01:01:19] Speaker B: Happy coding. Bye.