Data Engineering in Logistics

Tech Backstage / Ava Connolly Interview Transcript

Eddie Hudson(00:16):

Hey, y’all. Happy New Year. Carlos was still in the picture, but that's what happens when you're doing live podcasts. So, happy New Year to everybody. Welcome to the first episode of Tech Backstage of 2023, and I am joined today by Ava Conolly, a data engineer at Arrive Logistics. Ava, how you doing today?

Ava Connolly(00:35):

Doing great. Thank you so much for having me today.

Eddie Hudson(00:37):

Yeah, thanks for being on the show, and thanks for taking a little bit of time out of what I'm assuming is a very busy schedule coming back from the holidays to sit down and talk with me about all things data engineering and Arrive.

Ava Connolly(00:51):

Looking forward to it.

Eddie Hudson(00:53):

Yeah. Before we dive too much into the topic that we have today, I'd love to just introduce you to the audience and talk a little bit about you know, your backstory. I, when I was reading through your LinkedIn profile, I thought it was fairly interesting and I think that it'd be fun to share with everybody on the show. So, I'd love to hear a little bit about your origin story and how you ended up becoming a data engineer, and then also how you came to Arrive.

Ava Connolly(01:20):

Of course. So I went to college in the greater Boston area where I studied primarily biology and computer science. And during my senior year of college, I worked full-time as a data engineering intern at a health tech startup. And I've been at Arrive for about a year and a half as a data engineer. So that being said, all of my experience has been in hypergrowth startups, you know, in 2020 in health tech during the height of the pandemic to supply chain logistics where it's always busy. There's always something to learn.

Eddie Hudson(01:53):

Yeah, I love that. What, when you were going to school in Boston what was the thing that drew you to Data engineering?

Ava Connolly(02:02):

I think I've always wanted to help as many people as possible, and data engineering allows for that because of the nature of the positioning of the role in any given company. So, you know, I can complete a ticket and by executing that body of work, I can help a lot of downstream users in multiple different departments, which I think is really exciting.

Eddie Hudson(02:26):

Yeah. Absolutely, I agree. I actually started out more in like the JavaScript/React world, but I think I very quickly came to realize just how impactful data can be. Mm-Hmm. <Affirmative>. I think it's something that, it's almost like I don't wanna say the underdog cause it's definitely not an underdog, but it's kind of the magic behind the scenes, right? It's what really drives a lot of decision-making processes. We use a lot of data-driven processes in our business as well, but it's one of those things that I kind of, I fell in love with being a software engineer. Cause I really like that I could like change something in the code and see it over here on screen immediately. But I love your reference there to being able to execute a ticket and then that data and, and that, that body of work becoming immediately relevant to what the company does for its clients. I love that. Talk to me a little bit about your journey from Boston to Arrive, because I noticed that you did a few other things. I saw that you worked for the military a little bit, doing some UI/UX research and then in some other roles. What was that, what was that journey like for you? You know, from university to where you're at right now?

Ava Connolly(03:39):

Yeah, so I did not always believe I wanted to be in logistics specifically. You know, I primarily studied biology, right? And then added computer science later. So it has been quite the journey. And like many people in logistics and supply chain, I essentially just ended up here. I'm so glad I did, but it was not a linear path. So I, you know, I started out just kind of figuring my way, right? Like, I knew that I wanted to be in tech, so I started in UI/UX that gave me a really good foundation and I was an intern. That was during my sophomore year of college. So, I worked there at the US Department of State for about nine months, and I was doing education technology research. And so I really got to focus in on users and what they really need out of software products. And I think that really shaped my understanding of software and being, you know, a user first engineer, which is kind of how I identify today.

Eddie Hudson(04:44):

Yeah. I also love the fact that you were doing UI/UX research because I think one area where data is extremely important is on the user experience side, right? We use that a lot to drive decisions to make sure that the user journey is meaningful and that somebody's able to complete what they set out to do within an application, right? And if you don't have data telling you how users interact or use the application, you know, there's all those memos out there about people breaking stuff in QA, but really the first time when you get it out in the wild without any sort of user experience research or any sort of data behind it, it's hard for you to just guess what people want to do in an application, right? So, I think that's another like, really important area where I think data sometimes is overlooked because without that data you can't, you can't make a meaningful impact or meaningful changes to a user interface or, you know, a user experience in an application, right?

Ava Connolly(05:42):

Absolutely. And here at Arrive in data engineering, we are responsible for designing, building, and maintaining that data platform infrastructure, which enables our stakeholders and data consumers being, you know, our data scientists, analytics engineers, our department analysts, you know, business intelligence, et cetera. So data engineering has a really close relationship with our downstream users and, you know, it's my job to make sure that they are getting highly useful data that can help facilitate all of our goals and ambitions in the company.

Eddie Hudson(06:16):

Yeah. Talk to me, I want to talk a little bit too, I guess this can be a segue into the actual topic, right? Because you said that one, I kind of caught onto one thing you weren't sure that you wanted to work in logistics at least initially, right?

Ava Connolly(06:32):

No, that I was going to end up here. I just ended up here

Eddie Hudson(06:36):

<Laugh>. Yeah. But I, I mean, it, it's such an interesting thing to think about because logistics, I know a lot of other people don't, like, don't really think about this stuff on a daily basis, but logistics is really kind of at the heart of what drives our entire country, right? Absolutely. if you look at even going back to when the highways and freeway system was created, I think it was back with Eisenhower, right? Like the whole purpose behind that was to facilitate the transportation of goods and to, and to open up you know, the country, especially the west. You know, because it wasn't quite as far along as the East coast, but logistics really lies at the heart of everything that drives our country, especially in the trucking industry. And interested to understand a little bit about after you got into logistics tell me a little bit about how you felt, you know, coming to a, to a company like arrive and, and actually dedicating your time, you know, full-time to working in logistics.

Ava Connolly(07:37):

Yeah, I, I totally agree with the importance of logistics. I mean, one thing that really piqued my interests about it as an industry is that, you know, everything I'm looking at in front of me was delivered by a trucker. And to what you mentioned, hardly anyone thinks about that, but it's such a critical industry and it provides so many really awesome opportunities for data and analytics that are just really important to our infrastructure as a, you know, as a global industry. So I think it's really interesting. But to answer your question you know, actually, could you repeat your question, <laugh>?

Eddie Hudson(08:13):

Sorry. Yeah, I was just, I was kind of interested to understand a little bit about how you, you know, how you feel working at logistics now, you know, even though it wasn't something that you were necessarily aiming towards mm-hmm. <Affirmative>. tell me a little bit about your experience working in logistics as a data engineer.

Ava Connolly(08:28):

For sure. So, you know, I only have experience in logistics that arrive, so that's what I can speak to. But what I love about Arrive is our co-pilot approach, so, which I think is different from a lot of tech companies in that we don't automate every single thing, you know, just because we can automate something we don't. And I think, you know, ultimately, like sales and bis dev and non-technical people are paying my salary and I'm making products and goods that enable them, and business relationships and the relationships with our shippers and carriers are so important that that is the utmost priority for us. And me as a tech person, I'm just creating tools to help them and build their relationships and allow them to focus on what matters most, which is, you know, driving better business relationships. So I think that's, that dynamic is really interesting in that, you know, I'm really working for them. I'm truly building products that help them, which isn't common, I think, with a lot of tech and software.

Eddie Hudson(09:37):

Yeah. I, I think, I think that's actually a really holistic approach to, to not only logistics, but also to software and, and to engineering in general. I was reading a blog the other day, I forgot where it was coming from, but anyways, it was talking about how the SaaS model is broken because a lot of these companies out there are, they're talking a lot about solving productivity, right? So it's always about more output or always about being more productive. But it, what's essentially happened is kind of just this over-engineering and to the point where, you know, businesses will sign up for an application because it promises to help them reduce, you know, in this particular instance, oh, you know, reduce how long it takes the truck to get from San Antonio to Dallas. It's like, yeah, but there's kind of a floor to that, right?

Eddie Hudson(10:27):

But they're, they're still promising these, these, these productivity increases and companies end up going out and investing, you know, millions of dollars in these tools to get minimal return on investment in, in some cases. I'm not saying in all cases. So I love the approach that you're talking about. It's like, Hey, we don't need to over-engineer this, right? It's our job as engineers, not necessarily to just do it with code or, you know, with data or, or whatever you're working on specifically as an engineer. It's not our job to just do it because we can a certain way, it's our job to understand the entire problem and what's the best way to approach this. Like, you don't need to write a migration script if you only need to move two records, right? Like, <laugh>, it's completely, like, completely a waste of time. But, you know, the, I think that's a, I think that's a really important holistic approach to, to how you guys, or how all of us handle solving problems. What'd you guys, you called it the co-pilot program?

Ava Connolly(11:24):

Yeah, it's part of our messaging for our new release of our technology platform arrive now.

Eddie Hudson(11:31):

Awesome. I love that. Well, let's dive into, I wanna spend a little bit of time to, I know we've talked a lot about like logistics and just getting started, but I, I'd love to dive in a little bit around the topic. Today we're talking about data engineering and logistics, but we're talking about careers in, in data engineering. So I guess first up, I'd really love to talk about just, I guess specifically developing your career, right? I think there's a lot of things that, especially me being from a non-traditional background and, and kind of being self-taught there, there's a lot of things that I just kinda like had to go out and figure out on my own. But I'd love to talk about maybe some, or I guess talk, you know, about developing your career as a data engineer, right? What's that process look like? You know, what, what's kind of, what are some of the first steps that you can take? And it, it's, it's interesting because there's so many different ways you can get into it, right? There's the traditional path computer sciences, there's self-taught, there's a lot of like data boot camps out there. But talk to me a little bit about like, kind of first steps for getting started into data engineering.

Ava Connolly(12:44):

Yeah, I think that's a great question because data engineering really isn't a career that you initially think of. It's kind of a branch off of software engineering. Yeah. you know, I would say, well, let's be more specific. <Laugh>. Is this person, like in what stage are they at? Like, have they been a software engineer? Have they not?

Eddie Hudson(13:07):

I, you know, personally having worked with quite a few software engineers, I think it's really important to understand like, let's, let's say they're just starting to come outta college, right? Because one of the things for me is that when, especially when we're hiring engineers, for example, a computer science degree does not necessarily equate to a great engineer or a great data engineer, right? Yes. I think computer science degrees, this is my humble opinion, and for everybody that has a computer science degree, I'm not trying to be negative here, but I look at it as the theory, right? Like it's, it's, it's basically the understanding of, of how it all works, right? But to really be a great engineer, to be a great data engineer, you need to understand how to solve problems mm-hmm. <Affirmative>, and you need to be able to, I, I think again, look at the problem holistically and come up with a solution that that gets you to, to the end. And hopefully it's not spaghetti code, right? But let's talk about it from, from that perspective, like coming outta college and, and you're looking to get started in, in, in the field,

Ava Connolly(14:12):

Right? So I like to look at my career as a marble stone block. So I came from the CS degree, more traditional background, and, you know, that formal education for me was able to kind of chip off like four big blocks, right? But I have this vision of, you know, my ideal excellent engineer. And you know, it's up to me as an individual to spend time in carving those details of that marble block. You know, there's, there's the skills you were mentioning, right? Of CS or boot camp, what have you, that get you like 30% of the way there. But really your learning starts when you graduate or when you enter into enterprise software development. You know, it takes a lot of blood, sweat and tears essentially to carve out those details. And sometimes it's not pretty and you're not gonna have someone holding your hand doing it for you. So, you know, I would say just be prepared <laugh> to like part out those details.

Eddie Hudson(15:20):

You said there was some, some in, in, maybe we can go back to this, but you said that there's some, some specific things that you look for when you're talking to other engineers. Can you maybe elaborate a little bit more on what some of those qualities are that you're looking for in other engineers?

Ava Connolly(15:36):

Yeah, so I think, let's just say I was interviewing someone, you know, an entry level data engineering candidate at Arise. You know, I'm going to be probably asking for a get home portfolio. That would be something that I'd be looking for. You know, I wanna see projects that are not just the traditional common ones like hangman, you know, tic-tac toe, things like that. You know, I want real enterprise experience in cloud native services in, you know, like look at the job description and be able to translate those words into your GitHub portfolio and your resume. You know, I think that really covers a lot of what data engineering is and what it means to be a great data engineer, which is, you know, do you understand the assignment? Do you understand what this job description is asking of you? Even just that ability to translate a business need into an actual tangible result is something that I do as a data engineer on a daily basis and is something that you need to be really good at.

Eddie Hudson(16:45):

Yeah. I actually love that you brought up you know, talking a li like not just doing the, the traditional stuff like hangman or tic tac toe, but there was another thing that you brought up that I think is a really good piece of advice, right? Like if you go to an, into a data engineering role, I don't think anybody necessarily expects you to be a DevOps engineer as well. I mean, some places probably do, right? But you brought up a really good point that I think it's really important to understand things like cloud services, right? Because in, yeah, in some instances companies are building with on-prem solutions still but the majority of applications, at least the ones that I've worked on, are based in the cloud. And so I think it's important especially if you're starting out in your career that you don't just do the data engineering side, right? That you take a little bit un of time to understand what the underlying infrastructure can look like, what some of the underlying services that you might use to build those you know, those data pipelines or, or ETLs or, you know, even your machine learning programs, right? Because I think that's really important to understand that even if you're not gonna be the one putting that infrastructure together, I think it's really important to understand where the code that you write is gonna ultimately live, right?

Ava Connolly(18:03):

Yeah. And that's one thing I work on as a data engineer all the time, and something we strive for is really improving that cost and reliability of our production engineering systems. So a lot of what I do is actually, you know, migrating those analytical workflows into our data platform and snowflake using tools like Azure. And so to your point, I don't need to understand everything that's going on with Azure, but I do need to have enough context to be able to accomplish that so that my end user can be enabled and can, you know, that data can be highly useful for them.

Eddie Hudson(18:38):

Yeah, I love that. Right? Sizing right sizing services we call it making sure that you don't have a you know, like eight core, 900 gajillion gigabyte RAM system trying to run, you know, one simple data pipeline, you know, making sure that, that the instances or, or the services that you're using are right size for the workload and making sure that they're resilient and, you know, fail or there's failovers in place, you know, potentially making sure that there's like auto scaling involved. But again, that's not necessarily your job, but it's good for you to have a working knowledge of how those things kind of go together, right? Wow, that's also a good area to exercise your Cs degree in as well.

Ava Connolly(19:20):

Yeah, I think cost-driven development is something that more entry level data engineers and software engineers should be focused on because, you know, especially in a startup culture, cost driven development is so important and it's something that if I was interviewing a candidate who was entry level, it would really impress me.

Eddie Hudson(19:39):

Yeah, no, I, I a hundred percent agree. And I think that that's, that's something that we try to instill in our engineers as well as like, look, you're not necessarily the person that's gonna have to ultimately be responsible for this, but it's really good that they're able to actually look at what they're trying to do, understand kind of like what the, the bare minimum requirements are for the environment that they're trying to spin up and make sure they also understand like, what's kind of the maximum we expect that we need here, right? Like, you know, and, and being able to understand that because I think that gives them a better view of you know, of how the entire thing goes together. And that ultimately, at the end of the day, US engineers are also responsible for making sure that things are cost effective.

Eddie Hudson(20:21):

Mm-Hmm. <affirmative> I don't wanna spend too much time on cost and cloud infrastructure, but talk to me a little bit about love to talk about data engineering at arrive, because when we were talking about stage, there was a very specific thing that you mentioned that I don't wanna spoil it cause I'd like you to introduce it, but it was really interesting to me, especially the way that you guys do things that arrive, because I think it's a lot different than a lot of other technology companies right now. So can you talk to me a little bit about how you guys use data engineering at Arrive, but also how you guys support your clients?

Ava Connolly(20:59):

Yeah, so I mean, arrived I think is really unique in a way because our technology, you know, we work as a support services in a support services dynamic. So you know, we're really just focused on user-centric development, which I think is really exciting, you know, in that ability for us to talk to our users and collect feedback on a real-time basis let’s develop more skills like communication, you know, industry knowledge, documentation, things of that nature. Because I'm not only talking to engineers, I'm talking to non-technical users, which I think, especially as a younger engineer, is really just a great opportunity.

Eddie Hudson(21:44):

Yeah, no, I love that. Talk to me a little bit. I, I'd love to also dive in a little bit more and if we're going into the proprietary stuff, that's fine, but real-time data pipelines to me have always been super fascinating. It's, it's something that I think is I don't wanna say it's, it's necessarily just an art, but it, it's definitely something that I think to do it right, takes a lot of work. Talk to me a little bit about how you guys, I, I don't want to dive too much into the proprietary stuff, but talk to me a little bit about how you guys use real-time data and realtime analytics to support your clients.

Ava Connolly(22:23):

Well, I think you'd, you'd get like different opinions on if real-time actually exists, right? <Laugh>,

Eddie Hudson(22:31):

So good, good point.

Ava Connolly(22:33):

Yeah. But you know, I'm mean, we're so user-centric and user driven. You know, my stakeholders and the people that I fulfill bodies of work for are, are data scientists, our analytics engineers, you know, department analysts, business intelligence, things like that. So how it works with us is I receive a, you know, a ticket let's just say, so one example is analytics engineers. We have a department here for that. And one of their initiatives is providing certified data sets. So they'll come to me and say, Hey, I'd love to get, you know, this specific data, this logging data, for example, into our data platform in Snowflake, so I can create a power BI dashboard downstream. And so I'll walk through with them and ask them questions about, hey, you know, do you need a historical load plus your incremental? How often do you need that? What is your use case? Things of that nature. And it's a really dynamic back and forth, you know, environment where we kind of come up with a solution together and, you know, is one that is sustainable and maintainable and really does fulfill the need of my user as soon as possible, rather than having to go through iterations and iterations again.

Eddie Hudson(23:52):

Yeah. And I, I, I think that's really important as well because the iterations are what take time, right? Like you can get a ticket and then you get it finished and it, you, you can send it out for review, but if it has to come back multiple times mm-hmm. <Affirmative>, that's where you start losing a lot of the time, right. Especially off the original estimates as well. So I think that's, that's really important. We got about five minutes left so I'd love to just sit and talk because I know that we wanna focus a lot on data engineering careers. What sort of tips and tricks do you have for people that are out there, especially right now in the market? Like what are some of the tips and tricks that you have? Yeah, you'd mentioned one earlier about you know, them kind of going above and beyond that would really kind of set them apart from the rest of candidates. But talk to me a little bit about some of the other advice, maybe some advice that you'd give to yourself to you two or three years ago as you were starting to get into the field. What, what are some of the things that you can, can recommend to job seekers out there?

Ava Connolly(24:54):

I would say that don't use big tech or thing as your benchmark or standard for engineering excellence, particularly in interviews. I think that mindset is incredibly limiting. And I think the implications of that are that if you study in lead code and grind it out for months and get through that interview and you do great and you pass that, you can't scale that mindset, right? Yeah. And engineering is all about scaling. And so I would suggest to anyone searching for any job to take some time to create your own internal metric and standard of behavior and excellence for your own self because you can apply that to everything in your life, you know, even beyond your interview process. So for example, yeah, you know, I, if I'm doing something, I'm doing it a hundred percent, I am putting out my best work and I can apply that to my job interview, my work, the relationships I have with people. So if I focus on that, that is scalable into my interview process and that allows me to grow my knowledge in, you know, everything I'm learning as an engineer.

Eddie Hudson(26:07):

Yeah. And I love the point about fangs too, because there's such a small percentage of what are actually out there. Like what, what, what tech companies are actually out there and there's, I think there's so many people coming outta school and they're just shooting for the moon, right? But there's so many companies out there that are doing so much good in technology that aren't fangs A lot of us even fly under the radar. So you don't have to be working, you know, for a fang like meta to, you know, to get recognition as an engineer and to solve actual problems that have a meaningful difference on the world. You don't have to be there. And the other thing I think to your point as well is, I, I think that's a really good point on the sustainability side too, is, you know, talking about you know, burnout, it's, you, I think it's really important as an engineer, it's really easy, especially post pandemic.

Eddie Hudson(26:58):

I, I, I was talking to my wife about this over the weekend actually is like, it's still kind of hard to get out to the world cuz you just got so used to at least I got so used to just being in my office all day and that was how I interacted with people. But I think it's really important to understand that, you know, sometimes we need to make sure that we set boundaries and working at fangs that's, that's gonna be burnout culture, right? And a lot of times those guys are in, you talked a lot about it earlier, hyper-growth mode, and they're just trying to scale as fast as possible. And that's where a lot of the layoffs even recently come from because they come from companies that just did mass hiring when they saw a wave or you know, when they got a round of investment to try and meet some business goals and then all of a sudden now they're scaling back. And that's a very common tactic in business, which is, hey, we need to worry about growth right now. We can cut costs later. Right? So I definitely like that. Few more minutes here. Any other words of wisdom for the interview seekers out there? Looking to get a job with a company like Arrive?

Ava Connolly(28:04):

Yeah, I would say start your personal branding today if you haven't already. You know, I think I see a lot of posts like you mentioned, unfortunately, where people get laid off and, you know, your personal branding starts yesterday, it started years ago. And so, you know, I like to approach my personal branding as learning out loud. So I, I personally use LinkedIn as a kind of a public forum of learning and sharing, just sharing with people where I'm at and what I'm thinking about in my career now. And I think that the more people do that and the more you prioritize that, the better your network and your chances of getting a job will be. But you do have to do it consistently.

Eddie Hudson(28:50):

Yeah, I agree. And the point about personal branding is huge. As a recluse, I can tell you that I have to challenge myself to get on and do even do video calls and especially a podcast. But I think it's really important. It's, you know, especially now that even though there are companies that are starting to go back to the office, that's a lot of what we do is still remote or distributed and even if you are in the office, you're still likely meeting with clients or people that aren't directly there with you, right? So, I think personal branding's really important. I think I read your post and I was re I was, I was like, yeah, that's, yes, we need to do more of that. I probably need to do more of that.

Ava Connolly(29:30):

We all do

Eddie Hudson(29:30):

<Laugh>. Yeah, we all do. But the other thing too, a really important point I think, especially for the job seekers out there, your network is likely where your job is going to come from, right? Like, there is not gonna be a data engineering position that I mean, maybe there's one in the Austin Statesman, but I mean more, or you know, maybe it's on LinkedIn and maybe that's what gets you there, but more than likely you're probably gonna hear about it from a friend and they're gonna introduce you to a recruiter. And there was actually a really good piece of advice that there's a really good piece of advice that somebody gave me, which is, especially if you're looking to get into, I don't wanna say a high-level position, but for example, for a lot of these jobs, there's gonna be a ton of candidates, right?

Eddie Hudson(30:15):

Your best bet is a side door into the recruiting department. And if you have somebody that works at a company or you meet somebody whom you can befriend and, and they're able to introduce you to somebody that's gonna be kind of your, I don't wanna say shortcut, but you know what I mean, like that's how you get introduced to the people that make the decisions in a company. And I think that it's really important, especially as a young engineer trying to establish themselves in the industry that you're out there and meeting people and there's a ton of events, especially in Austin, they do 'em up at the capital factory all the time. There are startup events all the time over on sixth and anyways, there there's a huge network of events going on for people that are just getting outta school or people that are looking to get into the industry. So I think that's as a final point, I think that is a really good final piece of advice.

Ava Connolly(31:05):

Yeah, I got my job here at Arrive by messaging two people on the engineering team. I didn't go through the traditional way. I think I only submitted a resume as a formality at the end, but that's how I got my job, so I would recommend it

Eddie Hudson(31:21):

<Laugh>. Yeah, I agree. All right, well we are out of time, but Ava, thanks so much for taking the time to meet with us. I really appreciate it. And Carlos, do you have any announcements or are we good to go?

Carlos Ponce (31:36):

Yes, well actually Eddie more than announcements, it's an encouragement to all the viewers out there to actually, there's gonna be some, some updates to our, our calendar section and lemme just share the URL so that everyone can have it. Just chime in there within the next couple of days and you, there's gonna be a bunch of updates. And this is Eddie because we still are waiting for confirmation. I don't know why this is always happening in the early days of the year, but yeah, so everyone just stay tuned, right there on Upcoming on the Tech Backstage website right there - Upcoming. Just go there and stay tuned for those. And with that being said, Eddie, the only thing left for me to do is just thank you as our guest and remember right here on Tech Backstage, 12:00 PM Pacific every single day.

Eddie Hudson(32:25):

Thanks Carlos. Thanks Ava.

Ava Connolly(32:28):

Thank you.