Discover the essence of full-stack data science in this comprehensive article. Learn about the four hats of a full-stack data scientist and explore the entire machine-learning workflow from end to end. From diagnosing business problems to deploying machine learning solutions, uncover the key skills and principles essential for mastering this dynamic field. Join Sha as he delves into project management, data engineering, data science, and ML engineering, offering valuable insights and practical tips along the way. Subscribe for more insightful content on data science and entrepreneurship.
Table of Contents:
- Defining Full Stack Data Science
- Importance of End-to-End ML Solutions
2. The Versatile Roles of a Full-Stack Data Scientist
- Hat 1: Project Manager
- Hat 2: Data Engineer
- Hat 3: Data Scientist
- Hat 4: ML Engineer
3. Hat 1: Project Manager
- Role and Responsibilities
- Importance of Project Management in ML Workflow
4. Hat 2: Data Engineer
- Data Preparation for ML Solutions
- Key Skills and Tools for Data Engineering
5. Hat 3: Data Scientist
- Leveraging Data for Impact
- Model Training and Evaluation
6. Hat 4: ML Engineer
- Deploying ML Models into Solutions
- Containerization and API Integration
7. Principles for Becoming a Full-Stack Data Scientist
- Having a Reason to Learn
- Learning Just Enough
- Keeping Things Simple
8. Implementing a Machine Learning Project
- Building a Semantic Search System
- Walkthrough of Each Hat in Project Implementation
9. Conclusion and Future Directions
- Value from the Full Stack Data Science Approach
- Invitation for Feedback and Suggestions
10. Q&A and Community Engagement
- Addressing Comments and Suggestions
- Building a Learning Community around Full Stack Data Science
Introduction to Full Stack Data Science
Although it is common to delegate different parts of the machine learning workflow to specialized roles there are many situations that require individuals who can manage and Implement ml Solutions end to end.
I call these individuals full stack data scientists in this article I will introduce full stack data science and discuss its four hats and if you're new here welcome I'm icoversai I make articles about data science and Entrepreneurship.
If you enjoy this content please consider subscribing that's a great no-cost way you can support me in all the articles that I make starting with the basic question of what a full-stack data scientist the way.
I'll Define it here: a full-stack data scientist is someone who can manage and Implement an ML solution from end-to-end in other words they have a sufficient understanding of the entire ML workflow which gives them a unique ability to bring ML solutions to reality so typical ml workflow might look something like this.
Defining Full Stack Data Science
You'll start by diagnosing the business problem and designing an ML solution to that problem next with a design in mind you'll move on to sourcing and preparing the data for solution development then you'll develop the solution so in other words you'll train a machine learning model and then finally you will deploy.
Your solution so you'll integrate that machine learning model into existing workflows or into a product given the rise of specialized roles for each aspect of this machine learning workflow this idea of a full stag data scientist might seem a bit outdated and this was my thinking.
When I was working as a data scientist at a large Enterprise where we had a data engineering team and an ML engineering team and I was sitting on the data science team however over time the value of learning the entire Tex stack has become more and more obvious to me the spark for this realization and change in perspective for me happened around last year.
When I was interviewing top data science Freelancers on Upwork one of the key takeaways from these interviews was that data science skills alone provide no value while this might sound like a provocative statement think of it like this.
If I'm a freelancer I'm probably going to be talking with small to medium-sized businesses and most of the time these businesses don't have a data science function that's the whole reason they're hiring a freelancer so they often don't have the data infrastructure to provide the foundation for training machine learning models that means.
I want to come in as a data scientist and train a machine learning model I need to be able to extract the data prepare the data and make it available for training but it doesn't stop there once the model is trained it needs to be integrated into their existing workflows and again they probably don't have a machine learning engineer on staff.
Importance of End-to-End ML Solutions
That can do this work so for the value to be realized that's something I would need to do as a freelancer the data science skills the model training piece of the ml workflow is sandwiched in between the data engineering piece and the ml engineering piece.
While it is an important part of the workflow it can't even happen if the data aren't available for model training and it can't provide any impact if it's not implemented in the real world however freelance isn't the only context where knowing the full Tech stack is valuable even if you're not a freelancer working with small to medium-sized businesses say if you're a full-time employee at one of these companies.
They are often in the early stages of their data maturity and AI maturity so you might be the only resource or part of a small team of resources that are responsible for implementing the AI strategy of the company another situation might be your work at a large Enterprise but you're embedded in a team where you are the lone AI contributor.
The Versatile Roles of a Full-Stack Data Scientist
Hat 1: Project Manager
So in that situation, you may not have a ton of support on the data engineering side or the ML engineering side to implement machine learning Solutions and then finally if you're a Founder that wants to build a machine learning product you're going to need skills from all aspects of the Tex deck because often you're the only person in your company.
And it's on you to build the product from end to end so that brings us to what I like to call the four hats of a full stack data scientist and each of these corresponds to key parts of the machine learning workflow so hat one is the project man manager.
So diagnosing problems and designing Solutions piece of the workflow hat two is the data engineer so the sourcing and preparing of the data hat three is the data scientist training the machine learning model and hat four is the ML engineer which consists of deploying the ML solution starting with hat one the project manager the way.
I see the key role of a project manager is to answer three questions what why and how more specifically what are we building why are we building it and how are we going to build it while this might sound simple enough it's not uncommon for people to gloss over or skip this step entirely perhaps especially for technical folks.
Hat 2: Data Engineer
Who really wants to dive into the implementation and building the model so it might be like this Meme here the technical folks are more excited about coding than necessarily doing this project management work but the reason.
It's important that if you skip over this step you run the risk of spending a lot of time and money solving the wrong problem and even if you are solving the right problem you may solve it in an unnecessarily complex and expensive way.
All that to say taking some time at the outset of any project to stop and think about the problem you're trying to solve and the solution that you want to build can save you a lot of time and wasted effort so the way I see it the key skills involved with the project manager hat are communication and managing relationships.
Hat 3: Data Scientist
The reason for this first one is that as data scientists or full-stack data scientists, you're probably not going to be solving your own problems more often than not you're solving other people's problems and what this typically looks like is you're talking with stakeholders to better understand their problem.
And talk through potential Solutions the next key skill is the ability to diagnose problems and design Solutions Diagnosing problems comes down to finding the root cause for why something is going wrong and then designing Solutions isn't just about automatically throwing AI at problems.
But thinking through the value and the costs of each potential solution in making your decision and then the final key skill is being able to estimate project timelines costs and defining requirements again while this work may not seem as exciting as the technical stuff coding the implementation Etc doing this step right can save you a lot of headaches down the line next.
We have hat two which is that of a data engineer so in the context of full stack data science what data engineering is all about is making data readily available for model development or inference data Engineering in this context has one key difference.
Hat 4: ML Engineer
What we might call traditional data engineering at a large Enterprise at a large Enterprise The bulk of the data engineering work is often optimizing data architectures to support a wide range of business use cases on the other hand in the context of full stack data science the work is typically more product-focused.
While having some understanding of how to design flexible databases is important the type of data engineering work you would do in this full stat context is more concerned with building data pipelines so this will be creating ETL processes which stands for extract transform and load as well as data monitoring.
So giving visibility to data flowing through your pipeline and all the data related to your machine learning product some of the key skills that come up here are much more technical than what we saw on the previous slide so Python has really become a standard among data Engineers.
Python can be used for a wide range of tasks such as the extract process scraping web pages or working with APIs transforming the data so this could be things like d duplication exception handling feature engineering knowing SQL is a must especially if you're loading this data into a database which is going to be queried in some Downstream task also a basic understanding of command line interface tools.
While there are a lot of gooey-based applications for data engineering being able to use command line tools allows you to automate and scale processes a bit more easily next we have building data pipelines so this could be things like ETL or ELT again ETL is extract transform and load while elt is extract load and transform these will depend on the details of your use case that I'll talk about in a later article.
Principles for Becoming a Full-Stack Data Scientist
But common tools for building data pipelines are airflow which is an orchestration tool and Docker which is a containerization tool then finally you can definitely have your own servers and compute to acquire and store your data these days.
It's common to implement these data pipelines and data stores on some sort of cloud platform The Big Three are AWS GCP or Azure next we have Hat number three which is the data scientist hat my definition of a data scientist is someone who leverages regularities in data to drive impact.
Since computers are much better than us at finding regularities and patterns in data what this often boils down to is training a machine learning model what this typically looks like is you start with the real world which consists of things that you care about what we do is we'll collect data about those specific things.
We care about and then we'll use that data to train a model and the model can be used to make a prediction such as the probability that someone will buy our product based on their demographics or their behavior or something else it could be the probability that they don't pay back their credit card bill based on their credit score and other things and so on and so forth there are countless applications of machine learning models.
But the role of the data scientist doesn't just stop with training the model what's just as important if not more important is how one evaluates the model so defining performance metrics that are meaningful and in the best case scenario the performance metrics you use to evaluate the model can be tied back to key business performance metrics.
There's a clear mapping between model performance and business impact and the nature of training models is very experimental and iterative so what often happens is that you'll go through this whole workflow you'll evaluate the model and you'll learn something and that'll create a feedback loop for the data scientist to update the algorithm used to train the model.
Implementing a Machine Learning Project
The specific hyperparameters used or the specific features used in the model or the data scientist might realize there are inconsistencies in the data which requires to kind of go back to the data engineering step of the workflow and make some changes in how the data are transformed or they may realize that the data aren't sufficient so one has to go back and collect even more data.
Finally, you may evaluate the model and realize that some of your assumptions about how the real-world processes work were flawed for example at the project outset you may have assumed a particular variable was a key driver in something that you cared about but then upon training your model you may realize that that specific variable doesn't have the predictive performance
That you initially hoped so this requires you to kind of go back to the drawing board and rethink your assumptions about the specific use case this very iterative and experimental process makes data science in many ways more art than science which is one of the reasons I like doing it so much flexus one's creativity.
But it also introduces a fair amount of uncertainty into the model development process some of the key skills of the data scientist are probably first and foremost python some common data science libraries include pandas and powers which provide data structures for working with data and ways to manipulate data there's sklearn which is a popular machine learn learning library it has several machine learning algorithms readily available.
Then finally there are deep learning libraries like tensor flow or pytorch which allow you to train neural networks another key skill is exploratory data analysis even before you train the model there's usually a step in between here where one looks at the data looking at distributions looking how different variables track with one another if there any missing values if there any duplicates double checking the quality of the Data.
Before training a model on it and then finally model development and everything that goes into that one of the most important things is experiment tracking because there are so many knobs in dials that go into training a model such as the algorithm you choose the hyperparameters for the algorithm the train test split of the data the features used in your model the performance of the model and keeping track of all this information is very valuable.
Conclusion and Future Directions
When you're trying to discover the best-performing model the fourth and final hat is that of the ML engineer and what this consists of is turning your machine learning model into a machine learning solution what I mean by that is a model in of itself provides very little value it's probably going to be implemented in Python so it's going to require you to run a python script and provide a particular data and.
Then it'll spit something out and that output may not be inherently meaningful by itself so taking that model and embedding it into an existing workflow or a broader software solution is a critical part of this process so a very common way of deploying machine learning models is as follows you'll take your model and you'll containerize it which means
You'll create a modular version of the model that can be deployed in many different contexts and this is typically done using Docker however the model just sitting in this container still doesn't provide a whole lot of value so a simple way to allow this container to talk to external applications or external workflows is by adding an API to it what this allows is that if you have some AI app.
Let's say this is an internal website that allows employees to look up specific information users can write in a query into the user interface this will get sent to the machine learning model and then the API will spit back a response this is a very common and simple design you just have a container version of the model and you slap an API on top of it two popular tools.
For doing this Docker for the containerization bit and then fast API which is a Python library that allows you to create APIs for Python scripts however some use cases may not be so simple and require more sophisticated Solutions you know maybe something like this where you have your production model with the API SLA on top of it.
But you also want to do consistent model monitoring and you want to do some automated model retraining say every single month so if you want to do the retraining you're going to have to ingest data in an automated way and do the ETL process to put it into a database.
Maybe you also want to do some data monitoring so you slap that on top of the database as well this process data gets passed to a model retraining module which then gets pushed to the production model after some automated checks or something like that.
So for these more sophisticated Solutions, you're probably going to want to use an orchestration tool like airflow which provides an abstract way to connect these different pieces of software together the key skills for ML engineering is to containerize scripts using Docker and to build APIs using perhaps fast API another key skill is orchestrating multiple data and machine learning processes together so connecting data.
Q&A and Community Engagement
ML pipelines and a popular tool for that these days is airflow and then again while you could Implement all these Solutions on your local machine or some local hardware it's common practice these days to deploy these Solutions in the cloud.
While I've been doing data science for 5 years I would say I am just at the beginning of my journey toward becoming a full stack data scientist and while it might seem like this daunting and overwhelming task of learning the full Tex stack the way I think about it is that it's not about learning everything it's not about learning every single detail and skill involved in the machine learning workflow.
But rather it's about learning anything necessary to implement your particular solution and so the way I see it the best way to become a full stack data scientist is taking a more bottom-up approach as opposed to a top-down approach as problems arise learning just enough to solve that problem.
Here are three principles that I'm personally following the first is to have a reason to learn new skills there are many ways one can do this I'm personally building out my own projects and products both as a way to learn and as a way to solve specific problems that come up for me however there are other ways Beyond personal projects.
You know freelancing is a great opportunity instead of solving your own problems you're solving other people's problems which are going to require you to learn all aspects of the tech stack and indeed most of the Freelancers I know have skills across the entirety of the text de the second is to learn just enough to be dangerous this goes to this idea of not worrying about learning every single little detail but learning.
Whatever is necessary to solve the problem in front of you and then finally to keep things as simple as possible there are countless tools Technologies libraries Frameworks Solutions best practices for doing machine learning these days and it's easy to get so caught up in the best PR practices and what's scalable that you end up over-complicating the project so in my view Simplicity is the best guide for building machine learning Solutions.
This article is part of a larger Series in upcoming articles I will Implement a machine learning project end to end walking through each of the four hats discussed here so specifically I'm going to build a semantic search system that allows people to search across all of my YouTube articles.
I'll walk through each hat and I'll have an article for each one of these I'll do the project manager hat walking through AI project management estimating time and costs defining requirements I'll do the data engineering stuff which is walking through the data acquisition building the data Pipeline and creating the data store then hat three.
I'll walk through the solution development and the experimentation phase and then evaluate the solution finally I'll do an article on the ml engineering deploying the solution the container process and building an API that brings us to the end.
I hope you got some value from this article and the others in this series are all part of my own personal learning process toward that end if you feel like anything's missing or you have suggestions for future content I invite you to drop those in the comment section below those are very valuable to me personally and as always thank you so much for your time and thanks for reading.