We need to represent the real world as accurately as possible. If some situations are underrepresented we are introducing Sampling Bias.
Sampling Bias is nasty because we'll have high test accuracy, but our model will perform badly when deployed.
👇
Machine Learning in the Real World 🧠 🤖
ML for real-world applications is much more than designing fancy networks and fine-tuning parameters.
In fact, you will spend most of your time curating a good dataset.
Let's go through the process together 👇
#RepostFriday
We need to represent the real world as accurately as possible. If some situations are underrepresented we are introducing Sampling Bias.
Sampling Bias is nasty because we'll have high test accuracy, but our model will perform badly when deployed.
👇
Let's build a model to recognize traffic lights for a self-driving car. We need to collect data for different:
▪️ Lighting conditions
▪️ Weather conditions
▪️ Distances and viewpoints
▪️ Strange variants
And if we sample only 🚦 we won't detect 🚥 🤷♂️
👇
Now we need to clean all corrupted and irrelevant samples. We need to remove:
▪️ Overexposed or underexposed images
▪️ Images in irrelevant situations
▪️ Faulty images
Leaving them in the dataset will hurt our model's performance!
👇
Most ML models like their data nicely normalized and properly scaled. Bad normalization can also lead to worse performance (I have a nice story for another time...)
▪️ Crop and resize all images
▪️ Normalize all values (usually 0 mean and 1 std. dev.)
👇
Manual labeling is expensive. Try to be clever and automate as much as possible:
▪️ Generate labels from the input data
▪️ Use slow, but accurate algorithms offline
▪️ Pre-label data during collection
▪️ Develop good labeling tools
▪️ Use synthetic data?
👇
You will always have errors in the labels - humans make mistakes. Review and iterate!
▪️ Spot checks to find systematic problems
▪️ Improve labeling guidelines and tools
▪️ Review test results and fix labels
▪️ Label samples multiple times
👇
A recent study by MIT found that 10 of the most popular public datasets had 3.4% label errors on average (ImageNet had 5.8%).
This even lead authors to choose the wrong (and more complex) model as their best one!
https://t.co/dfZPz6xnU0
👇
Dealing with imbalanced data can be tricky...
Let's classify the color of the 🚦 - we can get 97% just by learning to recognize 🟢 and 🔴, just because 🟡 is severely underrepresented.
I have a separate thread on this topic:
https://t.co/R8z3AeDD2b
👇
Dealing with imbalanced datasets \U0001f401 \u2696\ufe0f \U0001f418
— haltakov.eth \U0001f30d \U0001f1fa\U0001f1e6 (@haltakov) February 10, 2021
Real world datasets are often imbalanced - some of the classes appear much more often in your data than others.
The problem? You ML model will likely learn to only predict the dominant classes.
What can you do about it? \U0001f914
Thread \U0001f447
This is the part that is usually covered by ML courses. Now is the time to try out different features, network architectures, fine-tune parameters etc.
But we are not done yet... 👇
In most real-world applications the bottleneck is not the model itself, but the data. After having a first model, we need to review where it has problems and go back to:
▪️ Collecting and labeling more data
▪️ Correcting labels
▪️ Balancing the data
👇
Deploying the model in production poses some additional constraints:
▪️ Speed
▪️ Cost
▪️ Stability
▪️ Privacy
▪️ Hardware availability and integration
We have to find a good trade-off between these factors and accuracy.
Now we are done, right? No...👇
The performance of the model will start degrading over time because the world keeps changing:
▪️ Concept drift - the real-world distribution changes
▪️ Data drift - the properties of the data change
We need to detect this, retrain, and deploy again.
Example 👇
We now have a trained model to recognize 🚦, but people keep inventing new variants - see what some creative people in Munich came up with 😄
We need a way to detect that we have a problem, collect data, label, and retrain our model.
👇
This is how a typical ML pipeline for real-world applications looks like. Please remember this:
▪️ Curating a good dataset is the most important thing
▪️ Dataset curation is an iterative process
▪️ Monitoring is critical to ensure good performance over time
If you are interested in seeing more, follow me @haltakov
.
More from haltakov.eth 🌍 🇺🇦
Let's talk about a common problem in ML - imbalanced data ⚖️
Imagine we want to detect all pixels belonging to a traffic light from a self-driving car's camera. We train a model with 99.88% performance. Pretty cool, right?
Actually, this model is useless ❌
Let me explain 👇
The problem is the data is severely imbalanced - the ratio between traffic light pixels and background pixels is 800:1.
If we don't take any measures, our model will learn to classify each pixel as background giving us 99.88% accuracy. But it's useless!
What can we do? 👇
Let me tell you about 3 ways of dealing with imbalanced data:
▪️ Choose the right evaluation metric
▪️ Undersampling your dataset
▪️ Oversampling your dataset
▪️ Adapting the loss
Let's dive in 👇
1️⃣ Evaluation metrics
Looking at the overall accuracy is a very bad idea when dealing with imbalanced data. There are other measures that are much better suited:
▪️ Precision
▪️ Recall
▪️ F1 score
I wrote a whole thread on
2️⃣ Undersampling
The idea is to throw away samples of the overrepresented classes.
One way to do this is to randomly throw away samples. However, ideally, we want to make sure we are only throwing away samples that look similar.
Here is a strategy to achieve that 👇
Imagine we want to detect all pixels belonging to a traffic light from a self-driving car's camera. We train a model with 99.88% performance. Pretty cool, right?
Actually, this model is useless ❌
Let me explain 👇
The problem is the data is severely imbalanced - the ratio between traffic light pixels and background pixels is 800:1.
If we don't take any measures, our model will learn to classify each pixel as background giving us 99.88% accuracy. But it's useless!
What can we do? 👇
Let me tell you about 3 ways of dealing with imbalanced data:
▪️ Choose the right evaluation metric
▪️ Undersampling your dataset
▪️ Oversampling your dataset
▪️ Adapting the loss
Let's dive in 👇
1️⃣ Evaluation metrics
Looking at the overall accuracy is a very bad idea when dealing with imbalanced data. There are other measures that are much better suited:
▪️ Precision
▪️ Recall
▪️ F1 score
I wrote a whole thread on
How to evaluate your ML model? \U0001f4cf
— Vladimir Haltakov (@haltakov) August 31, 2021
Your accuracy is 97%, so this is pretty good, right? Right? No! \u274c
Just looking at the model accuracy is not enough. Let me tell you about some other metrics:
\u25aa\ufe0f Recall
\u25aa\ufe0f Precision
\u25aa\ufe0f F1 score
\u25aa\ufe0f Confusion matrix
Let's start \U0001f447
2️⃣ Undersampling
The idea is to throw away samples of the overrepresented classes.
One way to do this is to randomly throw away samples. However, ideally, we want to make sure we are only throwing away samples that look similar.
Here is a strategy to achieve that 👇
Machine Learning Paper Reviews 🔎📜
Check out this thread for short reviews of some interesting Machine Learning and Computer Vision papers. I explain the basic ideas and main takeaways of each paper in a Twitter thread.
👇 I'm adding new reviews all the time! 👇
AlexNet - the paper that started the deep learning revolution in Computer Vision!
DenseNet - reducing the size and complexity of CNNs by adding dense connections between layers.
Playing for data - generating synthetic GT from a video game (GTA V) and using it to improving semantic segmentation models.
Transformers for image recognition - a new paper with the potential to replace convolutions with a transformer.
Check out this thread for short reviews of some interesting Machine Learning and Computer Vision papers. I explain the basic ideas and main takeaways of each paper in a Twitter thread.
👇 I'm adding new reviews all the time! 👇
AlexNet - the paper that started the deep learning revolution in Computer Vision!
It's finally time for some paper review! \U0001f4dc\U0001f50d\U0001f9d0
— Vladimir Haltakov (@haltakov) September 28, 2020
I promised the other day to start posting threads with summaries of papers that had a big impact on the field of ML and CV.
Here is the first one - the AlexNet paper! pic.twitter.com/QNLPIMZSIa
DenseNet - reducing the size and complexity of CNNs by adding dense connections between layers.
ML paper review time - DenseNet! \U0001f578\ufe0f
— Vladimir Haltakov (@haltakov) October 15, 2020
This paper won the Best Paper Award at the 2017 Conference on Computer Vision and Pattern Recognition (CVPR) - the best conference for computer vision problems.
It introduces a new CNN architecture where the layers are densely connected. pic.twitter.com/DuHytaoXia
Playing for data - generating synthetic GT from a video game (GTA V) and using it to improving semantic segmentation models.
Time for another ML paper review - generating synthetic ground truth data from video games! \U0001f3ae
— Vladimir Haltakov (@haltakov) October 5, 2020
I love this paper, because it pushes the boundaries of creating realistic synthetic ground truth data and shows that you can use it for training and improve your model.
Details \U0001f447 pic.twitter.com/fBgORYG8Lz
Transformers for image recognition - a new paper with the potential to replace convolutions with a transformer.
Another paper review, but a little different this time... \U0001f937\u200d\u2642\ufe0f
— Vladimir Haltakov (@haltakov) October 5, 2020
The paper is not published yet, but is submitted for review at ICLR 2021. It is getting a lot of attention from the CV/ML community, though, and many speculate that it is the end of CNNs... \U0001f447https://t.co/bh6wUxYfxu pic.twitter.com/dZGBYB8A5U
More from All
#ஆதித்தியஹ்ருதயம் ஸ்தோத்திரம்
இது சூரிய குலத்தில் உதித்த இராமபிரானுக்கு தமிழ் முனிவர் அகத்தியர் உபதேசித்ததாக வால்மீகி இராமாயணத்தில் வருகிறது. ஆதித்ய ஹ்ருதயத்தைத் தினமும் ஓதினால் பெரும் பயன் பெறலாம் என மகான்களும் ஞானிகளும் காலம் காலமாகக் கூறி வருகின்றனர். ராம-ராவண யுத்தத்தை
தேவர்களுடன் சேர்ந்து பார்க்க வந்திருந்த அகத்தியர், அப்போது போரினால் களைத்து, கவலையுடன் காணப்பட்ட ராமபிரானை அணுகி, மனிதர்களிலேயே சிறந்தவனான ராமா போரில் எந்த மந்திரத்தைப் பாராயணம் செய்தால் எல்லா பகைவர்களையும் வெல்ல முடியுமோ அந்த ரகசிய மந்திரத்தை, வேதத்தில் சொல்லப்பட்டுள்ளதை உனக்கு
நான் உபதேசிக்கிறேன், கேள் என்று கூறி உபதேசித்தார். முதல் இரு சுலோகங்கள் சூழ்நிலையை விவரிக்கின்றன. மூன்றாவது சுலோகம் அகத்தியர் இராமபிரானை விளித்துக் கூறுவதாக அமைந்திருக்கிறது. நான்காவது சுலோகம் முதல் முப்பதாம் சுலோகம் வரை ஆதித்ய ஹ்ருதயம் என்னும் நூல். முப்பத்தி ஒன்றாம் சுலோகம்
இந்தத் துதியால் மகிழ்ந்த சூரியன் இராமனை வாழ்த்துவதைக் கூறுவதாக அமைந்திருக்கிறது.
ஐந்தாவது ஸ்லோகம்:
ஸர்வ மங்கள் மாங்கல்யம் ஸர்வ பாப ப்ரநாசனம்
சிந்தா சோக ப்ரசமனம் ஆயுர் வர்த்தனம் உத்தமம்
பொருள்: இந்த அதித்ய ஹ்ருதயம் என்ற துதி மங்களங்களில் சிறந்தது, பாவங்களையும் கவலைகளையும்
குழப்பங்களையும் நீக்குவது, வாழ்நாளை நீட்டிப்பது, மிகவும் சிறந்தது. இதயத்தில் வசிக்கும் பகவானுடைய அனுக்ரகத்தை அளிப்பதாகும்.
முழு ஸ்லோக லிங்க் பொருளுடன் இங்கே உள்ளது https://t.co/Q3qm1TfPmk
சூரியன் உலக இயக்கத்திற்கு மிக முக்கியமானவர். சூரிய சக்தியால்தான் ஜீவராசிகள், பயிர்கள்
இது சூரிய குலத்தில் உதித்த இராமபிரானுக்கு தமிழ் முனிவர் அகத்தியர் உபதேசித்ததாக வால்மீகி இராமாயணத்தில் வருகிறது. ஆதித்ய ஹ்ருதயத்தைத் தினமும் ஓதினால் பெரும் பயன் பெறலாம் என மகான்களும் ஞானிகளும் காலம் காலமாகக் கூறி வருகின்றனர். ராம-ராவண யுத்தத்தை
தேவர்களுடன் சேர்ந்து பார்க்க வந்திருந்த அகத்தியர், அப்போது போரினால் களைத்து, கவலையுடன் காணப்பட்ட ராமபிரானை அணுகி, மனிதர்களிலேயே சிறந்தவனான ராமா போரில் எந்த மந்திரத்தைப் பாராயணம் செய்தால் எல்லா பகைவர்களையும் வெல்ல முடியுமோ அந்த ரகசிய மந்திரத்தை, வேதத்தில் சொல்லப்பட்டுள்ளதை உனக்கு
நான் உபதேசிக்கிறேன், கேள் என்று கூறி உபதேசித்தார். முதல் இரு சுலோகங்கள் சூழ்நிலையை விவரிக்கின்றன. மூன்றாவது சுலோகம் அகத்தியர் இராமபிரானை விளித்துக் கூறுவதாக அமைந்திருக்கிறது. நான்காவது சுலோகம் முதல் முப்பதாம் சுலோகம் வரை ஆதித்ய ஹ்ருதயம் என்னும் நூல். முப்பத்தி ஒன்றாம் சுலோகம்
இந்தத் துதியால் மகிழ்ந்த சூரியன் இராமனை வாழ்த்துவதைக் கூறுவதாக அமைந்திருக்கிறது.
ஐந்தாவது ஸ்லோகம்:
ஸர்வ மங்கள் மாங்கல்யம் ஸர்வ பாப ப்ரநாசனம்
சிந்தா சோக ப்ரசமனம் ஆயுர் வர்த்தனம் உத்தமம்
பொருள்: இந்த அதித்ய ஹ்ருதயம் என்ற துதி மங்களங்களில் சிறந்தது, பாவங்களையும் கவலைகளையும்
குழப்பங்களையும் நீக்குவது, வாழ்நாளை நீட்டிப்பது, மிகவும் சிறந்தது. இதயத்தில் வசிக்கும் பகவானுடைய அனுக்ரகத்தை அளிப்பதாகும்.
முழு ஸ்லோக லிங்க் பொருளுடன் இங்கே உள்ளது https://t.co/Q3qm1TfPmk
சூரியன் உலக இயக்கத்திற்கு மிக முக்கியமானவர். சூரிய சக்தியால்தான் ஜீவராசிகள், பயிர்கள்
ChatGPT is a phenomenal AI Tool.
But don't limit yourself to just ChatGPT.
Here're 8 AI-powered tools you should try in 2023:
1. KaiberAI
@KaiberAI helps you generate beautiful videos in minutes.
Transform your ideas into the visual stories of your dreams with this Amazing Tool.
New features:
1. Upload your custom music
2. Prompt Templates
3. Camera Movements:
Check here
https://t.co/ivnDRf628L
2. @tldview TLDV
Best ChatGPT Alternative for meetings.
Make your meetings 10X more productive with this amazing tool.
Try it now:
https://t.co/vOy3sS4QfJ
3. ComposeAI
Use ComposeAI for generating any text using AI.
It’s will help you write better content in seconds.
Try it here:
https://t.co/ksj5aop5ZI
4. Browser AI
Use this AI tool to extract and monitor data from any website.
Train a robot in 2 minutes to do your work.
No coding required.
https://t.co/nNiawtUMyO
But don't limit yourself to just ChatGPT.
Here're 8 AI-powered tools you should try in 2023:
1. KaiberAI
@KaiberAI helps you generate beautiful videos in minutes.
Transform your ideas into the visual stories of your dreams with this Amazing Tool.
New features:
1. Upload your custom music
2. Prompt Templates
3. Camera Movements:
Check here
https://t.co/ivnDRf628L
2. @tldview TLDV
Best ChatGPT Alternative for meetings.
Make your meetings 10X more productive with this amazing tool.
Try it now:
https://t.co/vOy3sS4QfJ
3. ComposeAI
Use ComposeAI for generating any text using AI.
It’s will help you write better content in seconds.
Try it here:
https://t.co/ksj5aop5ZI
4. Browser AI
Use this AI tool to extract and monitor data from any website.
Train a robot in 2 minutes to do your work.
No coding required.
https://t.co/nNiawtUMyO
You May Also Like
Following @BAUDEGS I have experienced hateful and propagandist tweets time after time. I have been shocked that an academic community would be so reckless with their publications. So I did some research.
The question is:
Is this an official account for Bahcesehir Uni (Bau)?
Bahcesehir Uni, BAU has an official website https://t.co/ztzX6uj34V which links to their social media, leading to their Twitter account @Bahcesehir
BAU’s official Twitter account
BAU has many departments, which all have separate accounts. Nowhere among them did I find @BAUDEGS
@BAUOrganization @ApplyBAU @adayBAU @BAUAlumniCenter @bahcesehirfbe @baufens @CyprusBau @bauiisbf @bauglobal @bahcesehirebe @BAUintBatumi @BAUiletisim @BAUSaglik @bauebf @TIPBAU
Nowhere among them was @BAUDEGS to find
The question is:
Is this an official account for Bahcesehir Uni (Bau)?
Bahcesehir Uni, BAU has an official website https://t.co/ztzX6uj34V which links to their social media, leading to their Twitter account @Bahcesehir
BAU’s official Twitter account
BAU has many departments, which all have separate accounts. Nowhere among them did I find @BAUDEGS
@BAUOrganization @ApplyBAU @adayBAU @BAUAlumniCenter @bahcesehirfbe @baufens @CyprusBau @bauiisbf @bauglobal @bahcesehirebe @BAUintBatumi @BAUiletisim @BAUSaglik @bauebf @TIPBAU
Nowhere among them was @BAUDEGS to find
