Think accuracy should be your primary metric? You’re sorely mistaken.
Most of the best metrics instead focus on how far your were from the correct answer. Think RMSE and MAE.
Others point to how well calibrated your model is, like F1.
The worst taught skill in machine learning is model validation.
If you can’t validate your models well, you have no idea if they will actually work.
Here are 3 steps I’d take if I was relearning model validation from scratch 🧵
Think accuracy should be your primary metric? You’re sorely mistaken.
Most of the best metrics instead focus on how far your were from the correct answer. Think RMSE and MAE.
Others point to how well calibrated your model is, like F1.
Before diving in too deep, make sure you understand the basics.
You can’t become an expert in validation in the classroom, but knowing what is out there (simple k-fold, stratified, grouped, roll forward, etc.) is crucial.
Every day, or multiple times a week, pick an old Kaggle competition.
Read every solution that is posted and skip to their validation schemes.
There are nuances to every dataset, and this is the best way to see how pros navigate them.
Get a dataset and create a random test set.
Then, build some simple models and switch validation strategies in and out and see how well your models generalize for each scheme.
This will cement the importance of validation.
You will only improve at validation if you apply it to a ton of datasets.
If you stop after step 2, your skills will not be good enough. Full stop.
Never rest on your laurels. There is always something new to learn, and some new trick you can use.
I also discussed them on a podcast with @bhutanisanyam1 here: https://t.co/AiGAe1zBH3
Follow me @marktenenholtz so that you don’t miss it!
More from All
How can we use language supervision to learn better visual representations for robotics?
Introducing Voltron: Language-Driven Representation Learning for Robotics!
Paper: https://t.co/gIsRPtSjKz
Models: https://t.co/NOB3cpATYG
Evaluation: https://t.co/aOzQu95J8z
🧵👇(1 / 12)
Videos of humans performing everyday tasks (Something-Something-v2, Ego4D) offer a rich and diverse resource for learning representations for robotic manipulation.
Yet, an underused part of these datasets are the rich, natural language annotations accompanying each video. (2/12)
The Voltron framework offers a simple way to use language supervision to shape representation learning, building off of prior work in representations for robotics like MVP (https://t.co/Pb0mk9hb4i) and R3M (https://t.co/o2Fkc3fP0e).
The secret is *balance* (3/12)
Starting with a masked autoencoder over frames from these video clips, make a choice:
1) Condition on language and improve our ability to reconstruct the scene.
2) Generate language given the visual representation and improve our ability to describe what's happening. (4/12)
By trading off *conditioning* and *generation* we show that we can learn 1) better representations than prior methods, and 2) explicitly shape the balance of low and high-level features captured.
Why is the ability to shape this balance important? (5/12)
Introducing Voltron: Language-Driven Representation Learning for Robotics!
Paper: https://t.co/gIsRPtSjKz
Models: https://t.co/NOB3cpATYG
Evaluation: https://t.co/aOzQu95J8z
🧵👇(1 / 12)
Videos of humans performing everyday tasks (Something-Something-v2, Ego4D) offer a rich and diverse resource for learning representations for robotic manipulation.
Yet, an underused part of these datasets are the rich, natural language annotations accompanying each video. (2/12)
The Voltron framework offers a simple way to use language supervision to shape representation learning, building off of prior work in representations for robotics like MVP (https://t.co/Pb0mk9hb4i) and R3M (https://t.co/o2Fkc3fP0e).
The secret is *balance* (3/12)
Starting with a masked autoencoder over frames from these video clips, make a choice:
1) Condition on language and improve our ability to reconstruct the scene.
2) Generate language given the visual representation and improve our ability to describe what's happening. (4/12)
By trading off *conditioning* and *generation* we show that we can learn 1) better representations than prior methods, and 2) explicitly shape the balance of low and high-level features captured.
Why is the ability to shape this balance important? (5/12)
