If you enjoyed this thread
- Rt and like the first tweet
- Follow @SeekLifeMastery for more!
More from All
How can we use language supervision to learn better visual representations for robotics?
Introducing Voltron: Language-Driven Representation Learning for Robotics!
Paper: https://t.co/gIsRPtSjKz
Models: https://t.co/NOB3cpATYG
Evaluation: https://t.co/aOzQu95J8z
🧵👇(1 / 12)
Videos of humans performing everyday tasks (Something-Something-v2, Ego4D) offer a rich and diverse resource for learning representations for robotic manipulation.
Yet, an underused part of these datasets are the rich, natural language annotations accompanying each video. (2/12)
The Voltron framework offers a simple way to use language supervision to shape representation learning, building off of prior work in representations for robotics like MVP (https://t.co/Pb0mk9hb4i) and R3M (https://t.co/o2Fkc3fP0e).
The secret is *balance* (3/12)
Starting with a masked autoencoder over frames from these video clips, make a choice:
1) Condition on language and improve our ability to reconstruct the scene.
2) Generate language given the visual representation and improve our ability to describe what's happening. (4/12)
By trading off *conditioning* and *generation* we show that we can learn 1) better representations than prior methods, and 2) explicitly shape the balance of low and high-level features captured.
Why is the ability to shape this balance important? (5/12)
Introducing Voltron: Language-Driven Representation Learning for Robotics!
Paper: https://t.co/gIsRPtSjKz
Models: https://t.co/NOB3cpATYG
Evaluation: https://t.co/aOzQu95J8z
🧵👇(1 / 12)
Videos of humans performing everyday tasks (Something-Something-v2, Ego4D) offer a rich and diverse resource for learning representations for robotic manipulation.
Yet, an underused part of these datasets are the rich, natural language annotations accompanying each video. (2/12)
The Voltron framework offers a simple way to use language supervision to shape representation learning, building off of prior work in representations for robotics like MVP (https://t.co/Pb0mk9hb4i) and R3M (https://t.co/o2Fkc3fP0e).
The secret is *balance* (3/12)
Starting with a masked autoencoder over frames from these video clips, make a choice:
1) Condition on language and improve our ability to reconstruct the scene.
2) Generate language given the visual representation and improve our ability to describe what's happening. (4/12)
By trading off *conditioning* and *generation* we show that we can learn 1) better representations than prior methods, and 2) explicitly shape the balance of low and high-level features captured.
Why is the ability to shape this balance important? (5/12)
You May Also Like
1/ Here’s a list of conversational frameworks I’ve picked up that have been helpful.
Please add your own.
2/ The Magic Question: "What would need to be true for you
3/ On evaluating where someone’s head is at regarding a topic they are being wishy-washy about or delaying.
“Gun to the head—what would you decide now?”
“Fast forward 6 months after your sabbatical--how would you decide: what criteria is most important to you?”
4/ Other Q’s re: decisions:
“Putting aside a list of pros/cons, what’s the *one* reason you’re doing this?” “Why is that the most important reason?”
“What’s end-game here?”
“What does success look like in a world where you pick that path?”
5/ When listening, after empathizing, and wanting to help them make their own decisions without imposing your world view:
“What would the best version of yourself do”?
Please add your own.
2/ The Magic Question: "What would need to be true for you
1/\u201cWhat would need to be true for you to\u2026.X\u201d
— Erik Torenberg (@eriktorenberg) December 4, 2018
Why is this the most powerful question you can ask when attempting to reach an agreement with another human being or organization?
A thread, co-written by @deanmbrody: https://t.co/Yo6jHbSit9
3/ On evaluating where someone’s head is at regarding a topic they are being wishy-washy about or delaying.
“Gun to the head—what would you decide now?”
“Fast forward 6 months after your sabbatical--how would you decide: what criteria is most important to you?”
4/ Other Q’s re: decisions:
“Putting aside a list of pros/cons, what’s the *one* reason you’re doing this?” “Why is that the most important reason?”
“What’s end-game here?”
“What does success look like in a world where you pick that path?”
5/ When listening, after empathizing, and wanting to help them make their own decisions without imposing your world view:
“What would the best version of yourself do”?
Department List of UCAS-China PROFESSORs for ANSO, CSC and UCAS (fully or partial) Scholarship Acceptance
1) UCAS School of physical sciences Professor
https://t.co/9X8OheIvRw
2) UCAS School of mathematical sciences Professor
3) UCAS School of nuclear sciences and technology
https://t.co/nQH8JnewcJ
4) UCAS School of astronomy and space sciences
https://t.co/7Ikc6CuKHZ
5) UCAS School of engineering
6) Geotechnical Engineering Teaching and Research Office
https://t.co/jBCJW7UKlQ
7) Multi-scale Mechanics Teaching and Research Section
https://t.co/eqfQnX1LEQ
😎 Microgravity Science Teaching and Research
9) High temperature gas dynamics teaching and research section
https://t.co/tVIdKgTPl3
10) Department of Biomechanics and Medical Engineering
https://t.co/ubW4xhZY2R
11) Ocean Engineering Teaching and Research
12) Department of Dynamics and Advanced Manufacturing
https://t.co/42BKXEugGv
13) Refrigeration and Cryogenic Engineering Teaching and Research Office
https://t.co/pZdUXFTvw3
14) Power Machinery and Engineering Teaching and Research
1) UCAS School of physical sciences Professor
https://t.co/9X8OheIvRw
2) UCAS School of mathematical sciences Professor
3) UCAS School of nuclear sciences and technology
https://t.co/nQH8JnewcJ
4) UCAS School of astronomy and space sciences
https://t.co/7Ikc6CuKHZ
5) UCAS School of engineering
6) Geotechnical Engineering Teaching and Research Office
https://t.co/jBCJW7UKlQ
7) Multi-scale Mechanics Teaching and Research Section
https://t.co/eqfQnX1LEQ
😎 Microgravity Science Teaching and Research
9) High temperature gas dynamics teaching and research section
https://t.co/tVIdKgTPl3
10) Department of Biomechanics and Medical Engineering
https://t.co/ubW4xhZY2R
11) Ocean Engineering Teaching and Research
12) Department of Dynamics and Advanced Manufacturing
https://t.co/42BKXEugGv
13) Refrigeration and Cryogenic Engineering Teaching and Research Office
https://t.co/pZdUXFTvw3
14) Power Machinery and Engineering Teaching and Research